Member-only story
A Gentle Intro to Streams
An overview of working with live data — and yet another reason to love Elixir
Most of us have watched Netflix, so we know conceptually what streaming data means. Working with streaming data in code can seem much more intimidating than catching up on past episodes of your favorite shows. Understanding the why of things will make dealing with streaming data in any language more intuitive.
Whenever data is live updated continuously, you have a stream. It could be something continuous like music or video. It could also be a contiguous series of live data updates such as a Twitter feed or stock prices. It could even be a series of mouse coordinates in a UI.
Either way, you can think of a stream as being sort of like an Array, but infinite in length. If you try to work with this data using the same tools you would reach for with for finite data, it’ll cause problems.
Suppose you have an Array of recent Apple stock prices. These can give you an idea of where the stock has been lately, but it won’t really answer the question of how much you would pay if you were to buy 5 shares right now. If you want to provide the most recent price, you will need a stream of live-updating data.
If you continue using an Array to store this data, you will end up with a fresh copy of the whole data set each time the price updates. Memory usage will spike, and your app will be slooooowww.
The ideal way for live updating data to be handled is with only the latest update, or the diff of the data, being stored. This is obviously more efficient than getting a whole new copy of the entire data set each time.
Another optimization for dealing with live data is lazy evaluation. Rather than doing a potentially costly calculation up front it waits to see if the time is right. So laziness can be good, under the right circumstances.
If you are working in JavaScript, an external library such as RxJS offers convenience methods for working with streams. In Elixir, there is a Stream abstraction built into the language.
Working with streams can feel like working with Arrays. In JavaScript you have functional methods for iterating over Arrays such as map and filter. These have analogs in RxJS that offer the benefits of…
