Last week, we organized another breakfast session at Understanding Data themed “From Business Question to Solution.” It’s now a regular event every last Thursday of the month. But behind the scenes, there’s a lot of work that goes into it. First, we stopped by the bakery to pick up some pastries. Then we set the table. Next, the first participants arrived, and the session could begin. The heat wave calls for some extra refreshment. Trigger warning: We noticed that the water pitcher was almost empty and needed to be refilled.
At first glance, these just look like a few atmospheric photos of breakfast. But they actually perfectly illustrate what data orchestration is.
Orchestration means organizing and managing various tasks so that everything happens at the right time, in the right order. So it’s not just a simple list of tasks. Because some steps can’t start until others are finished.
For example, the breakfast session didn’t start with participants already seated at the table.
First, the following had to happen:
Only then could the session truly begin.
It works exactly the same way in a data environment. Before you display a dashboard, for example, the following must first be done:
Imagine that everyone has already arrived while we’re still at the bakery. That would leave us with a rather strange breakfast session. The pastries aren’t there yet, the table is still empty, and everyone is just sitting there waiting. The same thing can happen with data. If a report refreshes before the underlying data has been processed, you’ll end up with incorrect figures. If a step fails and no one notices, the entire process continues based on unreliable data. Just like a conductor, orchestration ensures that processes are not only executed but also at the right time.
No one had planned in advance that the water level needed to be checked at 9:12 a.m. Something just happened. Someone noticed that the water was almost gone, and that became a signal, a trigger that set an action in motion. In modern data platforms, this happens all the time. A file comes in, and processing begins. A dataset is ready, and the dashboard is updated. An error occurs, and a warning is sent. Not everything has to wait for a fixed time. Sometimes you want to respond to what’s happening (this is also called “event-driven,” driven by events that occur at random moments).
So far, we’ve explained it using pastries and water. But how does that work in a data environment? This is where an orchestrator like Dagster comes into play. Dagster not only determines which tasks need to be performed, but it also knows which steps depend on one another, when something can start, what happens if something fails, which events should trigger a process, and so on… In our breakfast metaphor, for example, Dagster (or Perfect or Airflow) would say: “The session can’t start until we’re back from the bakery and the table is set,” and then: “Warning: the water is almost gone. Start process: refill water.”
A successful breakfast session seems simple. But behind the scenes, a surprising range of activities takes place. There are dependencies, sequences, checks, and unexpected events that trigger actions. Just like a modern data platform. And sometimes, a lesson in orchestration starts with someone simply going to the bakery to get pastries.
Contact us now, and learn how we can bring your organisation to the next level.
