Right-sizing the orchestrator
When Dagster earns its keep, and when a Python script is the more honest answer.
There is a moment, early in a data project, where someone reaches for an orchestrator. Dagster, Airflow, Prefect — it doesn’t really matter which one. The question is whether it’s the right thing to reach for.
We use Dagster a lot. It earns its place on the kind of work where assets need lineage, where dbt runs need to participate in a wider DAG, where on-call engineers need a UI to triage from. None of that is in question. But there is a smaller class of work where it is not the right tool, and the discipline is to notice when you are in that class.
The fixed-term campaign
A recent project was a four-week regulatory contact campaign. Three customer cohorts. Five Python entrypoints: a one-off load, a weekly send, a webhook receiver, a postal extract, a postal-returns ingest. State in one BigQuery dataset. Done.
The host stack had Dagster. The temptation was to integrate. The right answer was not to.
The orchestrator was going to live longer than the campaign. The campaign had its own deadlines and its own ops cadence. The cost of integrating — defining assets, building sensors, mapping the campaign’s state machine into the orchestrator’s — would have been measured in days. The value of integrating — a UI, lineage, sensors firing on schedule — would have been measured in hours, because the campaign’s schedule was weekly, manually triggered, and tied to specific business events.
We chose standalone Python. Five scripts, a shared library of state-derivation queries, a Cloud Run service for the webhook. The whole system was readable end-to-end by someone who knew BigQuery and a bit of Python. When the campaign ended, the artefacts were small enough to archive, and the audit trail was a BigQuery dataset that didn’t need anything running to be queryable.
The test
The test we ran the decision through was simple, and it generalises:
- Does the work outlive the orchestrator’s setup cost? A four-week build does not. A multi-year platform does.
- Does the work share infrastructure with other things the orchestrator already runs? If so, integration is cheap. If not, the orchestrator is an island carrying overhead for the rest of the team.
- Does the work need a UI for on-call? Some does, some doesn’t. A campaign with a Monday-morning console session does not.
- Does the work need lineage with other assets? Stand-alone work doesn’t.
Reach for the orchestrator when more than one of those answers is yes. Otherwise, reach for the smallest thing that solves the problem.
What we kept anyway
Some of the orchestrator’s habits are worth keeping even when you don’t use the orchestrator. State derived from events, not stored in status columns. Deterministic IDs for idempotency. Audit data as a first-class output of each step. None of those need Dagster to be true. They just need to be designed in from the start.
The orchestrator is a tool, not a posture. Use it when it pays for itself. Don’t, when it doesn’t.