Not every data pipeline needs Airflow. For smaller projects — two or three ingestion sources, one transformation step, a predictable schedule — a full orchestration cluster is a maintenance tax you don't need to pay.
Here's a pattern we've shipped repeatedly using only managed GCP services. Nothing to patch, nothing to cluster-manage, still observable and reliable.
The stack at a glance
Four pieces, all managed, all serverless.
- Cloud Run Functions for the ingestion jobs themselves.
- Google Cloud Workflows to sequence them.
- dbt Cloud for the transformation step.
- Cloud Scheduler to trigger the workflow on cron.
Execution logs, step timings, and failure alerts are all available in the Cloud Console by default.
Part 1 — Ingestion in Cloud Run Functions
Each source gets its own stateless HTTP function. The function does three things:
- Read the latest "watermark" timestamp from the target BigQuery table.
- Call the source API for records changed since that watermark.
- Insert the new records into BigQuery.
The watermark pattern keeps syncs incremental. You only fetch what's changed, you never duplicate, and if the pipeline breaks, you resume from the last successful timestamp automatically.
In practice we build one function per domain — sync_clients, sync_sales — each with its own staleness tolerance. A client-records function might alert if data is older than a day; a sales-records function might tolerate five days because of legitimate reporting lag.
Part 2 — Orchestration in Google Cloud Workflows
Workflows is a YAML-defined, fully managed orchestration service. The DAG is tiny by Airflow standards — a few sequential steps, some try/except blocks, a final notification — but that's the point. You don't need Airflow for this shape of work.
The full workflow typically looks like:
- sync_clients # HTTP call to Cloud Run Function
- sync_sales # HTTP call to Cloud Run Function
- trigger_dbt_cloud # HTTP call to dbt Cloud API
- notify_success # Teams / Slack webhookEach step has its own error handler. A failure in any step routes to a single alerting endpoint so on-call gets one Teams or Slack message, not a confusing cascade.
Why not Airflow?
For this size of pipeline, Airflow is overhead. You pay for the scheduler, the executor, the metadata database, and you pay in time every time you upgrade. Cloud Workflows gives you sequential execution, try/except, retries, and audit trails without any of that.
If your pipeline has dynamic task generation, parallel interdependent branches, or deep DAGs — that's when Airflow earns its keep. This pattern is for the 80% of pipelines that don't.
Part 3 — Scheduling
Cloud Scheduler runs on cron and hits the workflow's HTTP trigger. That's the entire schedule layer. No workers to run, no cluster to monitor.
What you get
| Concern | Handled by |
|---|---|
| Incremental ingestion | Watermark pattern in Cloud Run Function |
| Step sequencing | Cloud Workflows YAML |
| Error handling | Try/except per step, single Teams alert |
| Observability | Cloud Console step logs and timings |
| Transformation | dbt Cloud via REST API |
| Scheduling | Cloud Scheduler cron |
All of it managed. Nothing to patch.
When this pattern works — and when it doesn't
Works well when pipelines have a small number of clearly-named sources, predictable runtimes, and linear dependencies.
Doesn't work when you need dynamic task generation, fan-out/fan-in with real coupling, or compute-heavy transformations that don't belong in dbt. At that point, Airflow on MWAA — or Databricks Workflows — starts earning its keep.
What to do next
If your current pipeline is a handful of VM cron jobs or a manually-maintained Airflow install that nobody loves, this pattern is a very real simplification. We've migrated teams onto it in under two weeks. If that sounds like what you need, book a discovery call — we'll tell you if Workflows is the right fit or if you need something heavier.