Schedules, pipelines and CI¶
Once the model builds, three things keep it running: schedules say when each part is refreshed, pipelines for your orchestrator run them, and CI checks every change before it lands. Murakumo generates all three from the project, and can also export it as plain SQL for teams that do not use dbt, and pipeline docs for everyone else.
Schedules¶
Schedules live in murakumo.yaml:
orchestration:
timezone: UTC # any IANA name: Europe/London, America/New_York, ...
schedules:
- name: hourly_sales
cron: "0 * * * *" # minute hour day month weekday
select: [+domain:sales]
description: Order lines and customers, every hour
- name: nightly
cron: "0 6 * * 1-5" # no select: everything the other schedules leave out
A schedule's select picks tables:
| Write | Picks |
|---|---|
domain:sales |
every table in the sales domain |
layer:marts |
every table in that layer (raw, staging, intermediate, marts, semantic) |
source:tpch_erp |
every raw table of that source |
fct_orders |
one table |
+fct_orders |
that table and everything it is built from |
tpch_erp+ |
... and everything built from it |
One schedule may leave select out: it refreshes whatever no other schedule picks, so nothing is
forgotten. With no schedules at all, everything is refreshed once a day at 06:00.
murakumo schedules # each schedule, in words, with its next run
murakumo add schedule hourly_sales --cron "0 * * * *" --select +domain:sales
murakumo rename schedule hourly_sales sales_hourly
murakumo remove schedule sales_hourly
murakumo schedules --timezone Europe/London
murakumo build --schedule hourly_sales # run one schedule by hand
murakumo scheduler # run every schedule here, at its times
Running the schedules on this computer¶
An orchestrator runs the schedules in production. Before that, or for a project that lives on one
machine, murakumo scheduler does the same here: it sleeps until the next schedule is due, builds
what it refreshes (murakumo build --schedule NAME), and waits for the next, until stopped
(Ctrl+C). Each run is recorded in the run log with its schedule, so murakumo runs and the canvas
show when each last ran. --env dev builds an environment instead of your own schema;
--data TABLE=FOLDER loads the new files in a folder before each run. In the canvas, the Schedules
panel's Run them here starts it (beside any other job) and Stop stops it.
Cron times take *, numbers, ranges (1-5), steps (*/15), lists (6,18), month and day names
(jan, mon) and @hourly, @daily, @weekly, @monthly. murakumo validate checks them,
and checks that every selection names something that exists (MK601); a name used twice (MK602),
two schedules that both refresh "the rest" (MK603), and tables no schedule refreshes (MK604, a
warning) are reported too. The canvas's Schedules panel edits the same list, says each time in
words with its next run, and highlights the tables a schedule refreshes.
Each schedule becomes a dbt selector in generated/dbt/selectors.yml: its models carry a tag
(schedule_hourly_sales) and the selector selects that tag, so dbt build --selector hourly_sales
runs exactly that schedule. A schedule runs the tests on the tables it builds, including
relationship tests between two of them; the schedule that refreshes the rest also runs every
relationship test, so a test between tables in different schedules still runs once a day.
Pipelines¶
murakumo generate writes pipelines for the orchestrator in murakumo.yaml (generation.orchestrator,
or murakumo config --orchestrator) to generated/orchestration/<orchestrator>/, one per schedule,
with a README on deploying them:
| Orchestrator | Generated | Each run |
|---|---|---|
| Airflow, Cloud Composer | dags/<project>__<schedule>.py, requirements.txt |
source freshness, then the selector through Cosmos, one task per model |
| Dagster | definitions.py, requirements.txt |
the schedule's models as dbt assets, with their tests as asset checks |
| Prefect | flows.py, prefect.yaml |
a flow per schedule; deployments on its cron time |
| Databricks Workflows | databricks.yml (an asset bundle) |
a dbt task on a SQL warehouse; schedules deploy paused |
| Snowflake Tasks | tasks.sql |
a task per schedule running the dbt project deployed in Snowflake |
| AWS Step Functions | main.tf, state_machine.asl.json, Dockerfile, run.sh |
A state machine runs the schedule as a Fargate task; an EventBridge Scheduler schedule per schedule starts it |
| AWS Glue | main.tf, glue_dbt.py, run.sh |
A Python shell job runs dbt from a zip of the project in S3; a scheduled trigger per schedule, in a workflow |
| Google Cloud Workflows | main.tf, workflow.yaml, Dockerfile, run.sh |
A workflow runs the schedule as a Cloud Run job and waits; a Cloud Scheduler job per schedule starts it |
| Azure Data Factory | pipeline.json, triggers/, Dockerfile, run.sh |
A pipeline starts a Container Apps job, waits, and fails when the build fails; a schedule trigger per schedule |
Pipelines find the dbt project at generated/dbt, beside them; set MURAKUMO_DBT_DIR if you copy
them elsewhere, and MURAKUMO_TARGET to build another environment than prod. Warehouse
credentials come from the same environment variables as profiles.yml. Freshness is checked for
sources that say how fresh they should be (freshness on the source or table).
Fetching and loading first¶
When sources have a connection, the Airflow, Dagster and Prefect pipelines fetch and load before they build, for the raw tables each schedule reads (stack.md):
- With
stack.extract: python, one extract step per source (python generated/ingestion/python/extract.py sources/<source>.json). - With
stack.load: build, a load step:murakumo load --env <target>. - Then the dbt build, as before.
The pipelines find the project through MURAKUMO_PROJECT_DIR when copied elsewhere. Lambda and Glue
fetch on their own schedules, and Snowpipe and Auto Loader load continuously, so with those the
pipelines skip the steps that are done elsewhere.
The Airflow, Dagster and Prefect pipelines have been run for real on DuckDB with the commerce
example. To try an Airflow DAG without its scheduler, run airflow dags test <dag_id> with
AIRFLOW__COSMOS__ENABLE_DATASET_ALIAS=False: on Airflow 2.10, dags test cannot save Cosmos's
dataset aliases (scheduled runs can).
Plain SQL¶
The project as scripts that need nothing but the warehouse: 00_landing.sql (schemas and landing
tables), then one numbered script per model in the order they must run, and tests/<model>.sql,
where each query returns the rows that fail a test. The scripts are the same SQL as the dbt
project with dbt's parts written out: views and tables are re-created; an incremental model is
built in full once, then adds the rows past its watermark, replacing those with the same key; an
SCD2 snapshot closes the current version of each changed row and adds a new one, with
dbt_valid_from and dbt_valid_to. Run them in order in one session.
On the same data, the plain SQL builds the same tables as dbt, row for row, on a first and a second
run: Murakumo's tests check this on DuckDB. Set plain_sql: true under generation in
murakumo.yaml to keep the scripts in generated/sql/<warehouse>/, written and checked with
everything else.
Pipeline docs¶
murakumo generate writes generated/docs/index.html, one page that needs nothing else, and
generated/docs/<table>.md for reading in the repository. They cover the domains and their
owners, the schedules and what each refreshes, the sources, and for every table: its description,
grain, load, schedule, the tables it is built from and feeds, and each column with its type, keys,
PII and tests, where it comes from (with its expression) and what uses it. murakumo docs opens the
page as the model is now; so does Docs in the canvas. The docs contain no dates, so they
change only when the project does.
CI¶
murakumo ci # .github/workflows/murakumo.yml at the repository root
murakumo ci --provider gitlab # .gitlab-ci.yml
murakumo ci --deploy # also deploy each environment when its branch changes
murakumo ci --dry-run # print it instead
murakumo ci --run # run the same checks here
The workflow runs on every pull request, and on every push to an environment's branch (dev,
stage, main), in the project's folder:
murakumo validate --strict: the model has no errors or warnings;murakumo generate --check:generated/matches the model;murakumo migrate --check: every table change has its migration;murakumo verify: the dbt project builds and its tests pass, on DuckDB with made-up rows.
With --deploy, a job per environment deploys it after the checks pass, when its branch changes:
murakumo migrate --apply --env dev, then murakumo build --env dev on a push to dev, and so
on up to prod from main. On GitHub each job uses a GitHub environment of the same name (add
required reviewers to prod under Settings → Environments, so each production deploy waits for
approval); on GitLab, prod is a manual job and the others run on their own. The warehouse
credentials come from the CI's secrets (SNOWFLAKE_ACCOUNT, SNOWFLAKE_USER and
SNOWFLAKE_PRIVATE_KEY, the key's contents, for Snowflake), and each environment's role from
SNOWFLAKE_ROLE_DEV, SNOWFLAKE_ROLE_PROD... (murakumo env provision creates those roles). A
project that builds into DuckDB files has nothing to deploy to.
On every pull request, steps before the checks make sure it follows the git flow (murakumo
branch check: new work into dev, dev into stage, stage into main) and write the model
changes and their impact (murakumo diff --format markdown) to the run's summary page; see
teamwork.md. Branch names reach those steps as variables, never written into the
script, since whoever opens a pull request chooses its branch's name. --preview adds a job that
builds each pull request on a clone of the environment it goes into, with its new migrations, and
drops it when the pull request closes (Snowflake).
The file is yours once written. If it differs from what Murakumo would write, murakumo ci shows
the difference and changes nothing; --force replaces it. The canvas's CI panel does the
same, and its Run checks now runs the four checks on your computer, showing each as it
passes or fails.