Murakumo against established practice¶
A review of Murakumo against the practices data teams rely on, from four sources: Kimball's dimensional modelling, dbt's own guidance, Snowflake's, and the principles in Designing Data-Intensive Applications. Each practice says what Murakumo does about it today (0.9) and, where it falls short, what would close the gap. The review works from the published practices; it does not quote the books.
Done means Murakumo does it for you. Partly means it can be done, by hand or only in some cases. Not yet is a gap, with the milestone it fits.
Dimensional modelling (Kimball)¶
| Practice | Murakumo | Status |
|---|---|---|
| Declare each fact table's grain before its columns | grain: on an entity; tested as a unique combination; the canvas shows it |
Done |
| Surrogate keys for dimensions | Hash keys (modelling.surrogate_keys: hash), computed from the natural key |
Done |
| Slowly changing dimensions (type 1, 2) | history: scd1 | scd2 per entity; SCD2 by dbt snapshot, with valid_from, valid_to, is_current |
Done |
| Types 3, 4 and 6 | Type 3 keeps each tracked column's previous value beside it; type 4 puts the current version in the dimension and every version in <name>_history; type 6 is type 2 with each tracked column's current value on every version (1.0) |
Done |
| Role-playing dimensions (pickup and drop-off zone) | Relationships carry a role; one dimension joined twice |
Done |
| Conformed dimensions shared across facts and domains | conformed: true on a dimension; a fact using another domain's dimension that is not conformed is reported (MK316), and so is a copy of a conformed one (MK317) (0.9) |
Done |
| Bus matrix (which facts use which dimensions) | Drawn from the relationships, in the replication guide and the canvas's Bus matrix panel (0.9) | Done |
| Fact table types: transaction, periodic snapshot, accumulating snapshot | Transaction facts; cumulative tables (a periodic, per-key history); accumulating snapshots, a row per key that keeps each milestone's date and the time between them (0.9) | Done |
| Late-arriving dimensions: an "unknown" member, so facts never lose rows | An unknown (-1) row in each dimension, and facts point a missing key at it (modelling.unknown_members, on for new projects; per dimension too) (0.9). A key that is present but not yet in the dimension is still reported by the relationship test |
Done |
| Late-arriving facts | Picked up by load time since 0.7 (incremental_by: load_time), with a look-back |
Done |
| Degenerate dimensions (an order number on the fact) | A fact column with no dimension behind it | Done |
| Junk dimensions (flags gathered into one small dimension) | Not generated | Not yet |
| An audit dimension: which load and run each row came from | _ingested_at, _batch_id on raw rows; _loaded_at, _updated_at on modelled ones; the run log |
Done (as columns rather than a dimension) |
| Data Vault (hubs, links, satellites) | Hubs, links and satellites, insert-only, with hash keys of the trimmed, upper-cased business keys, load_date and record_source; a satellite adds a row when its hashdiff changes, and <satellite>_current has the latest per key (1.0) |
Done |
dbt¶
| Practice | Murakumo | Status |
|---|---|---|
| Staging, intermediate and marts layers, each with its job | The model's layers; staging generated one-to-one from each raw table | Done |
| Sources declared, with freshness | sources.yml per source, loaded_at_field: _ingested_at, freshness from the model |
Done |
| Every model tested: unique and not-null keys, relationships, accepted values | Generated from keys, relationships and column rules | Done |
| Model contracts on what others read | Enforced on marts (contracts.enforce_on); every table's contract written in ODCS v3.0.2, contracts from others imported, each load checked against its source's contract (1.1) |
Done |
| Bad rows quarantined, not dropped or let through | Per rule: stop, warn or set rows aside in <staging>__quarantine with the rule they failed; reconciliation that every raw row is accounted for (1.1) |
Done |
| Write-audit-publish | quality.write_audit_publish: build and test in wap_ schemas, publish only when every test passes (1.1) |
Done |
| One set of rules, run by the tool a team uses | The same rules as dbt tests, PySpark and Polars checks, Great Expectations suites and Soda checks (1.1) | Done |
| Groups and access (private, protected, public) | A dbt group per domain, and access: per table (0.9) |
Done |
| Incremental models that are idempotent | Replaced by unique key (delete+insert, or merge on BigQuery and Databricks); load-time increments re-run safely |
Done |
| Snapshots for history | Generated for SCD2 entities | Done |
| Exposures (dashboards and apps that read the marts) | model/exposures.yaml, written into dbt, lineage and the guide; edited in the canvas (0.9) |
Done |
| Semantic layer (metrics defined once) | dbt semantic models and metrics (MetricFlow) from model/metrics.yaml: simple, derived and filtered metrics, SCD2 dimensions joined by validity, and the time spine (1.0) |
Done |
Slim CI: build only what changed (state:modified+ --defer) |
murakumo verify --changed-since <branch> builds what the model's changes reach, by lineage; the generated CI uses it on pull requests (1.0). Previews on a warehouse still build the whole project |
Done |
| Documentation that stays current | murakumo docs and the pipeline docs, from the model |
Done |
| One project, one source of truth, no hand edits to generated code | generated/ written from the model; custom/ for hand-written code; generate --check in CI |
Done |
Snowflake¶
| Practice | Murakumo | Status |
|---|---|---|
| A database per environment | Since 0.7: TAXI_DEV, TAXI_STAGE, TAXI_PROD, and MURAKUMO for people's own schemas |
Done |
| Role-based access: a role per job, granted to people and services | murakumo env provision: a deployer role per environment, a developer role |
Done |
| Functional and access roles kept apart | Access roles per database (<DB>_READ, <DB>_WRITE) hold the privileges; the developer, deployer and unmasked roles are made of them, in the provisioning SQL and in Terraform (1.0) |
Done |
| Key-pair authentication for services | The profiles and CI use key pairs; keys stay in files and secrets | Done |
| Warehouses that suspend themselves | Provisioning creates an x-small warehouse that suspends after 60 seconds | Done |
| Resource monitors to cap spend | targets.warehouse.monthly_credits: provisioning creates a monitor that notifies at 75% and 90% and suspends at 100% (0.8) |
Done |
| Zero-copy clones for testing | Pull-request previews and copies into your own schema clone | Done |
| Time travel and fail-safe settings per layer | targets.warehouse.time_travel_days per layer, set on raw tables and by dbt after each build; MK703 when a transient layer asks for more than a day (0.8) |
Done |
| Transient tables for what can be rebuilt (staging, intermediate) | transient_layers (staging and intermediate by default); marts and snapshots kept permanent, where dbt alone would make every table transient (0.8) |
Done |
| Clustering keys on large tables | cluster_by: on an entity |
Done |
| Masking and row-access policies for personal data | Masking policies from PII and classification (governance.masking), set after each build, with the roles that see the real values (0.9). Row-access policies from row_access: on a table: which values of a column each role sees (1.0) |
Done |
| Loading with COPY and Snowpipe | COPY for files from the build; with stack.load: snowpipe, a pipe per raw table from an S3 stage through a storage integration (0.8) |
Done |
Designing data-intensive applications¶
| Principle | Murakumo | Status |
|---|---|---|
| Keep raw data immutable; derive everything else from it | Raw tables append (or merge, as declared); every other table is rebuilt from them | Done |
| Make every step idempotent, so it can safely run again | Loads remember files by content; incremental models replace by key; backfills re-run days | Done |
| Event time is not processing time: late data happens | Increments by load time, with a look-back; cumulative tables by event day | Done |
| Reprocessing: rebuild derived data from the raw history | build from scratch, backfill by day; full refresh by dropping a table |
Done |
| Evolve schemas compatibly; know what a change breaks | Typed diffs and impact reports; migrations planned and applied per environment | Done |
| Exactly-once results without distributed transactions | Through idempotence (above), not transactions | Done |
| Lineage: know where every value comes from | Column lineage in the model, the canvas and the docs | Done |
| Validate at the edges: reject or quarantine bad input | Rejected rows counted with the reason; raw columns can be kept as text and converted in staging | Done |
| Backpressure and streaming | Kafka sources, read by the extract script on each schedule, or continuously by Flink, Spark Structured Streaming or Snowpipe Streaming (1.0). Backpressure is the engine's own | Partly: no exactly-once into the warehouse |
| Observability: know what ran, what it did, and when it failed | The run log: runs, models, tests, rows in and rejected, by schedule | Done |
What this says about the roadmap¶
The gaps fell in three groups:
- Platform settings, closed in 0.8 with the stack choice:
- resource monitors
- retention and time travel per layer
- transient tables for rebuildable layers
- Snowpipe
- Modelling patterns, closed in 0.9 with the replication guide:
- unknown members for late-arriving dimensions
- accumulating snapshots
- conformed dimensions and the bus matrix
- access levels
- exposures
- masking policies (row-access policies are still to come)
- Scale, closed in 1.0: Slim CI that builds only what a change reaches.