Skip to content
dbprint

dbprint format specification — v1

dbprint is a portable on-disk format for "what this database actually looks like" — DDL, statistics, relationships, and human descriptions, designed for offline consumption by humans and AI agents alike.

This document specifies the v1 format. Any tool that produces or consumes the format MUST comply with the requirements below. Conformance MUST be mechanically verifiable. The reference JSON Schemas SHALL be located at spec/v1/. The reference pytest-based conformance suite SHALL be located at tests/conformance/.

0. Scope and terminology

0.1 What this spec covers

  • On-disk directory layout for a "print" of one or more databases.
  • The schema and field semantics of each artifact (DDL, statistics, relationships, description, manifest, diff).
  • Column classifications and inferred semantics.
  • What an absent field, block or file means, collected for a reader (§7).
  • Versioning and compatibility rules.
  • Conformance criteria.

0.2 What this spec does NOT cover

  • Producer implementation details (how a producer connects to a database, computes statistics, infers relationships).
  • Consumer behavior (how a consumer renders, queries, or assembles fragments from the format).
  • The dbprint CLI surface — that's documented separately.

0.3 Terminology

  • Producer: any tool that writes dbprint-format artifacts. The dbprint CLI is one producer; future producers may include dbt, Atlas, IDE integrations.
  • Consumer: any tool that reads dbprint-format artifacts. AI coding agents, drift dashboards, CI gates, the dbprint context command itself.
  • Print: a directory tree containing artifacts for one or more tables in a database connection.
  • Connection: a named scope inside a project, typically mapping 1:1 to a database/warehouse credential set.

0.4 Requirement levels

This document uses MUST, MUST NOT, SHOULD, SHOULD NOT, and MAY per RFC 2119.

1. Directory layout

1.1 Project root

A project root contains:

  • A .dbprint.yaml config file (project-level).
  • A prints/ directory containing one subdirectory per connection.

1.2 Connection root

prints/<connection_name>/ contains:

  • manifest.yaml (REQUIRED) — index over all per-table artifacts.
  • diff.yaml (REQUIRED once manifest.yaml records any table) — latest structured diff.
  • reading.md (REQUIRED) — generated consumer guide (§1.2.1).
  • manifest.annotations.yaml (OPTIONAL; user-authored; §2.7.3) — connection-grain human notes.
  • One or more namespace path directories below.

1.2.1 The consumer guide

reading.md teaches a consumer how to read the rest of the print — not the format grammar (this document already does that), but which fields to trust for what, and the questions a reader can answer without querying the database. It is a generated file: a producer MUST overwrite it in full on every generate, byte-identical run to run unless the producer's own generator version has changed. A consumer MUST NOT hand-edit it — the next generate discards any edit without warning, the same terms manifest.yaml and diff.yaml are held to.

1.3 Namespace path

Per-adapter hierarchy:

AdapterPath shapeExample
Snowflake<database>/<schema>/arboretum/seedbank/
duckdb<database>/<schema>/arboretum/seedbank/
PostgreSQL<database>/<schema>/arboretum/seedbank/
Redshift<database>/<schema>/arboretum/seedbank/
Databricks<catalog>/<schema>/arboretum/seedbank/
MySQL<database>/arboretum/
ClickHouse<database>/arboretum/
BigQuery<dataset>/seedbank/

The number of segments is the adapter's, not the format's; a table's own table field (§2.2.1) carries the same shape as its path.

A table's FQN is its namespace path's segments and its own name joined by ., in path order: arboretum/seedbank/accession/ is arboretum.seedbank.accession. Because no segment contains . (§1.5.1), a manifest key names exactly one directory and a directory exactly one key. An FQN naming an object outside the print — a target_table, referencer_table or depends_on entry absent from manifest.tables — is that object's folded parts joined by ., with no allowlist applied. A consumer MUST treat such an FQN as opaque and MUST NOT derive segments or a path from it.

Path segments MUST be lowercase. Producers MUST lowercase identifier segments when constructing paths. SQL identifier content inside .sql files MAY remain in native case (e.g., Snowflake's GET_DDL typically emits UPPERCASE).

1.4 Per-table directory

Each table/view/matview is a directory at the leaf of its namespace path. The directory contains the table's artifacts:

<namespace_path>/<object_name>/
├── ddl.sql                        (REQUIRED for all object types)
├── statistics.yaml                (REQUIRED for all object types; catalog-only where no query was issued, §2.2.15)
├── relationships.yaml             (REQUIRED for tables and matviews; MAY be absent for plain views)
├── description.md                 (OPTIONAL; user-authored)
├── statistics.annotations.yaml    (OPTIONAL; user-authored; §2.7.1)
└── relationships.annotations.yaml (OPTIONAL; user-authored; §2.7.2)

§7.3 collects what each of these files means when it is absent.

When a table leaves the manifest, a producer removes the producer-written files in its directory (ddl.sql, statistics.yaml, relationships.yaml) and leaves the user-authored ones in place; a directory holding only user-authored files stays where it is, and a table that returns under the same path picks them up again.

1.5 Identifier handling

The format maintains a 1:1 bijection between SQL identifiers and filesystem path segments, modulo case folding. Producers MUST reject identifiers that would break this bijection rather than escape, hash, or otherwise transform them.

1.5.1 Character allowlist

After lowercase normalization (§1.3), a path segment MUST match the regex:

^[a-z0-9_][a-z0-9_-]*$

That is:

  • First character: ASCII lowercase letter, digit, or underscore ([a-z0-9_])
  • Subsequent characters: those plus hyphen (-)
  • Allowed character set (ASCII only): a-z, 0-9, _, -

Producers MUST reject identifiers that lowercase to a path segment not matching this regex, including:

  • All ASCII non-alphanumerics except _, -
  • All Unicode (BMP and beyond)
  • All control characters (0x00–0x1F)
  • All whitespace (space, tab, newline, etc.)
  • Any .: it is the FQN separator (§1.3), and a segment containing it would make one FQN name two objects

Leading hyphen (-taxon) and leading digit (9taxon) are allowed.

dbprint identifiers are ASCII-only. Production databases overwhelmingly use ASCII identifiers; Unicode is exotic and adds cross-platform / git / search complexity for little AI-context utility gain.

1.5.2 Case-collision detection

At scan time, after applying the allowlist check, producers MUST compute the lowercased path for every matched identifier and reject the run when two distinct SQL identifiers produce the same lowercased path.

Example: a Snowflake schema with both quoted "Taxon" (stored as-is) and unquoted TAXON (stored UPPERCASE) → both lowercase to taxon. Rejected before any file is written.

Rare in practice but the format's bijection invariant depends on it.

Column grain. Two columns of one table whose names lowercase to the same columns map key (§2.2.1) cannot both be represented. A producer MUST NOT publish that table's statistics with either column dropped, merged, or keyed under another spelling; it MUST refuse the table with the §1.5.5 message at column grain. The collision is confined to one table's file, so the producer refuses that table alone and continues with the rest — unlike a table collision, which breaks the layout and so rejects the run.

1.5.3 No length enforcement

dbprint does not enforce a maximum identifier length. Filesystem natural limits apply (typically 255 bytes per segment on Linux ext4 / macOS APFS / Windows NTFS; Snowflake's 255-char identifier limit matches). Producers SHOULD NOT pre-emptively reject on length grounds.

Long total paths may exceed Windows' default 260-char path limit. Documented as a known cross-platform consideration but not actively rejected.

1.5.4 Why reject rather than escape or hash

Escape (e.g., URL-encode weird name → weird%20name):

  • Pollutes git diffs with %XX sequences
  • Loses greppability — grep -r 'weird name' prints/ doesn't find it
  • Breaks the "path == identifier" mental model
  • Path becomes ambiguous: encoded taxon%20v2 vs a literal SQL identifier named taxon%20v2
  • Quoting a dotted segment inside the FQN (nursery."beds.v2") would force a quote-aware parser on every consumer and every selector

Hash (e.g., weird name → weird_a3f9b/):

  • Loses identity — ls shows hashes, not names
  • Hash collision risk (low but non-zero for short hashes)
  • Renames trigger cascading filesystem changes even when the identifier didn't change

Reject + exclude: selectors as the escape hatch:

  • Preserves the 1:1 bijection between SQL identifier and filesystem path (modulo case folding)
  • Forces the user to make an explicit choice ("this table exists; leave it unprofiled")
  • Preserves human readability and greppability
  • Real-world rare — production warehouses almost never have weird identifiers
  • Error message is actionable: rename in DB, OR add an exclude: pattern

1.5.5 Error message contract

Producers MUST surface rejection with:

ERROR: Table identifier rejected: <FQN>
  Reason: <reason-code>
  Detail: <character / collision-target / specific cause>
  Resolution: Either rename the identifier in the database, OR exclude it in .dbprint.yaml:
    tables:
      exclude:
        - "<pattern that matches this identifier>"

When the committed print still holds the refused table — written by a release that accepted its name — the message adds one line naming that directory, since excluding the table carries its old print forward instead of removing it:

  Stale print: <directory>/ - after excluding the table, delete this directory; the next generate drops its entry

At column grain the header names the table's FQN and the column's map key, the reason names the other catalog spelling, and the detail names this column's:

ERROR: Column identifier rejected: <FQN>.<map key>
  Reason: case-collides-with-<other catalog spelling>
  Detail: <this column's catalog spelling>
  Resolution: Either rename one of the columns in the database, OR exclude the table in .dbprint.yaml:
    tables:
      exclude:
        - "<pattern that matches the table>"

Reason codes:

  • contains-unsafe-character — identifier contains a disallowed character
  • contains-period — identifier lowercases to a segment containing ., the FQN separator
  • case-collides-with-<other-FQN> — two SQL identifiers produce the same lowercased path

Producers MUST exit with code 1 (generic error / configuration issue) on a table-grain identifier rejection. Not code 4 (connection error) — this is a config/scoping problem, not a DB-availability issue. A column-grain rejection fails its table like any table the producer could not profile.

2. Artifacts

2.1 ddl.sql

Native-dialect SQL DDL for the object — the canonical SQL representation of the object's schema-shape, post-normalization for diff stability.

This file is NOT versioned with a format_version header — it's SQL, and the manifest entry carries the format reference.

2.1.1 Per-adapter source SQL

AdapterObject typeSource
SnowflaketableSELECT GET_DDL('TABLE', '<fqn>')
Snowflakeview / matviewSELECT GET_DDL('VIEW', '<fqn>')
PostgreSQLtablepg_dump --schema-only --table=<fqn> <db> (shell-out)
PostgreSQLviewpg_dump --schema-only --table=<fqn> <db>
MySQLtableSHOW CREATE TABLE <fqn>
MySQLviewSHOW CREATE VIEW <fqn>
duckdbanySELECT sql FROM duckdb_tables(), UNION ALL the same over duckdb_views()
ClickHouseanySELECT create_table_query FROM system.tables
RedshifttableSHOW TABLE <fqn>
RedshiftviewSHOW VIEW <fqn>, the fallback when SHOW TABLE returns nothing or errors
DatabricksanySHOW CREATE TABLE <fqn>
BigQueryanySELECT ddl FROM <dataset>.INFORMATION_SCHEMA.TABLES

Postgres uses shell-out to pg_dump. Reconstruction from pg_catalog is not implemented.

2.1.2 Normalization philosophy

ddl.sql holds the text the source command in §2.1.1 returned, modified only by the strip list and the credential masks in §2.1.3. Everything else about the file — keyword casing, indentation, line breaks, statement style, quoting — is the database's own, which is why it differs from one adapter to the next.

dbprint applies light normalization: strip well-defined adapter noise to enable diff stability, preserve everything semantically meaningful, keep adapter idioms intact. Producers MUST NOT reflow / canonicalize / reorder / reformat the SQL — the DDL stays in the adapter's native style. The strip list and the credential masks (§2.1.3) are exhaustive; producers MUST NOT strip or mask more, MUST NOT strip or mask less. A mask is the one change inside a statement: it replaces the characters of a secret and keeps the statement whole.

2.1.3 What MUST be stripped (per adapter)

PostgreSQL (pg_dump output):

  • Leading header block (-- PostgreSQL database dump, -- Dumped from database version X.Y, -- Dumped by pg_dump version A.B, -- Started on TIMESTAMP)
  • Trailing footer (-- PostgreSQL database dump complete)
  • All session-control SET statements (SET statement_timeout, SET lock_timeout, SET idle_in_transaction_session_timeout, SET client_encoding, SET standard_conforming_strings, SET check_function_bodies, SET xmloption, SET client_min_messages, SET row_security, SET search_path)
  • SELECT pg_catalog.set_config(...); calls
  • Per-object banner comments (-- Name: X; Type: Y; Schema: Z; Owner: -) pg_dump emits ahead of every statement — the banner never carries information the statement immediately below it doesn't already state, and stripping it does not touch the statement itself, including COMMENT ON (see 2.1.4)
  • GRANT / REVOKE statements (access-control, not schema)
  • CREATE TRIGGER, CREATE RULE (behavior, not schema-shape — out of scope for DDL)
  • \restrict / \unrestrict psql meta-commands — client directives, not schema, and their token is regenerated on every invocation, so keeping them would break the §2.1.6 diff-stability guarantee
  • Multiple consecutive blank lines collapsed to a single blank between logical statements

MySQL (SHOW CREATE TABLE output):

  • AUTO_INCREMENT=<N> clause from the table-options trailer — the counter is volatile (changes on every INSERT) and produces noise without semantic content
  • AUTO_INCREMENT keyword on COLUMN definitions is PRESERVED (it defines the auto-increment column)

ClickHouse (system.tables.create_table_query):

  • The DDL is read with the server's secret masking in force, so every secret argument of an engine or table function reads [HIDDEN]; where a session has turned format_display_secrets_in_show_and_select on, the read sets it off, and a server that refuses the setting fails that table rather than publish the secret.

Snowflake (GET_DDL output):

  • Trailing whitespace per line (minimal — Snowflake output is already clean)

All adapters:

  • Trailing whitespace per line
  • Final newline at end of file ensured (POSIX convention) — added if the adapter output lacks it
  • Credentials are masked: each of the following is replaced by the literal [HIDDEN], and nothing around it changes — the password of a URL's user part (scheme://user:password@host), wherever it appears; a value in one of the secret formats §4.4.3 recognises for credential, wherever it appears; inside a string literal, the value of a key=value, key: value or key => value pair, or of a key followed by a quoted value (password "p"), whose key names a secret — compared without case or separators, password, passwd, pwd, pw, secret, token, api_key, access_key, private_key, or a name §4.4.1 makes a credential on its own; and outside a string literal, the content of a literal following such a key and =, => or whitespace (REMOTE_PASSWORD='p', WHERE api_key = 'p', OPTIONS (password 'p')). An empty value, null, true and false are never masked, and no identifier, type or operator is rewritten. The mask is fixed text, so the file stays byte-identical across runs (§2.1.6) and a rotated secret changes nothing in it. redact rules (§2.2.9) do not reach ddl.sql: a DEFAULT or CHECK constant is schema, not a cell value. The same masks apply wherever a print repeats catalog text: sql_type, the expression of a physical_layout or merging key, and the comment_changed and column_default_changed events of diff.yaml.

An adapter named in no block above strips nothing beyond these two and masks nothing beyond the credentials above: a source that hands back a catalog column or a single recreate-DDL statement carries none of the surrounding noise a dump utility emits.

2.1.4 What MUST be preserved

  • Full CREATE TABLE / CREATE VIEW / CREATE MATERIALIZED VIEW statement(s)
  • All constraints (PK, FK, UNIQUE, CHECK, DEFAULT, NOT NULL)
  • All COMMENT ON statements (table and column comments — semantic info, distinct from the user-authored description.md artifact)
  • Native casing of identifiers (Snowflake UPPERCASE typical, Postgres lowercase typical, MySQL backtick-quoted)
  • Native quoting style (Snowflake unquoted/quoted, Postgres unquoted/quoted, MySQL backtick)
  • Engine / charset / collate clauses (MySQL — semantic table options)
  • Constraint names (adapter-native)
  • CREATE OR REPLACE (Snowflake idiom)
  • All custom SQL types referenced
  • Index definitions inline with CREATE TABLE (Postgres / MySQL inline index syntax)
  • CREATE INDEX statements appearing after CREATE TABLE in pg_dump output

2.1.5 Multi-statement DDL

pg_dump typically emits multiple statements per table: a CREATE TABLE, several ALTER TABLE ADD CONSTRAINT, optionally COMMENT ON, optionally CREATE INDEX. The ddl.sql file SHALL contain all of them in pg_dump's emission order, separated by semicolons and blank lines.

Producers MUST NOT split multi-statement DDL into multiple files. One ddl.sql per table — multi-statement when the adapter emits multiple statements.

2.1.6 Diff stability guarantee

After normalization, running dbprint generate twice on the same unchanged schema MUST produce byte-identical ddl.sql files. The strip rules in §2.1.3 are exactly what's required to make this guarantee hold against the noisy elements each adapter introduces.

This is a normative correctness criterion for any conforming producer.

Adapter-version-stable normalization (DDL stays identical across upgrades of the underlying DB engine that change pg_dump / GET_DDL output) is out of scope; that conformance enhancement is not implemented.

2.1.7 Edge cases

CaseResolution
Table has zero constraintsDDL is shorter; emit as-is post-normalization. Cross-artifact signal is empty refers_to/referenced_by in relationships.yaml.
Table with no commentsDDL has no COMMENT ON. The user-authored description.md is a separate artifact.
View with complex query bodyPreserve the full view definition. No normalization of the SELECT clause beyond §2.1.3's credential masks.
CREATE INDEX in pg_dump outputPreserved. This is the source of truth for secondary indexes (per §2.6.7).
Adapter version produces different DDL after upgradeNormalization does not stabilize across adapter version changes. Consumers compare DDL files as opaque strings; cross-version stability is not implemented.
GRANT / REVOKE / CREATE TRIGGER / CREATE RULE in pg_dumpStripped — not schema-shape. dbprint captures schema, not behavior or access control.

2.2 statistics.yaml

Column-level statistics designed for SQL-writing utility. Every field is one of:

  1. Directly usable by an LLM in writing a query (e.g., distinct values for WHERE).
  2. A signal flag that changes the consumer's approach (e.g., inferred.candidate_key: true).

The reference JSON Schema SHALL be at spec/v1/statistics.schema.json. The normative content below is authoritative; the JSON Schema is its mechanical companion.

2.2.1 Top-level fields

FieldTypeRequiredNotes
format_versionintALWAYS1 for v1 artifacts
tablestringALWAYSFully-qualified name as its adapter reports it: db.schema.table (PostgreSQL, Snowflake), db.table (MySQL). The number of parts is the adapter's, not the format's, and selectors and directory paths follow it
typeenumALWAYStable | matview | view. A view file carries catalog_only exactly when its producer issued no query for it (§2.2.15).
profiled_atstringALWAYSISO 8601 with explicit UTC offset (2026-03-09T14:27:36Z). Producers MUST normalize to UTC; no local-time-with-offset strings.
catalog_onlyboolOPTIONALPresent and true only when no query was issued for this object at all; see §2.2.15
externalboolOPTIONALPresent and true only when the object's rows live in another system, so every query against it reads that system; see §2.2.20
row_countint ≥ 0CONDITIONALTotal rows in the table, including nulls. Not affected by scope. REQUIRED unless catalog_only is present, in which case MUST NOT be emitted; see §2.2.15
row_count_methodenumCONDITIONALexact (COUNT(*)) | approximate (system-table estimate). REQUIRED unless catalog_only is present, in which case MUST NOT be emitted; see §2.2.15
scopemapOPTIONALPresent only when the statistics describe part of the table; see §2.2.8
null_patternsmapCONDITIONALWhich columns are null on the same rows. REQUIRED when any column reports a non-zero null_count — unless the file's own unmeasured or disabled list names it — and MUST NOT be emitted otherwise. See §2.2.10
physical_layoutmapOPTIONALThe table's declared clustering/partitioning key. Absent means not clustered/partitioned, never "not checked" — unless the file's own unmeasured list names it, which is how a producer records that the read failed — every producer MUST report on this, except that a catalog_only file's absence needs no further explanation: a producer MAY still emit the block there from catalog metadata alone. See §2.2.11, §2.2.15
mergingmapOPTIONALThe table's merging engine and sorting key (ClickHouse only). Absent means the engine combines no rows, never "not checked" — unless the file's own unmeasured list names it. See §2.2.19
grainmapOPTIONALWhat identifies a row: declared keys always, plus a bounded measured probe. A conforming producer MUST emit it, keys possibly empty, never silently omitted. See §2.2.12
dependencieslistOPTIONALWhich columns determine which, measured over the scanned rows. A conforming producer MUST emit it, possibly empty, never silently omitted, unless catalog_only is present or the file's own unmeasured list names it, in which cases it MUST NOT be emitted. See §2.2.13, §2.2.15
timelinemapOPTIONALOne column's activity, bucketed over time. Absent means no eligible anchor column exists, the file carries scope, row_count is 0, or the file's disabled list names it. See §2.2.16, §2.2.21
depends_onlistCONDITIONALThe objects a view/matview's own definition reads directly, catalog-derived. ALWAYS present on a view or matview whose catalog was read, possibly empty; MUST NOT be emitted on a table. Absent on a view/matview means the dependency read did not happen. See §2.2.17
unmeasuredlistOPTIONALNames the blocks among physical_layout, merging, null_patterns and dependencies this run tried and failed to measure — the four whose absence is otherwise a positive claim (not clustered; no merging engine; no column has nulls; nothing determines anything). Each named block MUST be absent. grain is never named: it is REQUIRED, and grain.search.exhausted (§2.2.12) already says a measured search was incomplete. depends_on is never named either: its absence already means the dependency read did not happen (above). Never names a block disabled names. Omitted entirely when the run measured every block the file should carry. See §2.2.4's column-level twin
disabledlistOPTIONALNames the optional measurements this producer's configuration switched off for this object, from looks_like, null_patterns, parts, relationships.infer, sketches, timeline, sorted ascending. Every field and block a named measurement produces (§2.2.21) MUST be absent, and a reader treats each as not measured, never as absent. Omitted entirely when nothing was switched off. See §2.2.21
columnsmapALWAYSKeyed by column name; values are per-column stats objects per §§2.2.2–2.2.4

The map key is always lowercase. Producers MUST lowercase every column name for this key, on every adapter, regardless of the case the catalog reports it in — the same normalization §1.3 requires for path segments, applied here at column-name grain so detection (§4.4.3), statistics.annotations.yaml keys (§2.7.1) and this map itself agree on one spelling for a schema ported between engines. Where a column's catalog-reported spelling differs from its lowercased key, the producer MAY carry it forward as physical_name (§2.2.4) so a consumer can still address the column directly. Two columns whose names lowercase to one key are refused, never merged (§1.5.2). A part of a column (§2.2.18) is never a key here; it lives in its column's parts map.

2.2.2 Universal per-column fields

Required on every column except unsupported, and except the three cardinality fields on spatial and vector, which never carry them, and on composite, which carries them where its engine compares whole values (see §2.2.3):

FieldTypeNotes
sql_typestringNative dialect type as the database reports it (VARCHAR(64), TIMESTAMP_NTZ(9), JSONB) (§2.1.3's credential masks applied)
nullableboolFrom DDL, not inferred from data
null_countint ≥ 0Count of NULL rows in this column
null_ratefloat [0, 1]null_count / rows_scanned; 0 when rows_scanned == 0; floored/ceilinged at the two boundaries per §2.2.6
cardinalityint ≥ 0Distinct non-null values, under the collation the source compares them with — see below
cardinality_ratiofloat [0, 1]cardinality / rows_scanned; 0 when rows_scanned == 0; floored at the lower boundary per §2.2.6
cardinality_methodenumexact (COUNT(DISTINCT)) | approximate (a measurement not obtained by counting — see below)
classificationenumOne of the 12 values per §3

approximate names more than one measurement. It covers a live sketch computed over the scanned rows (APPROX_COUNT_DISTINCT, an HLL estimate this run's own read produced) and a stored planner statistic of unbounded staleness (a catalog's own last-ANALYZE estimate, taken at an unrelated time over the whole table rather than the scanned set). A consumer cannot tell which kind a given approximate is from this field alone — both name "not counted", not "counted this way". Producers MAY publish either under the same token; a producer publishing the catalog-statistic kind SHOULD re-probe a column exactly once its estimated ratio nears the §4.2 candidate-key threshold, so an estimate's own imprecision cannot cost a column its candidate_key verdict.

Distinctness is collation-relative. For a string-valued column, cardinality, the values list (§2.2.4) and distribution (§2.2.5) are computed under whatever collation the source compares that column's values with — the format states this rather than pinning one collation across engines, so the read a producer already issues is what the artifact describes. Two prints of one logical schema taken through different engines, or through the same engine under different column-level collations, can disagree on cardinality for a text column for this reason alone; the disagreement is explicable from the manifest's default_collation (§2.5) and a column's own collation (§2.2.4) when it overrides that default, never from the number by itself.

How much of that relativity is measurable, on the columns where it matters. normalized_cardinality (§2.2.4) states, for a bounded population of join-key columns, how many of the collation-relative distinctions above would disappear under a case- and whitespace-insensitive read: the gap between it and cardinality is the count of values that would merge across a differently-collated system. A column where the two counts agree is portable across collations; a gap on a join key is exactly the number of ways a join to one can silently miss.

2.2.3 Required / optional / forbidden field matrix per classification

Cell legend: R = REQUIRED, O = OPTIONAL (emit if applicable), — = MUST NOT emit, R†, R‡, R¶, R‖, R※ and R◊ = REQUIRED unless the marked footnote's condition holds, under which the field MUST NOT be emitted (a cell carrying two markers, e.g. R¶‖, is REQUIRED only while both conditions are unmet), O¶ = OPTIONAL unless the ¶ footnote's condition holds, under which it MUST NOT be emitted, R (scoped) = REQUIRED when the file's top-level scope block is present (§2.2.8), MUST NOT emit otherwise — uniform across every classification, unlike the per-column conditions above.

Fieldunsupportedbooleanspatialvectorjsoncompositeforeign_key_candidatecategoricaltemporalnumericbinarytext
sql_typeRRRRRRRRRRRR
nullableRRRRRRRRRRRR
null_countRRRRRRRRRRRR
null_rateRRRRRRRRRRRR
cardinality—R——RORRRRRR
cardinality_ratio—R——RORRRRRR
cardinality_method—R——RORRRRRR
classificationRRRRRRRRRRRR
physical_name (§2.2.4)OOOOOOOOOOOO
collation (§2.2.4)OOOOOOOOOOOO
physical_layout_key (§2.2.11)OOOOOOOOOOOO
derived_from (§2.2.4)OOOOOOOOOOOO
rows_scanned (§2.2.8)R (scoped)R (scoped)R (scoped)R (scoped)R (scoped)R (scoped)R (scoped)R (scoped)R (scoped)R (scoped)R (scoped)R (scoped)
populated (§2.2.4)OOOOOOOOOOOO
inferred.looks_like——————OO——OO
inferred.sampled (§4.1.3)——————OO——OO
inferred.matched (§4.1.3)——————OO——OO
inferred.looks_like_candidate (§4.1.3)——————OO——OO
inferred.looks_like_candidate_share (§4.1.3)——————OO——OO
inferred.sensitivity—OOOOOOOOOOO
inferred.epoch_unit (§4.5)——————OO—O—O
redacted—OO———OOOOOO
inferred.candidate_key (§4.2)—O——OOOOOOOO
inferred.candidate_key_exception (§4.2)—O——OOOOOOOO
inferred.fk_candidate (reserved; see §4.3)——————O—————
values—R————RRRR—R‡
values_coverage—R————RR———R‡
values_coverage_method (§2.2.4)—O————OO———O
distribution——————RRRR—R‡
frequencies (§2.2.4)————————RR——
range (min, max)————————R†R†——
range.span_days————————R†———
percentiles————————R†R†——
mean—————————R¶——
sum—————————R¶——
zero_count———R—O———R¶——
negative_count—————————R¶——
non_finite_count (§2.2.4)—————————O¶——
empty_count—————O————R¶R¶
quantized_count (§2.2.4)————————R¶※R¶——
length (§2.2.4)——————R¶‖R¶‖——R¶R¶
normalized_cardinality (§2.2.4)——————OO———O
freshness————————R———
unrepresentable————————O———
unmeasured (§2.2.4)OOOOOOOOOOOO
sketch (§2.2.14)—O————OOOOOO
geometry (§2.2.4)——R—————————
extent (§2.2.4)——R¶◊—————————
dimension (§2.2.4)———R◊————————
norm (§2.2.4)———R◊—O——————
parts (§2.2.18)————OR——————
parts_found (§2.2.18)————OR——————
size (§2.2.18)————OO——————
types (§2.2.4)————O———————
occurrences (§2.2.18)————————————

Producers MUST emit exactly the fields marked R, except where a field is named in the column's own unmeasured list (§2.2.4) — the one condition under which a required field is legitimately absent, and the only one a reader can tell apart from the structural causes §7.2 lists. They MAY emit O fields when applicable. They MUST NOT emit — fields. The inferred sub-object MUST be omitted entirely when it has no sub-fields. inferred.candidate_key is set whenever cardinality_ratio clears the SPEC 4.2 threshold, independent of classification — it is not required by any row, since a column's ratio may fall short of it regardless of type. sketch is O everywhere a join-key column's classification can land, per §2.2.14's own emission rule (edge participation, canonical type, redaction and scope), never per this matrix alone — a categorical or numeric column carries one only when §2.2.14's own conditions hold, the same way redacted is O here but gated by whether a rule actually matched. normalized_cardinality is O on the same three classifications for the identical reason, per its own §2.2.4 emission rule (join-key population, string-valued type) rather than this matrix alone.

Every field this matrix marks anything but R can therefore be absent from a conforming column, usually for more than one reason. §7.2 lists what each absence can mean, and is derived from the rows above.

† Bounds under a dropped column. range, its span_days, and percentiles are cell values, so a column declaring redacted: drop emits none of them: that primitive emits no literal at all, and a bound is nothing but a literal (§2.2.9). For every other column of these classifications the three cells are REQUIRED; for a dropped one they MUST NOT be emitted, and neither may range or percentiles on a numeric column with no finite value (§2.2.4). freshness stays REQUIRED under every primitive, drop included: a derived value is a cell value whenever the derivation can be run backwards, and max_age_days can be — range.max = profiled_at - max_age_days, from any artifact carrying a redacted marker. §2.2.9 states the coarsening this requires rather than an omission.

‡ A prose column publishes no value list. A text column whose inferred.looks_like is prose (§4.1.1) MUST NOT emit values, values_coverage or distribution; every other text column MUST emit all three. The top entries of a free-text column describe nothing a consumer can act on, while the grouped scan that finds them is among the most expensive statements a producer issues — so the format exempts the scan deliberately, and a producer SHOULD skip the query rather than discard its result. distribution is covered because for this classification it is derived from the value list, and computing it another way would re-issue the scan the exemption exists to avoid.

This is the only cell conditional on a field of inferred, and it reaches text alone. categorical runs the same detection (§4.1.5) and can also report prose, but its cardinality is bounded by construction: the scan is cheap there, and the enumeration is what the classification is for. A column that is prose in one run and not the next therefore changes its emitted field set, which diff reports like any other change.

Withheld under redaction. REQUIRED unless the column carries any redacted marker (§2.2.9), under which it MUST NOT be emitted. Each of these fields is computed from what the column's values are rather than how many rows share them, so published beside the per-value counts it solves for the literals the marker withholds; §2.2.9 states the rule these cells follow and the fields it reaches beyond this matrix's R rows. mean and sum MUST NOT be emitted either on a numeric column with no finite value (§2.2.4).

‖ length follows the value's type, not the classification, and needs a non-null value to describe. categorical (priority 7) and foreign_key_candidate (priority 6) match before any type-based branch runs (§3.2), so either can carry a boolean, JSON, temporal, numeric, or string-valued column alike. length is REQUIRED on one of these two classifications only where the column's sql_type could hold a string value — not boolean, not JSON, not a temporal type, not a numeric type, not a binary type — by the same elimination text's own fallback (priority 11) already applies, or is itself a binary type, whose length counts bytes (§2.2.4), AND the scanned set holds at least one non-null value: an all-null column (§2.2.7, §3.3) reaches categorical at priority 7 regardless of type, leaving nothing for the aggregate to describe. A categorical column backed by INTEGER or TIMESTAMP MUST NOT emit length, and neither may an all-null one of any type; one backed by VARCHAR, UUID, BINARY(16), or any type outside those four families, carrying at least one non-null row, MUST. text and binary need no footnote of their own here: priority 11 only matches after eliminating the same four families and binary, priority 10 matches binary alone, and both come after the cardinality-0 fallthrough to categorical, so every text column already qualifies on both counts.

A part's block follows its own classification's column above, with one population field added and a fixed list removed (§2.2.18). A part MUST carry occurrences and MUST NOT carry nullable, physical_name, collation, physical_layout_key, derived_from, rows_scanned, populated, sketch, normalized_cardinality, inferred.candidate_key, inferred.candidate_key_exception, parts or parts_found; every other cell reads as it does for a column, with occurrences in place of the scanned rows. occurrences is — above because a column never carries it.

◊ extent, dimension and norm need a value to describe. A spatial column whose non-null values are all empty geometries (geometry.empty_count equal to the non-null count), or that has no non-null value at all, has no box to publish and MUST NOT emit extent; every other spatial column MUST, unless a redacted marker withholds it (¶). A vector column with no non-null value MUST NOT emit dimension, and one whose non-null values are all zero vectors (zero_count equal to the non-null count) MUST NOT emit norm; every other vector column MUST emit both.

※ quantized_count follows the value's day resolution, not the classification alone. A temporal column is REQUIRED to emit it unless its sql_type is DATE or DATE32 (both already their own day-truncation — a published count would be a truism) or carries no date at all (TIME, TIME WITH TIME ZONE, or MySQL's YEAR). Every other temporal type — every TIMESTAMP variant, timezone-aware or not — MUST emit it. numeric carries no such exception: every numeric type is REQUIRED to emit it, integer-typed columns included, on the same unconditional footing zero_count and negative_count already carry.

2.2.4 Sub-object schemas

physical_name (string, OPTIONAL):

The column's identifier as its catalog reports it, in the case a producer would have to quote to address the column directly. Producers MUST omit this field when it is identical to the map key (§2.2.1) — the common case on every adapter that already reports lowercase identifiers, and on any column whose name happened to be written in lowercase to begin with. A reader who does not find it MAY assume the map key addresses the column unchanged.

columns:
  firstname:
    physical_name: firstName        # catalog spelling; the map key is always lowercase
    ...

collation (string, OPTIONAL): the catalog's own name for the collation this column compares under, when the catalog reports one explicitly set. Producers MUST omit this field when the column carries no explicit collation, or when its collation is identical to the connection's default_collation (§2.5) — the common case, since most columns inherit the connection default. A reader who does not find it MAY assume the column compares under default_collation.

columns:
  country_code:
    collation: utf8mb4_bin          # overrides the connection default recorded in manifest.yaml
    ...

derived_from (list, OPTIONAL): the columns this column's defining SELECT copies or computes it from, as a producer parsed them from a transformation tool's compiled SQL — for a dbt project, each model's compiled code. Each entry carries table, the source's FQN, and column, its lowercase name, both REQUIRED; entries are sorted by (table, column) and unique. A column counts as a source wherever it appears inside the projection, passed through any expression, function or aggregate. Only direct parents are recorded: the source of a source is that source's own derived_from. A source outside the print is still named by FQN, on the same footing as §2.2.17's out-of-scope dependencies. It is a fact about the object's definition, not a row read, so it sits beside physical_name and is allowed under catalog_only (§2.2.15). Omitted whenever no source column was traced — no transformation tool configured, the object absent from its manifest, a definition the parser could not read, a column it could not resolve, or one computed from no column at all (a literal, count(*)) — and these causes are not told apart; there is no derived_from: []. A human's derived_from in statistics.annotations.yaml replaces this list for its column (§2.7.1).

columns:
  email:
    derived_from:                   # traced from the model's compiled SELECT
    - table: arboretum.seedbank.collector
      column: email
    ...

geometry (spatial only, REQUIRED):

geometry:
  kinds:                  # ALWAYS; the OGC kind of each non-null value
    - kind: point         # lowercase, no ST_ prefix, no Z/M suffix
      count: 9120
  srids:                  # ALWAYS where the type carries a reference system; absent on ClickHouse
    - srid: 4326          # the engine's own identifier; an unset reference system is 0
      count: 9120
  dimensions:             # ALWAYS; xy | xyz | xym | xyzm
    - dimensions: xy
      count: 9120
  empty_count: 0          # ALWAYS; values holding no coordinates
  invalid_count: 0        # ALWAYS where the engine can hold an invalid value and test for one

What a spatial query needs to know before it is written. kinds picks the predicate (points want a distance test, polygons a containment or intersection test) and says whether ST_X is legal; srids says whether the column needs a transform before it meets another and prevents the "mixed SRID" class of error; dimensions says whether a 3D or measure-aware function applies; invalid_count says whether to repair values before a GEOS predicate, which raises on an invalid input.

  • Each list is ordered by count descending, then by its key's text. The counts in kinds, srids and dimensions each sum to the column's non-null count, and empty_count and invalid_count are each at most that count.
  • A kind is one of the seven OGC kinds (point, linestring, polygon, multipoint, multilinestring, multipolygon, geometrycollection), or a PostGIS curve or surface kind as PostGIS spells it. A ClickHouse Ring counts as linestring; BigQuery reports an empty GEOGRAPHY as geometrycollection, which is published as reported.
  • srid is the engine's own identifier, the literal a transform function takes: an integer on every engine except duckdb, whose reference system is a string such as OGC:CRS84. An unset reference system is 0 on every engine.
  • invalid_count is absent on an engine that cannot store an invalid value (BigQuery GEOGRAPHY) or cannot test validity (ClickHouse).

geometry describes no coordinate, so a redacted marker leaves it published (§2.2.9).

extent (spatial only, REQUIRED unless the ◊ or ¶ condition holds):

extent:
  min_x: -8.610001
  min_y: 41.139999
  max_x: -8.579999
  max_y: 41.170001

The smallest box, in X and Y only, containing every non-empty value — read in the engine's own axis order, so it also reveals whether a column is longitude-first and whether its units are degrees or metres. Each bound is rounded outward to six decimals (a minimum floored, a maximum ceiled), so the published box still contains every value — the rounding §2.2.6 would otherwise apply can move a bound inside the data. On a geodetic type the box is the engine's own: BigQuery documents that a box crossing the antimeridian may carry a bound outside [-180, 180], so such a box is not portable across engines. min_x <= max_x and min_y <= max_y always hold.

An extent is literal: each bound is some value's coordinate, and on a column holding one point it is that point. A column carrying a redacted marker MUST NOT emit it (§2.2.9).

dimension (vector only, REQUIRED unless the ◊ condition holds):

dimension:
  min: 768
  max: 768

The fewest and most elements among the column's non-null vectors. min == max is the dimension a query embedding must have; min < max says the column mixes dimensions, which an untyped pgvector vector and MySQL's VECTOR(N) (up to N entries) both admit, so the declared type alone does not settle it. A sparse vector reports its declared dimension, not its count of non-zero entries.

norm (vector only, REQUIRED unless the ◊ condition holds):

norm:
  min: 0.999999
  max: 1.000001

The smallest and largest Euclidean norm among the column's non-null, non-zero vectors, rounded per §2.2.6. Both bounds near 1 mean the vectors are unit-normalized, so inner product ranks exactly as cosine; a spread means the two rank differently. No tolerance is fixed here: a normalized column stored in single precision rounds to bounds a few millionths either side of 1. Zero vectors are excluded and counted in zero_count instead, so one placeholder row cannot drag min to 0.

On a vector column, zero_count counts the non-null vectors whose norm is 0 — the same meaning it carries on numeric, a value writers use for absence without a NULL. Cosine distance is undefined on them. No element value of a vector column reaches any field: an embedding can be inverted to approximate its source text, and dimension, norm and zero_count are aggregates no reader can run backwards to an element, so a vector column carries no redacted marker (§2.2.9).

types (json only, OPTIONAL):

types:
  OBJECT: 3
  ARRAY: 1
  VARCHAR: 1
  NULL: 1

How many values hold each type, keyed by the engine's own name for the type of one document value — the literal its per-value type function returns (duckdb json_type, Postgres jsonb_typeof, MySQL JSON_TYPE, Snowflake TYPEOF, BigQuery JSON_TYPE, Redshift JSON_TYPEOF, Databricks schema_of_variant up to its first < or (), so an agent writes the same name in its own type test. On a column it counts every value that is not SQL NULL, a document that is the JSON literal null under the engine's null name, so the counts sum to the scanned rows less null_count; on a part (§2.2.18) a JSON null is already a null occurrence, so the counts sum to occurrences - null_count and the null name never appears. It is a count, not a literal: a redacted marker leaves it alone. A producer whose engine names no per-value type omits it (§7.2).

inferred (omit when all sub-fields absent):

inferred:
  looks_like: <pattern>           # per §4.1; omit when no match
  sampled: <int>                  # per §4.1.3; the draw looks_like was scored against
  matched: <int>                  # per §4.1.3; how much of the draw agreed with looks_like
  looks_like_candidate: <pattern> # per §4.1.3; best-scoring pattern below the verdict bar
  looks_like_candidate_share: <float>  # per §4.1.3; that pattern's share of the same draw
  candidate_key: true             # only ever true; omit when not set
  candidate_key_exception: measured_duplicates | estimated  # per §4.2; omit at ratio 1.0
  sensitivity: <category>         # per §4.4; omit when nothing detected
  epoch_unit: seconds | milliseconds  # per §4.5; omit when not detected
  fk_candidate: { ... }           # reserved, not defined (see §4.3)

sampled and matched are emitted together, and only beside looks_like: both MUST be omitted when looks_like is absent, even on a classification where a sample was drawn and cleared no pattern. Producers MUST NOT emit either field alongside epoch_unit or sensitivity alone — both axes may read the same draw, but the pair describes looks_like's own verdict exclusively, per §4.1.3. looks_like_candidate and looks_like_candidate_share follow the identical pairing rule, in looks_like's absence rather than its presence — see §4.1.3.

values (boolean, categorical, foreign_key_candidate, numeric, temporal, text):

values:
  - { value: <scalar>, count: <int> }
  - { value: <scalar>, count: <int>, spelling_of: <scalar> }   # OPTIONAL; see below

One ordered list describes every column that carries value data. Entries are ordered by count DESC, with ties broken by ascending code-point order of the text of value as written in this file (the YAML scalar after unquoting, e.g. 0.0000001, 09:00:00, true) — never a driver's or language's own string conversion of the value — deterministic across runs. Values MUST be strings, numbers, or booleans. On a classification other than numeric, temporal or boolean, a value of a non-character SQL type (a network address, a range, an interval, a set) is the engine's own text rendering of it — the same text length and empty_count are measured over. A value of a binary SQL type (§3.1) is the exception: every engine renders bytes differently, and some not as text at all, so it is written as lowercase hexadecimal with no prefix (0aff), two digits per byte, the spelling every engine's binary literal syntax accepts ('\x0aff'::bytea, X'0aff', FROM_HEX('0aff')). NULL MUST NOT appear (NULL is tracked separately via null_count). On numeric and temporal, value is rendered exactly as range/percentiles render it (§2.2.4's domain-rendering rule below), and the list is drawn from the same top-N fetch frequencies (below) already issues — a producer MUST NOT issue a second statement to obtain it.

How much of the column the list describes is decided by cardinality, not by classification. The bound is enumeration_threshold (config; default 50), or top_n_values (config; default 20) where that is larger. When the column's distinct count is at most the bound the list is exhaustive and carries every distinct non-null value; above it, the list carries the top_n_values most frequent entries. The two settings therefore divide one question between them: enumeration_threshold is what makes a domain closed (§3.1), so it is how far a closed domain is published, and top_n_values is how much of an open one is sampled. A producer MUST NOT decide this from the classification: a low-cardinality column is enumerated in full whether it is boolean, categorical, foreign_key_candidate or text.

An exhaustive list's entry count equals cardinality. "Carries every distinct non-null value" means the number of entries IS the distinct count — a producer publishing values_coverage: 1.0 with a list shorter or longer than cardinality has broken this obligation regardless of what its row-level counts sum to.

A column of numeric SQL type lists numbers on every classification, never their string spellings: a categorical DECIMAL(4,1) lists -19.5 and -20, as a numeric one does (§2.2.6's exact-numeric cell rule).

A list rather than a map, because YAML and JSON mappings are unordered by definition — an ordering rule on a mapping asks consumers to honor something their parsers may discard.

spelling_of (OPTIONAL, on an entry): the listed value this entry is another spelling of. A column filled from more than one source holds Active, ACTIVE and active as three values, and a consumer reading three entries reads three categories; this field says which of them are one category stored several ways.

  • Two entries group when their values are strings that fold to one key: trimmed and case-folded, the same normalization normalized_cardinality (§2.2.3) measures with. A producer MUST NOT group under any other key — a consumer that reads a group as one category is entitled to the same rule the count beside it was measured under.
  • The canonical member is the group's most frequent spelling; ties go to the order §2.2.4 already fixes. It carries no spelling_of; every other member carries one naming it.
  • Grouping applies to the entries a producer lists, exhaustive and sampled alike, and never changes what is listed: both spellings stay entries, and both count toward cardinality and values_coverage. A consumer wanting the category adds their counts; one wanting the literal has each spelling as stored.
  • A producer MUST NOT emit spelling_of on a redacted column (§2.2.9): the field names another entry's literal, which is what redaction withholds.
  • Where the recorded collation (§2.2.3) already folded two spellings before the count, there is one entry and no group — spelling_of describes what the producer listed, never what the database considers equal.

values_coverage (present alongside values on every classification that admits it except numeric and temporal, where the matrix forbids it — frequencies.total (below) already carries the listed counts' sum, and a second name for that number is the rederivation §2.2.4 elsewhere exists to avoid):

  • Float in [0, 1]
  • = sum of values counts / (rows_scanned — null_count)
  • Exactly 1.0 when the list is exhaustive — carries every distinct non-null value the column has — regardless of what the raw division would give. A producer MUST NOT let rounding report 1.0 for a list it truncated, and MUST emit 1.0 for a column with no non-null rows. Under scope (§2.2.8), 1.0 states that the list is exhaustive over the scanned set, not over the whole table; the same column block's rows_scanned says what that set was, so a consumer reading values_coverage alongside it never mistakes scanned-set completeness for a table-wide guarantee

values_coverage_method (enum, OPTIONAL; measured | bounded): whether the emitted values_coverage is the raw quotient (measured) or the clamp the rule above describes (bounded), because the value list and the population it is measured against were not read at the same instant — the same distinction §2.2.10's coverage_method draws, applied to this field.

Emitted only for an exhaustive list a producer's own truncation did not cut short; a truncated list is short by design, which is a different, already-explained condition this field does not cover. measured states the listed counts agreed with the non-null rows they were scanned against; bounded states a producer detected the two disagreeing — the listed counts exceeding the non-null rows, the phase A / phase B disagreement a live table taking writes between the two reads can produce. The detector is one-sided, catching only an overrun: a row deleted between the two reads shrinks the listed total below the scanned count instead, which reads as measured the same as a genuine same-read agreement would.

Day counts. range.span_days and freshness.max_age_days are the only statistics a producer derives by arithmetic rather than by reading. Both are the count of whole elapsed days between two instants:

day_count(earlier, later) = floor( (later - earlier) in seconds / 86400 )

Producers MUST compute elapsed time, NOT the number of calendar-date boundaries crossed. The two differ in both directions and are not interchangeable: a span from 01:00 to 21:00 elapses 20 hours and crosses no date boundary, while a span from 23:50 to 00:10 elapses 20 minutes and crosses one. Counting boundaries would report 1 and 0 respectively; counting elapsed whole days reports 0 for both, which is what this rule requires.

Flooring rather than rounding is what makes a sub-day span report 0 (§2.2.7) without a special case: any interval shorter than 24 hours has zero whole days in it.

Elapsed time is exact, fractional seconds included. A producer MUST NOT truncate either instant to whole seconds, count second boundaries, or compute the count through floating point or rounded decimal division: 2024-01-01T00:00:00.5 to 2024-03-01T00:00:00.25 is 59 days, not 60, and a 9999-12-31T23:59:59.999999 maximum is counted to the microsecond.

range:

range:
  min: <val>                      # rendered in the column's own domain; see below
  max: <val>
  span_days: <int>                # TEMPORAL ONLY; day_count(min, max)

percentiles (numeric, temporal):

percentiles:
  p01: <val>                      # key format: `p` + zero-padded 2-digit integer percent
  p25: <val>
  p50: <val>
  p75: <val>
  p99: <val>

Percentile values in .dbprint.yaml configuration MUST be representable as integer percentages (multiples of 0.01); fractional percentiles like 0.999 are REJECTED by config validation. Emitted values are rendered in the column's own domain, exactly as range is.

A temporal percentile is a value the column holds. It is the nearest rank — of the column's n non-null values in ascending order, the one at position CEIL(p × n), as PERCENTILE_DISC defines it — or, from a producer using an approximate construct, a value of the column near that rank. It is never interpolated between two instants: an instant carried through a floating-point number to interpolate can come back a microsecond or more from any value the column holds, in another time zone, or in a year the column does not reach.

Percentiles MUST ascend with their keys (non-decreasing, not strictly ascending — a single-valued column legitimately publishes the same value at every key), and, when range is present, every percentile MUST lie within [range.min, range.max]. Both bounds and every percentile come from one statement per column, and a temporal percentile is itself a value of the column, so a producer cannot correctly emit a percentile outside its own range or out of order with its neighbors.

Domain rendering. range.min, range.max and every percentiles entry are emitted in the column's own domain — the form a predicate against that column would use. For most temporal types that is an ISO 8601 string ('2026-03-09T14:27:36Z', '2026-03-09'). For types that carry no date it is not:

Column typeEmittedA consumer then writes
TIMESTAMP / DATE'2026-03-09'WHERE seen_at >= '2026-03-09'
TIME'08:00:00'WHERE run_at >= '08:00:00'
YEAR1960WHERE made BETWEEN 1960 AND 2019

The rule exists so a value read out of an artifact goes back into a query unchanged. The ISO form of a YEAR — '1960-01-01' — does not: MySQL evaluates a YEAR-to-date comparison to NULL, and the predicate returns no rows without reporting an error.

mean, sum (numeric only):

mean: 42.5
sum: 850000

Two additional expressions on the same statement that already computes range and percentiles — no second scan. mean is the arithmetic mean of the column's values; sum is the total the percentiles alone cannot recover, and the denominator behind every share a consumer computes from the column. Both are rounded per §2.2.6, and neither is domain-rendered: unlike range/percentiles, a numeric column's own domain already is a plain number.

temporal never carries either — a mean instant is definable and useless, and the dialects disagree on how to express one; range and percentiles already answer where a temporal column sits.

Withheld under redaction. A column carrying a redacted marker emits neither field (§2.2.9, the ¶ footnote above).

zero_count, negative_count, empty_count (numeric: the first two; vector: the first, see norm above; text and binary: the third; an array, see §2.2.18):

zero_count: 6000        # numeric and vector
negative_count: 12      # numeric only
empty_count: 4000       # text and binary only

A fixed count of the values commonly used to encode absence without a NULL — 0, a negative sentinel, the empty string. Each is an exact COUNT of scanned non-null rows equal to that value, bounded by rows_scanned - null_count (§2.2.3's conformance check), and asserts nothing about what the count means: a column that is 60% zero may be recording a real quantity, a coalesced absence, or both, and this field states the arithmetic fact, never the interpretation. empty_count counts the empty string exactly, as the column's own collation compares it (§2.2.2); a whitespace-only value is a different value and is not counted here. On a binary column it counts zero-length values, and on an array (§2.2.18) the non-null instances holding no element.

No second statement: each is an additional expression on the same per-column pass that already computes null_count. A column carrying a redacted marker emits none of the three: each count names how many rows hold one particular value, which solves for the literal beside the per-value counts (§2.2.9).

quantized_count (numeric; temporal where the column's sql_type has a day to truncate to, per the ※ footnote above): the count of scanned non-null values equal to their own truncation — to an integer on numeric, to the day on temporal.

quantized_count: 382   # numeric: values with no fractional part
quantized_count: 7     # temporal: values that are exactly midnight

An exact COUNT of scanned non-null rows, bounded by rows_scanned - null_count (§2.2.3's conformance check), on the same per-column pass that already computes null_count — no second statement. It states the arithmetic fact and nothing about a float's binary representation or a number's intended unit: dollars, cents, and a percentage already multiplied by 100 all publish the same count if their values happen to be whole. A column carrying a redacted marker emits none (§2.2.9): how many values sit on a day boundary is a fact about what the values are.

Truncation is evaluated in a frame the data or the DDL fixes, never in a session's or server's zone. A value whose type stores its own offset (Snowflake TIMESTAMP_TZ) is truncated in that offset; a value whose column declares a zone (ClickHouse DateTime('Europe/Berlin')) is truncated in that zone; every other timezone-aware value is an absolute instant and is truncated in UTC, the instant range and percentiles render (the domain-rendering rule below). A type with no zone (TIMESTAMP WITHOUT TIME ZONE, TIMESTAMP_NTZ, DATETIME) is truncated in its own reading, unconverted. So the same rows count the same on every host, and a UTC-frame column's local midnights in another zone are not day-aligned:

AdapterNumeric expressionTemporal expressionTimezone-aware, u the frame's reading
Postgrescol = TRUNC(col)col = date_trunc('day', col)u = date_trunc('day', u), u = col AT TIME ZONE 'UTC'
MySQLcol = TRUNCATE(col, 0)col = CAST(col AS DATE)col = CAST(col AS DATE), col read in a session whose time_zone is +00:00
Snowflakecol = TRUNC(col)col = DATE_TRUNC('day', col)TIMESTAMP_TZ: col = DATE_TRUNC('day', col); TIMESTAMP_LTZ: u = DATE_TRUNC('day', u), u = CONVERT_TIMEZONE('UTC', col)::TIMESTAMP_NTZ
duckdbcol = TRUNC(col)col = date_trunc('day', col)u = date_trunc('day', u), u = col AT TIME ZONE 'UTC'
ClickHousecol = trunc(col)— (every DateTime is an instant)toStartOfDay(col, z) = col, z the column's declared zone, else 'UTC'
Redshiftcol = TRUNC(col)col = DATE_TRUNC('day', col)u = DATE_TRUNC('day', u), u = col AT TIME ZONE 'UTC'
Databrickscol = FLOOR(col)col = DATE_TRUNC('DAY', col)u = DATE_TRUNC('DAY', u), u = CAST(CONVERT_TIMEZONE(CURRENT_TIMEZONE(), 'UTC', col) AS TIMESTAMP_NTZ)
BigQuerycol = CAST(col AS INT64)col = DATETIME(DATE(col))col = TIMESTAMP(DATE(col, 'UTC'), 'UTC')

non_finite_count (numeric, optional): the exact count of scanned non-null values that are NaN, +inf or -inf, emitted when it is nonzero and only then.

non_finite_count: 2   # two scanned values were NaN or an infinity

Every other numeric statistic reads the finite values alone. range, percentiles, mean, sum, negative_count and quantized_count are computed over the scanned non-null values that are finite; a column with no finite value emits none of range, percentiles, mean or sum. A mean, sum or percentile the engine returns non-finite over finite inputs — a sum that overflows binary64 — is named in unmeasured, never published. values lists a non-finite cell as measured (.nan, .inf, -.inf), since §2.2.7 forbids altering one. Engines disagree on where NaN sorts and on what MIN/MAX return for it, and one NaN otherwise erases a column's maximum, mean and sum. A column carrying a redacted marker emits no non_finite_count: beside the per-value counts it solves for which listed literal is NaN (§2.2.9). Under mask or hash, where the withheld count leaves no reader able to see that no finite value exists, such a column names range and percentiles in unmeasured instead.

length (text and binary; categorical and foreign_key_candidate when the column's sql_type carries a string value or is a binary type, per the ‖ footnote above):

length:
  min: 2
  max: 4000
  avg: 187.4
  p95: 512.0

Character length, not byte length: a value's encoding does not change how many characters it holds, and characters are what a display width, a context budget, and a SELECT decision are all sized against. A binary type (§3.1) is the one exception: it holds bytes, not characters, so on a column of binary SQL type — classified binary, or categorical/foreign_key_candidate — length counts bytes, and min == max == 16 is how a sixteen-byte key reads. min, max and avg are computed in the same per-column statement every adapter already issues; p95 is one more expression on it everywhere except MySQL, which reads it from a second statement over a ranked derived table — MySQL 8 defines no PERCENTILE_CONT aggregate and MariaDB 10.11 rejects the WITHIN GROUP form. Producers MUST satisfy min <= avg <= max.

Producers MUST measure the column's own character-length function, never a byte length, so two producers reading one column agree:

AdapterFunction
PostgresLENGTH(CAST(col AS text))
MySQLCHAR_LENGTH(CAST(col AS CHAR))
SnowflakeLENGTH(TO_VARCHAR(col))
duckdbLENGTH(CAST(col AS VARCHAR))
ClickHouselengthUTF8(toString(col)) — length() is bytes on this engine. Undefined (not an error) on invalid UTF-8 input; the engine gives no guaranteed result to fall back to
RedshiftLENGTH(col::varchar)
DatabricksLENGTH(CAST(col AS STRING))
BigQueryLENGTH(col)

On a binary SQL type the byte-length function is measured instead, and an empty count counts values of byte length 0:

AdapterFunction
PostgresOCTET_LENGTH(col)
MySQLLENGTH(col)
SnowflakeLENGTH(col) — bytes on a BINARY value
duckdbOCTET_LENGTH(col)
RedshiftOCTET_LENGTH(col)
DatabricksOCTET_LENGTH(col)
BigQueryLENGTH(col) — bytes on a BYTES value

An empty string measures 0 — a measurement, not an error (§2.2.7). A column with null_rate: 1.0 emits no length, the same terms range follows.

Withheld under redaction. A column carrying a redacted marker emits no length: its bounds are the withheld literals' own lengths (§2.2.9, the ¶ footnote above).

normalized_cardinality (int ≥ 0, OPTIONAL; categorical, foreign_key_candidate, text when the column's sql_type could hold a string value, and only for the population §2.2.14 defines for sketch — a single-column FK edge, a declared single-column unique key, or inferred.candidate_key — minus that section's own cardinality-exhaustive widening bullet, which this field does not carry forward):

normalized_cardinality: 4102   # cardinality: 4118 - 16 values merge under folding

cardinality, trimmed of leading and trailing whitespace and case-folded — the one normalization this field defines, so two producers reading one column agree:

AdapterFunction
PostgresLOWER(TRIM(CAST(col AS text)))
MySQLLOWER(TRIM(CAST(col AS CHAR))); a BIT column LOWER(TRIM(CAST(col AS UNSIGNED))), folding the integer its other statistics read rather than its raw bytes
SnowflakeLOWER(TRIM(TO_VARCHAR(col)))
duckdbLOWER(TRIM(CAST(col AS VARCHAR)))
ClickHouselowerUTF8(trimBoth(toString(col)))
RedshiftLOWER(TRIM(col::VARCHAR))
DatabricksLOWER(TRIM(CAST(col AS STRING)))
BigQueryLOWER(TRIM(CAST(col AS STRING)))

Accent folding is deliberately excluded — it varies far more by engine than case does, and including it would make the field incomparable across adapters. normalized_cardinality MUST NOT exceed cardinality: folding can only merge two values into one, never split one into two. It carries the column's own cardinality_method rather than one of its own: an approximate cardinality makes the comparison approximate on both sides.

A violation is a warning, because the bound holds over one read and the two counts need not come from one. The folding argument is about values, not about time: a table taking writes between the statement that measured cardinality and the statement that measured normalized_cardinality can leave the later count the larger, with no producer defect and nothing a producer reading a live table can do about it. §6.3 therefore reports the excess rather than gating on it, on the terms stats.values-sum-mismatch already sets for the same disagreement. A gap still counts the values that merge under folding, except that a negative one reflects read order.

Unlike sketch, a scoped table stays eligible. normalized_cardinality is computed over the scanned set cardinality is itself defined against (§2.2.8) — a scanned-set count like any other, present under scope on the same terms cardinality is, not withheld the way §2.2.14 withholds sketch there. Where a producer copies a sampled draw, both counts read that one copy and they describe the same rows; where nothing is copied, they are two reads of the same table and the paragraph above governs. A column carrying a redacted marker emits none: how many spellings merge under case-folding is a fact about what the values are, not how many rows share them (§2.2.9).

populated (map, OPTIONAL, every classification; requires timeline (§2.2.16) in the same file):

populated:
  from: '2026-03-04T00:00:00'   # min(timeline.column) over rows where this column is non-null
  to: '2026-08-27T14:02:11'     # max(timeline.column) over rows where this column is non-null

The window over which the data shows this column non-null, dated against timeline's own anchor — the fact a bare null_rate cannot carry: a column that is 20% null at random and one that is null on every row before a date look identical at one number, and only diverge here. Both instants render through the anchor's own domain rule (this section's rendering convention), the same rule range uses.

Omitted where it would restate a fact already on the file. A fully-populated column (null_count: 0) has nothing to say the anchor's own range does not already say, and an all-null column (null_rate: 1.0) has no populated rows to date at all — a producer MUST NOT emit populated in either case.

MUST NOT be emitted without timeline. The pair is readable only beside the anchor's name, which timeline.column publishes (§2.2.16); a file with no timeline block has nowhere for a reader to look that up, so populated MUST be absent from every column wherever timeline itself is — under scope, on an empty table, with no eligible anchor, or where a producer's own configuration disabled the block.

Both instants lie inside the anchor's own range, and a violation is a warning. They are drawn from the anchor column, so no populated row can be dated outside the anchor's own bounds — over one read. The window and the bounds are measured by separate statements, and an anchor that advances between them leaves to past a range.max recorded earlier. §6.3 therefore reports the excursion rather than gating on it, on the terms stats.values-sum-mismatch sets for the same disagreement. Inserts cannot move range.min, so an excursion on from is the one worth reading closely — but the artifact does not record which statement ran first, and the check does not distinguish them.

Redaction follows the anchor, not the subject column. The two instants are cell values of the anchor column (§2.2.9); a redacted column is never chosen as an anchor in the first place (§2.2.16), so this case cannot arise from a conforming producer. A redacted subject column still carries populated unaffected — the pair discloses only when the column had a value, never what that value was.

States what the data shows, never a claim about schema history. A backdated write moves from earlier without the column having existed earlier at that instant; the pair records what the rows say, not a schema-migration timeline a reader might otherwise infer from it.

One additional statement per table, not one per column. A single batched statement covers every eligible column of a table in one round trip — two conditional aggregates per column, MIN/MAX of the anchor filtered to rows where that column is non-null.

frequencies (numeric, temporal):

frequencies:
  top: <int>       # the most frequent listed count
  bottom: <int>    # the least frequent listed count
  listed: <int>    # how many counts were listed
  total: <int>     # the listed counts added up

A fixed-size summary of the same top-N frequency fetch distribution (§2.2.5) is computed from. Both classifications carry a values list (above), but values_coverage is forbidden on either (above), so a validator has no exhaustive list to recompute distribution from. frequencies is what a validator checks it against instead: an exact, four-integer summary of the same fetch, never a share, so a consumer recomputes any ratio itself against the non_null and cardinality the column already publishes rather than trusting a rounded one.

unrepresentable (temporal only, optional):

A temporal column MAY carry unrepresentable: a list naming the emitted fields — min, max, and any emitted percentiles key — whose value lies outside the years 0001 through 9999 inclusive, proleptic Gregorian. The boundary is stated as this calendar range, not as any one library's or driver's limits, because two conforming producers reading the same column MUST mark the same fields regardless of the runtime they were written in.

The domain-rendering rule above already makes the text form correct for such a value — a database's own rendering of an instant beyond the standard calendar is exactly what belongs in the file. unrepresentable is what lets a consumer feeding that value to a typed parser degrade deliberately instead of crashing.

created_at:
  classification: temporal
  sql_type: TIMESTAMP_NTZ
  range:
    min: "1970-01-01T00:00:00"
    max: "52030-01-01T00:00:00"
    span_days: 15376234
  percentiles:
    p50: "2026-01-01T09:30:00"
    p95: "11000-07-05T00:00:00"
  unrepresentable: [max, p95]

Entries are ordered as the fields are emitted (min, max, then percentile keys in ascending order) and carry no duplicates. The key is omitted entirely when no emitted value qualifies.

The marker names a field the format carries, not a claim about the value's correctness: a column whose maximum is year 52030 probably holds a mis-scaled epoch, and this field does not say so — it carries the value and flags that it will not parse. A column carrying any redacted marker MUST NOT emit it: which withheld bound lies outside years 0001–9999 is a fact about what the literal is (§2.2.9).

unrepresentable deliberately introduces no diff change kind of its own: it is derived from the same bound its own statistic_changed event already reports, so a value moving in or out of the representable range is already visible as a change to range.min, range.max, or the percentile in question.

unmeasured (any classification, optional):

A column MAY carry unmeasured: a list naming the fields the §2.2.3 matrix marks R for this column's classification that this run attempted and could not obtain, or that its engine has no expression for. Every named field MUST be absent, and a field the matrix does not require for this classification MUST NOT be named — that absence is already structural, and §7.2 explains it without a marker. Two optional fields are the one exception: on a classification §4.1.5 samples, a column whose sample draw failed names inferred.looks_like and inferred.epoch_unit, because their absence otherwise states that the draw found nothing (§4.1, §4.5). The verdict's evidence fields (sampled, matched, looks_like_candidate, looks_like_candidate_share) are not named; they describe that verdict and read as unmeasured with it.

This is the one absence §7.2 cannot otherwise account for. Every cause it lists is a property of the column: the classification forbids the field, a redaction withheld it, the type has no day to truncate to. A query that was issued and failed is none of those, and without a marker for it a producer must either omit the field — which states whichever structural cause a reader infers — or publish a value it did not measure. Both are false statements about the data; this marker lets a producer state the true one.

logged_at:
  classification: temporal
  sql_type: TIMESTAMP
  cardinality: 4096
  cardinality_ratio: 0.41
  cardinality_method: exact
  null_count: 0
  null_rate: 0.0
  unmeasured: [distribution, freshness, frequencies, percentiles, quantized_count, range, values]

Entries are the field names as the matrix spells them, sorted, without duplicates. The key is omitted entirely when the run measured every field the column should carry — which is every column of a print whose reads all succeeded.

A named field is REQUIRED to be absent rather than merely permitted to be: emitting a value and declaring it unmeasured are contradictory claims, and a consumer cannot tell which of the two is true. §6.4 carries both violations as errors.

unmeasured deliberately introduces no diff change kind of its own, on unrepresentable's own terms: a field either side names is not compared at all, so nothing about it can produce an event. A field that stops being unmeasured between two runs surfaces as the ordinary statistic_changed for the field itself, once both sides carry a value again.

Date-less temporal types. A type carrying a time of day but no date — TIME, and TIME WITH TIME ZONE — has no instant to measure from. Producers MUST emit range.span_days: 0 (every such value falls inside one day) and freshness.max_age_days: 0. Producers MUST NOT derive either quantity by arithmetic against the current timestamp; the operation is undefined for these types and the dialects fail differently, some silently.

freshness (temporal only):

freshness:
  max_age_days: <int>             # max(0, day_count(max(column), profiled_at))
  classification: live | stale | dormant

max_age_days is clamped at 0. A column whose newest value has not happened yet — a queue of future-dated work, an expiry timestamp, a sentinel birth date — would otherwise produce a negative age, which this field does not carry. Producers MUST emit 0 for such a column, and it therefore classifies live. This is the answer for a column where only some values are in the future and for one where all of them are: the rule is applied to the maximum, so the two cases are the same case.

The clamp is deliberately lossy. freshness answers "how stale is this data", and data that has not arrived yet is not stale, so 0 is the honest reading of that question — but it does not distinguish a column written a moment ago from one scheduled a century out. Consumers needing that distinction MUST read range.max, which always carries the true maximum.

The day count is computed in UTC. profiled_at is always UTC (§2.2.1). A zone-aware range.max is an absolute instant, so day_count(range.max, profiled_at) is unambiguous. A timestamp without time zone or MySQL DATETIME value carries no zone; producers MUST treat its rendered reading as UTC when computing max_age_days, the same assumption the value's own domain rendering makes. This can differ from the value's true civil age by the server's UTC offset — at most one day, and only when the newest value sits within that offset of midnight, where it can shift the live/stale/dormant bucket below.

Thresholds (hardcoded; not configurable):

  • live: max_age_days < 7
  • stale: 7 ≤ max_age_days < 90
  • dormant: max_age_days ≥ 90

A column carrying a redacted marker emits max_age_days floored to the nearest 90 rather than the exact count — see §2.2.9, which states the reason and the identical rule for range.span_days. classification is still computed from the true, uncoarsened age, never re-derived from the floored integer.

2.2.5 distribution rules

Enum: uniform | imbalanced | dominant_value | long_tail.

Producers MUST evaluate the rules in this priority order; first match wins:

  1. dominant_value — top value's count / (rows_scanned — null_count) ≥ 0.95
  2. long_tail — cardinality > the value-list bound (§2.2.4) AND sum of the listed counts / (rows_scanned — null_count) < 0.30
  3. imbalanced — max-frequency / min-frequency > 2× (over non-null values)
  4. uniform — fallthrough (max/min ratio ≤ 2×)

Where the values list is exhaustive (values_coverage of 1.0), long_tail is undefined and producers MUST skip step 2: a complete enumeration has no tail beyond itself. Steps 3 and 4 read the true minimum frequency in that case, and the least frequent listed entry otherwise.

A validator was not present at the scan and cannot see rows_scanned directly, so it checks steps 1 and 2 only where values is exhaustive, against the sum of the listed counts — which then equals rows_scanned - null_count exactly, since an exhaustive list already accounts for every non-null value the scan produced. A truncated list is not checked against this rule at all.

numeric and temporal carry a values list (§2.2.4), but values_coverage is forbidden on both, so the check above, which fires only where the list is exhaustive, cannot reach them; frequencies (§2.2.4) exists to close that gap. Its four integers reproduce the same priority order over the scanned set (non_null) and cardinality the column already publishes: top decides step 1, total decides step 2's ratio, and the top-to-bottom spread decides steps 3 and 4. A validator reads the fetch's own exhaustiveness from listed == cardinality, when cardinality_method is exact — an approximate cardinality is not guaranteed to equal the fetch's own count of distinct groups, so a validator MUST NOT check distribution against frequencies on such a column.

2.2.6 Numerical precision

Numeric stats (range bounds, percentiles, mean, sum, ratios; never non-finite, §2.2.4) MUST be rounded to 6 decimal places before emission. This stabilizes git diffs across runs where adapter floating-point precision varies slightly, while leaving a value's magnitude intact — a maximum of 12345678.9 is emitted as it stands, not rewritten to a rounder number of the same size.

Rounding MUST NOT turn a nonzero measurement into zero. Six decimal places round any value below half of one of them to zero, which does not leave the magnitude intact: a column of measurements around 4e-07 would publish range.min, range.max, every percentile and mean as 0.0, beside a sum that disagrees with all of them. Where rounding to six decimals would produce 0.0 from a value that is not zero, a producer MUST emit six significant figures instead. This is a floor on the rounding rule, not a second rule: any value large enough to survive six decimals is unaffected. It governs the measured fields named above — the two ratios are bounded by 1 and carry their own floor, stated below. A spatial extent is the one exception: its bounds round outward to six decimals so the box still contains every value (§2.2.4), and a minimum of 4e-07 floors to 0.0 there by design.

A listed value is not among the rounded statistics. The fields above are computed over a scan, and rounding them stabilizes a diff across runs. A values entry (§2.2.4) is a cell read from the table, and §2.2.7 forbids altering one except under a declared redaction — so it is emitted as measured, at whatever precision the column holds. Rounding them also merges two distinct values that agree to six decimals into one repeated literal, which contradicts what the list is.

An exact numeric cell is written exactly. A value of an exact numeric type (DECIMAL, NUMERIC, NUMBER, BIGNUMERIC and their kin) that is integral is written as an integer literal (-20, 18446744073709551616); any other is written positionally with every significant digit and no trailing zeros (0.123456789012345678), never with an exponent and never passed through binary floating point. A statistic whose input is exact — range bounds, sum, mean, percentiles, length.avg where the engine returns an exact value — is rounded as above in decimal arithmetic (half to even) and written exactly; one the engine itself computes in binary floating point stays a float. A reader decoding YAML floats as binary64 recovers a non-integral value past about 17 significant digits only approximately; the artifact text itself is exact.

null_rate and cardinality_ratio (floats in [0,1]) follow the same rule.

Both ratios are recomputable and checked, with a floor and a ceiling. null_rate MUST equal null_count / rows_scanned rounded per this rule, and cardinality_ratio MUST equal cardinality / rows_scanned rounded the same way — the same identities §2.2.2 states, with a conformance check behind them. Two exceptions to plain six-decimal rounding: a nonzero numerator (null_count or cardinality) MUST NOT round down to 0.0 — it emits 0.000001, the smallest representable value at this precision, instead. null_rate additionally MUST NOT round up to 1.0 unless null_count == rows_scanned — a nonzero non-null count emits 0.999999 instead, because null_rate: 1.0 is a defined sentinel (§2.2.7, §3.3) that rounding must not produce; cardinality_ratio carries no such ceiling. rows_scanned == 0 is the one case this floor does not touch: both ratios stay exactly 0 there, per §2.2.2, since the ratio is undefined rather than merely small. A rounding boundary the producer's own rule lands on — including the floor and the ceiling — is not a violation; a value that disagrees with the recomputed identity is.

A consumer recomputing either ratio by hand from null_count or cardinality and rows_scanned will not reproduce the floored or ceilinged value at the extremes — the published number and the raw division agree everywhere else, but diverge deliberately at the two boundaries this rule exists to avoid publishing.

Counts are exact integers — no rounding. null_count, cardinality and every values entry count are measured over the scanned set (§2.2.8); row_count is a separate, table-scale count and MUST NOT be read as measured over the same population.

Rendering. Every float in an artifact MUST be written in positional decimal notation. Exponent form (6.6e-05) is not permitted anywhere, including inside a values list, and the conversion MUST be lossless — the emitted text MUST parse back to the identical float. This is a producer obligation stated in prose rather than a schema constraint, because a validator sees parsed numbers and cannot observe the notation they arrived in.

Rendering notation is not a value change, so it does not conflict with §2.2.7's prohibition on altering listed values: an exact positional expansion of 1e-09 denotes the same number.

2.2.7 Edge cases

CaseResolution
Empty table (row_count = 0)File emitted with row_count: 0. Per-column: null_count: 0, null_rate: 0, cardinality: 0, cardinality_ratio: 0. classification derived from sql_type via §3 priority; at cardinality = 0 this resolves to categorical (or foreign_key_candidate / boolean by FK/type). The value fields are emitted EMPTY rather than omitted: values: [] + values_coverage: 1.0 — a column with nothing to list covers all of it — plus distribution for the classifications that require it. unsupported columns stay minimal (cardinality: null, no value fields). A row_count: 0 carrying a non-zero scope.rows_scanned is NOT this case: §2.2.8 lets an estimate that undershot stand as the estimate it is, so that pair reads as inverted rather than as an empty table, and a consumer MUST read the two fields together before concluding a table holds nothing.
All-null column (cardinality = 0, null_rate = 1)Same empty-collection behavior as the empty table for that column. classification per §3.3.
Single-value column (cardinality = 1)Classification = categorical. values: { sole_value: count }. distribution: dominant_value.
Sub-day temporal spanrange.span_days: 0 — a consequence of the day-count rule (§2.2.4), not an exception to it: an interval shorter than 24 hours contains zero whole days however it sits against the calendar.
Long string values in valuesEmitted as-is. No truncation. Producers MAY log warnings when total YAML size exceeds an adapter-defined threshold. Producers MUST NOT alter values except under a declared redaction (§2.2.9), which is the one sanctioned substitution and is always announced by the redacted marker.
Approximate vs exact methodsAdapter chooses based on table size / cost. row_count_method and cardinality_method document the choice.
values ties at the cutoff boundaryWhich of the tied values survive the cutoff is the producer's choice and MUST be deterministic across runs — a database orders them by its own collation and its own text of the value, which need not match this file's. The surviving entries are then ordered as §2.2.4 says, by ascending code-point order of the text of value as written in the file.
Empty scanned set (scope.rows_scanned = 0, row_count > 0)A filter that matched no rows. Per-column shape follows the empty-table row above, but row_count keeps the table's own count — the two cases are distinct and MUST NOT be collapsed.

The empty-collection shapes these rows produce read like omissions and are not; §7.4 states what each one can mean.

2.2.8 scope — statistics over part of a table

By default every statistic in this file describes every row of the table. A producer that reads only part of it — because a row predicate was configured, or because the table was sampled to bound cost — MUST say so with a top-level scope block. Its absence is the assertion that nothing was skipped.

A statistic measured under this block is present, not partial; §7.1 states why that is the absence readers mistake most often.

scope:
  rows_scanned: <int ≥ 0>       # ALWAYS when scope is present; rows the statistics were computed over
  sample: <float (0, 1]>        # OPTIONAL; the fraction the producer asked the database to read - see below
  filter: <string>              # OPTIONAL; the row predicate applied, verbatim

rows_scanned is the denominator for every scanned-set-relative field, not only the two ratios. That set is every §2.2.3 matrix cell outside sql_type, nullable and classification that is not a dash on every classification: null_count, null_rate, cardinality, cardinality_ratio, every values entry count, values_coverage's denominator, range.min, range.max, percentiles, mean, sum, zero_count, negative_count, empty_count, quantized_count, length, dimension, norm, freshness.max_age_days, and every frequencies integer. row_count is the one required field this rule does not touch — it is a count over the whole table, not the scanned set (see below). When scope is absent, rows_scanned is defined to equal row_count, so one rule covers both cases. sum in particular is measured over the scanned set only and is NOT rescalable to table grain by the reader without assuming the sample is representative — a consumer reading it under a narrowed scope reads a partial total, not the column's true sum.

The same denominator governs every table-level block outside the §2.2.3 matrix too, each stated in its own section rather than repeated here: null_patterns.coverage (§2.2.10) and timeline.buckets[].count/timeline.coverage (§2.2.16) are both counts over rows_scanned, on the identical terms this paragraph sets for a per-column cell.

Every column of a scoped file echoes rows_scanned. A reader of one column block recomputes null_rate, cardinality_ratio and values_coverage without leaving it, and needs only the file head's row_count to rescale a row count to table grain. The field is REQUIRED on every column — unsupported included, since its null_rate is scanned-set-relative too — whenever the top-level scope block is present, and MUST NOT be emitted on any column otherwise (§2.2.3). A file whose rows_scanned equals row_count still emits the marker on every column; omitting it there would make its absence mean two different things again.

row_count describes the table, not the slice. It counts the rows in the table whether or not the read narrowed. The two fields are read together: rows_scanned of row_count.

rows_scanned / row_count is the sampling fraction the rescaling uses, not sample. A row count (below) is scaled up to table grain by MULTIPLYING it by the reciprocal, row_count / rows_scanned — stated in that direction because the ratio itself is less than one, and applying it as a multiplier shrinks the very figure it is meant to raise. sample is the fraction the producer asked the database to read: passed through as configured, or, under a max_rows_scanned ceiling, snapped to a power of the producer's own ceiling grid against a catalog estimate — not a measurement of what the scan touched. rows_scanned and row_count are both required fields already, rows_scanned is already the stated denominator above, and naming their ratio as authoritative costs nothing further. Wherever row_count_method is approximate, this rescaling ratio is itself an estimate: rows_scanned is exact, but row_count is not.

Only a row count rescales, and only under sample. A row count counts scanned rows: null_count, every values entry count (a boolean's true/false split included), zero_count, negative_count, empty_count, quantized_count, frequencies.top, frequencies.bottom, frequencies.total, every null_patterns entry count (§2.2.10), and a part's occurrences (§2.2.18). Under sample a reader MAY multiply a row count by row_count / rows_scanned for an estimate at table grain. A reader MUST NOT rescale a field that is not a row count, by that ratio or any other: cardinality and frequencies.listed count distinct values, which do not grow in proportion to the rows read; null_rate, cardinality_ratio, values_coverage and null_patterns.coverage are ratios; range.min, range.max, percentiles, mean, length, dimension, norm and freshness.max_age_days are bounds and summaries; and sum is a partial total (above). Under filter nothing rescales: a predicate chose the rows read, so they are no sample of the table.

filter is provenance, not a query language. Producers record the predicate verbatim and MUST NOT parse, rewrite, normalize, or validate it. Consumers MUST treat it as opaque text — it exists so a reader can judge whether these numbers answer their question, not so a tool can reconstruct the query.

A table is narrowed one way or the other. sample and filter are mutually exclusive, and a block carrying both is invalid. Composing them expresses nothing a wider predicate does not already express, while costing an ordering rule every producer has to honor and forcing the dialects whose sampling construct binds to a base table to wrap that table in a subquery — which is where a sample stops being reproducible. Producers MUST NOT record how the sample was drawn; the sampling constructs available differ per dialect and the difference is not actionable for a consumer.

Rules:

  • sample and filter MUST NOT both be present. Each describes the whole narrowing the producer applied.
  • A producer MUST omit the block entirely when it read every row. A block whose rows_scanned equals row_count and which carries neither sample nor filter asserts nothing and SHOULD NOT be emitted.
  • rows_scanned MUST NOT exceed row_count, except where row_count_method is approximate and the estimate undershot — in which case the producer SHOULD emit the exact scanned count and leave row_count as the estimate it is.
  • rows_scanned: 0 with row_count > 0 is well-formed: a predicate that matched nothing.

2.2.9 redacted — values withheld

A print enumerates a column's values, and for a low-cardinality column that enumeration is exhaustive — so a small set of personal values is written out in full and committed. The counts are what a consumer needs; the literals are what must not leave the database.

A producer MAY replace or omit cell values, and MUST declare it with a column-level redacted marker naming the primitive applied:

PrimitiveBehaviour
maskEvery literal replaced by the placeholder [redacted]
dropNo literal emitted; each entry keeps its count
hashA salted digest emitted in the literal's place, written as 16 lowercase hexadecimal characters

A redacted column publishes its count profile, and nothing else computed from its values. Every per-column statistic on a column carrying a redacted marker MUST be unchanged by any one-to-one renaming of the column's distinct non-null values within its SQL type. The count profile satisfies this: null_count, null_rate, cardinality, cardinality_ratio, cardinality_method, every values entry's count, values_coverage, distribution, frequencies and populated, and a consumer may rely on each being the true measurement. mean, sum, length, zero_count, negative_count, empty_count, quantized_count, non_finite_count, normalized_cardinality, unrepresentable, sketch and a spatial column's extent do not — each is computed from what the values are rather than how many rows share them — and a column carrying any marker MUST NOT emit them. A spatial column's geometry is the one named exception to the renaming invariant: it publishes no per-value count to solve against, and a kind, a reference system, a coordinate dimension, an empty count and a validity count locate nobody, so it stays published under every primitive. Two groups are governed by their own paragraphs below instead: detection, and the derived day counts, which are coarsened.

No threshold on a column's size makes such a statistic safe. With per-value counts published, sum is one equation over the withheld values and the predicate counts fix which of them are zero, negative or whole; a column of 0 and one other value divides back to that value, and a count-like column whose values grow rarer as they grow larger publishes the least sum its counts allow, which exactly one assignment of values reaches.

Bounds and percentiles are cell values too. Under mask and hash, range.min, range.max and every percentiles entry carry the substituted form; under drop the range and percentiles fields are omitted, the one primitive where redacting a bound and omitting it coincide. A consumer MUST NOT order, compare, or perform arithmetic on a bound in a column carrying the marker — a masked maximum still looks like a maximum, and two hashed bounds sort by digest rather than by value, so min may sort above max. The literals a marker governs are each values entry's value, range.min, range.max and every percentiles entry: under mask and hash each takes the primitive's form, under drop none is emitted.

A derived value is a cell value whenever the derivation can be run backwards. freshness.max_age_days and range.span_days (§2.2.4) are computed by arithmetic rather than read from a cell, but arithmetic that inverts is not exempt on that account: range.max = profiled_at - max_age_days, and profiled_at is always present (§2.2.1), so an uncoarsened max_age_days recovers the maximum a mask/hash/drop marker declares withheld — to the day, from any artifact. range.min = range.max - span_days recovers the minimum the same way wherever range is still present (mask/hash; drop omits it already). A temporal column carrying any redacted marker MUST therefore emit both fields floored to the nearest 90 days: coarsened = 90 * floor(value / 90). This holds under every primitive, drop included — freshness stays REQUIRED there (§2.2.3's matrix footnote), so without the coarsening a fully dropped column would still reveal a birth date to the day. freshness.classification is unaffected and still reads the true age: the bucket is coarse enough on its own (§2.2.4) that coarsening it too would only relabel a dormant column, not protect one.

The marker is mandatory wherever a redaction rule covers the column. An artifact that substitutes values silently is worse than one that omits them, because a consumer cannot tell measurement from fabrication — the same principle the scope block (§2.2.8) rests on. The marker states that a rule covers the column, not what the column still publishes: a covered column carries it even where it publishes no literal at all — a text column detected as prose (§2.2.3), or one naming values in its own unmeasured (§2.2.4) — so a reader can tell a withheld column from an ordinary one wherever both publish nothing, and so the rules that read the marker (§2.2.14) read a reliable signal. json and unsupported are outside it: their §2.2.3 row carries no cell value to withhold and forbids the marker itself. A values entry carrying no value in a column that declares no marker is an error.

A rule reaches a part through its column, through the part's own detections, or through the detections of any part it sits inside. A rule covers a part when it covers the part's column — a columns glob over <table>.<column>, or the column's own sensitivity/looks_like — or when its sensitivity/looks_like lists the part's own detection or the detection of any part the part sits inside (every proper prefix of its path, [*] steps included). Declaration order still decides, and the last covering rule wins. A covered part carries its own redacted marker and follows this section in full; the column above it carries none. A columns glob never names a part. Where a rule covers a part whose key set is data — a map's keys, a document's keys — the member parts whose paths spell those keys are withheld and are never read, since the statement reading one would spell its key; parts_found still counts them, but not the parts below them.

Detection is unaffected. looks_like and sensitivity are computed over sampled values that are never persisted, so a hashed email column still reports looks_like: email. That is correct: the shape claim describes the column, not the emitted literals.

A salt is a precondition of hash, not an option. An unsalted digest of an email is reversible by dictionary attack, so a producer offering hash MUST require a configured salt and MUST NOT default one. The salt belongs with credentials, never in a committed project config, and MUST be stable per project or every redacted column churns on every diff.

2.2.10 null_patterns — which columns are null together

null_rate describes one column at a time. It cannot say that two columns are null on the same rows, and a consumer that assumes they are not will read a filter on one as narrowing by a dimension the other already narrowed — and will read a LEFT JOIN whose null-producing side is null on exactly those rows as a wider query than it is.

A file whose data carries nulls records which columns carry them together in a top-level null_patterns block:

null_patterns:
  coverage: <float [0, 1]>            # ALWAYS; the listed counts over rows_scanned
  coverage_method: measured | bounded # OPTIONAL; whether the pattern counts agreed with rows_scanned
  patterns:                           # ALWAYS; may be empty
    - columns: [<column name>, ...]   # ALWAYS; the columns null in these rows, empty for none
      count: <int ≥ 0>                # ALWAYS; rows carrying exactly this combination

A pattern names columns, never positions. Every entry in columns MUST name a column present in this file's columns map. A bitstring or an index list would carry its meaning in a column order stated somewhere else, and adding a column to the table would silently change what every artifact written before it meant.

An entry is an exact combination, not a superset. A row counted under columns: [a] has a null and every other column populated, so the entries partition the rows they cover and no row is counted twice.

The all-populated pattern is listed. Rows carrying no null at all appear as an entry whose columns is empty. Omitting it would leave coverage describing a subset of the scanned rows without saying which.

Each combination appears at most once. Two entries naming the same columns would count the same rows twice, and the reconciliation rule below would fail with neither count wrong on its own.

Entries are ordered by count descending, ties broken by ascending lexicographic order of the columns array — the same shape §2.2.4 sets for values, and for the same reason: two runs over unchanged data MUST emit the same bytes. Which entries survive a producer's cap is a separate question from the order they are published in: at a count tie straddling the cap a producer MAY keep either, provided it keeps the same one on every run over unchanged data. §2.2.7 draws the same distinction for values.

coverage is the share of the scanned rows the listed entries account for: the sum of their count values over rows_scanned, rounded per §2.2.6. A producer that caps how many entries it lists MUST publish the coverage that cap leaves. 1.0 asserts the listed patterns are complete, and then the listed counts MUST sum to rows_scanned exactly.

The counts reconcile against each column's own null_count. For any column, the sum of count over the entries naming it MUST NOT exceed that column's null_count, and MUST equal it exactly where coverage is 1.0. This is the one arithmetic identity in the format that crosses from a table-level object to a per-column one, and it is what catches two figures that came from different reads of the same table.

coverage_method (enum, OPTIONAL; measured | bounded): whether an untruncated list of patterns agreed with rows_scanned (measured) or a producer detected the two disagreeing (bounded), because the pattern counts and the row count they are measured against were not read at the same instant — the same distinction §2.2.4 draws for values_coverage_method, applied to this block. Emitted only for a list a producer's own cap did not cut short; a truncated list is short by design, which is a different, already-explained condition coverage_method does not cover. A validator reads that truncation from the block itself: coverage below 1.0 with coverage_method absent. Such a list covers a subset of the scanned rows, so it can sum above rows_scanned, or name a column more often than its null_count, only where the table took writes between the two reads — the disagreement bounded discloses — and is reported with the same -bounded warnings. A sampled table whose statements each redraw an unmaterialized sample can show the identical symptom from a different cause — a working fix already exists for that one (a materialized draw removes it entirely, reading measured) — and coverage_method states the symptom it observed, not which of the two causes produced it.

Absence is a claim about the data, not about the producer. The block MUST be omitted when no column in the file carries a null, and MUST be present when any does — unless the file's own unmeasured list names it (§2.2.1), which is how a producer whose pattern count failed records that rather than asserting the table has none, or its disabled list names it (§2.2.21), which is how one switched off records that. Both cases are checkable against the null_count of every column in the same file, so an absent block with no marker never leaves a reader deciding between "no nulls" and "not measured".

A pattern is a measurement, not a constraint. That b was null on every scanned row where a was null is an observation over the rows that were read, not a rule the database enforces, and a consumer MUST NOT treat it as one — the same terms §2.3.8 sets for an inferred edge. Under a scope block it says less still: a combination occurring on few rows may not have been drawn at all, so the absence of a pattern is not evidence that the combination does not occur.

Redaction does not interact. A pattern discloses which columns were null together and on how many rows, never a cell value, so §2.2.9's markers neither apply to it nor alter it.

2.2.11 physical_layout — the declared clustering or partitioning key

A row count and per-column cardinality say nothing about how a table is physically laid out, and on a warehouse that clusters or partitions, layout is the difference between a pruned scan and a full one. That fact already exists in ddl.sql, recoverable only by parsing a dialect's own clustering or partitioning syntax; physical_layout states it as structured data instead.

physical_layout:
  mechanism: cluster | partition | sort # ALWAYS; named honestly per adapter
  keys:                               # ALWAYS; ordered, may not be empty
    - expression: <string>            # ALWAYS; the declared expression, verbatim (§2.1.3's credential masks applied)
      column: <string>                # OPTIONAL; the base column, when the expression resolves to one

mechanism names the mechanism, not a judgment. cluster for Snowflake's clustering key, partition for Postgres declarative partitioning and MySQL PARTITION BY, sort for Redshift's SORTKEY — all three say the same thing to a consumer sizing a query (these columns prune, the rest do not), so one field carries them rather than forcing a consumer to know which dialects use which word. Redshift's DISTKEY fails that test — it decides which node a row lands on, a join-colocation fact rather than a pruning one — and is deliberately left unexpressed; a table declaring only a DISTKEY emits no physical_layout block at all.

keys is ordered, because the order is not decoration. A multi-column key prunes far more on its first component than its last (cluster by (vault_id, reading_id) prunes almost entirely on vault_id), and the array's own order is that ranking — a producer MUST NOT reorder it.

column is what a predicate matches against; expression is what was declared. cluster by (logged_at::date) yields the expression logged_at::date and the column logged_at — a consumer needs the column to match a WHERE clause against, and the expression to know what was actually declared rather than assume. column MUST be omitted when the expression resolves to no single column (a function call over more than one column, or one dbprint cannot parse); the expression alone is still recorded. When present, column MUST name a column present in this file's columns map, on the same precedent §2.2.10 sets for null_patterns.

The two surfaces agree, or the artifact is inconsistent. Every column named as a column in physical_layout.keys MUST carry physical_layout_key: true, and every column carrying that marker MUST be named there — the table-level list and the per-column flag are two views of one fact, and a producer emitting one without the other has written a self-contradicting file.

Declared, never measured. physical_layout states what the schema declares, never how well the table is actually clustered at any moment — Snowflake's automatic clustering can leave a declared key poorly maintained, and clustering depth is a measurement that costs a query and goes stale immediately. A producer MUST NOT publish a cost or pruning claim anywhere in this block; naming the key is a schema fact, asserting what it saves is not.

§7.3 sets this absence beside the others a reader has to interpret.

Absence means not clustered, never not checked. Every producer MUST answer whether a table declares a physical layout key. An adapter that cannot express the concept for a given engine surface, or a table that genuinely declares none, both emit no physical_layout block — the two are indistinguishable from the artifact alone, which is the same absence-has-one-meaning discipline §2.2.10 applies to null_patterns. A producer whose read failed says so through the file's own unmeasured list (§2.2.1) instead, which is the one absence here that is not this claim.

physical_layout_key (bool, OPTIONAL, per column): true on every column named as a column in the table's own physical_layout.keys, omitted otherwise. A declared catalog fact, not a detection, so it sits beside physical_name rather than under inferred. Carries no order — a consumer wanting the pruning priority reads physical_layout.keys at the table level; this marker exists so the fact is visible where a reader is already looking, in the per-column statistics.

2.2.12 grain — what identifies a row

inferred.candidate_key (§4.2) is single-column by construction. A table whose rows are identified by more than one column — (vault_id, logged_at), (taxon_id, trial_year) — has no field to say so, and a consumer is left re-deriving the grain from a GROUP BY or, worse, assuming a column that merely looks like a key is one.

grain:
  keys:                                # ALWAYS; may be empty
    - columns: [<column name>, ...]    # ALWAYS; one or more, in declaration order for a `declared` entry
      detection: declared | measured   # ALWAYS
  search:                              # OPTIONAL; present only when the measured probe ran
    exhausted: <bool>                  # ALWAYS once `search` is present

A conforming producer always emits the block. Declared-key introspection is catalog metadata and costs nothing, so a producer MUST include grain on every table it emits, with keys an empty array rather than the block omitted when nothing declared and nothing measured applies — the block itself is the answer "nothing identifies a row here", not its absence.

Every declared unique key is emitted, at every arity, in declaration order. A table declaring a single-column PRIMARY KEY still gets a grain entry for it, even though inferred.candidate_key says the same thing on that column: the two are independent axes — one declared, one measured — and a consumer reading only grain gets the full answer without cross-checking inferred on every column. A declared key comes from a constraint or unique index whose members are all columns: an index with any expression member declares no key, and an index's included columns are never part of one.

A measured entry is a probe result, never a constraint. It states that the named columns were jointly unique over the rows read at profiled_at, on the same footing §2.2.10 gives a null pattern and §2.3.8 gives an inferred foreign key: an observation, not a guarantee the database enforces. A producer and a consumer MUST NOT call it a key.

The measured search is bounded, column-pairs only, and arithmetic-pruned before any statement is issued. A producer MAY search for an undeclared two-column grain among columns carrying no null (COUNT(DISTINCT a, b) diverges on nulls across dialects, so the candidate space excludes them rather than special-case each one), restricted to pairs where cardinality(a) * cardinality(b) >= row_count — necessary, not sufficient, and free to compute from fields already on disk. A producer MUST cap how many candidate pairs it actually tests; a three-or-more-column grain is out of reach.

search.exhausted distinguishes "nothing found" from "the search stopped early". true when the search tested every arithmetic-pruned candidate; false when a per-table cap cut the search short before it could. Both are measurements, and neither is the same bytes as search being absent entirely, which means the measured search never ran at all — because the file carries scope, row_count or rows_scanned is 0, or some column already carries inferred.candidate_key with no inferred.candidate_key_exception (SPEC 4.2) and a pair search would answer a question that column's own exact measurement already settled. A column whose candidate_key carries an exception is a near-uniqueness ratio, not the answer a pair search gives, and suppresses nothing. A consumer reading an empty keys list MUST check search before concluding no grain exists: absent means the search did not run, exhausted: false means it stopped before testing every candidate, and only exhausted: true means the search itself found nothing.

Never emitted under scope, and never on an empty table. Uniqueness measured over a sample is not uniqueness (§2.2.8) and every combination is trivially unique on a table with no rows — a measured entry MUST NOT appear in either case, though declared entries are unaffected: they are a schema fact, not a measurement, and hold regardless of how much of the table was read.

§7.3 covers an absent block and §7.4 an empty keys list.

No per-column marker. Unlike physical_layout_key, membership in a grain key is not restated on the column — inferred.candidate_key already carries the single-column case, and a multi-column one has no single column to mark without implying an ordering or a weight the combination does not have.

2.2.13 dependencies — which columns determine which

A GROUP BY status, status_label and a GROUP BY status return the same rows whenever every status value pairs with exactly one status_label, and a print with no way to say so leaves a consumer re-deriving the redundancy from the data or, worse, assuming a wider grouping than the table actually has.

dependencies:                        # ALWAYS; may be empty
  - determinant: <column name>       # ALWAYS
    dependent: <column name>         # ALWAYS
    strength: <float (0, 1]>         # ALWAYS

A conforming producer always emits the block, [] for nothing found — the same "measured, not skipped" convention §2.2.12 sets for grain. The one exception is a run whose probe failed: it MUST omit the block and name it in the file's own unmeasured list (§2.2.1), since [] would state that nothing determines anything.

strength is cardinality(determinant) / cardinality(determinant, dependent). 1.0 when every determinant value pairs with exactly one dependent value (the exact case); each additional distinct pairing under one determinant value lowers it. Both operands are cardinalities already published elsewhere in this file, computed under the same collation (§2.2.2). This is a group-level measure — how many distinct pairings exist — not a row-weighted share of consistent rows; a table where one determinant value carries a thousand exceptions and one where it carries a single exception can report the same strength if the count of distinct wrong pairings matches.

Only a pair clearing the 95% confidence bar §4.1.3 already sets is worth publishing. A producer MUST NOT emit an entry below that threshold — two independent columns with any cardinality asymmetry between them will still show some fraction less than the threshold, and publishing every candidate a producer measured would flood the block with noise a consumer cannot act on.

A determination is a measurement, not a constraint. It states that the named columns paired this way over the rows read at profiled_at, on the same footing §2.2.10 gives a null pattern and §2.3.8 gives an inferred foreign key. A producer and a consumer MUST NOT call it a rule the database enforces.

Never emitted under scope, and never on an empty table. A dependency measured over a sample is not a dependency (§2.2.8), and every combination is trivially functional on a table with no rows — dependencies MUST be empty in both cases. A consumer checking whether the search ran at all reads scope and row_count, the same signals §2.2.12 points a grain reader to, plus the file's own unmeasured list (§2.2.1) for the run whose probe failed outright.

Direction is not interchangeable. determinant names the column whose value fixes the other; a pair may hold in one direction, both (a mutual dependency, published as two entries), or neither. Producers MUST NOT publish a pair in the direction cardinality rules out: cardinality(determinant) >= cardinality(dependent) is necessary for the direction to be possible at all, since a function's image has no more distinct values than its domain.

No new privacy surface. A dependency discloses which columns move together and how strongly, never a literal value; §4.4's markers neither apply to it nor alter it, the same terms §2.2.10 sets for a null pattern.

2.2.14 sketch — whether two key sets overlap, without a second scan

relationships.yaml states that an edge exists; it cannot state whether the referencing column's values actually fall inside the referenced column's, because that requires reading both tables together, and the two may not even share a print. sketch answers a narrower, offline question instead: a fixed-size summary of one column's distinct value set, from which a consumer computes an approximate overlap against another column's summary — same print, a different print, or a different database entirely — with no query against either source.

sketch:
  method: kmv_md5_lo64      # ALWAYS; identifies the hash, k and encoding as one bundle
  values: <base64 string>   # ALWAYS; packed sketch payload, see below

What it is. A k-minimum-values (KMV) sketch: the k smallest values of hash(v) over the column's distinct non-null values, hash and k fixed by method. kmv_md5_lo64 is k = 1024, MD5, low 64 bits of the digest read as an unsigned big-endian integer, no salt and no seed — the hash is unkeyed and deterministic by construction, so two producers hashing the same value always agree. A KMV sketch, unlike HyperLogLog, supports set intersection and union directly from two sketches' own minimums, which is the property this field exists for; a producer MUST NOT substitute a HyperLogLog or any cardinality-only sketch under this field.

values encoding. The retained hashes, ascending, each packed as 8 bytes big-endian unsigned, concatenated, then base64-encoded (standard alphabet, per RFC 4648 §4). Decoded length is a multiple of 8; dividing by 8 gives the retained count, which is k exactly when the column has at least k distinct values and the exact distinct count otherwise — a decoded values shorter than k * 8 bytes is not truncated, it is the whole distinct set, and the sketch it produces is exact rather than estimated.

Canonical encoding. Two producers must hash the same logical value to the same bytes, or their sketches are incomparable regardless of which database either read from. Before hashing, a value is rendered to a canonical UTF-8 byte string per its SQL type — the byte form MUST NOT vary by adapter, by the column's native type spelling, or by locale:

KindSQL typesCanonical form
integersmallint, integer, bigint, int, tinyint, mediumint (any dialect spelling)Decimal string, no leading zeros, ASCII - for negative, no + for positive (42, -7, 0)
exact decimaldecimal, numeric, numberSPEC §2.2.6's positional-decimal rendering, scale-preserving (trailing zeros within the column's own scale are kept: a numeric(10,2) value of 4 renders 4.00)
textvarchar, text, char, character varying, character, string, uuid, fixedstring, tinytext, mediumtext, longtextRaw UTF-8 bytes, unmodified. A native uuid column is cast to text first, which every supported adapter renders in lowercase dashed form (xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx) by default — the same form §4.1.1's uuid pattern matches — so no separate UUID rule exists. Inherits §2.2.2's collation dependence: two engines that disagree on a text column's distinct set (a case- or accent-sensitivity difference) produce different sketches for it whatever this encoding does; the format does not resolve that, only states it
booleanboolean, boolASCII true or false
binarybytea, blob, tinyblob, mediumblob, longblob, binary, varbinary, varbyte, binary varying, bytes, imageThe value's own bytes, unencoded — no hex or base64 spelling first. Every engine hashes a binary value directly, so no adapter needs a cast before hashing
temporaldate, date32, time, timestamp, timestamp with time zone, timestamp without time zone, time with time zone, time without time zone, timestamp_ntz, timestamp_ltz, timestamp_tz, timestamp_ns, timestamp_ms, timestamp_s, datetime, datetime64, yearISO 8601: YYYY-MM-DD for a date-only value; YYYY-MM-DDTHH:MM:SS[.ffffff] for a value with no timezone concept (nothing to normalize, so nothing to mark); YYYY-MM-DDTHH:MM:SS[.ffffff]Z for a timezone-aware value, normalized to UTC first; HH:MM:SS[.ffffff] for a time of day, never a fabricated date, and the same normalized to UTC with a trailing Z for time with time zone; YYYY for a year. The fraction is all six digits or none. This binds every adapter the same way regardless of that adapter's own range.min/range.max rendering - §2.2.4 permits one adapter to omit Z there even on a timezone-aware type (a display choice), but the sketch cannot: two adapters hashing the same instant to different bytes breaks cross-adapter agreement

A floating-point type (real, double precision, double, float, money) has no canonical encoding and MUST NOT carry a sketch — equality-based set membership on a floating-point value is not stable across engines, and a join key of this type is not a case this field is built for. A type outside every row above — json/jsonb/variant and unsupported types among them — also has no canonical encoding and MUST NOT carry a sketch.

Test vectors (MD5, low 64 bits, unsigned big-endian — every adapter's hash expression MUST reproduce these):

KindCanonical bytesLow-64-bits value
integer4215584161582054922406
integer-75585573698858288085
exact decimal4.0010071140874863616590
texthello13362634815750784402
booleantrue317521853213362953
binarythe two bytes 0x0a 0xff8458408032758825719
temporal2026-03-09T14:27:36Z9552359349123505343
temporal (time of day)00:00:3710604805012337869669

Which columns carry one. Unless the file's disabled list names sketches (§2.2.21), a producer MUST emit sketch on every column, in a table read unscoped, whose SQL type falls in the table above, is not redacted (see below), and that meets at least one of:

  • it is, on either side, a column/target_column (declared or inferred refers_to) or referencer_column/column (referenced_by) entry in that table's own relationships.yaml;
  • it carries a single-column PRIMARY KEY or a sole single-column UNIQUE constraint;
  • its cardinality is at most k (the method's own retained size — 1024 under kmv_md5_lo64), so the sketch it would carry is exact rather than estimated;
  • it carries inferred.candidate_key: true (§4.2).

Composite keys sketch per member column — there is no joint sketch for a multi-column key, on the same footing §2.3.4 gives every other per-edge measurement; a composite key's own membership in the second bullet above is therefore never satisfied, only a single-column key's. A producer MAY sketch other columns beyond this set; this specification does not require it, and does not require a producer to expose a way to opt into it.

Redacted columns carry no sketch. A KMV sketch is a set of hashes of real cell values; an unsalted one is a dictionary-attackable enumeration of a column someone deliberately withheld, and a salted one is incomparable across projects — the property this whole field exists for. Neither is worth shipping, so sketch is absent from a redacted column under every primitive, not only drop.

A sketch of a scoped column is not shipped either. scope (§2.2.8) narrows a table's read to less than the whole; a sketch built from a re-drawn narrowing is not reproducibly the same rows a second generate would draw, and §2.2.7's determinism expectation does not hold for it the way it does for a full scan. A producer MUST NOT emit sketch for a column whose file carries a top-level scope block: the absence and its cause are stated, rather than a value that cannot be reproduced.

Determinism. Two runs over unchanged data MUST emit byte-identical sketch.values — the hash is unkeyed, so nothing about the run itself can move it, and the k smallest values of a fixed set are themselves fixed.

The error bound is a function of what is actually comparable, not a flat percentage. Two sketches agree only below the tighter of their two retained thresholds — the count of hashes genuinely answerable is that intersection's size, not k, and it varies per edge. A truncated sketch's estimate carries roughly 1/sqrt(answerable count); a published overlap is not a data-quality signal until it is read against that edge's own margin, and a small answerable count widens the margin sharply. Only an edge where both sketches are exhaustive needs no such margin — neither side truncated, so the shared hashes are the whole comparable set, not a sample of it. An exhaustive sketch measured against a truncated one still measures over a sample: the truncated side's own retained threshold, not the exhaustive side's completeness, decides how much of it was comparable, and the same margin applies to that measurement as to any other truncated one. observed.answerable_count (§2.3.10) publishes that count on every sketch-measured edge, so a consumer computes the margin from the print alone rather than reading it as a stated bound.

Membership is answerable per sketch, not per field. A sketch whose decoded values length is below k is exhaustive, so hashing a candidate value's canonical bytes and testing it against the retained set answers membership exactly — absence is proof, presence is certain up to a 64-bit collision. A decoded length of k or more is truncated, the same test the encoding rule above already draws that line with, and membership for a value the sketch does not itself hold is not answerable at all: only a value whose hash falls below the retained threshold is checkable from the sketch. A consumer wanting to test a value against a truncated sketch needs a Bloom filter, which this field is not. The candidate value is canonicalized per this section's encoding and matched on the same collation terms §2.2.2 gives the column — a case- or accent-folded miss reads as a real absence, not evidence either way. Membership is as of profiled_at, the same staleness every other statistic in this file carries; it is not a live check against the column.

A sketch is not a privacy leak by itself, and it still discloses something: a value's presence in an exhaustive sketch is testable, as above. §4.4's sensitivity detection and its redaction requirement above are the boundary; nothing else in this section changes what §4.4 already requires.

2.2.15 catalog_only — no query was issued

Every field above this line assumes a producer queried the object it describes. row_count assumes a COUNT(*) or a system-table estimate; every per-column statistic assumes a read of the rows themselves. Neither exists for an object a producer describes from catalog metadata alone — introspecting a view's declared output columns without reading through it, for one such case, or an object whose rows live in another system (§2.2.20), whose read would reach that system, for another.

catalog_only: true       # OPTIONAL; present and true only when no query was issued

The marker is independent of type. A producer MAY query a plain view; a view it queried carries no marker and follows every rule above exactly as a table's file does. type says what the object is, catalog_only whether it was read.

Presence states the fact; there is no false value. Like scope (§2.2.8), the field exists to be present or absent, not to hold a boolean either way — a producer MUST NOT emit catalog_only: false. Absence is the ordinary case: a query was issued, exactly as every other field in this section already assumes.

Permits the absence of row_count and row_count_method. Both are REQUIRED unless catalog_only is present, in which case a producer MUST NOT emit either (§2.2.1) — an object nothing was queried for has no count to report, exact or estimated.

Also permits the absence of physical_layout, merging and dependencies. Each otherwise states a finding by its absence (§2.2.11, §2.2.19, §2.2.13). dependencies requires a query to measure and MUST NOT be emitted under the marker, the same rule row_count follows. physical_layout and merging are declared schema facts rather than measurements, so a producer MAY still emit either here from catalog metadata alone (§2.2.15's per-column allowance already treats physical_layout_key the same way) — only its absence needs the marker, never its presence.

depends_on is the opposite case: this is exactly where it is expected, not merely tolerated. A plain view described without a query carries catalog_only, and depends_on is catalog-derived on the same footing as physical_layout — never a query result, so catalog_only permits nothing about it that the field's own §2.2.17 rule does not already state on its own terms. A matview file, which never carries catalog_only, follows the same §2.2.17 rule unmodified, and so does a plain view its producer queried: it carries depends_on all the same.

A column carries only catalog facts. sql_type, nullable, classification, physical_name, collation, physical_layout_key and derived_from — schema and DDL facts, not measurements — are the only fields a column may carry while the file's catalog_only is present. Every other per-column field states something read from the object's rows; a producer MUST NOT emit any of them under the marker, classification itself still following the ordinary §3.2 priority order on every input available without a query — sql_type and a foreign key, declared or naming-inferred, both catalog-derived (§3.3).

Not a redaction and not a failure. catalog_only states that no query was issued, never that one was attempted and came back withheld or errored — a masked value and an unattempted one are different gaps, and this marker covers only the second.

2.2.16 timeline — one column's activity over time

range and percentiles describe a temporal column's extremes; neither says whether the most recent period is complete or falling off. A table refreshed nightly and read the morning after has a partial final day no single-value statistic can distinguish from a genuine drop in volume, and a month with no activity at all leaves no trace in either field.

timeline:
  column: <column name>            # ALWAYS
  unit: day | week | month         # ALWAYS
  buckets:                         # ALWAYS; may be empty, ascending by `start`
    - start: <instant>             # ALWAYS
      count: <int >= 0>            # ALWAYS
  coverage: <float [0, 1]>         # ALWAYS

One anchor column per table, chosen by a fixed rule, never left to the reader to guess. A producer MUST pick column as follows, in order, and MUST NOT emit the block once no step applies:

  1. A temporal, non-redacted column whose raw type carries a calendar date, named in physical_layout.keys (§2.2.11) — the column the table is itself organized by.
  2. Otherwise, among the remaining temporal, non-redacted, calendar-typed columns, the one with the lowest null_rate, ties broken by the higher cardinality, then by column name.
  3. No such column exists, or every temporal column is redacted: timeline MUST be absent.

A TIME-only or YEAR-only column never qualifies at either step, however low its null rate: neither carries a calendar day, week, or month to bucket into.

unit is chosen from the anchor's own measured span, not configured. A producer picks the coarsest unit that still resolves recent activity — finer for a short span, widening as the span grows — so the bucket list stays a manageable size on a table spanning years rather than months. The exact thresholds are a producer's own choice, the same latitude §2.2.12 leaves the measured grain search's own cap; what this section requires is that the unit is fixed once per table, every bucket the same width, never adapted bucket to bucket.

A bucket exists only where at least one row does. buckets lists exactly the day/week/month spans containing at least one non-null anchor value, ascending by start; a span with none is simply absent from the list, never published as a zero-count entry. A consumer reading two consecutive buckets whose start values are not adjacent at unit's own width has found a gap — no row fell in between.

start is the bucket's own beginning, in the anchor column's domain rendering. The same ISO 8601 rule §2.2.4 gives range.min/range.max applies here: a DATE-typed anchor renders a bare date, a TIMESTAMP-typed one a full instant, both UTC-normalized.

A timezone-aware anchor is bucketed in its column's frame: the zone its type declares (ClickHouse DateTime('Europe/Berlin')), and UTC otherwise — including a type whose values each store an offset (Snowflake TIMESTAMP_TZ), since a timeline needs one calendar for the whole column. Each start is the instant that frame's day, week or month begins, UTC-normalized like every other instant in the file, and adjacency at unit's width is read in that frame: in a declared zone that observes daylight saving, two adjacent day buckets' starts are 23 or 25 hours apart across the change.

coverage is the listed counts over rows_scanned, the same scanned-set-relative convention values_coverage (§2.2.4) and null_patterns.coverage (§2.2.10) use. It reaches 1.0 only when every non-null anchor value landed in a listed bucket; a null anchor value counts toward rows_scanned but no bucket, which is the one way coverage falls short of 1.0 here.

The share is bounded, and 1.0 means exhaustive rather than merely rounding to it. The bucket counts and rows_scanned are read by separate statements, so a table taking writes between them can leave the counts summing above the rows they are divided by. A producer MUST NOT publish a ratio above 1.0: it emits the largest value below it that this precision can carry, exactly as §2.2.4 requires of a truncated values_coverage, and reserves 1.0 for a bucket total that genuinely equals rows_scanned. A validator recomputes coverage through that same bounded share, deriving exhaustiveness from the two counts rather than from the published field — so a 1.0 over a partial sum is still rejected — and reports the overrun itself as stats.timeline-buckets-exceed-rows-scanned, a warning, on the terms stats.values-sum-mismatch (§2.2.4) sets for the same disagreement — unconditionally, since timeline carries no coverage_method with which a producer could disclose it.

Never emitted under scope, on an empty table, or where the file's disabled names timeline. A bucketed count over a sample is not a timeline (§2.2.8), and there is nothing to bucket on a table with no rows — timeline MUST be absent in both cases, the same rule §2.2.12 and §2.2.13 already state for grain and dependencies. A file whose disabled names timeline (§2.2.21) carries neither the block nor any column's populated.

No new privacy surface. A bucket count discloses volume over time, never a literal value from the anchor column itself beyond its own rounded start; §4.4's markers neither apply to it nor alter it, the same terms §2.2.10 sets for a null pattern.

§7.3 covers an absent block, §7.4 an empty buckets list.

2.2.17 depends_on — what a view reads

A warehouse built with a transformation tool routinely has more views than tables underneath them, and the layer an analyst queries is the interface, not the substrate ddl.sql alone describes. Which other objects a view or matview reads is already captured DDL text — depends_on publishes it as data instead, the same move §2.2.11 makes for a clustering key buried in CREATE TABLE syntax.

depends_on:
  - <fully-qualified name>

Every view/matview carries the key, or explains why it does not. The list comes from the catalog exclusively — pg_depend/pg_rewrite on PostgreSQL, information_schema.VIEW_TABLE_USAGE on MySQL 8, ACCOUNT_USAGE.OBJECT_DEPENDENCIES on Snowflake — never from parsing ddl.sql. A dependency list that is sometimes exact and sometimes a guess, with nothing in the artifact saying which, is the same encoding defect §2.3.8's naming-inference rule exists to avoid on relationships.yaml. A table MUST NOT carry this field under any circumstance; it names what a view reads, and a base table reads nothing.

Two encodings, so absence keeps one meaning. depends_on: [] states that the catalog was read and this object reads no other object in the print. The key omitted entirely states that the dependency read did not happen — no grant to the catalog table, no such catalog table on this engine version, or the read failed for any other reason. A producer MUST NOT collapse the two: emitting [] for an object whose catalog was never read would give [] two meanings on every engine to cover one engine's gap.

Direct dependencies only. A view over a view names that view, never the view's own upstream sources — the same one-hop rule regardless of how many views a chain of CREATE VIEW ... AS SELECT * FROM other_view stacks. A consumer wanting the transitive set walks the graph itself, one depends_on list at a time; publishing both the direct and the transitive set under one name would store two different facts under one field.

An out-of-scope dependency is named by its FQN like any other. A view reading a table the connection's selectors exclude from the print still lists that table's fully-qualified name — the same footing §2.3.7's External FK target row already gives a refers_to entry whose target_table is not itself in the print. A consumer checks the manifest for whether the name resolves locally; its absence there means external, not that depends_on was wrong to list it.

Object grain, not row grain — never confused with a foreign key. relationships.yaml models a relation between rows at column grain; depends_on states that one object's definition reads another object at all, with no column, no cardinality, and no join implied. §2.3.5 states this explicitly as a pointer from the other direction.

Catalog latency is a source's own limitation, not this field's. Snowflake's ACCOUNT_USAGE.OBJECT_DEPENDENCIES lags up to three hours behind a freshly created view, which can return no rows for an object the catalog has not yet indexed and so publish depends_on: [] where the true answer is non-empty. That is a stated limitation of the source, not a defect this field's own encoding needs a third state for — the two-encoding rule above stands regardless of which source was read or how current its data is.

A dependency the catalog cannot fully resolve is reported as the source gives it, never as a guessed name. A broken or dropped dependency inside a view's definition is an edge case a catalog read may not resolve cleanly; a producer publishes whatever the source reports rather than inferring a name from context, and the entry's absence for one unresolvable object is not a claim that the view reads nothing — the same object-grain "dependency read did not happen" state above, applied to one dependency inside an otherwise complete list.

§7.3 covers an absent block, §7.4 an empty depends_on list.

2.2.18 parts — what a column holds

An array, a record, a map or a JSON document holds values a query reaches inside it — tags[*], items[*].sku, attrs->>'status' — and none of them is a column the catalog lists. parts publishes what a producer found inside such a column, under the column, one entry per part:

columns:
  items:
    sql_type: STRUCT(sku VARCHAR, qty INTEGER)[]
    classification: composite
    parts_found: 3
    size: {min: 1, max: 5, avg: 2.5, p95: 5.0}
    parts:
      "[*]":
        sql_type: STRUCT(sku VARCHAR, qty INTEGER)
        classification: composite
        occurrences: 250
        null_count: 0
        null_rate: 0.0
      "[*].sku":
        sql_type: VARCHAR
        classification: categorical
        occurrences: 250
        # ... the categorical block, over its 250 occurrences

One flat map, keyed by path. A part's key is its path relative to the column, whatever its depth, and a part never carries parts of its own; its parent is its longest proper prefix, which the map also lists. A part is not a column: no table-level block (null_patterns, grain, dependencies, timeline, physical_layout) names one, and it never becomes a columns key, a path segment on disk or an annotation key.

The path grammar is dbprint's own, relative to the column:

path    = 1*step
step    = member / element / keys
member  = "." name / "[" json-string "]"
name    = (ALPHA / "_") *(ALPHA / DIGIT / "_")      ; ASCII only
element = "[*]"                                     ; every element of an array
keys    = "[keys]"                                  ; the key set of a map

A member step names a record field, a union member, a document key or one map key's value. It takes the .name form exactly when the name matches name, and the bracket form otherwise; json-string is the RFC 8785 serialization, escaping only ", \ and U+0000–U+001F, with the short forms \b \t \n \f \r where one exists and lowercase \u00hh otherwise. The spelling is therefore unique: two paths name one part exactly when their strings are equal, and a producer MUST emit that spelling. Member names keep their case and are never folded — a document key is data, and §1.5's folding applies to path segments on disk, which a part never becomes. A non-string map key is written as the text §2.2.4 publishes for that value (["3"]). A reader shows a part as its column key followed by its path: items[*].sku, attrs["user-id"], labels[keys].

A part's block is the ordinary column block. It is classified by the §3.2 order with no foreign key (no edge ends at a part), gets the §2.2.3 row its classification selects — values, range and the rest included — and follows the part delta stated beneath that matrix. A part of a part that was itself descended into classifies composite and lists nothing of its own; its parts are its siblings in the column's map. One row more is removed on a part that is an array's floating-point elements (a path ending in [*] whose sql_type is a binary floating-point type) classifying numeric: it carries no values, frequencies or distribution, since the list pools every position of every array into counts no predicate reads; its range, percentiles and the rest stay. Such a part with few distinct values still reaches categorical first and lists them.

occurrences (int ≥ 0, REQUIRED on a part) is the number of instances of the part in the scanned rows: every element of every non-null array for [*], and every non-null parent instance holding the member for a member step. It is the population every ratio in the block divides by — null_rate = null_count / occurrences, cardinality_ratio = cardinality / occurrences — and a values list sums to occurrences - null_count. A member absent from an instance is not an occurrence; a member present with a null value is a null occurrence. The share of parent instances holding a member is occurrences over the parent's non-null count (the column's scanned rows for a top-level part), so a consumer derives it rather than reading it.

size (OPTIONAL on a composite or json column and on a part of either) is the number of immediate parts each non-null instance holds — the elements of an array, the entries of a map, the members of an object — in length's shape (§2.2.4). It is a count, not a literal, so a redacted marker leaves it alone. A record has a fixed member count and carries none.

An array is read through [*]. A producer descends into an array by its element part, [*], whose rows are the elements of every non-null array — the rows an UNNEST, arrayJoin, explode or FLATTEN yields — so a NULL element is a null occurrence and a NULL or empty array contributes none. An array of arrays reaches [*][*] where the engine types the inner array as one; Postgres does not (integer[] is every integer array, whatever its dimensions), so a Postgres array has one part, [*], over every element in storage order, and its size is cardinality. A Snowflake ARRAY declares no element type: its [*] takes the one scalar kind every non-null element holds, read through a cast to it, or VARIANT when the kinds differ. An array column and an array part carry size and empty_count (the non-null instances holding no element); size.min == 0 says empty arrays exist, empty_count how many. Where the engine counts distinct arrays, the column carries cardinality too. An array of floating-point elements whose non-empty instances all have one length also carries norm and zero_count with §2.2.4's meaning, so an embedding stored as an array says whether inner product ranks as cosine; it stays composite, never vector.

A record or a union is read through its members. A producer descends into a record (a struct, a composite type, a tuple) or a tagged union by one member step per declared member. A record field occurs in every non-null record instance, its value null or not — a record whose fields are all null is still an instance, not an absent one — so each field's occurrences is its parent's non-null count, and a field whose every parent instance is null is not a part — sql_type still declares it. A union member occurs in the instances whose tag names it, so the members' occurrences sum to the parent's non-null count, a member holding a null value is a null occurrence, and a member no scanned instance holds is not a part. A tuple element with no name is named by its 1-based position (["1"]), and a union whose members are told apart by type alone — a ClickHouse Variant or Dynamic — names each member by its type (.String, ["Array(Int64)"]). A record or a union carries no size and no cardinality: its parts describe it.

A map is read through [keys] and one member step per key. A producer reads a map (a duckdb, Snowflake or ClickHouse MAP, a Databricks map, a Postgres hstore) as its entries, one row per key each non-null map holds, never through a lookup by key — a lookup cannot tell an absent key from a null value, and on ClickHouse not from a stored default. [keys] has one occurrence per distinct key of each non-null map: its occurrences is the sum over maps of their distinct keys, its null_count is 0 (no engine admits a null key), and a key's count in values is the number of maps holding it; its sql_type is the key type. A member step ["k"] names one key's values: one occurrence per non-null map holding k, so a held key with a null value is a null occurrence and a map without k contributes none; its sql_type is the value type, and a value that is itself a map, array or record descends by the same rules. A map column and a map part carry size (entries per non-null map; a ClickHouse map counts a repeated key at each entry, while [keys] counts it once) and empty_count (the non-null maps holding no entry), and never a whole-map cardinality. Key candidates are pre-cut to the top parts.max by occurrences, as unbounded candidate sets may be (below).

A key is a cell value, and a member step's path spells it. A path is never redacted — it is a parts key, a diff's part field and a context row — so while a map's [keys] part carries a redacted marker, or is not listed, no member step of that map is published, nor any part below one. parts_found still counts them, except below a member whose [keys] carries a marker: a producer MUST NOT issue a statement that spells a key its key set withholds, so it never reads that member and never finds what lies below it. A rule covering the column covers [keys] as it covers any part (§2.2.9), and [keys] runs its own looks_like and sensitivity detection over the keys, so a map keyed by email addresses meets an existing looks_like: [email] rule.

A JSON document is read through [keys], its members and [*], typed value by value. A producer descends into a document column (Postgres json/jsonb, MySQL JSON, Snowflake VARIANT/OBJECT, BigQuery JSON, Redshift SUPER, Databricks VARIANT, duckdb and ClickHouse JSON), and the column stays json. Its object instances offer [keys] and one member step per key, read exactly as a map's (above: one [keys] occurrence per distinct key of each object, one member occurrence per object holding the key, a key repeated within one object counted once), and its array instances offer [*], one occurrence per element; a scalar instance holds no part. A member or element whose value is the JSON literal null is a null occurrence. size counts an object's distinct keys or an array's elements. A part's sql_type is the engine's type name (types above) shared by every non-null occurrence; where all are numeric it is the widest of them (integer, then decimal, then floating point); otherwise it is the engine's general document type (duckdb JSON, Postgres the column's own json or jsonb, MySQL JSON, Snowflake and Databricks VARIANT, BigQuery JSON, Redshift SUPER), the part classifies json and carries its own types. Values are read through the matching scalar reader, so a part of strings profiles as text and one of numbers as numbers; a part whose values are objects or arrays is itself descended. Key withholding applies to a document's [keys] as to a map's; [*] spells no key and stays listed. ClickHouse stores a JSON null and an absent path alike, so there a member's occurrences counts only non-null values and its null_count is 0, and a key holding a dot is a nested path (.a.b).

parts_found (int ≥ 0, REQUIRED wherever parts is) is the number of distinct parts the producer found within the depth bound. Greater than the number of entries means the cap below cut the rest, or redaction withheld parts whose paths spell cell contents (a map's keys, above); it is to parts what values_coverage is to a value list.

Which parts are kept. A producer bounds a descent by two numbers it records in the manifest's profile (§2.5): parts.max caps the parts per column, and parts.max_depth caps the steps in a path. A producer walks the candidates by occurrences descending, ties broken by path text in code-point order, and keeps a path only once its parent part is kept — a child that outnumbers its parent, as an array's elements can, waits for it — until parts.max are kept. Where a candidate set is unbounded (map keys, document keys), a producer MAY pre-cut it to the top parts.max by occurrences, which can differ from this order only among candidates tied at the boundary.

A column a producer did not descend into carries no parts. Descent is per column and best-effort: one that fails leaves the column as it would otherwise be — unsupported, or json without parts — and the run goes on. parts is never named in unmeasured. A file whose descent was switched off names parts in disabled (§2.2.21). §7.2 lists what an absent parts can mean.

2.2.19 merging — rows a merge has not yet combined

ClickHouse's merging engines combine the rows that share a sorting key only when a background merge joins the parts holding them, and parts in different partitions never merge. A plain read returns the stored rows — duplicate keys, partial sums, a state and its cancel row — so row_count, cardinality, grain and every statistic in the file describe those, not the rows a query with FINAL or GROUP BY the key returns. The block says so:

merging:
  engine: ReplacingMergeTree      # the engine family, a Replicated or Shared prefix removed
  key:                            # the sorting key, in order; empty for ORDER BY tuple()
    - expression: order_id        # verbatim (§2.1.3's credential masks applied)
      column: order_id            # the base column, when the expression is one
  one_row_per_key: true
  rows: stored                    # stored | merged

Which engines. ReplacingMergeTree, SummingMergeTree, AggregatingMergeTree, CoalescingMergeTree, CollapsingMergeTree and VersionedCollapsingMergeTree, each also under its Replicated or Shared prefix. A plain MergeTree has a sorting key and combines nothing; a Distributed or Merge table and a matview writing TO another table merge nothing themselves, and a matview with inner storage carries its inner table's block. key is the sorting key as the catalog spells it, split at top-level commas, in physical_layout's {expression, column} shape (§2.2.11), and every column names a column in columns.

one_row_per_key is true where a full merge leaves one row per key (Replacing, Summing, Aggregating, Coalescing), so FINAL returns one row per key; false for the two collapsing engines, whose merge keeps up to a state row and a cancel row per key. A producer derives it from the engine, so a consumer needs no list of engines.

rows says which rows the statistics describe. A producer reads with a plain SELECT, so it is stored — unless the session itself reads every table as FINAL (ClickHouse's final setting, which a user profile can turn on), when it is merged. A materialized sample copy (§2.2.8) is a plain table filled by one SELECT the setting does not reach, so a table read through one is stored either way.

Declared, never measured. No share of unmerged rows is published: the next insert changes it, and a reader trusting "0 duplicates" would drop FINAL from a query that needs it. grain keeps describing the stored rows: the sorting key is never a declared grain entry, since over those rows it is not unique.

Absence means no merging engine, never not checked. A read that fails omits the block and names merging in the file's unmeasured list (§2.2.1). Every engine but ClickHouse returns committed rows to a plain read and never carries the block.

2.2.20 external — rows that live in another system

A table can be a name in this database for rows another system holds: a foreign table over a remote database, a table over files in object storage, a table whose engine fetches its rows from elsewhere on every read. Every query against it — a producer's and an agent's alike — reaches that system, at that system's cost and freshness.

external: true       # OPTIONAL; present and true only when the object's rows live in another system

Presence states the fact; there is no false value. Like catalog_only (§2.2.15), a producer MUST NOT emit external: false. Absence means the rows are stored by the database the connection reads. Every producer can answer at listing time, from the catalog row that names the object's kind, so absence is never "not checked" and unmeasured never names this field.

Independent of catalog_only. A producer that described the object without a query carries both markers; one that read through it carries external alone, and the file's statistics then describe rows fetched from the other system. external never changes type: the object is a table to every query an agent writes.

A SHAPE fact. An object that becomes external, or stops being, between two runs is reported as external_changed (§2.6.6) — the cost and freshness of every query against it changed even where its columns did not.

2.2.21 disabled — measurements switched off

A producer MAY let its configuration switch an optional measurement off for one object. The file then names it in disabled, so an absence the measurement would otherwise explain is never read as a finding about the data.

disabled: [null_patterns, timeline]   # OPTIONAL; sorted; omitted when nothing was switched off
NameAbsent from this file, and from nowhere else unless stated
looks_likeinferred.looks_like, inferred.sampled, inferred.matched, inferred.looks_like_candidate, inferred.looks_like_candidate_share, and inferred.epoch_unit on the classifications §4.1.5 samples, on every column and part. No sample is drawn: inferred.sensitivity rests on the name alone (§4.4.3), and a text column publishes values under the ordinary §2.2.3 rule, since no prose verdict exempts it
null_patternsthe null_patterns block
partsparts, parts_found and size on every column. A column only a descent would classify composite classifies unsupported (§3.2)
relationships.inferevery detection: inferred entry in this object's own refers_to, and its mirror in the target's referenced_by. The object stays a target of other objects' inferred edges, and eligible_target (§2.3.1) is published as usual
sketchessketch on every column. Also absent: every detection: measured edge (§2.3.11) with an endpoint in this object, in either endpoint's relationships.yaml, and observed.containment/observed.answerable_count on every edge with an endpoint here, whose target_coverage takes its cardinality-derived form (§2.3.10)
timelinethe timeline block, and populated on every column

Switched off is not failed. unmeasured (§2.2.1, §2.2.4) names a measurement this run attempted and did not get. disabled names one it never attempted. No block or field is named by both.

The list states configuration, not applicability. A producer names a measurement whenever its configuration switched it off for this object, including where scope or catalog_only would already forbid its output. A reader reaching the same absence through two causes reads either one: both say not measured.

Presence states the fact. There is no disabled: []. A consumer MUST tolerate a name it does not know and read nothing into it (§5.3). A validator reports one (§6.3).

2.3 relationships.yaml

Two-section relationship graph: refers_to (outgoing FKs from this table) and referenced_by (incoming FKs from other tables). Both sections list foreign keys, each carrying its own detection: a key declared in the catalog, one inferred by the producer from column naming, or one measured from value containment between two columns' sketches (§2.2.14) — see §2.3.11.

Inference exists because a warehouse routinely declares none — Snowflake does not enforce foreign keys and plenty of analytics schemas skip them — so a print of one would otherwise carry an empty graph. An inferred edge is a producer's claim about the schema, not a constraint the database will honour, and a consumer MUST NOT treat it as one.

What an inferred edge permits. A consumer MAY use it as a join candidate when writing a query — naming, declared-uniqueness and type evidence together are the best signal available in a schema that declares nothing — but MUST NOT treat the join as cardinality-guaranteed the way a declared FK is, and SHOULD prefer a declared edge over an inferred one wherever both describe the same relationship. This is the normative floor for a graph where every edge is inferred, which §2.3.8's eligibility rule makes common: a warehouse-wide absence of declared keys leaves nothing for a consumer to prefer, and the inferred graph is what there is to query against. A human MAY reject a specific inferred edge, readable from this table's own directory — see relationships.annotations.yaml (§2.7.2).

What a measured edge permits, and what it does not. A measured edge is stronger evidence than a name — it states that the referencing column's values were contained in the referenced column's values at profiled_at, over the sketches' answerable range, within §2.2.14's own margin — and a consumer MAY use it as a join candidate on the same footing an inferred edge already has: not cardinality-guaranteed, and a declared or inferred edge describing the same relationship SHOULD be preferred where one exists. It is not a stronger CLAIM about the schema than an inferred edge is; it is a stronger claim about the DATA at the instant of the read. §2.3.11 states the floors that must hold before a producer publishes one.

The reference JSON Schema SHALL be at spec/v1/relationships.schema.json.

2.3.1 Top-level fields

FieldTypeRequiredNotes
format_versionintALWAYS1 for v1 artifacts
tablestringALWAYSFQN of this table
profiled_atstringALWAYSUTC ISO 8601 with Z suffix (same convention as statistics.yaml)
eligible_targetboolOPTIONALWhether an inferred edge could ever point at this object, per §2.3.8's eligibility rule. Emitted with both true and false; absent only when a producer never evaluated it (no object in the run had naming inference switched on) — see §2.3.7
refers_toarrayALWAYSOutgoing FKs from this table. Explicit [] when empty; never omitted.
referenced_byarrayALWAYSIncoming FKs to this table. Explicit [] when empty; never omitted.

2.3.2 refers_to entry schema (outgoing FK: this → other)

FieldTypeRequiredNotes
columnarray of stringRThis table's column(s) participating in the FK; always array (length 1 for single-column FKs)
target_tablestringRFQN of the referenced table
target_columnarray of stringRReferenced column(s); same length as column
on_deleteenumR (declared only)NO ACTION | CASCADE | SET NULL | SET DEFAULT | RESTRICT. A guessed edge — inferred or measured — declared nothing to report here — see §2.3.8, §2.3.11
on_updateenumR (declared only)same enum
detectionenumRdeclared — read from the catalog | inferred — derived from column naming, see §2.3.8 | measured — derived from value containment, see §2.3.11
constraint_namestringOAdapter-native; preserved in adapter case (Snowflake UPPERCASE typical, Postgres lowercase typical, MySQL adapter-dependent)
observedobjectOMeasured cost of joining across this edge — see §2.3.10

2.3.3 referenced_by entry schema (incoming FK: other → this)

Mirror of refers_to with reversed perspective. The two entries of one edge are addressed by referencer table, referencing columns, referenced table and referenced columns, each spelled from its own side. Every other field both entry schemas define (detection, on_delete, on_update, constraint_name, observed) MUST carry the same value on both sides, and a field absent on one side MUST be absent on the other. path/target_path are refers_to-only (§2.3.9) and are not mirrored.

FieldTypeRequiredNotes
columnarray of stringRThis table's column(s) being referenced (typically PK)
referencer_tablestringRFQN of the table that holds the FK
referencer_columnarray of stringRReferencing column(s) on the referencer table; same length as column
on_deleteenumR (declared only)From the referencer's FK constraint; absent when the referencer's edge is inferred or measured
on_updateenumR (declared only)
detectionenumR
constraint_namestringOThe referencer's FK constraint name
observedobjectOSame block as refers_to's, from the referencer's side — see §2.3.10

2.3.4 Always-array convention for column lists

column, target_column, and referencer_column are ALWAYS arrays of strings:

  • Single-column FK: column: [taxon_id]
  • Composite FK: column: [vault_id, shelf_code]

The corresponding pair (column ↔ target_column in refers_to; column ↔ referencer_column in referenced_by) MUST have the same length. Position i in one corresponds to position i in the other.

Composite FKs MUST be emitted as a SINGLE entry — not split into multiple per-column entries.

2.3.5 NOT modeled

The following Postgres-specific FK attributes are NOT captured:

  • DEFERRABLE / INITIALLY DEFERRED
  • MATCH FULL / MATCH PARTIAL / MATCH SIMPLE

Producers MUST NOT emit fields for these; they don't affect AI SQL-writing utility at this maturity level and are reserved for potential future additions.

A view's depends_on is a different relation, not a fourth omission here. statistics.yaml's depends_on (§2.2.17) states that one object's definition reads another object at all — object grain, no column, no cardinality, no join. relationships.yaml models foreign keys — row grain, between two named columns. Neither file's schema restates the other's fact, and a producer MUST NOT infer a refers_to/referenced_by entry from a depends_on entry or vice versa.

2.3.6 Scope limits of referenced_by

referenced_by is exhaustive only within the print's scope. The producer's two-pass algorithm (extract all FKs from all profiled tables → build reverse map) cannot discover external tables (outside the include selectors) that reference this table.

If a table outside the include selectors references this table, that incoming FK is NOT captured in referenced_by. Consumers SHOULD be aware of this limit when reasoning about a table's full incoming FK graph.

§7.3 records this as an absence with no marker of its own; the manifest's selectors (§2.5) is what names the objects left out.

Within the print, a refers_to entry whose target_table is in the manifest and declares a relationships.yaml MUST have its mirror in that file's referenced_by. A target outside the manifest is external (§2.3.7) and has none.

2.3.7 Edge cases

CaseResolution
No FKs at all, eligible targeteligible_target: true, refers_to: [], referenced_by: []. File still emitted. A measurement: the producer evaluated this object's eligibility and nothing references it.
Ineligible targeteligible_target: false, referenced_by: []. The empty list is not a measurement here — no single-column PRIMARY KEY or sole single-column UNIQUE means no inferred edge could ever resolve to this object (§2.3.8), so nothing was left to find. Distinct from the row above: same referenced_by: [], different eligible_target.
Eligibility not evaluatedeligible_target absent, referenced_by: []. Only when the producer never ran the declared-keys pre-pass (no object in the run had naming inference switched on) — a fourth case, collapsed into neither of the two above.
Self-referential FK (e.g., taxon.parent_taxon_id → taxon.taxon_id)Normal emit. target_table equals the top-level table. Both refers_to and referenced_by get an entry.
External FK target (target not in print scope)Emitted in refers_to normally. Consumer checks the manifest for whether target_table is present; absent means external. No special flag — schema stays clean.
Composite FK (FOREIGN KEY (a, b) REFERENCES other(x, y))Single entry with column: [a, b] and target_column: [x, y].
Plain viewsrelationships.yaml MAY be omitted entirely (per §1.4). If emitted, both sections may be empty arrays. A view MAY originate an inferred foreign key (§2.3.8): naming evidence on a view's column is the same evidence as on a table's, and the target's referenced_by names the view as the referencer_table, so a consumer can always tell a virtual referencer from a real one. A view is never the target of one, so its own eligible_target is always false when evaluated. Materialized views are the same on both counts.
UNIQUE constraintsNot FKs; do NOT appear in relationships.yaml. The DDL file captures them. statistics.yaml's structured analog is grain (§2.2.12) for a declared key at any arity, and inferred.candidate_key for a single column's measured uniqueness.

The first three rows are one encoding read three ways; §7.4 sets them beside the other empty collections the format emits, and §7.3 covers the incoming edge §2.3.6 leaves out of scope.

2.3.8 Inferred foreign keys

A producer MAY infer a foreign key that the catalog does not declare, and MUST mark it detection: inferred.

Inference is permitted only on evidence the producer can state. A conforming producer MUST require all of:

  • the column is named <name>_id, where <name> or its regular plural is the object name of an eligible base table in the print's scope, matched against the real table list. Eligible means the object can supply the declared-unique column the next bullet's selection rule requires. A view and a materialized view never can, and neither can a base table that declares no single-column PRIMARY KEY and no sole single-column UNIQUE constraint — the type check and the key check are the same test, checked two ways. A producer MUST NOT resolve a stem to an ineligible object. The exclusion is total rather than a preference — an ineligible object MUST NOT take a name that would otherwise resolve to an eligible one, nor make a stem ambiguous that would otherwise resolve. Creating a view or a materialized view MUST NOT change any table's inferred edges, since neither is ever eligible; a base table's own declared keys are a schema statement, and a change to them MAY retarget an edge that resolves outside its namespace, since an inference rule built on declared keys is meant to follow what they say. The exact stem is tried first, so a schema holding both person and persons resolves person_id to the one the column names. Only the mechanical English plural forms are permitted (+s, +es after a sibilant, y to ies); a producer MUST NOT stem, singularize, or consult a dictionary of irregulars, so person_id does not reach people;
  • the candidate target column is declared unique on that table, by PRIMARY KEY or UNIQUE constraint. Measured uniqueness MUST NOT be substituted: it is a property of the data at one moment, and using it would make the same schema infer differently depending on when it was profiled;
  • the two column types are compatible;
  • no declared foreign key already covers the column.

Eligibility is published, not left for a consumer to re-derive. A producer that evaluates the naming-inference pre-pass for a table MUST record the result as the top-level eligible_target field (§2.3.1): true when the object could supply the declared-unique column above, false when it never can. Both values are measurements and MUST be emitted whichever the answer is; the field is absent only when the pre-pass did not run at all. Without this field, an object's own referenced_by: [] is ambiguous between "nothing references it" and "nothing could" — the same bytes for two different facts about the schema.

Which declared-unique column the edge targets. A table routinely declares more than one, so the acceptability test above does not by itself name a target. A conforming producer MUST select, in this order:

  1. the single-column PRIMARY KEY, where the table declares one;
  2. otherwise the sole column declared unique by a single-column UNIQUE constraint, where exactly one such column exists;
  3. otherwise nothing — several qualifying columns and no primary key is an ambiguity, and MUST infer no edge, exactly as an ambiguous stem does.

Two consequences worth stating. A column carrying both a PRIMARY KEY and a UNIQUE constraint is one column reported twice, never an ambiguity. And a composite key never participates: it cannot be the target of a single-column edge, so a table whose only PRIMARY KEY spans several columns has no single-column primary key and falls to rule 2 rather than to "no primary key exists, therefore any unique column will do".

The order is normative because the alternative is producers disagreeing about a schema neither of them is reading wrongly. collector(collector_id PRIMARY KEY, email UNIQUE) — a namespaced surrogate key beside a natural one, which is the commonest table shape in the wild — has two qualifying columns, and a producer left to choose emits the edge, the other edge, or none at all.

An inferred edge carries no on_delete or on_update semantics, because none were declared; producers SHOULD omit both fields rather than emit NO ACTION as filler — a value of that shape reads identically to a real referential action a declared edge reports (§2.3.2/§2.3.3). It carries no constraint_name.

Composite keys MUST NOT be inferred — naming evidence for a multi-column key is too weak. Self-references are permitted, and the permission is between two columns: taxon.parent_taxon_id targeting taxon.taxon_id is an ordinary edge that happens to stay inside one table. An edge whose target table and target column are both the source column's own is a different thing and MUST NOT be inferred. It asserts nothing a schema can express, and it is what a table whose primary key is named after itself — collector(collector_id uuid PRIMARY KEY) — satisfies every requirement above for: the stem resolves, the target is declared unique, the types match, and no declared foreign key covers the column. Producers would otherwise disagree about every table following that convention while both conformed, which is the disagreement this section exists to prevent. A stem resolving inside the source table's own namespace takes that table where that table is eligible; otherwise resolution proceeds as though the local name were not there. A stem that resolves only outside the source's namespace, in more than one namespace among eligible candidates, is ambiguous and MUST infer nothing, because an edge pointing at the wrong schema is worse than no edge.

An inferred edge appears in the target's referenced_by exactly as a declared one does, so §2.3.6's reciprocity holds for both.

Inference switched off for one object. A file whose disabled list names relationships.infer (§2.2.21) carries no detection: inferred entry in its own refers_to. The object stays a target of other objects' inferred edges and publishes eligible_target as usual, since dropping it from the universe could make another stem unambiguous and manufacture an edge.

What this deliberately misses. Only regular plurals are matched, so an irregular one is out of reach: person_id reaches person or persons, never people. The boundary is drawn at what a reader can apply by eye — a producer that stemmed or carried an irregular dictionary would infer edges this specification cannot describe, and two conforming producers would disagree about the same schema.

2.3.9 Path-valued relationship endpoints

column and target_column name whole columns. A warehouse that stores a join key inside a semi-structured column (Snowflake VARIANT, Postgres/MySQL JSON) has no column to name for one side of the edge — refers_to's ordinary shape cannot express it, and the format's encoding for "no representable edge" is the same bytes as "no FKs at all" (§2.3.7).

The structured endpoint. A refers_to entry MAY carry path, a sibling of column, and target_path, a sibling of target_column — each an ordered array of string keys naming a path inside the named column's semi-structured value:

refers_to:
- column: [collector]
  path: [collector_id]
  target_table: arboretum.seedbank.collector
  target_column: [collector_id]
  detection: declared

Both are OPTIONAL, and each is legal only where its partner array (column / target_column) names exactly one column — a path endpoint on a composite key is not representable and MUST be rejected. Neither field changes what column / target_column mean; they narrow further, into the value the named column holds.

No wire-level rendering is defined. The three engines spell a path differently (col:key Snowflake, col->>'key' Postgres, col->"$.key" MySQL); publishing any one of them as the wire form would read as endorsing that vendor and force an escaping rule for a key containing the delimiter. A structured endpoint has nothing to parse. A consumer building a predicate from {column: collector, path: [collector_id]} constructs the native form for its own target engine; the three constructions below are informative, not the artifact's wire format:

EngineConstruction
Snowflakecollector:collector_id
PostgreSQLcollector->>'collector_id'
MySQLcollector->"$.collector_id"

No producer inference. A conforming producer emits no path-valued endpoint on its own; the vocabulary exists so a human can state one through relationships.annotations.yaml (§2.7.2), where an entry naming a path and matching no producer-emitted edge is the human-authored addition that section's layering rule describes.

Scalar paths only. The array case (readings[*]:reading_id, one-to-many) is not representable under this shape either — refers_to carries one target per entry, and a one-to-many relationship is a different shape, not a longer path.

Ordinary column-to-column edges are unaffected: path and target_path are absent, and nothing about their absence changes.

2.3.10 observed (per-edge measured cost)

A refers_to or referenced_by entry MAY carry observed: what joining across this edge costs, computed from the two endpoints' own statistics.yaml — no additional query. The referencing ("child") side always supplies the fanout numerator and the referenced ("parent") side always supplies the coverage denominator, regardless of which of the two files states the edge — a referenced_by entry's observed describes the same measurement its mirror refers_to entry does, from the referencer's own column, and is identical to it field for field (§2.3.3).

FieldTypeRequiredNotes
fanout_avgnumberR(Child row_count - child column's null_count) / child column's cardinality, rounded per §2.2.6 — average rows per distinct key among the referencing rows that carry one; a null referencing value joins to nothing and is not counted. The numerator is never taken below cardinality, since each distinct key has at least one row. Both operands describe the same rows only where the child was read whole, which is what scope_compatible guarantees: under a scope (§2.2.8) row_count counts the table while cardinality counts the rows scanned, and their ratio answers neither question
fanout_maxnumberOThe referencing column's own values[0].count (§2.2.4 orders by count descending) — the true worst-case group size, not the mean. Absent wherever values itself is (§7.2)
target_coveragenumberRThe fraction of the parent's distinct values this edge actually reaches. Measured from both endpoints' sketch (§2.2.14) when both carry one and the measurement has evidence to report; cardinality-derived (child cardinality / parent cardinality) otherwise — the same field, silently upgraded to the sharper number when the sketches exist to support it, never a second field carrying the other formula
containmentnumberOThe fraction of the child's distinct values found in the parent's set, measured from both endpoints' sketch (§2.2.14) — the question target_coverage cannot answer, since a small target_coverage and a containment near 1 both hold whenever a few child values reach a much larger parent. Present only when both endpoints carry a sketch and the measurement has evidence to report; no cardinality-derived fallback exists for it
answerable_countintegerOThe count of the child's own retained hashes below the shared threshold both sketches cover (§2.2.14) — the denominator §2.2.14's 1/sqrt(answerable count) margin is sized against, so a consumer computes that margin from the print alone. Present exactly where containment is; absent under the same fallback
coherentboolOfalse when the child's cardinality exceeds the parent's — arithmetically impossible for a real containment. Present only when both sides measured cardinality_method: exact (§2.2.6); a scoped or sampled comparison cannot support the claim either way, so the field is omitted rather than guessed
scope_compatibleboolRfalse whenever either endpoint carries a scope block (§2.2.8). Every other field in this table is absent whenever this is false; no ratio is ever published across a mismatched pair — and a scoped endpoint is a mismatch even where both sides carry the same sample, since two draws at one rate are still two draws and a value-identity comparison across them is not measurable. That is the position §2.2.14 already takes in withholding sketch from a scoped table; a consumer wanting these numbers profiles the two endpoints unscoped

Absence of the whole block. A composite edge (column/target_column longer than one) and an edge where either endpoint carries no cardinality, or the referencing column no null_count — a catalog-only file (§2.2.15), or an object this run could not measure one for — carry no observed block at all; see §7.3.

Sketch-measured fields and their error. containment and a sketch-measured target_coverage are derived from the two endpoints' sketch (§2.2.14), and which computation applies depends on whether the child sketch is exhaustive.

An exhaustive child sketch (§2.2.14's decoded-length case) needs no scaling: containment is the match rate over the child's own answerable subset — the hashes falling below the parent's own retained threshold, the only region both sketches cover — measured directly against the parent's set. target_coverage then derives from that same ratio applied to the child's own cardinality. When the answerable subset is empty (every child hash sits above the parent's threshold), neither field is published from the sketches at all — target_coverage falls back to its cardinality-derived form, and containment is absent, the same fallback a missing sketch already causes.

Exactness depends on the parent too, not only the child. When the parent's sketch is also exhaustive, the answerable subset is the child's whole distinct set, and the match rate against it is exact — no error bound to state. When the parent's sketch is truncated, its own retained threshold covers only part of the hash space, so the answerable subset is a hash-uniform sample of the child rather than its whole set: the published value is the same measurement, but it carries §2.2.14's margin over that subset's own size, on the same footing as any other truncated comparison. observed.answerable_count publishes that subset's size, so a consumer computes the margin (§2.2.14) from the print alone rather than reading only that the value is not exact.

A truncated child sketch instead goes through the standard bottom-k intersection estimator: the smaller of the two sketches' own retained thresholds bounds the hash range both cover, the observed match rate inside that range is scaled to the full hash space, and the result is divided by each side's own published cardinality (measured exactly, never re-estimated from the sketch). This estimate's error is roughly 1/sqrt(answerable count) — the count of hashes genuinely below the shared threshold, which varies per edge and shrinks toward a much wider margin as that count falls, never the flat percentage a fixed k alone would suggest (§2.2.14). A value near 1 states that the sketches found no evidence against full containment or coverage within that margin, not a guarantee finer than the margin allows; a value of exactly 0 or 1 from a truncated sketch is still an estimate, on the same footing as an exhaustive child's own value against a truncated parent — both are 1.0/0.0 and neither is exact. Only both endpoints exhaustive produces the exact form. Neither field is ever computed from only one endpoint's sketch — the same fallback as an empty answerable subset applies.

An endpoint with sketches switched off. Where either endpoint's file names sketches in disabled (§2.2.21), it carries no sketch, so containment and answerable_count are absent and target_coverage takes its cardinality-derived form.

Not a cost claim. observed states row-count ratios measured on the print; it never predicts query latency or execution-plan cost, the same boundary §2.2.11 draws for a clustering key.

2.3.11 detection: measured — an edge proposed from value containment

A producer MAY compare two columns' sketches (§2.2.14) across every table pair it profiled and propose an edge from what their value sets show. A column's name is consulted at one point only, rule 1's measured-key clause; everything else is decided by values. No query is issued for the comparison itself — both sketches are already on disk from §2.2.14's own pass.

An object whose file names sketches in disabled (§2.2.21) carries no sketch, so it proposes and receives no measured edge. A pair is proposed only where all three hold:

  1. The referencing ("child") column is a reference, not a category, a key of its own table, or a timestamp. Its cardinality exceeds enumeration_threshold. A column enumerable in full is a lookup value, and a containment of 1.0 over a handful of values is as likely to be coincidence as evidence. Measured uniqueness does not change that: on a table of a few rows every column is a candidate key. The child MUST NOT be declared unique on its own table by a single-column PRIMARY KEY or UNIQUE constraint, and its canonical encoding kind (§2.2.14) MUST NOT be temporal or boolean: surrogate-key sequences of unrelated tables overlap by construction, and so do the instants two tables were written at, so containment between them is evidence of nothing. A child carrying inferred.candidate_key qualifies only where its name ends in _id — the naming convention §2.3.8 already acts on. Values alone cannot tell a one-to-one reference from a table's own surrogate key, both being unique and drawn from the same kind of sequence; the name is the one signal that separates them. Where the parent is an integer column whose range is a dense sequence — range.max - range.min + 1 equals its exact cardinality — the child qualifies only where its name ends in _id: every small integer falls inside such a range, so containment in it shows only that the child's values are in range. The child MUST NOT already carry a declared or inferred single-column edge in the same file: its parent is known, and containment in a second key is coincidence.
  2. The referenced ("parent") column is key-like. It carries a single-column PRIMARY KEY, a sole single-column UNIQUE, or inferred.candidate_key — the population §2.2.14 already sketches for this reason.
  3. The measurement can carry the claim. Either both sketches are exhaustive and containment (§2.3.10) is exactly 1.0, which an exhaustive comparison makes exact with no margin at all; or answerable_count is at least 400 and containment is at least 0.95. 400 is where §2.3.10's own 1/sqrt(answerable_count) margin falls to 0.05 — the slack the 0.95 floor leaves.

The direction is child-into-parent; the parent's own cardinality MUST be strictly the larger of the two. This is a tighter floor than observed.coherent (§2.3.10), which is false only where the child's cardinality exceeds the parent's: equal cardinalities are coherent, and would admit the same pair in both directions, each end naming the other as parent. A producer MUST NOT publish a candidate that duplicates an edge the same file already carries as declared or inferred — the asserted edge keeps its own detection and the candidate is dropped, not published twice. Only columns whose canonical encoding kind matches (§2.2.14's type table) are ever compared; a column under a scope block carries no sketch (§2.2.14) and proposes and receives nothing, and neither does a redacted column, for the same reason.

on_delete, on_update and constraint_name are absent from a measured edge on the same terms §2.3.8 already sets for an inferred one — a producer publishes no referential action a database never declared.

Never compared by dbprint diff. The comparison depends on sketches a diff run never computes (§2.6.6), so a detection: measured entry on either side of that comparison is dropped before it runs; such an edge appearing or disappearing between two runs produces no relationship_added, relationship_removed or relationship_modified event, and a reader sees the change only in relationships.yaml itself.

2.4 description.md

Free-form Markdown narrative authored by humans. The format imposes no structure on this file — it's user content. Producers MUST NOT write to description.md; once a user creates one, it's preserved across regenerations.

This file is NOT versioned with a format_version header — it's Markdown.

Precedence against the measured layer. On any question statistics.yaml answers, use the measured statistic — it describes the run that produced it, current as of that run, where description.md may not be. description.md is authoritative only on what a statistic cannot express: table grain, units, derivation, deliberate exclusions. This rule is about which source is current, not about the prose's accuracy — prose written correctly can still describe a schema that a later run recorded as changed, and the rule does not ask a consumer to distrust it on that account, only to prefer the statistic where the two disagree. A checkable claim belongs in statistics.annotations.yaml (§2.7.1) rather than here, where a consumer — human or automated — can verify it against the statistic it describes; nothing checks the prose in this file, so a claim placed here can become false without any check reporting it. Absence of description.md carries no meaning beyond "no narrative was written."

2.5 manifest.yaml

The index file at the connection root. Source of truth for which tables are present and what artifacts each has, and which the last run could not profile.

format_version: 1
generated_at: <ISO8601>
connection: <connection_name>
adapter: <string>                         # non-empty; the producer's own name for itself
dbprint_version: <semver>
profile:                                  # the connection's resolved profile, before any per-table rule
  read:
    tables: {mode: full | sample | filter | catalog, ...}   # only the parameters of its mode
    views: {mode: full | filter | catalog, ...}
  statistics: {enumeration_threshold: <int>, top_n_values: <int>, percentiles: [<int>, ...]}
  looks_like: {enabled: <bool>, sample_size: <int>}        # a switched-off feature records {enabled: false} alone
  null_patterns: {enabled: <bool>, top_n: <int>}
  parts: {enabled: <bool>, max: <int>, max_depth: <int>}
  timeline: {enabled: <bool>}
  sketches: {enabled: <bool>, columns: required | all}
  relationships: {infer: <bool>}
selectors:                                # the connection's own include/exclude, as configured
  include: [<glob>, ...]
  exclude: [<glob>, ...]
redaction_rules_configured: <int>         # count of `redact` rules in force; 0 means none configured
default_collation: <string>               # the connection's own name for its default comparison collation
failed_tables: [<fqn>, ...]               # OPTIONAL; tables this run attempted and could not profile
manifest_annotations: manifest.annotations.yaml   # present only if user-authored (§2.7.3)
tables:
  <fqn>:
    type: table | view | matview
    path: <relative_dir>
    artifacts:
      ddl: ddl.sql
      statistics: statistics.yaml
      relationships: relationships.yaml   # may be absent for plain views
      description: description.md         # present only if user-authored
      statistics_annotations: statistics.annotations.yaml       # present only if user-authored
      relationships_annotations: relationships.annotations.yaml # present only if user-authored
    row_count: <int>                      # absent where the table's statistics.yaml carries catalog_only
    columns: <int>
    profiled_at: <ISO8601>
    max_age_days: <int ≥ 0>               # OPTIONAL; the freshness threshold this table was judged against
    profile:                              # OPTIONAL; only the blocks that differ from the connection's, each whole
      <block>: {...}                      # e.g. null_patterns: {enabled: false}
    max_rows_scanned: <int ≥ 1>           # OPTIONAL; the row-count ceiling that governed this table

failed_tables names every table the writing run's scope matched and the target listed whose profiling failed, plus any such mark carried forward for a table outside this run's scope that the connection's own selectors still cover. The list is sorted and omitted when empty. An FQN may also be a key of tables; that entry is then an earlier run's, dated by its own profiled_at. A producer MUST NOT record a failure cause here, because a driver's message quotes cell values past every redaction rule; the cause belongs to the run's own output. A producer SHOULD re-attempt a named table on its next run regardless of max_age_days. A consumer MUST NOT read an FQN named here and absent from tables as absent from the database: the table exists and is unprofiled.

max_age_days is the threshold the producer resolved for that table on the run that wrote the entry — the number the run itself used to decide whether the table needed re-reading. It is a whole number of days and MUST NOT be negative; 0 states that the table is re-read on every run, and no age can satisfy it. Consumers judging freshness SHOULD read it rather than re-deriving a threshold from their own configuration, which may have changed since the print was written, and which cannot express a threshold that depended on anything but the table's name. It is carried for every object the run resolved a threshold for, views included. An entry omits it where the run resolved none; consumers fall back to their own configuration for those entries alone, since a manifest may mix both.

max_rows_scanned is the row-count ceiling that governed the table on the run that wrote the entry, recorded as the ceiling itself rather than the fraction it resolved to: the fraction moves whenever the table's size estimate moves, the ceiling does not. It is recorded wherever a ceiling governed the table, including a table smaller than the ceiling that was therefore read whole, and omitted where none did — no ceiling in force, a read mode other than sample, or an object described without a query (§2.2.15).

Provenance: the parameters that produced the print, recorded in the print itself. Four top-level fields, all REQUIRED, carry the parameters that decide what every number underneath them means:

  • profile is the connection's resolved profile for this run — how each object type's rows are read, the parameters that decide how much of a column's domain a values list carries, which columns classify categorical, how much evidence a looks_like verdict rests on and which percentile keys exist, and which optional measurements are switched on. A switched-off measurement records {enabled: false} ({infer: false} for relationships) and no parameters; a read block records its mode and that mode's parameters only. Freshness thresholds and redaction rules are not part of it. Per-table rules can override any of it: a table whose resolved profile differs carries its own profile naming only the blocks that differ, each written whole — under read, each object type's block — so its effective profile is the connection's with those blocks replaced. A table with no override carries none.
  • selectors is the connection's own configured include/exclude glob set — the scope a regeneration of this print would cover. A CLI narrowing (--include intersects, --exclude unions) applies to one invocation and is NOT recorded: two glob lists have no glob intersection, so no pattern list can express the narrowed scope, and a reader who saw one could not tell it from a configuration. A run narrowed that way therefore scans fewer tables than this value implies, and the artifact does not say so. This is the same information diff.yaml's target.selectors (§2.6.3) carries for a live comparison; the manifest is where a consumer asks "what was deliberately left out of this print" without needing to know the drift-detection protocol. The two MUST NOT disagree where both are present (§6.3).
  • redaction_rules_configured is the count of redact rules in force for this connection. It tells a consumer whether a column's absent redacted marker means "no rule matched" (rules exist, none applied here) or "no rules configured" (the connection redacts nothing at all) — a distinction the column itself cannot express. The redaction salt and the rules' own shapes are never recorded here.
  • default_collation is the connection's own name for the collation a string column compares under when it carries no explicit override (§2.2.2, §2.2.4). Every producer resolves and records one, even where the source engine has no session- or database-level concept of a default — the field then states what an unspecified column behaves as.

A manifest missing any of the four does not describe what produced it, and a validator MUST report it (§6.3).

The effective profile and the file agree. Every measurement a table's effective profile switches off is named in its statistics.yaml's disabled list (§2.2.21), and no other (§6.3).

Carried entries state what their table was measured under. An entry the producer carries forward without re-reading its table keeps its profiled_at, max_rows_scanned and effective profile: its profile delta is restated against the new connection-level block, so the effective profile still describes the files on disk. A producer that judges freshness re-reads every table whose effective profile differs from the one it would now resolve, and every table of a manifest that records no profile. A carried entry declares the user-authored files present in its table's directory at the run that writes it. A producer re-reads, rather than carries, a table whose declared producer-written file is absent or does not parse. A run that carries forward any table recorded under another dbprint_version keeps that value, so the next run that covers those tables re-reads them.

path is the table's directory relative to the connection root: one or more segments joined by /, each matching the §1.5.1 allowlist, and each artifacts value is its kind's own filename as listed above. A consumer does not follow an entry whose path starts with /, \ or a drive letter or has a .. segment, or whose directory or declared file resolves outside the connection root through a symbolic link, opens no artifact under any other filename, and reads no connection-root file that resolves outside the connection root; a validator reports each such entry and file (§6.3).

If the manifest disagrees with on-disk files (missing required artifact, orphaned file), consumers SHOULD treat the print as inconsistent. A producer that drops a table from tables removes the producer-written files it wrote for it and leaves user-authored files in place (§1.4).

The reference JSON Schema SHALL be at spec/v1/manifest.schema.json.

2.6 diff.yaml

Structured change-event stream describing what changed between a baseline (typically the committed prints on disk) and a target (typically the live database). Produced by every successful generate run; also emitted by dbprint diff --format json|yaml as a standalone artifact.

The reference JSON Schema SHALL be at spec/v1/diff.schema.json.

2.6.1 Top-level fields

FieldTypeRequiredNotes
format_versionintALWAYS1 for v1 artifacts
generated_atstringALWAYSUTC ISO 8601 with Z suffix — when the diff was computed
connectionstringALWAYSConnection name from .dbprint.yaml
adapterstringALWAYSNon-empty; the producer's own name for itself
baselineobjectALWAYSSee §2.6.2
targetobjectALWAYSSee §2.6.3
summaryobjectALWAYSSee §2.6.4
changesarrayALWAYSExplicit [] when no changes; never omitted

2.6.2 baseline sub-object

The "before" side. Always the committed prints on disk.

baseline:
  source: committed_prints            # the only defined value; a future value may add `live_database` for retrospective diffs
  path: prints/<connection_name>/     # relative to project root
  generated_at: <ISO8601 | null>      # null when no prior generate exists (first-ever run)
  dbprint_version: <semver | null>    # null when no prior generate

2.6.3 target sub-object

The "after" side. Always the live database.

target:
  source: live_database               # the only defined value; a future value may add `committed_prints` for snapshot-to-snapshot diffs
  scanned_at: <ISO8601>
  selectors:                          # the connection's own include/exclude, as configured
    include: [<pattern>, ...]
    exclude: [<pattern>, ...]
  tables_scanned: <int>               # informational; count of tables matched by selectors

selectors records the connection's own configured scope, matching manifest.yaml's own copy (§2.5). A CLI --include / --exclude narrows the invocation and is not recorded here either: it cannot widen what the config allows, and no pattern list expresses the intersection of two glob lists.

2.6.4 summary sub-object

Every counter naming an event kind is exact — the count of those events in the changes array — and none is threshold-filtered. tables_modified, unchanged_tables and unevaluated_tables count objects rather than events, and the last two count objects that produced none:

summary:
  tables_added: <int>
  tables_removed: <int>
  tables_modified: <int>              # tables with ≥ 1 event of any kind but table_added/table_removed (type, column, spelling, collation, stat, row-count, rel, index, comment, ...)
  columns_added: <int>
  columns_removed: <int>
  columns_type_changed: <int>
  columns_nullable_changed: <int>
  columns_default_changed: <int>
  statistics_drifted: <int>           # total statistic_changed events
  relationships_changed: <int>        # relationship_added + _removed + _modified
  indexes_changed: <int>              # index_added + _removed + _modified
  comments_changed: <int>
  unchanged_tables: <int>             # objects compared and found equal
  unevaluated_tables: <int>           # objects the diff had no basis to compare

unchanged_tables counts a comparison, unevaluated_tables counts the absence of one. An object the diff could not read produces no event, and counting an object that produced no event as "unchanged" publishes a definition as though it were a measurement. The populations that reach unevaluated_tables:

  • an object whose statistics.yaml carries catalog_only (§2.2.15) rather than a measurement — a plain view by default, for which this format defines no DDL comparison, so its whole body can be rewritten with nothing to detect it;
  • a carried-forward object from a run that did not re-read it, whose current state is its own committed state and therefore equal to itself by construction;
  • an object the run listed and could not read, and with no committed print to compare against — named in the manifest's failed_tables (§2.5).

A producer MUST decide this from whether it had a comparable artifact on both sides, never from type: a materialized view carries statistics.yaml and belongs in unchanged_tables. An object that produced at least one event is tables_modified whichever population it came from — something was compared, and it moved.

The four table counters partition target.tables_scanned:

tables_modified + unchanged_tables + unevaluated_tables + tables_added == tables_scanned

tables_removed is outside the identity: a removed object is absent from the target and so was never scanned.

2.6.5 changes array — common shape

Each entry has a kind discriminator (one of the kinds enumerated in §2.6.6, plus possible future additions). Consumers MUST tolerate unknown kinds. Order within changes is producer-defined but SHOULD be stable across runs (grouped by table, then by kind, then deterministic within kind).

2.6.6 Per-kind field schemas

table_added
- kind: table_added
  table: <FQN>
  type: table | view | matview
table_removed
- kind: table_removed
  table: <FQN>
table_type_changed
- kind: table_type_changed
  table: <FQN>
  before: table | view | matview
  after: table | view | matview

The object at one FQN is a different kind of object on the two sides — a table replaced by a view or a matview, or the reverse. Its FQN, relationships and annotations persist, so it is not reported as a removal and an addition. Compared only where both sides' manifest entries record a type; an entry without one compares as unknown.

external_changed
- kind: external_changed
  table: <FQN>
  before: <bool>
  after: <bool>

The object's rows moved between this database and another system (§2.2.20) — a local table replaced by a foreign one of the same name, or the reverse. An absent external on either side reads as false, so a file written before the field existed compares as stored locally. The object counts in tables_modified.

column_physical_name_changed
- kind: column_physical_name_changed
  table: <FQN>
  column: <name>                      # the lowercase map key (§2.2.1)
  before: <catalog spelling>
  after: <catalog spelling>

A column's catalog spelling moved with its key unchanged — only a case-only rename can do that; any other rename changes the key and reports as column_removed + column_added. before/after are the effective spelling: physical_name where present, else the key itself (§7.2), never null. Compared on every column both sides carry, catalog_only included (§2.2.15).

column_collation_changed
- kind: column_collation_changed
  table: <FQN>
  column: <name>
  before: <collation>
  after: <collation>

The effective collation moved: collation where present, else that side's manifest default_collation (§2.2.2). Compared only on a column where at least one side carries an explicit collation, and skipped where either side's effective value cannot be resolved, so a move of the connection default alone reports nothing. Compared whatever gates the column's statistics, catalog_only included.

column_added
- kind: column_added
  table: <FQN>
  column: <name>
  sql_type: <native_type>
  nullable: <bool>
column_removed
- kind: column_removed
  table: <FQN>
  column: <name>
column_type_changed
- kind: column_type_changed
  table: <FQN>
  column: <name>
  before: <native_type>
  after: <native_type>
column_nullable_changed
- kind: column_nullable_changed
  table: <FQN>
  column: <name>
  before: <bool>
  after: <bool>
column_default_changed
- kind: column_default_changed
  table: <FQN>
  column: <name>
  before: <scalar | null>             # null = no default on baseline
  after: <scalar | null>              # null = no default on target
statistic_changed
- kind: statistic_changed
  table: <FQN>
  column: <name>
  part: <path>                        # OPTIONAL; the part of the column the stat belongs to (§2.2.18)
  stat: <dot_path>                    # e.g., "cardinality", "percentiles.p99", "distribution"
  before: <value>                     # type matches stat's data type
  after: <value>
  delta: <numeric>                    # OMITTED for non-numeric stats (distribution, classification, values, cardinality_method), or either side is not a finite number
  delta_pct: <float>                  # OMITTED when before == 0 (division by zero), stat is non-numeric, or either side is not a finite number

Valid stat dot-paths: the measured fields under a column's stats per §2.2 — never physical_name, collation (their own kinds above), physical_layout_key (it restates physical_layout.keys, reported by physical_layout_changed) or derived_from (parsed from a transformation tool's SQL, never read from the database, so a live read has none to compare). For values, before and after carry the full list (per-value granularity not implemented).

A part compares as its own block. Each part both sides list is compared stat by stat, as a column is, with its sql_type and classification among the ordinary stat paths, and its events carry part. A part only one side lists produces no event: under parts.max a path leaving the kept set is not a removal, and parts_found moving is what reports a real change in the count. The redaction rule below applies per part.

A redacted marker withholds that column's cell values from the comparison. Where either side carries the marker (§2.2.9), a producer MUST NOT emit a statistic_changed event for values, range, percentiles, or any field §2.2.9 withholds from a marked column (mean, sum, length, zero_count, negative_count, empty_count, quantized_count, non_finite_count, normalized_cardinality, unrepresentable, sketch) on that column. §2.2.9 states the same prohibition from the consumer's side, and a diff is a consumer of the print it reads; a side written by an unredacted run may still carry some of them, and comparing it would republish its numbers. The marker is itself an ordinary stat path and still reports, which is what tells a reader why the column's other statistics produce no events; the count profile — the counts, ratios, values_coverage, frequencies and distribution — keeps comparing, since §2.2.9 leaves it unaffected.

table_row_count_changed
- kind: table_row_count_changed
  table: <FQN>
  before: <int>
  after: <int>
  delta: <int>                        # after - before
  before_method: exact | approximate
  after_method: exact | approximate

Table-grain, not column-grain — meaningful even when every column statistic in the same file is sample-scale (§2.2.8 scope), since row_count describes the whole table regardless of scope. Fires whenever row_count differs between baseline and target; a before_method/after_method mismatch alone does not emit an event, since the number may be unchanged while what it means shifts. A consumer reading before_method: approximate, after_method: approximate MUST NOT treat delta as measured growth — it is the difference of two estimates.

grain_changed
- kind: grain_changed
  table: <FQN>
  before: { keys: [ { columns: [<col>, ...], detection: declared|measured }, ... ], search: { exhausted: <bool> } }  # search OPTIONAL
  after:  { keys: [...], search: {...} }                                                                            # same shape

Fires whenever grain.keys (as a set of (columns, detection) pairs) or grain.search.exhausted differs between baseline and target (§2.2.12). before/after carry the block's full shape, never a computed add/remove delta: a column combination whose detection changes (declared <-> measured) is neither added nor removed, and a delta would misreport it as a spurious removal paired with a spurious addition. search is OMITTED on a side exactly where §2.2.12 omits it from statistics.yaml — the measured probe never ran there. Table-grain, on the same footing as table_row_count_changed: a change here means the columns identifying a row moved, not that a value did.

physical_layout_changed
- kind: physical_layout_changed
  table: <FQN>
  before: { mechanism: cluster|partition|sort, keys: [ { expression: <string>, column: <string> }, ... ] } | null
  after:  { ... } | null

Fires whenever physical_layout differs between baseline and target (§2.2.11). null on a side states that side confirmed no clustering/partitioning key, never "not checked" — the same absence-has-one-meaning discipline §2.2.11 itself sets, and the reason an absent block compares as null rather than suppressing the comparison the way an absent grain does: §2.2.11 already treats every absent block as one fact regardless of cause, so the transition to a genuine key reads as a real gain rather than as an unevaluated pair. A side whose own unmeasured list names the block (§2.2.1) is the exception, and the comparison is suppressed rather than reported: that side confirmed nothing.

merging_changed
- kind: merging_changed
  table: <FQN>
  before: { engine: <string>, key: [ { expression: <string>, column: <string> }, ... ], one_row_per_key: <bool>, rows: stored|merged } | null
  after:  { ... } | null

Fires whenever merging differs between baseline and target (§2.2.19). null on a side states that side's engine combines no rows. A shape change on physical_layout_changed's footing: recreating a MergeTree table as ReplacingMergeTree changes what every existing COUNT(*) against it means, and ddl.sql is not compared. A side whose unmeasured list names the block suppresses the comparison.

depends_on_changed
- kind: depends_on_changed
  table: <FQN>
  before: [<FQN>, ...]
  after:  [<FQN>, ...]

Fires whenever depends_on (§2.2.17) differs between baseline and target, on a view or matview whose catalog was read on both sides. Table-grain, on the same footing as grain_changed and physical_layout_changed: which objects a view's own definition reads is a declared fact, not data drift, so redefining seedbank.germination_by_taxon_mv to read seedbank.germination_reading instead of seedbank.germination_trial reports here even though neither table's own row counts moved. No event where either side never carries the field at all (§2.2.17's own two-encoding rule: the key omitted means the dependency read did not happen) — absent on both sides, as on every adapter that cannot resolve view dependencies at all, reads as nothing to compare, never as a removal.

relationship_added
- kind: relationship_added
  source_table: <FQN>
  source_column: [<col>, ...]          # always array per §2.3
  target_table: <FQN>
  target_column: [<col>, ...]          # same length as source_column
  on_delete: <enum>                    # declared edges only
  on_update: <enum>                    # declared edges only
  detection: declared | inferred

An inferred edge omits both referential actions, the same write-time rule §2.3.8 states for relationships.yaml: none were declared, and NO ACTION as filler reads identically to an action a declared edge reports. A producer MUST NOT emit them as null either — absence is how this format says a guessed edge has none. detection: measured never appears here at all, per the comparison rule above.

relationship_removed
- kind: relationship_removed
  source_table: <FQN>
  source_column: [<col>, ...]
  target_table: <FQN>
  target_column: [<col>, ...]

An inferred edge whose referencing object's disabled names relationships.infer on either side (§2.2.21) produces neither this event nor relationship_added: switching inference off is configuration, not a change in the database.

relationship_modified
- kind: relationship_modified
  source_table: <FQN>
  source_column: [<col>, ...]
  target_table: <FQN>
  target_column: [<col>, ...]
  on_delete: { before: <enum>, after: <enum> }    # OPTIONAL; present only when on_delete changed
  on_update: { before: <enum>, after: <enum> }    # OPTIONAL; present only when on_update changed

A relationship_modified event MUST carry at least one of on_delete or on_update. If neither changed, the event MUST NOT be emitted.

An operand is null where that side of the comparison declared no action. An inferred edge that gains a real constraint reports on_delete: {before: null, after: <enum>}, and one that loses it reports the reverse — an edge is matched on its columns and target alone (§2.6.6), so a change of detection is a modification of the same edge, not a removal paired with an addition. The null here says the side had no action to report, which is the same fact absence carries in relationship_added; it appears in an operand slot because a before/after pair has no way to omit one half.

No event of any of the three kinds ever names a detection: measured edge. The comparison a producer runs to find one (§2.3.11) depends on sketches (§2.2.14) a diff run never computes, so a detection: measured entry is dropped from both sides of the relationship comparison before it runs — the live side of that comparison would otherwise carry none at all and every measured edge on a committed print would report as removed. A measured edge appearing or disappearing between two runs is visible only in relationships.yaml itself.

index_added
- kind: index_added
  table: <FQN>
  index_name: <string>
  columns: [<col>, ...]                # ordered as declared in the index
  unique: <bool>
  type: <string>                       # adapter-native (btree, hash, gin, gist, brin, spgist, ...)
index_removed
- kind: index_removed
  table: <FQN>
  index_name: <string>
index_modified
- kind: index_modified
  table: <FQN>
  index_name: <string>
  before: { columns: [...], unique: <bool>, type: <string> }
  after: { columns: [...], unique: <bool>, type: <string> }
comment_changed
- kind: comment_changed
  table: <FQN>
  target: table | column
  column: <name>                       # REQUIRED when target=column; OMITTED when target=table
  before: <string | null>              # null = no comment on baseline
  after: <string | null>               # null = no comment on target

2.6.7 Index events — secondary indexes only

index_added, index_removed, and index_modified events cover secondary indexes only — explicit CREATE INDEX statements.

Indexes implicit to PRIMARY KEY or UNIQUE constraints are out of scope for these events:

  • PK changes manifest via column_* events (a PK redefinition typically involves columns)
  • UNIQUE constraint changes are not separately tracked; the data-derived signal is inferred.candidate_key in statistics.yaml (§2.2.4), and the structural definition is in the ddl.sql artifact

A v1 print records secondary indexes only in ddl.sql, which v1 does not parse, so no index_* event fires against a committed-prints baseline (§2.6.10); a future normative indexes.yaml artifact would give that baseline indexes to compare.

2.6.8 Scoping rules

Events are emitted only for tables in target.selectors scope. Tables outside the selectors are NOT reported as added / removed / modified — they're unknown to this diff.

  • target.selectors mirrors .dbprint.yaml on every run, narrowed or not (§2.6.3) — a CLI --include / --exclude applies to one invocation and is recorded nowhere, since no pattern list can express the intersection of two glob lists. A run narrowed that way therefore produces events for fewer tables than the recorded scope covers, and the artifact does not say so.
  • For partial runs (e.g., --include arboretum.fieldwork.*), only matching tables produce events; unrelated tables are not in this diff (even if their committed prints differ from live).

Consumers reading a partial diff understand: "this is what changed within the scope listed in target.selectors; the rest is unknown to this artifact."

2.6.9 Threshold behavior (machine vs human render)

statistic_changed events fire for every measured drift — including sub-percent changes. The diff.yaml artifact contains ALL of them. Machine consumers (JSON / YAML output) always receive the full data.

Human rendering (terminal output without --format json|yaml) filters by per-stat thresholds configured in .dbprint.yaml:

diff:
  stat_change_threshold:
    cardinality_ratio: 0.02
    percentile_pct: 0.05
    values_coverage: 0.05
    default: 0.01

The CLI --threshold FLOAT flag overrides all per-stat thresholds for one run (human-render only).

summary counts (§2.6.4) are NOT threshold-filtered — they always reflect the actual event count.

2.6.10 Edge cases

CaseResolution
First-ever generate (no prior prints)baseline.generated_at: null, baseline.dbprint_version: null, baseline.path still set. All tables appear as table_added events.
--dry-runNo diff.yaml written. (CLI spec §generate.)
--force on unchanged tablesZero statistic_changed events for those tables (file rewrite isn't a semantic change).
Empty diffchanges: [], summary all-zero. File still emitted.
Scoped run with --includeOnly events for in-scope tables. Out-of-scope tables not reported.
An object changes type (table <-> view/matview)table_type_changed; the object counts in tables_modified, never as removed plus added.
Case-only column renamecolumn_physical_name_changed with the effective spelling on each side; never statistic_changed.
Connection default collation movesNo event on a column whose collation is implicit on both sides; every column the move affects carries an explicit collation on one side and reports through column_collation_changed.
Repartitioningphysical_layout_changed only; physical_layout_key on each key column is not compared (§2.2.11).
Case-only table renameNo event: the table's catalog spelling is recorded only in ddl.sql, which v1 does not parse.
Column default, secondary index or comment changedNo event: v1 records column defaults, indexes and comments only in ddl.sql, which it does not parse, so a committed-prints baseline (§2.6.2) cannot know them. column_default_changed, index_* and comment_changed stay defined for a baseline that records them; until one exists, columns_default_changed, indexes_changed and comments_changed are 0.
statistic_changed with non-numeric stats (distribution, classification, values, cardinality_method)delta and delta_pct OMITTED; only before and after.
statistic_changed with before: 0delta_pct OMITTED (division by zero); delta still present.
Comment removedcomment_changed with after: null.
Comment added (never existed in baseline)comment_changed with before: null.
grain absent on either sideNo grain_changed event — absence on either side suppresses the comparison entirely, the same rule every other optional table-level field follows, rather than reporting every declared/measured key as newly added.
physical_layout absent on a side, or a table confirmed unclusteredBoth parse identically to "no layout" (§2.2.11's own absence-has-one-meaning rule); physical_layout_changed fires whenever a side's real state differs from the other's, including the transition from either kind of "no layout" to a genuine key.
A field or block either side's disabled covers (§2.2.21)No event for it — a statistic_changed, relationship_added/relationship_removed for an inferred edge, or a part appearing — since a switched-off measurement has no reading to compare, the same rule unmeasured follows. Nor does what a switch re-derives. Where the two sides' disabled lists differ, a column's classification and every field only one side carries produce no event, and neither does inferred.sensitivity where exactly one side's classification draws a sample (§4.1.5) — switching relationships.infer moves a column whose key was inferred between foreign_key_candidate and the classification its type gives (§3.2), and the fields that classification alone carries move with it. A field both sides carry is the same reading of the same column and compares as usual. A column whose classification moves between composite and unsupported because either side disables parts produces no event at all, and where either side disables looks_like, inferred.sensitivity and the value list of a column either side read as prose produce none.
A side whose unmeasured list names physical_layout (§2.2.1)No physical_layout_changed event — that side never read the block, so there is nothing to compare it against, the same absence-suppresses-the-comparison rule grain follows.
depends_on absent on either side (a table, which never carries it; or a view/matview whose catalog was not read, on an adapter that never resolves the field at all)No depends_on_changed event — the same absence-suppresses-the-comparison rule grain follows, never a reported removal.
PK / UNIQUE indexesOut of scope for index_* events. A PK change that moves a column appears via column_* events; any other change is visible only in ddl.sql.
Multi-connection auto runEach connection produces its own prints/<connection>/diff.yaml. No project-level aggregate.

2.7 Human-authored annotation files

The human layer: per-artifact files a person writes and the producer never touches, layered over the artifact each one annotates.

Naming. <artifact>.annotations.yaml is the human layer for <artifact> — statistics.annotations.yaml for statistics.yaml (§2.7.1), relationships.annotations.yaml for relationships.yaml (§2.7.2), and manifest.annotations.yaml for manifest.yaml (§2.7.3). The suffix makes authorship legible from the name alone: a print root holds both producer-written files and human-written ones, and a reader tells them apart without knowing which section introduced either.

Layering. A per-table annotation file carries the schema of the artifact it annotates, with every field OPTIONAL except the ones that address one entry, plus two extras: note (free-form Markdown; the format imposes no structure on it) and, where a per-artifact section below defines one, a field letting a human reject what the producer measured. An entry that resolves against the base artifact annotates it. An entry that resolves against nothing is content a human added that the producer did not, and structurally could not, emit — the same file covers both without a second shape. manifest.annotations.yaml (§2.7.3) has no addressable entries — a connection has nothing narrower than itself to key against — so it carries notes alone rather than this per-entry shape.

Producers MUST NOT write to an annotation file. Once a user creates one, it is preserved across regenerations, on the same terms as description.md. Unlike description.md, every annotation file IS versioned with a format_version header, since its shape is a mapping rather than free-form prose.

Every annotation file's root is closed. additionalProperties: false at the top level of all three schemas: a key this document does not name is rejected (schema.type-mismatch, §6.3), never silently accepted. Silent acceptance would leave a human with no signal that what they wrote reaches no consumer.

A file that does not match its schema is not read. An annotation file that fails its JSON Schema — any schema.* or version.* error, including a mapping key that is not a string (1:, true: or null: written unquoted), which no schema can accept — is reported per §6.3, and no cross-file check runs over it. A consumer treats it exactly as a file that does not parse: it renders nothing from it and names the file unreadable. One malformed field withholds the whole file, so what a reader is shown never depends on where the bad field sits.

The bound. An annotation MAY correct an inference and MAY add knowledge a measurement cannot express. It MUST NOT be used to contradict a measurement — the artifact's own arithmetic and catalog reads take precedence over prose written at another moment (§2.4's precedence rule, restated here at file grain). This is why these files are named .annotations., never .override..

Scope. The rule covers every human-written KEYED file — a mapping whose entries a producer can resolve against something it emitted. description.md is outside it: unstructured prose has no artifact to mirror and no entries a producer could check, so it keeps its own name and its own section (§2.4).

2.7.1 statistics.annotations.yaml

Per-table user-authored notes about individual columns, keyed by column name.

format_version: 1
columns:
  <column_name>:
    note: <markdown_string>          # OPTIONAL - free-form prose
    claims:                          # OPTIONAL - checkable predicates
      <stat>: <predicate>
    values:                          # OPTIONAL - notes on individual domain members
    - value: <scalar>
      note: <markdown_string>
    derived_from:                    # OPTIONAL - the columns this one's values come from
    - table: <fqn>
      column: <column_name>
grain:                               # OPTIONAL - a human-stated key the producer measured none for
  keys:
  - columns: [<column_name>, ...]
    note: <markdown_string>          # OPTIONAL - free-form prose

Keys MUST match a column's name as it appears in that table's statistics.yaml columns map (§2.2) — always lowercase (§2.2.1), not necessarily the form a user typed against the source database. A key naming a column not present there is stale: consumers SHOULD report it rather than fail on it, and it MUST NOT block generation or validation.

A column's entry carries note, claims, values, derived_from, or any combination — an entry with none of them is legal but carries no annotation. note is free-form Markdown. claims is zero or more <stat>: <predicate> pairs in the statistic-assertion grammar ASSERTIONS.md §2 specifies, scoped to this column: <stat> is any stat that grammar allows asserting on a column, <predicate> any form it defines. A claim states something a statistic can check — it MUST NOT be used to record what a statistic cannot express; that belongs in note or in description.md (§2.4). Consumers evaluate claims against the table's own statistics.yaml, offline, the same way the assertion DSL evaluates a statistic assertion in offline mode. A claim that contradicts the statistic it names, or that cannot be evaluated at all (an unassertable or unemitted stat, a malformed predicate, a redacted column), is reported per §6.3 — never silently accepted, and never failed as an error: this axis is advisory, per §2.4's precedence rule.

values carries what a domain member means, not what a statistic can measure. Each entry pairs a value — a scalar addressed the same way statistics.yaml's own values list addresses one (§2.2.4) — with a note explaining it: a sentinel (-1 meaning "count not recorded at collection, not a real zero"), a reserved code, anything a human knows about that one member of the domain that the count beside it cannot say. A value naming a member the column's own exhaustive values list (values_coverage of 1.0) does not have is stale, reported the same way a stale column key is (§6.3); under a truncated list the value may occur unlisted, so an unmatched note is treated as an addition, not stale. A redacted column (§2.2.9) has no literal a note could name, so values there is unassertable rather than checked.

derived_from names the columns this column's values are copied or computed from, so a note written once on a source reaches every table built from it. Each entry carries table, the source's FQN, and column, its name in that table's statistics.yaml; both are REQUIRED. A consumer rendering the column for a reader who writes queries carries each source column's note and value notes beside the column's own, labelled with the source they come from. They never replace the column's own entry, and inheritance is one step: a source's own derived_from is not followed. The field states lineage, not a join — it adds no edge to relationships.yaml and asserts no foreign key. An entry naming a table absent from manifest.yaml's tables, or a column that table's statistics.yaml does not have, is stale, reported at warning severity like a stale column key (§6.3). It layers over the producer's own derived_from (§2.2.4): where an annotation entry carries the key, its list replaces the producer's for that column wholesale, never merged — and derived_from: [] states that the column has no source worth carrying, suppressing the producer's list.

grain states a candidate key by the columns that make it one — the same identity statistics.yaml's own grain.keys[] addresses (§2.2.12), so a human writes the exact columns they mean rather than inventing a second naming scheme. Each entry carries columns (REQUIRED, the addressing tuple) and note (OPTIONAL). This is additive, never a correction: the producer's own measured or declared keys are unchanged and still rendered, and a human states a key the search never found or a declared one the catalog does not carry a constraint for — the two lists are shown together, the annotated one marked as such. A columns entry naming a column the table's statistics.yaml does not have is stale, reported the same way a stale columns key is (§6.3) — any one unknown column invalidates the whole key, since the tuple addresses a set, not a single field.

This file carries no table-level key, so it never conflicts with description.md, which is table-grain narrative. An empty columns mapping is legal and carries no annotations.

Plain views MAY carry this file. A view's own statistics.yaml (§2.2.15) names every column its catalog read found, so a key naming a column absent from that list is stale the same way it is for a table.

The reference JSON Schema SHALL be at spec/v1/statistics_annotations.schema.json.

2.7.2 relationships.annotations.yaml

Per-table human channel over relationships.yaml, in the source table's own directory — the side a consumer joins from.

format_version: 1
refers_to:
- column: [taxon_id]
  target_table: arboretum.seedbank.taxon
  target_column: [taxon_id]
  verdict: rejected                # OPTIONAL - marks a producer-inferred edge wrong
  note: <markdown_string>          # OPTIONAL - free-form prose
  claims:                          # OPTIONAL - checkable predicates
    <stat>: <predicate>

An entry addresses an edge by column / target_table / target_column — the same triplet refers_to (§2.3.2) carries, and REQUIRED here for the same reason: without it there is nothing to resolve against. Every other field is OPTIONAL, per §2.7's layering rule.

An entry that resolves against an edge relationships.yaml emits annotates it. verdict: rejected states that a human has determined the edge is wrong — typically an inferred one the naming rule (§2.3.8) resolved to a column that shares nothing but a name with the source. note carries the reasoning. A verdict MUST NOT address a detection: declared edge: a declared edge is read from the catalog, and an annotation may correct an inference or add knowledge a measurement cannot express, never contradict a measurement (§2.4's precedence rule). A conforming producer preserves this file byte-identical across regenerations and never writes to it, on the same terms as statistics.annotations.yaml (§2.7.1).

An entry that resolves against nothing is a human-authored edge the producer did not, and structurally could not, emit - §2.7's layering rule covers this file the same as any other. This is the channel a path-valued endpoint (§2.3.9) is authored through: a warehouse storing a join key inside a semi-structured column has no producer-emitted edge to annotate, so the entry is stated here directly, with path / target_path alongside column / target_column. A consumer surfaces a human-authored edge beside the measured ones, on both tables it joins, tagged as annotated — never as declared, since nothing in the catalog or the data stands behind it.

Rejecting an edge does not remove it from the print. refers_to and referenced_by are producer measurements (§2.3); a verdict is a consumer-facing correction layered over them, and the table's own graph on disk is unchanged. A consumer handing the graph to an agent that writes queries withholds a rejected edge rather than marking it: dbprint context and dbprint serve omit it from every rendering, both sides of it, and from the relationship events of diff.yaml they return, so a reader never meets it as a join path. A browsing surface may still draw it, marked as rejected. A verdict addressing a detection: declared edge is void: it withholds nothing and marks nothing, and every consumer, a browsing surface included, renders that edge as it renders one no verdict addresses.

A verdict addressing an edge absent from relationships.yaml is stale: reported at warning severity, the same treatment statistics.annotations.yaml staleness gets (§2.7.1). An entry carrying no verdict needs no counterpart to resolve against, so a human-authored addition is never stale.

claims checks an edge's own observed block (§2.3.10) the same way statistics.annotations.yaml's claims checks a column (§2.7.1) — zero or more <stat>: <predicate> pairs in the ASSERTIONS.md §2.1 predicate grammar, where <stat> is any of observed.fanout_avg, observed.fanout_max, observed.target_coverage, observed.containment, observed.coherent, observed.scope_compatible, observed.answerable_count — this document's own edge-claim vocabulary, distinct from ASSERTIONS.md §2.4's column vocabulary, which that document scopes to .dbprint.yaml assertions only. Consumers evaluate claims against the edge's own refers_to entry, offline. A claim naming an address refers_to carries no counterpart for at all has nothing to resolve against, the same as any stat absent from a real edge — unassertable, never a stale-edge finding of its own. A claim that contradicts the measurement, or cannot be evaluated (an unassertable stat, a malformed predicate, a composite or scope-incompatible edge carrying no observed block), is reported per §6.3 — advisory, per §2.4's precedence rule, exactly as a column claim is.

The reference JSON Schema SHALL be at spec/v1/relationships_annotations.schema.json.

2.7.3 manifest.annotations.yaml

Connection-grain human notes, at the connection root beside manifest.yaml — the home for a fact true of the whole warehouse rather than one table, which the per-table description.md (§2.4) has no place to carry without repeating it once per table.

format_version: 1
notes: <markdown_string>          # OPTIONAL - free-form prose

notes is the only field. Free-form Markdown, on the same terms description.md gives table-grain narrative (§2.4) — the format imposes no structure on it, and nothing checks it against a measurement. An empty or absent notes key is legal; the file existing at all is what a consumer checks for, per the presence rule below.

Optional, and its own presence is the signal. Unlike manifest.yaml, diff.yaml and reading.md (§1.2), this file is OPTIONAL — most connections carry no warehouse-wide fact that does not already fit description.md. A producer that finds one at the connection root records it in manifest.yaml's top-level manifest_annotations field (§2.5); absence of that field means no human authored one, on the same absence-means-unauthored terms statistics_annotations (§2.7.1) and relationships_annotations (§2.7.2) already carry at table grain.

A consumer reading one table carries the notes too. A fact the file states is one a reader of any single table may need to read that table's columns — a code a column note names, a unit many columns share — so a single-table rendering carries the notes in its own right, and a rendering over several tables carries them once rather than once per table.

Producers MUST NOT write to it. Once a user creates one, it is preserved across regenerations byte-identical, on the same terms description.md and every other file in this section already carry (§2.7's layering rule).

Distinct from the producer-authored consumer guide. reading.md (§1.2.1) is generated, producer-written, and overwritten every run; manifest.annotations.yaml is human-written and never touched by a producer. The two sit at the same connection root and the suffix — .annotations. on one, none on the other — is what tells a reader which is which, the same distinction the naming rule above states for every file in this section.

The reference JSON Schema SHALL be at spec/v1/manifest_annotations.schema.json.

3. Column classifications

Every column has a single classification tag — the most useful one-word summary for AI consumers. Producers MUST assign exactly one classification per column.

3.1 Defined classifications

ClassificationDefining ruleIndicative fields
booleanSQL type is explicitly BOOLEANvalues
jsonSQL type is a document type: JSON / JSONB / VARIANT / OBJECT / SUPERsql_type, null_rate, types, parts
compositeThe producer descended into the column's parts — an array, a record or a map (§2.2.18). Defined by descent, not type name: a column of the same type a producer did not descend into is unsupported, and a JSON column descended into stays jsonparts, parts_found, size
spatialSQL type holds OGC simple-features geometries (PostGIS geometry/geography, MySQL POINT/POLYGON/..., GEOGRAPHY, GEOMETRY, ClickHouse Point/Ring/...)geometry, extent
vectorSQL type is a native embedding type (pgvector vector/halfvec/sparsevec, MySQL and Snowflake VECTOR, ClickHouse QBit) — never a plain array of floatsdimension, norm, zero_count
foreign_key_candidateHas a foreign key, declared or inferred (§2.3.8)inferred.fk_candidate, values
categoricalcardinality <= enumeration_threshold (default 50)values, values_coverage, distribution
temporalSQL type is date / time / timestamp AND cardinality > enumeration_thresholdrange, percentiles, freshness, distribution
numericSQL type is numeric AND cardinality > enumeration_thresholdrange, percentiles, distribution
binarySQL type is binary (BYTEA / BLOB / BINARY / VARBINARY / VARBYTE / BYTES) AND cardinality > enumeration_thresholdlength (bytes), empty_count, inferred.candidate_key, sketch
textSQL type is character (VARCHAR / TEXT / CHAR) or UUID, or any other type the producer measured a cardinality for, AND cardinality > enumeration_thresholdvalues, values_coverage, distribution
unsupportedThe producer could not measure a cardinality for the column at all — array, composite (RECORD/STRUCT), or any other type a producer's adapter declines to profilesql_type, null_rate (only)

Uniqueness is not a classification. inferred.candidate_key (§4.2) is set whenever cardinality_ratio clears its own threshold, independent of which row above a column matches — a unique column is classified by type or foreign-key status like any other, and carries the flag alongside whatever fields that classification requires.

The boundary between text and unsupported is representability, not a type-name list. A type a producer can print, compare and enumerate is summarisable — INET, MACADDR, INTERVAL, a native network or interval type no rule above names — and MUST classify by measurement like any other type, falling to text when nothing more specific matches. To compare means the engine can test two values for equality, which counting distinct values requires; Postgres's XML and geometric types have no equality operator. A type that cannot be summarised at all — an array, a composite record, a type the engine cannot compare — is unsupported, and a producer states this by declining to measure a cardinality for it rather than by the engine matching a type name: cardinality absent is what the fallthrough at priority 12 below reads.

A binary value is counted, never listed. Every engine with a binary type compares, groups and hashes it, so a binary column is measured like any other: a declared or inferred foreign key classifies foreign_key_candidate, a low-cardinality column categorical, each listing its values as lowercase hexadecimal (§2.2.4). Every other binary column is binary, which publishes its count profile, its byte length and its count of zero-length values, but no value list: the top values of a column of keys or uploaded files are neither a domain nor a predicate an agent writes, and the column needs no redaction rule to stay quiet. A future type neither this document nor a producer's own adapter names lands on the right side by this rule, without a maintainer having to extend a membership list first.

3.2 Priority order (first match wins)

When a column matches multiple defining rules, producers MUST walk the list top-to-bottom and assign the first match:

  1. boolean — explicit SQL type
  2. json — explicit SQL type
  3. composite — descended into
  4. spatial — explicit SQL type
  5. vector — explicit SQL type
  6. foreign_key_candidate — foreign key, declared or inferred
  7. categorical — low cardinality regardless of SQL type
  8. temporal — SQL type, high cardinality
  9. numeric — SQL type, high cardinality
  10. binary — SQL type, high cardinality
  11. text — fallback for character types, UUID, and any other type the producer measured
  12. unsupported — fallback for a type the producer could not measure at all

Example: an integer column with values 1–5 has SQL type INT (would match numeric on type) but cardinality = 5 (matches categorical first at priority 7). Classification is categorical. The value distribution gets enumerated explicitly via values:.

Example: a VARCHAR(64) column holding unique slug strings, cardinality above the enumeration threshold, has SQL type character (matches text at priority 11). Classification is text, and it publishes its value list like any other text column. inferred.candidate_key is set separately, since cardinality_ratio = 1.0 clears the SPEC 4.2 threshold — uniqueness did not decide the classification, and does not withhold the fields this one requires.

3.3 Edge cases

All-null columns (cardinality = 0, null_rate = 1.0): classification follows the §3.2 priority order. At cardinality = 0 this resolves to categorical (or foreign_key_candidate / boolean by FK/type), not the type-based numeric / temporal / text branches. The value fields are emitted EMPTY per §2.2.7 (values: [] + values_coverage: 1.0, plus distribution where required), not omitted. Range, percentiles, and freshness remain absent. Consumers see null_rate: 1.0 and read accordingly.

Single-value columns (cardinality = 1): match categorical (1 ≤ enumeration_threshold). distribution: dominant_value is implied.

A unique foreign key (1:1 relationships): classifies foreign_key_candidate (priority 6) regardless of cardinality_ratio — SPEC 3.1's defining rule does not exclude a unique column. inferred.candidate_key is set alongside it whenever the ratio clears the SPEC 4.2 threshold, so a consumer reads uniqueness from that field, not from the classification.

Generated / computed columns (Postgres GENERATED ALWAYS AS, Snowflake virtual columns): classified normally per the rules above. dbprint captures generated-ness only via the DDL file (which records the AS (...) expression). There is no generated: true flag.

A type that stores another type's values under its own name (ClickHouse's SimpleAggregateFunction(f, T)): classifies, measures and sketches exactly as the stored type T would, while sql_type publishes the declared spelling. A type that stores a partial aggregation state instead (AggregateFunction, Redshift's HLLSKETCH) holds no value of any column type, and is unsupported.

A spatial type is never counted. Its values are compared, grouped and hashed unevenly across engines — some refuse to group them at all — and an agent joins geometries by a spatial predicate, never by equality. So a spatial column carries no cardinality, no value list and no sketch at any cardinality, and a column of three distinct points is still spatial, never categorical. Postgres's own geometric types (point, box, circle, ...) are not OGC geometries and stay unsupported.

A vector type is never counted either. Equality exists unevenly across engines, and nobody filters or joins an embedding by it, so a vector column carries no cardinality, no value list and no sketch at any cardinality: a column holding two distinct embeddings is still vector. A plain array of floats used as an embedding (duckdb FLOAT[n], ClickHouse Array(Float32)) is an array, not a vector; only a native embedding type reaches this classification.

SQL types the producer cannot model (array, composite): in a queried file, the producer's adapter declines to measure a cardinality for these, and a column left unmeasured for this reason MUST be classified unsupported regardless of its type name (§3.1). The producer emits exactly the fields §2.2.3's matrix marks REQUIRED for unsupported — sql_type, nullable, null_count, null_rate and classification, plus rows_scanned when a scope block applies (§2.2.8) — and no other statistic. Consumers MUST handle unsupported gracefully — treat as opaque.

A type no rule above names, that the producer DID measure (a native network or interval type; any vendor type this document does not name): classifies text per priority 11, never unsupported — the fallthrough follows the measurement, not a second membership list neither side of a producer is obligated to keep in step. inferred.looks_like and inferred.sensitivity reach it exactly as they would any other text column.

An array, record or map column a producer descends into is composite, not unsupported, and carries its parts (§2.2.18); composite never occurs under catalog_only, where nothing is descended into.

A column of a file carrying catalog_only (§2.2.15): cardinality is absent for a different reason than either row above — no query was issued for the object at all, not that this type resists summarising. Classification still follows the ordinary §3.2 priority order, from every input a catalog read supplies — sql_type, and a foreign key whether declared or naming-inferred (§2.3.8), since inference reads only column names, not the object's rows. The fallthrough differs from a queried file's, though: an array or composite column still reaches unsupported, matched before either fallthrough runs, a binary column reaches binary, a spatial column spatial and a vector column vector, each by its type alone, but an otherwise-unmatched type — inet, interval, xml, anything §3.1 does not name — lands on text here rather than unsupported, since nothing was attempted that could have failed to summarise it. categorical cannot occur under the marker; it needs a cardinality the marker forbids, so a low-cardinality column seen through a base table and again through a catalog-only view over it classifies differently in each.

3.4 Classification reserves

The following are reserved for potential future additions; producers MUST NOT emit them:

  • geographic (lat/lon, geohashes, country codes) — not spatial: that classification covers a geometry type, this reserve the semantics of a pair of numbers or a code
  • monetary (amounts with currency context)
  • enum (explicit ENUM SQL types — distinct from inferred categorical)

Consumers MUST tolerate unknown classification values for forward compatibility (see §5.3) — these names are not the only candidates.

4. Inferred semantics

4.1 looks_like patterns

Unless the file's disabled list names looks_like (§2.2.21), producers MUST run regex/parser-based detection over a sample of distinct non-null values for the classifications §4.1.5 names, assigning each sampled value the first pattern it matches (priority order in §4.1.4) and setting inferred.looks_like to a pattern assigned to ≥ 95% of sampled values.

Assignment reads each value on its own, with one exception: §4.1.3 names the column-level evidence — the sample's size, the column's name, its classification and how often its values repeat — that withholds a pattern from every value of a column that cannot support it. A withheld pattern is skipped exactly as if the value had failed it, so the value falls to the next pattern in priority order.

The field is OPTIONAL — a sample where no pattern reaches the threshold carries none, and its absence is not an error. What is required is the detection, not a result; §4.1.2 requires it of every producer that samples the column, without exception for cost or scale.

Every pattern in §4.1.1 is defined over a string, except the six binary content kinds (png, jpeg, gif, pdf, gzip, zip), which are defined over a binary value's own leading bytes and are the only patterns a binary value is assigned; a string value is never assigned one. A sampled value that is not already a string or a binary value — a driver's native uuid, decimal, or date type, for example — MUST be coerced to its string form (str(v) or the producer's language equivalent) before assignment, applying no locale-dependent formatting and no per-driver special-casing: only the value's own default string rendering is used. This is what makes uuid reachable on a database's native UUID column type, numeric_string on a native decimal column, and iso8601_date / iso8601_datetime on a native date or timestamp column classified categorical (§3.2) — the one classification §4.1.5 runs detection on that a temporal SQL type can still reach despite temporal itself being excluded.

4.1.1 Pattern definitions

PatternDefining rule
uuidCase-insensitive regex ^[0-9a-f]{8}-[0-9a-f]{4}-[1-7][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$. Dashed canonical form; versions 1–7; RFC 4122 variant [89ab]. Braced ({...}), compact (no dashes), and URN (urn:uuid:...) forms NOT matched.
emailRegex ^[A-Za-z0-9._%+\-]+@[A-Za-z0-9.\-]+\.[A-Za-z]{2,}$. Conservative — catches typical emails; rejects garbage. Not a full RFC 5322 validator (no quoted local-parts, no comments, no IP literals).
urlRegex ^https?://[^\s]+$. Only http:// / https:// schemes matched. Other schemes (ftp://, file://, custom) NOT matched.
urnRFC 8141 shape, case-insensitive: urn:, then a 1-31 character namespace identifier ([A-Za-z0-9][A-Za-z0-9-]*), then :, then a non-empty, whitespace-free namespace-specific string. Ranked above path, which would otherwise claim urn:example:weather/today — one / is all path requires. Clears uuid by that pattern's own text excluding the URN form (urn:uuid:...), and clears url because that pattern is anchored on https?://.
ipA bare IPv4 or IPv6 address, parsed by the producer's standard library address type rather than a hand-written regex. Both families assign the same value, including an IPv4-mapped IPv6 address (::ffff:192.0.2.1) — a mixed column of both families reports ip at 100%, which two separate values could never clear the threshold at. NOT matched: a CIDR block (10.0.0.0/8), a host:port pair, a comma-separated forwarded-for chain, a bracketed form ([2001:db8::1]), a zone-qualified form (fe80::1%eth0), and an IPv4 octet carrying a leading zero (010.1.1.1) — each is transport decoration or an ambiguous rendering, not a bare address.
mac_addressSix two-character hex octets joined by one consistent separator, colon or hyphen (00:1b:63:84:45:e6, 00-1B-63-84-45-E6). NOT matched: a separatorless run (001b638445e6) — claimed by hex instead, which is the honest answer since nothing in the value says "address" without the grouping. Ranked above phone: a hyphenated all-digit MAC (00-11-22-33-44-55) is twelve digits with separators, indistinguishable from a national phone number by that pattern's own arithmetic.
jsonjson.loads(value) parses successfully AND the top-level result is a dict or list (not a primitive). Restricting to structured tops filters noise — bare primitives like "42" or "true" parse as JSON but are not what consumers want flagged.
hexOptional leading # (color codes), then 6 or more characters from [0-9a-fA-F], with at least one letter (a-f) among them. The letter requirement excludes an even-length digit id from reading as hex instead of numeric_string, and a compact postal code (postal_code outranks hex) from losing its own pattern. NOT matched: fewer than 6 characters, or an all-digit run of any length — a 6-character code drawn from the closed hex alphabet is self-limiting the way country_code is, but a longer all-digit run is not, so the letter requirement is not optional.
jwtRFC 7515 compact serialization: three .-joined base64url segments, where the first non-empty and decodes to a JSON object carrying alg. The third (signature) segment MAY be empty — the alg: none unsecured form (RFC 7515 §3.1) — and is never decoded, since a signature is bytes and decoding it proves nothing. NOT matched: a five-segment compact JWE, and any other three-dot-segment token whose first part does not decode to a JOSE header — a purely lexical rule would claim any dotted opaque token, which is the kind of exclusion clause this axis's patterns are defined to avoid needing.
base64Length ≥ 16 AND character set is standard base64 (A-Z, a-z, 0-9, +, /, padding =) OR URL-safe variant (-, _ instead of +, /) AND at least one uppercase and one lowercase character AND decodes without error. Length floor avoids matching short numeric or hex strings; the case-mixture requirement is what separates a real encoded token from an ordinary lowercase word or an all-uppercase code, since base64 output is near-uniform over a case-split alphabet. An unpadded URL-safe candidate is matched only when its own length is already a multiple of four — padding a short remainder would admit any label whose length is not congruent to 1 mod 4, which is most of them. hex outranks base64 — every digest length in circulation (32, 40, 64) is a multiple of four drawn from a subset of the base64 alphabet, so base64 would otherwise claim every cryptographic hash. An opaque identifier is NOT matched. A lowercase vendor prefix of one to eight characters, an underscore and an alphanumeric id (lot_K4m9...) is refused outright, and a value made of letters and digits alone — no +, /, -, _ or padding — is matched only when its decoded bytes are UTF-8 text or begin with one of the binary signatures below: a mixed-case alphanumeric id satisfies every other rule whenever its length is a multiple of four, and only the decode tells an encoded payload from it.
latlonTwo decimal numbers, comma-separated with optional following whitespace, each carrying a fractional part, the first in [-90, 90] and the second in [-180, 180]. A space-separated pair is NOT matched — the comma is the evidence, on phone's own precedent that a separator is signal rather than noise. A stated gap: nothing in the values separates a coordinate pair from any two small decimals (1.5,2.5 still matches), and no value test can.
iso8601_durationP[nY][nM][nD][T[nH][nM][nS]], or the PnW week form, requiring at least one component — a bare P or PT is NOT matched.
iso8601_date^\d{4}-\d{2}-\d{2}$, gated by a calendar-validity check — 2024-02-31 is regex-shaped but NOT matched. The basic form (20240115, no hyphens) is NOT matched — the hyphens are the evidence that separates a date from any other eight-digit run, on the same precedent phone's separator requirement uses.
iso8601_datetime^\d{4}-\d{2}-\d{2}[T ]\d{2}:\d{2}(:\d{2}(\.\d{1,6})?)?(Z|[+-]\d{2}:\d{2})?$, same calendar-validity gate on the date portion. Seconds and the zone designator are both optional. A single space is accepted alongside T — not RFC 3339, but what timestamp::text renders in Postgres. Lowercase t/z are NOT matched — legal under RFC 3339, but no database engine emits them, so accepting them would cost a second regex for a form nothing produces. A column mixing dates and datetimes reports neither — iso8601_date and iso8601_datetime are different cast targets, and a column that genuinely mixes them has no single correct one to hand a consumer.
numeric_stringRegex ^-?\d+(\.\d+)?$. Signed integer or decimal. No scientific notation, no thousand separators, no currency markers.
semverSemantic Versioning 2.0.0's own grammar: three dot-separated non-negative integers with no leading zeroes, an optional - prerelease of dot-separated alphanumeric identifiers, an optional + build metadata of the same. Accepts an optional leading v — a form semver.org's own grammar refuses, but that git tags and dependency manifests store constantly, and one that takes no column from any other pattern since it reports nothing today. A two-part version (1.2) is NOT matched and stays numeric_string; a three-part one already fails that pattern's single-dot grammar, so the two never compete for a column.
country_codeMembership of the ISO 3166-1 alpha-2 set, case-insensitive. A closed 249-entry list, and on a sample of eight or more distinct values membership is the whole test; on fewer, the column's name MUST also carry a country word (§4.1.3), since a short list of two-letter tokens — direction codes, initials, a JSON key named id — falls inside the set by chance. Alpha-3 codes are NOT matched — a second hand-maintained list is a second silent-error surface, and alpha-2 is overwhelmingly what databases store. The closed set is self-limiting in the useful direction: a column of US state abbreviations overlaps it by roughly a third and never reaches the threshold. Two-letter ISO 639-1 language codes are a stated overlap, not a defect: 110 of the 184 language codes are also alpha-2 country codes, and a column of them (de, es, it, pt) reports country_code — there is nothing in the values to tell the two apart, and a language_code pattern is refused for exactly that reason (undecidable against this one, not a maintenance concern).
currency_codeMembership of the ISO 4217 alpha set, case-insensitive. A closed 178-entry list, snapshot dated 2026-01-01 and amended two to four times a year — a missing code (a newly minted currency) reads as a known staleness gap, not a bug. Includes precious-metal, fund and test codes (XAU, XDR, XXX) alongside national currencies; none is excluded. Alpha-3 country codes are NOT matched by this pattern — only BTN, CHE, MKD and SLE sit in both sets, four entries out of 249, so a column of alpha-3 codes never approaches the threshold.
postal_codePostal formats that identify themselves by carrying a letter: United Kingdom, Canadian and Dutch shapes. All-digit formats are deliberately NOT matched — a five-digit US ZIP is indistinguishable from numeric_string without a locale, and dbprint has no locale to consult. A column of US ZIPs therefore reports numeric_string, which is the honest answer from values alone. This is a stated gap, not an oversight.
isbnClaimed only on a sample of at least five distinct values (§4.1.3). The 13-digit ISBN form: prefix 978 or 979, GS1 mod-10 check (weights 1 and 3 alternating from the right of the payload). The 10-digit form: nine digits plus a check character (0-9 or X), weights 10 down to 1 from the left, sum divisible by 11. Hyphenated forms are stored constantly; hyphens and spaces are stripped before the check runs, the same precedent card_number set. Outranks ean: the 13-digit form is the 978/979 prefix subset of the 13-digit EAN and carries the identical check, so the subset is claimed first. Both outrank card_number, so a barcode column is not diluted by the tenth of it that also happens to be Luhn-valid.
eanClaimed only on a sample of at least five distinct values (§4.1.3). GTIN forms of eight, twelve, thirteen or fourteen digits: a bare digit run of one of those four lengths, GS1 mod-10 valid. No prefix table — unlike isbn, a GTIN carries no reserved leading digits. A value failing the check is NOT matched. Collides with card_number by arithmetic alone (each check passes about one random digit run in ten of the other's length), and ean ranks above it for exactly that reason — see card_number's own row for the reciprocal case.
vinSeventeen characters from the alphabet A-Z and 0-9, excluding I, O and Q, with the check digit at position 9 computed per the federal VIN regulation (49 C.F.R. 565.15): each letter transliterates to a value (Table III), each position carries a weight (Table IV), and the weighted sum mod 11 MUST equal the check digit — a remainder of 10 renders as X. Ranked above base64 defensively: a 17-character alphanumeric VIN clears that pattern's floor and alphabet, and escapes it today only because 17 is not a multiple of four — arithmetic luck, not a rule.
imeiClaimed only on a sample of at least five distinct values (§4.1.3). Exactly fifteen digits, Luhn-valid, first two digits (the Reporting Body Identifier) a member of the set GSMA has ever allocated to a device manufacturer — eighteen two-digit codes, six of them still active (GSMA's IMEI allocation guidelines, Annex A). NOT matched when the RBI is not in that set, even if the value is otherwise Luhn-valid at device length — the column falls through to card_number instead. A stale RBI mislabels rather than misses: neither 34 nor 37 (Amex's issuer prefixes) has ever been allocated as an RBI, which is what keeps the two patterns from colliding; a future allocation into either prefix would need this list updated.
card_number13-19 digit PAN, spaces and hyphens optional, carrying a valid Luhn (mod-10) check digit. No IIN prefix table. Claimed only on a sample of at least five distinct values, as is every checksum pattern (§4.1.3). A value failing Luhn is NOT matched — a column of order ids or account numbers sharing a card-length digit run scores far below the threshold (about 10% pass Luhn by chance) and reports no pattern. A system that Luhn-checks its own account numbers is indistinguishable from a real PAN from the values alone and will be claimed — a stated gap, no value test can separate it. isbn, ean and imei all outrank this pattern, resolving the checksummed-shape overlaps at the values that would otherwise be claimed here: a fifteen-digit Luhn-valid run reports imei when its Reporting Body Identifier is allocated and card_number otherwise — an Amex number (issuer prefix 34/37) is never in that allocated set, so the two never collide in practice.
ibanISO 7064 mod-97-10: two-letter country code, two check digits, up to 30 further alphanumeric characters, total length 15-34, spaces optional. The first four characters move to the end, each letter substitutes for its position value plus 9 (A=10 .. Z=35), and the resulting integer mod 97 MUST equal 1.
bicISO 9362: six letters (institution + country code) then two alphanumerics (location), with an optional three-character branch code (8 or 11 characters total). Positions 5-6 MUST be a valid ISO 3166-1 alpha-2 country code, from the same set country_code uses.
phoneE.164 (+ then 8-15 digits, separators optional), OR a national form carrying at least one separator (space, hyphen, or parenthesis) with 10-15 digits once separators are removed. A bare digit run is NOT matched — a phone number and an account id are indistinguishable digit runs, and the separator is the only evidence that tells them apart. The dot is NOT a recognized separator — 555.123.4567 is also every version string and every multi-part decimal. The two digit floors exclude a US SSN (9 digits) and an ISO date stored as text (8 digits) arithmetically rather than by a special case. A 15-digit separator-bearing run that also passes Luhn is claimed by card_number instead, which outranks phone; one that does not remains phone.
timezoneExact, case-sensitive membership of the producer's zoneinfo.available_timezones() set — a stdlib call, not a hand-maintained literal. Covers Area/Location names (Europe/London), multi-level names (America/Indiana/Knox) and short aliases (UTC, GMT, EST) uniformly, including a column mixing all three forms. Lowercased names are NOT matched (europe/london) — IANA zone names are case-sensitive, and lowercasing would trade a closed set for a fuzzy one on this axis's precision bias (§4.4). A stated host-variance gap: the set reflects whatever tzdata the runtime resolves, so two producers on different hosts can disagree on rarely-used aliases; this does not break any reproducibility promise this specification makes, and it is smaller than the run-to-run variance §4.1.2's random sampling already permits.
content_typeA registered RFC 6838 top-level type (application, audio, example, font, image, message, model, multipart, text, video, or an x- experimental type), a slash, a restricted-name subtype, and optional ; parameters. Case-insensitive. The top-level type is the closed registry rather than a free token, because any two-segment path (a/image.png) satisfies the free-token grammar and the two patterns would stop being distinguishable.
pathA POSIX filesystem path: at least one / separator with a non-empty segment before it, no whitespace in any segment, and no URI scheme. Absolute and relative forms both match. Windows forms are NOT matched — a drive letter or a backslash separator is reserved, so a column of C:\... values reports no pattern rather than a wrong one. A network block is NOT matched — CIDR notation (10.0.0.0/8) satisfies the same shape (a segment, a slash, a segment), so a value that also parses as a network address reports no pattern rather than path. A chain of all-digit segments is NOT matched — a slash date (07/11/2025, 2025/11/07, 7/11/25), a fraction (3/4) and a slash-joined numeric id satisfy the same shape, and a date is a date, not a location: no pattern claims a slash date (iso8601_date requires hyphens), so such a column reports no pattern rather than a wrong one. A path needs at least one segment carrying a non-digit character.
filenameA bare name with an extension and no separator: ^[^/\\\s]+\.[A-Za-z][A-Za-z0-9]{0,9}$. The extension MUST begin with a letter, which is what keeps an IPv4 address and a decimal number from reading as filenames. semver outranks it, so a prerelease or build form (1.0.0-alpha.beta) reports that pattern instead. A stated gap: a hostname (example.com) is not separable from a filename by value alone, and no list resolves it. Neither carries a path separator, both end in an alphabetic label, and a TLD-length or TLD-membership rule fails on its own evidence — .zip, .mov, .sh, .py and .md are all delegated top-level domains and all common file extensions, so archive.zip and example.zip stay indistinguishable whichever list is consulted. Consumers reading filename on a column of domains have the honest answer available from the values themselves.
pngA binary value whose bytes begin with the PNG signature 89 50 4E 47 0D 0A 1A 0A.
jpegA binary value whose bytes begin with FF D8 FF.
gifA binary value whose bytes begin with the ASCII GIF87a or GIF89a.
pdfA binary value whose bytes begin with the ASCII %PDF-.
gzipA binary value whose bytes begin with 1F 8B.
zipA binary value whose bytes begin with 50 4B 03 04, 50 4B 05 06 (an empty archive) or 50 4B 07 08 (a spanned one). A container format — an .docx, .xlsx or .jar is a zip, and is reported as one.
proseFree-running text: at least 5 whitespace-separated tokens, at least one common function word, and matching none of the structural patterns above. The function-word list is English; prose in other languages is not detected, and its absence is not an assertion that a column is structured. A closed picklist is NOT matched — a categorical column whose values repeat (§4.1.3) holds a fixed set of sentence-like labels, which a predicate is written from, not free text.

4.1.2 Sampling strategy

Producers MUST sample up to looks_like.sample_size (default 1000) DISTINCT non-null values, selected RANDOMLY across the distinct value set of the scanned set (§2.2.8) — the rows a scope leaves, which are the rows every other statistic in the file describes. Random selection avoids the bias of top-N-by-frequency, which would over-weight common values.

If the column's distinct count is less than the sample size, producers MUST sample all distinct non-null values.

The sample size comes from the configuration in force for the table being profiled — looks_like.sample_size in .dbprint.yaml, which a producer MAY let a project set per connection, per table, or both. A producer that resolves it per table MUST use the value resolved for the table it is sampling.

Adapter implementations choose how to obtain the sample efficiently (Snowflake SAMPLE, Postgres TABLESAMPLE, MySQL WHERE RAND() < x); the spec requires only that the sample is a uniformly random selection of distinct non-null values.

A producer MAY approximate the distinct-value draw above a row-count threshold of its own choosing, trading the guarantee above for bounded query cost: drawing a uniformly random sample of rows first (Snowflake SAMPLE, Postgres TABLESAMPLE, MySQL ORDER BY RAND()) and taking the distinct values among them, rather than scanning the full distinct set to draw from directly. A full distinct-first random draw costs more as the column's distinct-value count grows, independent of row count, which a row-level sample bounds by construction. The approximation reintroduces exactly the bias the first paragraph's random-selection requirement exists to avoid: a value appearing in more rows is more likely to survive the row-level draw, so the sample over-represents common values in proportion to their frequency. Below the threshold, the full distinct set is read and drawn from directly, satisfying the requirement above without approximation.

A producer that samples an eligible column MUST publish the verdict it reached. There is no permission to decline detection on cost grounds alone: switching detection off is configuration, recorded in disabled (§2.2.21), never a silent omission. Where it is not switched off, an absent inferred.looks_like on a classification §4.1.5 names states only that no pattern reached the threshold, never that the column went unmeasured. A draw that failed is not a verdict: the column names inferred.looks_like in its own unmeasured list (§2.2.4).

4.1.3 Match threshold

looks_like is set to a pattern when ≥ 95% of sampled values are assigned that pattern under the priority order of §4.1.4. The threshold is uniform across all patterns, and a value assigned no pattern counts toward the denominator — the share is of what was sampled, not of what matched something. This holds regardless of a value's native type: §4.1.1's coercion happens before assignment, never before counting, so a minority of non-string values inside an otherwise-structured sample dilutes the share rather than being excluded from it.

The 95% threshold tolerates typical data-quality noise (legacy rows with placeholder values like "deleted" or "unknown") while remaining strict enough that consumers (LLMs writing SQL) can rely on the hint.

Below twenty sampled values the threshold is unanimity as an arithmetic consequence of the 95% rule, not as a separate requirement: the best non-unanimous score on n values is (n-1)/n, which reaches 0.95 only at n = 20. Unanimity on a handful of values is weak evidence for a pattern a value can satisfy by chance, so three column-level conditions withhold a pattern from every value of the column (§4.1), each a producer constant, never configuration:

  • The checksum floor. isbn, ean, imei and card_number are withheld from a sample of fewer than 5 distinct values. One digit run in ten passes a mod-10 check, so a one-value column of epoch nanoseconds would otherwise publish card_number — and through §4.4.3, sensitivity: financial_account. The value falls through to numeric_string.
  • The country-code floor. country_code is withheld from a sample of fewer than 8 distinct values unless the column's name carries one of the words country, countries, nation, nations or nationality as a whole name token (the same token split §4.4.3 applies, so shipToCountry qualifies and destination does not). 249 of the 676 two-letter tokens are alpha-2 codes, so a short list of them is not evidence on its own. A value the floor withholds from country_code is not reassigned to timezone either, although IANA keeps GB and NZ as zone aliases: it falls past both.
  • The closed-list rule. prose is withheld from a categorical column whose non-null scanned rows number at least 5 times its cardinality — every value repeating on five rows on average is a picklist, not free text.

The sample is of distinct values (§4.1.2), so its size is the distinct count the two floors read. Nothing else sets a minimum: a verdict resting on as few as two values is published exactly like one resting on ten thousand, and the next two paragraphs are what let a reader tell them apart.

Producers MUST publish the evidence a verdict rests on. Alongside inferred.looks_like, a producer emits inferred.sampled (the number of values drawn) and inferred.matched (how many of them the winning pattern claimed) — the two operands of this section's own threshold. A consumer recomputes matched / sampled to judge how much a verdict is worth trusting; a two-of-two match and a nine-thousand-five-hundred-of-ten-thousand match both satisfy the 95% rule but are not the same claim, and only the pair lets a reader distinguish them. Neither field is a second gate: the verdict already cleared the threshold before either is written, and a producer MUST NOT withhold looks_like because the pair looks thin.

sampled and matched describe looks_like alone, never epoch_unit or sensitivity even where those axes read the same draw (§4.1.5, §4.4.4) — each pattern matches a different subset of one sample, so one count could not describe more than one verdict without becoming ambiguous about which. A consumer reading a small sampled on epoch_unit or sensitivity is reading nothing, since the pair is never emitted for them.

A sample that clears no pattern MAY still have come close to one. Where no pattern reaches the 95% threshold, producers emit inferred.looks_like_candidate (the highest-scoring pattern, ties broken by the §4.1.4 priority order) and inferred.looks_like_candidate_share (that pattern's share of the sample) whenever the share is at least 50% — a producer constant, never configuration, so two prints stay comparable on it. Below 50% the column's dominant character is not the pattern, and producers MUST omit both fields rather than publish a share too small to be informative. The pair is read from the same per-pattern tally the verdict test already computed; no pattern's own matching rule, the §4.1.4 priority order, or the 95% verdict threshold changes on this account.

looks_like_candidate and looks_like_candidate_share are emitted together, and only in looks_like's absence: the near-miss and the verdict are mutually exclusive, and a column MUST NOT carry both. A share at or above 95% is a contradiction — it would have cleared the verdict threshold and been looks_like instead — and producers MUST NOT publish one there.

4.1.4 Priority order when multiple patterns match

When a sampled value matches multiple patterns, producers MUST walk this list top-to-bottom and assign the first match:

  1. uuid
  2. email
  3. url
  4. urn
  5. content_type
  6. ip
  7. mac_address
  8. country_code
  9. currency_code
  10. postal_code
  11. isbn
  12. ean
  13. imei
  14. card_number
  15. iban
  16. bic
  17. phone
  18. timezone
  19. json (top-level structured)
  20. hex
  21. jwt
  22. vin
  23. base64 (length ≥ 16)
  24. latlon
  25. iso8601_duration
  26. iso8601_date
  27. iso8601_datetime
  28. numeric_string
  29. semver
  30. path
  31. filename
  32. prose
  33. png
  34. jpeg
  35. gif
  36. pdf
  37. gzip
  38. zip

A column-level looks_like value is the pattern assigned to ≥ 95% of sampled values when each value is evaluated under this priority order.

Seventeen orderings in this list resolve a conflict between patterns, and a producer MUST NOT reorder them — iso8601_date and iso8601_datetime are not among them: nothing ranked above them claims their shape, and nothing between them and numeric_string claims what they decline, so their placement is a family grouping rather than a conflict resolution. currency_code is not among them either: no pattern above it can claim a three-letter alphabetic token, so its rank is free rather than a conflict resolution. Neither is jwt nor vin: nothing above base64 claims a dotted token or a 17-character alphanumeric run today, so both ranks above it are defensive rather than a conflict resolution — they stay robust against a future widening of the base64 alphabet, at no cost to any pattern that exists now.

  • url outranks path and filename, so a link never reports as either.
  • content_type outranks path, so image/png is not read as a two-segment path.
  • path outranks filename, so the presence of a separator decides between them: image.png is a filename, a/image.png is a path.
  • path and filename sit below the patterns whose character sets they subsume. A path is only "no whitespace and a separator", which a long base64 token satisfies — the slash is in its alphabet — and which compact json carrying a URL satisfies too. A filename is subsumed the same way by every dotted address and every decimal. Ranked higher, either would take columns that belong elsewhere.
  • semver outranks path and filename, so a prerelease or build form (1.0.0-alpha.beta) is not read as either — this is the entire fix for that row, since a dotted, hyphenated version string satisfies filename's shape (its final segment starts with a letter and clears the ten-character ceiling) below either point.
  • phone outranks base64, so a 15-digit E.164 number (+ plus 15 digits is 16 characters) is not read as base64 — its character set and length both clear that pattern's floor.
  • card_number outranks phone and base64, so a 15-digit separator-bearing Amex number and a 16-digit bare PAN both read as card_number rather than falling to either — this is the entire fix for both rows, since below either point the column would still report a pattern and the defect would go uncaught.
  • isbn outranks ean, so the 978/979-prefixed subset of the 13-digit EAN reads as isbn rather than the superset pattern it also satisfies. Both outrank card_number, so a barcode column is not diluted by the tenth of it that also happens to be Luhn-valid — the loser of that arithmetic collision reports nothing rather than the wrong shape.
  • imei outranks card_number, so a fifteen-digit Luhn-valid run with an allocated Reporting Body Identifier reads as imei rather than card_number — this is the entire fix for that row, since card_number carries no IIN table to exclude device identifiers on its own.
  • hex outranks base64, so a cryptographic hash of any length in circulation (32, 40, 64 hex characters) reads as hex rather than as base64-encoded data. postal_code in turn outranks hex, so a compact UK postcode (EC1A1BB, coincidentally all-hex) still reads as postal_code.
  • mac_address outranks phone, so a hyphenated all-digit MAC address is not read as a national phone number.
  • urn outranks path, so a URN carrying a / in its namespace-specific part is not read as a filesystem path.
  • timezone outranks path and base64, so Europe/London is not read as a filesystem path and Pacific/Auckland (16 characters, a base64-alphabet subset) is not read as base64.
  • country_code outranks timezone: a handful of IANA backward aliases (GB, NZ) are also alpha-2 country codes, and the ordering keeps the answer host-independent for the one pattern where the collision is possible.

Because each value is assigned before anything is counted, these orderings hold for every sample rather than only for uniform ones. A column of 90 image/png values and 10 POSIX paths reports neither pattern: the media types are assigned content_type and the paths path, and at 0.90 and 0.10 neither clears the threshold. Counting each pattern independently over the whole sample would report path at 1.00 — the outcome the second ordering above exists to forbid.

prose is the last string pattern because it is the fallthrough for anything textual carrying no structure. The six binary kinds follow it and compete only with each other: a binary value is assigned one of them or nothing, and no two of their signatures share a prefix, so their order resolves no conflict. Its definition (§4.1.1) also excludes every structural pattern, so the predicate answers for a value on its own rather than only in the position the priority order gives it.

4.1.5 Classifications that trigger looks_like detection

Unless the file's disabled list names looks_like (§2.2.21), producers MUST run looks_like detection on columns classified as:

  • categorical
  • text
  • foreign_key_candidate
  • binary — the six binary content kinds only, read from each sampled value's leading bytes; a fixed-length key needs no kind of its own, since length.min == length.max already says what it is

A part (§2.2.18) of one of these classifications is sampled and detected exactly as a column of it is.

Producers MUST NOT run looks_like on columns classified as:

  • boolean, numeric, temporal — the SQL type already conveys the shape; looks_like adds no information
  • json — the column IS JSON; looks_like: json would be tautological
  • unsupported — opaque by definition

A numeric_string verdict is additionally withheld on a numeric SQL type, even on a classification that runs detection. A categorical or foreign_key_candidate column whose SQL type belongs to the same numeric type family §3.2 classifies on (a numeric-typed column reaches either classification via a declared key or a low cardinality, independent of its type) restates its own declared type by construction if numeric_string is allowed to publish there — the same cost the classification-level exclusion above pays, paid here for a numeric-typed column that classifies elsewhere. Producers MUST withhold numeric_string in that case, along with the sampled/matched evidence pair that would have qualified it (§4.1.3); every other looks_like pattern remains eligible on the same column, detection still runs, and only this one verdict is suppressed. The same withholding applies where numeric_string is the near-miss rather than the verdict: producers MUST NOT publish looks_like_candidate: numeric_string on a numeric SQL type, and MUST NOT fall back to the next-best pattern in its place.

This exclusion is known to hide value shapes the SQL type does not convey, on numeric alone. For boolean and temporal the rationale is complete: a boolean sample can match nothing new, and every textual pattern looks_like defines is either the tautological ISO rendering of the column's own type (§4.1) or undecidable from the values (§4.4.1's national_id paragraph). A numeric column is different — BIGINT conveys "an integer this wide" and says nothing about whether the integer is a quantity, a card number, an epoch, or a national id, so the exclusion trades a real detection gap (a card number in a BIGINT is permanently unreachable by looks_like) for the certainty of never publishing looks_like: numeric_string on the overwhelming majority of numeric columns, which hold ordinary quantities and would report it on every one by construction. The trade is deliberate, not an oversight: widening costs one sample_values query per newly-eligible column per run, and the recall the axis needs is available more cheaply on sensitivity (§4.4.3), which already reads the column name for free on these two classifications.

4.1.6 looks_like reserves

Empty. Every pattern name this format uses is defined in §4.1.1 and ranked in §4.1.4.

Consumers MUST tolerate unknown looks_like values (see §5.3) — a pattern can be added without a MAJOR version bump, reserved here or not.

4.2 candidate_key

inferred.candidate_key: true MUST be set when cardinality_ratio ≥ 0.9999 (effectively unique), regardless of declared PRIMARY KEY / UNIQUE constraints. The flag is a threshold, not a proof of uniqueness: a column at ratio 0.9999 can carry measured duplicates, and candidate_key: true alone does not say whether it does.

inferred.candidate_key_exception distinguishes a genuinely unique column from one inside the threshold band. Emitted only on a column carrying candidate_key: true whose cardinality_ratio is strictly below 1.0 — a column at exactly 1.0 emits nothing here, unchanged from today. Two values:

  • measured_duplicates — cardinality_method is exact and cardinality is less than the non-null scanned count (rows_scanned - null_count). The producer counted every distinct value and some rows share one.
  • estimated — cardinality_method is approximate. The estimator's own error can straddle the threshold in either direction, so a ratio below 1.0 here is not a count of duplicates; it is a statement that the count itself is not exact.

The two are not interchangeable: an approximate count showing no shortfall against rows_scanned still MUST NOT be read as measured_duplicates, since the estimate could as easily have overcounted the true distinct value set as undercounted it.

4.3 fk_candidate

Reserved; not defined. Cross-table overlap-based foreign key detection — comparing a column's values against a candidate target's — is defined, but as relationships.yaml's third detection value, measured (§2.3.11), on the same footing naming-based inference already has there, not as a per-column statistics.yaml field. A producer MUST NOT emit inferred.fk_candidate; neither naming-based nor value-derived inference is ever reported through it.

4.4 sensitivity — data that must not leave the database

looks_like answers "what shape is this value" and is tuned for precision: a wrong shape hint makes an agent write a wrong query. sensitivity answers a different question — "does this column hold data that must not leave the database" — whose errors are asymmetric in the opposite direction. Over-flagging costs one over-masked column. Under-flagging leaks a customer's name, home address, or a live credential into a git repository.

Most categories on this axis are personal data in the ordinary sense — a name, an address, a date of birth. One is not: credential (§4.4.1) flags a secret whose disclosure is a security incident rather than a privacy one. The axis is named for what it gates, not for a single legal or ethical framing; SPEC 4.4.2 already forecloses reading it as a compliance claim, and that forecloses reading it as a privacy-only claim too.

The two are therefore separate fields with separate error budgets, and they are not alternatives: an email column carries looks_like: email and sensitivity: contact at the same time. A consumer gating redaction reads one field rather than maintaining its own shape-to-sensitivity mapping.

4.4.1 Categories

CategoryMeaning
personal_nameA natural person's name, in whole or in part
postal_addressA street address or a line of one
geolocationA coordinate, a coordinate pair, or a geohash
date_of_birthA person's date of birth
national_idA number a government issued to identify a person — a social security, tax, or passport number
financial_accountA bank or card account — an IBAN, a bank account number, a card number, a card security code
credentialA secret — a password hash, an API key, a session token, a private key
healthClinical data — a diagnosis, a blood type, a medication, an allergy, a disability
demographicAn attribute widely treated as needing extra care — ethnicity, religion, sexual orientation, gender identity, political or union affiliation
employmentA person's compensation — salary, wage, bonus, commission
contactA means of contacting a person — email, telephone
online_identifierA network address, device or session identifier that ties activity to a person — an IP address, a MAC address, a device or advertising identifier, a session or cookie identifier

health and demographic are separate categories, not one merged value. RedactRule.covers() targets by sensitivity value, so the vocabulary is the granularity of the redaction control — a merged value would force both or neither, where a project may legitimately need diagnoses withheld while ethnicity stays readable for equity reporting, or the reverse. Both are name evidence only; a diagnosis_notes text column reporting looks_like: prose still carries health even though its prose exemption (§2.2.3's second footnote) means it publishes no values list to withhold — the flag is not conditional on there being something to redact. condition, treatment, procedure/procedure_code, test_result and status are deliberately excluded from health — each is a rules engine's predicate, an experiment arm, a stored procedure, a CI run, or an ordinary status column at least as often as a patient's. Bare gender/sex, nationality/country (join keys, §4.4.1's postal_address row) and marital_status/language are deliberately excluded from demographic — each is the breakdown an analytics print is most often written to support, so a false positive there costs the most.

A spatial column carries geolocation by its type. A geometry is a set of coordinates, which is this category's own definition, so the type is evidence (§4.4.3) whatever the column is named — a column of customers' home points named geom is flagged as surely as one named location.

geolocation does not re-open §3.4's reserved geographic classification. That reserve is a classification, which replaces a column's whole emitted field set; a latitude column flagged geolocation keeps its numeric classification, its range and its percentiles — sensitivity is additive and displaces nothing, so the reasoning that keeps geographic reserved does not reach this axis. Two evidence paths: a latitude/longitude pair in separate numeric columns is name evidence only (no sample is drawn for numeric - §4.4.3 below), and a coordinate pair stored in one text column is flagged by its latlon values (§4.1.1, §4.4.4), independent of the column's name. city, region, postcode and zip are deliberately excluded: a redact rule on this value would silently empty the single most-read categorical enumeration in a schema, and a postcode is postal_address's question, not this one's.

date_of_birth is name evidence only, and deliberately excludes age. No sample is drawn for a temporal column (§4.1.5 names only categorical/text/foreign_key_candidate), so the column name is the whole test — unambiguous tokens (date_of_birth, dob, birth_date, ...) need no value agreement, the same treatment first_name gets. An age column is numeric, publishes a coarser range (ages, not dates), and the word means cache, account, or file age at least as often as a person's — it does not join this category. redacted: drop does not remove freshness: max_age_days is derived by arithmetic rather than read from a cell (§2.2.3's matrix footnote), so a fully dropped date_of_birth column still publishes it — coarsened to the nearest 90 days rather than to the day (§2.2.9), which is what keeps the youngest person's birth date from being recoverable through the one field drop cannot remove.

national_id carries no looks_like value. A dashed US SSN (123-45-6789) is a hyphenated all-digit string, locale-bound, and indistinguishable from a ZIP+4 or an internal reference code by value alone — the same reasoning that keeps postal_code off all-digit formats (§4.1.1). Its check is corroboration for this axis only (§4.4.4), never a shape published to looks_like.

A tax identifier held by an organisation is not a national_id. tax_id and vat_number are issued to companies as often as to people, so either token whose immediately preceding name token names an organisation — business, company, corporate, customer, merchant, organisation/organization, partner, reseller, seller, supplier, vendor — is not evidence: supplier_vat_number and merchant_tax_id report nothing. A bare tax_id, an employee_tax_id, and every other token in the category (ssn, passport_number, national_insurance_number, ...) still need no value agreement.

financial_account is the one category whose value corroboration is a checksum rather than a heuristic. A column whose sampled values are at least half iban or card_number (§4.1.1, §4.4.4) carries financial_account whatever its name is — a mod-97 or a Luhn check under the value, the same shape-corroboration §4.4.3 already gives contact for email/phone. Institution identifiers (bic, a SWIFT code, a UK sort code, a US routing number) are refused rather than unconsidered: they identify a bank, not the account itself, and flagging a public institution code would redact data nobody needed redacted.

credential is the one category not defined by personal data, and its leak shape depends on the column's classification. A password_hash column that is unique per row classifies text (SPEC 4.2 — uniqueness does not withhold the value list), so its top-N hashes publish exactly like any other text column's, live values and all. A small api_key table classifies categorical and its values list is exhaustive. Both deserve the category, and both carry live literals a redact rule is needed to withhold — a reader cannot tell which risk applies without reading the matrix. Strong tokens (password, passwd, pwd, password_hash, password_digest, password_salt, api_key, secret_key, secret_access_key, access_token, refresh_token, session_token, private_key, client_secret) need no value agreement. token or secret ending a longer name (api_token, remember_token, webhook_secret) needs no value agreement either: the words before it name what the secret opens, and a pagination cursor so named (next_page_token) is flagged too, the over-masking §4.4.2 accepts. Weak tokens (key, token, secret, hash — ordinary names: a settings table's key, a cache key, a content hash) are corroborated by shape: jwt values flag unconditionally, independent of the name, the same mechanism §4.4.3 already gives contact; hex values corroborate only a weak name, since an unconditional hex flag would wrongly catch a checksum column — each at §4.4.4's 50% share. Each is weak only as the whole name; key and hash stay unflagged at the end of a longer name too, since cache_key and content_hash are ordinary. A detected sensitivity: credential never gains a redacted marker on a column whose classification carries no cell value (§2.2.9's marker rule) — a credential stored in a json column stays unmarked even under a matching redact rule, which is the rule doing nothing rather than failing, while its parts take their own markers (§2.2.9).

employment is name evidence only, and its reach is deliberately narrow. No sample is drawn for a numeric column, so the name is the only evidence, the same constraint geolocation and date_of_birth share. Only drop protects a flagged column: mask and hash leave range and percentiles in place with a substituted scalar (§2.2.9), so redact: [{sensitivity: [employment], with: drop}] is the one rule that removes a salary's bounds. performance_rating, termination_reason, manager_notes, job_title and department are deliberately excluded — each is an ordinary column an HR dashboard groups by, and a false positive there would silently empty an analytic column a print exists to describe (§4.4.2). The category is a person's pay, so money paid to or by an organisation is not in it. A token whose immediately preceding name token is affiliate, agency, company, customer, event, merchant, partner, platform, refund, reseller, supplier, total or vendor is not evidence — affiliate_commission, agency_commission and refund_compensation report nothing, while sales_rep_commission and employee_bonus still report employment. A bare trailing pay is not a token at all: it ends a form field (preferred_way_to_pay) as often as it names a wage; the compounds that do (pay_rate, base_pay, gross_pay, net_pay) remain.

online_identifier reads the ip and mac_address shapes (§4.4.4), and is checked last. The chain is most-specific-first (§4.4.3): an email address is an online identifier in the broad sense too, so online_identifier is checked after contact to keep such a column reporting contact. A server's IP and a client's IP are the same shape, and dbprint cannot tell them apart from a column name alone — a false positive on an infrastructure inventory is the accepted cost (§4.4.2). user_id, customer_id, account_id and every other internal key are refused rather than unconsidered: they identify a person as surely as a cookie does, and flagging them would put a redaction rule over every foreign key in the database. session_token is deliberately excluded here even though it identifies a session — a live session token is a bearer credential first, and sensitivity: credential is where it is reached, so the two vocabularies do not claim the same token. username joins neither this category nor personal_name: a handle identifies an account rather than a device, and nothing separates a real name from a pseudonym in one. Whether the column is unique or repeats on an event log, it classifies text and its top-N most frequent handles ship — a stated residue, not a gap this category is meant to close. A native INET or MACADDR column is reachable in every cardinality band. §3.2's categorical check is type-agnostic and runs before any type-based branch, so a low-cardinality native column classifies normally and stays reachable by name evidence; a unique one reaches text the same way any other high-cardinality measured column does (SPEC 4.2). A column at moderate, non-identifying cardinality with no declared FK matches no named type-based branch either, and classifies text rather than unsupported: the producer measured a cardinality for it, and the fallthrough follows what was measured (§3.2) — so inferred.sensitivity reaches it there like any other text column, with no gap left to state.

4.4.2 Recall bias is normative

Producers SHOULD resolve ambiguity toward flagging. A consumer MUST NOT read sensitivity as a precise claim, and MUST NOT read its absence as an assertion that a column is safe: absence means "not detected", never "safe to publish". The field exists to gate redaction and to direct a human's attention; it does not make a producer a compliance tool, and nothing in this specification claims completeness for it. §7.2 carries this rule alongside every other absence a reader has to interpret.

A column that names its own category and publishes it anyway is a conformance warning, not silence. privacy.unredacted-sensitive (§6.3) fires when a column carries inferred.sensitivity, or a part sits inside a column or part that does, and publishes at least one of values, range, percentiles, extent with no redacted primitive covering it. The code is a warning and never moves the conformance verdict or an exit code — the axis inherits its recall bias, so the warning fires on every false positive sensitivity produces, and silencing a real one costs redacting a column a reader wanted to see.

4.4.3 Evidence

The column name is the primary evidence and a producer SHOULD use it. It is catalog metadata, so consulting it costs nothing at profiling time and is unaffected by sampling or by a row filter (§2.2.8). It is also the only evidence that separates a name column holding people from a name column holding companies, which values alone cannot settle. Detection MUST run against the catalog's own spelling (§2.2.4's physical_name, where the map key (§2.2.1) folded it to lowercase), not the lowercased map key: a token-boundary detector reads firstName and first_name alike but not firstname, so evaluating the lowercased key would miss every camelCase or PascalCase column on an adapter that folds case for the key.

Values corroborate where the name is ambiguous. A producer MUST NOT require value agreement for an unambiguous column name: a first_name column holding initials, placeholders, or a script the producer does not model is still a name column.

A boolean column carries no category. Two values cannot hold anything §4.4.1 names, so a flag named for what it toggles — opted_in_email, prefers_phone, wants_postal_address, enforce_strong_password — reports none, whatever its name says.

A head naming something other than a person vetoes a personal_name match. Organisation, place and label names read as people's names to a value test — "Lakeside Botanical Society" and "Awaiting Review" are capitalized two- and three-word strings exactly as "Ada Lovelace" is — so the name decides first: a name, display_name or full_name whose immediately preceding name token is one of account, branch, brand, building, campaign, category, channel, city, cluster, column, company, country, domain, event, field, file, group, host, list, location, merchant, metric, office, organisation/organization, partner, plan, product, program, project, region, report, reseller, segment, site, stage, status, store, supplier, survey, table, team, template, tier, type, vendor, venue or workflow is not evidence (supplier_name, site_name, status_name). The veto is final: a vetoed full_name or display_name is not judged again by the name it ends in (company_full_name, site_display_name). Every other name token names a person attached to the thing rather than the thing, and the word before it never vetoes it: project_owner, event_contact, campaign_recipient, report_author and account_contact_name are judged exactly as owner, contact, recipient, author and contact_name are. A bare name column is judged by its table's name the same way: when the table's last name token, singular or plural, is one of those words (sites, stores, accounts, suppliers, companies), name is not evidence either. An unlisted head neither vetoes nor qualifies, and the list holds English qualifier nouns only — never names of people, which the next paragraph forbids.

Producers MUST NOT ship a dictionary of given names for this purpose. Such a list is culturally biased by construction, and the column name provides the signal more cheaply and more accurately.

Where the file's disabled names looks_like (§2.2.21), no column's values corroborate, because none are drawn, and the value-shape route the next paragraph describes is unavailable: the name is the only evidence.

On numeric and temporal, values cannot corroborate, because none are drawn. §4.1.5 samples only categorical/text/foreign_key_candidate, so a numeric or temporal column reaches this axis on the column name alone or not at all — the corroboration this section otherwise describes is unavailable there, not merely unexercised. This is why date_of_birth, geolocation and employment (§4.4.1) carry no weak, value-agreement tier: there is no sample to agree with, so every token in those vocabularies is treated as unambiguous by construction.

A part (§2.2.18) is detected as a column of its classification is, reading its last member name as its name, or its column's name when it has no member step. A spatial column's SQL type is evidence for geolocation on its own (§4.4.1). contact, financial_account, online_identifier, geolocation and credential additionally read the sample's value shapes — the pattern §4.1 assigns each sampled value, not the column's looks_like verdict. Each category's shapes are email and phone for contact, iban and card_number for financial_account, ip and mac_address for online_identifier, latlon for geolocation, and jwt for credential. A column carries the category whatever its name is when the sampled values assigned any of its shapes, counted together, reach the share §4.4.4 states. For contact, a value in which an email address (§4.1.1's email rule, unanchored) occurs anywhere also counts — a list of addresses or a Name <address> form matches no pattern whole and leaks every address it holds. personal_name and postal_address are not defined over looks_like — their evidence is the column name and, where the name is ambiguous, its values, per the corroboration rule above. credential also reads the sampled values themselves: a column any of whose sampled values contains a recognised secret format — a modular-crypt or pbkdf2_ password hash ($2a$, $2b$, $2y$, $argon2, $scrypt$, $5$, $6$), a key with a fixed issuer prefix and body (sk_live_, sk_test_, rk_live_, whsec_, ghp_, gho_, github_pat_, glpat-, xoxb-, xoxp-, AKIA, AIza), a PEM private-key header, or a JWT anywhere in the value (the §4.1.1 jwt check applied to one token, so no 95% share is needed) — carries sensitivity: credential whatever its name. The formats publish nothing to looks_like; like national_id's SSN check they corroborate this axis only.

4.4.4 Thresholds

The 95% threshold §4.1.3 defines for looks_like is a precision instrument and does NOT apply here. Each detector states its own, and they need not agree with one another. A producer SHOULD document the thresholds it uses.

The value-shape route (§4.4.3) requires at least 50% of sampled values for each of contact, financial_account, online_identifier, geolocation and credential, over the denominator §4.1.3 uses — a value assigned no pattern counts toward it. The 95% looks_like verdict does not gate it: a column whose sample is 90% email addresses publishes 90% of its addresses, and this field exists to gate their redaction. 50% is the floor national_id's weak tier and looks_like_candidate (§4.1.3) already use, so a column publishing looks_like_candidate: email carries contact unless an earlier category claims it.

date_of_birth, health, demographic, online_identifier, geolocation and employment use no corroboration threshold on their names: every token in each vocabulary means what the category names — employment's once its disqualifying heads (§4.4.1) are applied — so there is nothing for a sample to corroborate — the same economy §4.4.3 already states for personal_name's strong tier.

credential's weak tier (key, token, secret, hash) is corroborated when at least 50% of sampled values are assigned jwt or hex, counted together; hex corroborates only a weak name, since an unconditional hex would catch a checksum column.

credential's value route needs one sampled value. Each recognised format is a fixed prefix with a fixed or minimum body length, or a JWT whose header must decode to a JOSE object, and no ordinary value reaches one by accident, and a column where only some rows hold a live key still publishes whichever of them reach its values list, so no share is computed.

national_id's weak tier (a generic column name such as id_number) requires at least 50% of sampled values to match a dashed US SSN shape (area, group and serial ranges the SSA never issues excluded) before the category is reported. This is a single-locale corroborator, stated as a gap rather than left to be rediscovered: a UK National Insurance Number, a dashed Canadian SIN, or any other country's identifier grammar will not corroborate a generic column name, and a per-country table of such grammars is the hand-maintained literal §4.1.1 already declines to carry for postal_code. A column name unambiguous on its own (ssn, passport_number, ...) needs no value agreement at all, the same treatment §4.4.3 already gives personal_name.

financial_account's generic column names (account_number, account_no, pan) are in no vocabulary: such a column is flagged exactly when its value shapes reach the share above.

4.4.5 Conformance

Membership is the schema's job. The validator MUST NOT inspect sensitivity values for plausibility, and an unrecognized value degrades to the warning schema.unknown-sensitivity exactly as an unrecognized looks_like value does (§5.3).

4.5 epoch_unit — instant vs quantity

An integer column can be a quantity or an instant: a BIGINT of Unix epoch seconds or milliseconds looks the same as an ordinary large number, and range/percentiles publish it as one without saying so. inferred.epoch_unit names the storage form when the evidence supports it, so a consumer reading min: 1704067200 knows to read it as 2024-01-01T00:00:00Z.

epoch_unit is a third independent axis alongside looks_like and sensitivity (§4.4): a column may carry any combination the matrix (§2.2.3) permits, and none is a fallback for another.

4.5.1 Windows

UnitWindowCalendar span
seconds1e9 .. 2e9 inclusive2001-09-09T01:46:40Z .. 2033-05-18T03:33:20Z
milliseconds1e12 .. 2e12 inclusivethe identical 32-year span, scaled by 1000

The test is a plausibility check over a 32-year calendar window, not a proof: an ordinary large integer can fall inside it by chance, and this specification does not claim otherwise. Microsecond (1e15..2e15) and nanosecond (1e18..2e18) epochs are a stated gap — out of scope for v1, in the register §4.1.1 uses for postal_code's all-digit gap, not an oversight.

4.5.2 Two evidence rules under one field

Each states its own threshold, the shape §4.4.4 already permits for the sensitivity detectors — the two need not agree with one another, and a column reached by one is never reached by the other:

  • Bounds rule, for numeric columns: range.min and range.max are both integral and fall inside the same window. No sample, no threshold — the test is over the two required bounds and it is unanimous or it is nothing. Requiring BOTH bounds inside the window is what excludes most ordinary large integers without further evidence: a count, an amount or a byte size almost always has a minimum orders of magnitude below its maximum, while an epoch column's does not. An id sequence seeded near 1e9 is the one shape whose minimum is large by construction, and it is a named, accepted false positive rather than an excluded case.
  • Per-value rule, for the three classifications §4.1.5 samples (categorical, text, foreign_key_candidate): at least 95% of sampled values stringify to an integer inside the same window, reusing looks_like's own match threshold (§4.1.3) rather than a second number nothing argues for. No sample is drawn where the file's disabled names looks_like (§2.2.21), so this rule does not run there.

The two rules never compete for the same column: numeric is never sampled (§4.1.5 names the other three classifications only, and no range is required on any of them - §2.2.3's matrix marks it -), so a column reached by the bounds rule has no per-value verdict to disagree with, and vice versa.

epoch_unit MUST NOT be derived any other way. It is not a looks_like value: §4.1.3's per-value assignment is normative there, and a bounds test over two aggregates has no per-sampled-value verdict to make — subsuming numeric_string on every epoch-shaped digit run would also need the kind of exclusion clause §4.1's patterns are defined to avoid needing. It is not a note inside range either: range is a bounds object shared with temporal, unreachable for the sampled epoch columns (categorical, text, foreign_key_candidate), none of which emit range at all, and it disappears under redacted: drop — exactly the column where the marker is worth the most (§4.5.3).

4.5.3 Interaction with redaction

epoch_unit is not a cell value (§2.2.9 scopes redaction to cell values only), and detection runs over values that are never persisted, so it is unaffected by redaction: a redacted: drop column emits no range at all yet still carries inferred.epoch_unit. This mirrors a redacted looks_like-bearing column still reporting its shape.

epoch_unit is not a targetable redact vocabulary. sensitivity and looks_like are the two closed vocabularies a redact rule may target (§2.2.9); an epoch unit is not personal data, and this specification defines no third.

4.5.4 Conformance

Membership is the schema's job, the same as looks_like and sensitivity: the validator MUST NOT inspect epoch_unit values for plausibility, and an unrecognized value degrades to a warning exactly as an unrecognized looks_like value does (§5.3).

5. Compatibility

5.1 Versioning

Two independent semvers:

  • format_version (1, 2, …) — the format spec's MAJOR number, embedded in every artifact YAML.
  • dbprint_version (full semver) — the implementation version, embedded in manifest.yaml.

The format MAJOR bumps independently of any implementation version. A dbprint 0.5.0 can emit format_version: 1, and a dbprint 3.0.0 can still emit format_version: 1.

5.2 Format semver rules

  • MAJOR (1 → 2): breaking. Required field removed, semantics of an existing field changed. Consumers MUST branch on MAJOR.
  • MINOR (1.0 → 1.1): backwards-compatible additions. New optional fields, new change-kinds in diff, new column classifications, new looks_like values. Consumers MUST tolerate unknown fields (forward-compat).
  • PATCH (1.1.0 → 1.1.1): clarifications, typo fixes, no field changes.

Artifacts carry only MAJOR (format_version: 1); the full MAJOR.MINOR.PATCH triple identifies a spec revision.

5.3 Forward compatibility

Consumers MUST tolerate:

  • Unknown top-level keys in any YAML artifact.
  • Unknown kind values in diff.yaml changes.
  • Unknown classification values in statistics.yaml.
  • Unknown looks_like values in inferred.
  • Unknown sensitivity values in inferred.
  • New artifact files added to per-table directories.

5.4 Stability commitment

Within format_version: 1, MINOR bumps are unrestricted.

6. Conformance

6.1 Severity model

The conformance suite emits two severity levels:

LevelMeaningEffect on result
errorThe directory does NOT conform. Producer is buggy or artifact is corrupt.At least one error → DOES NOT CONFORM
warningDirectory conforms but has anomalies — typically forward-compat values (unknown classifications, unknown change kinds) or recoverable issues.No errors, any warnings → CONFORMS WITH WARNINGS

A directory CONFORMS iff validate_print() returns zero error-severity issues.

6.2 Issue document shape

Each issue emitted by the conformance suite has the structure:

- code: <stable_identifier>          # e.g., "manifest.missing-artifact"
  severity: error | warning
  path: <relative_path>              # from print root, e.g., "arboretum/seedbank/accession/statistics.yaml"
  detail: <human_readable>           # explanation including specific values
  spec_ref: <SPEC_section>           # e.g., "§2.5"

Consumers MUST tolerate unknown code values (forward-compat for MINOR catalog additions).

6.3 Error catalog

Grouped by concern. E = error, W = warning.

Directory layout (§1)

CodeSevTrigger
layout.missing-manifestEConnection root missing manifest.yaml
layout.missing-reading-guideEConnection root missing reading.md (§1.2)
layout.missing-diffEConnection root missing diff.yaml though manifest.yaml records a table (§1.2)
layout.invalid-path-segmentEDirectory segment fails the §1.5.1 allowlist regex (artifact file names are not segments)
layout.unknown-fileWFile inside per-table dir not in the canonical artifact list (extensibility tolerance)
layout.unexpected-directory-levelEA producer-written artifact (ddl.sql, statistics.yaml, relationships.yaml) in a directory no manifest.tables[*].path declares

YAML / schema validity (§§2.2–2.6)

CodeSevTrigger
schema.invalid-yamlEFile isn't valid YAML
schema.missing-required-fieldERequired field absent per JSON Schema
schema.type-mismatchEField has wrong scalar type
schema.invalid-percentile-keyEPercentile key doesn't match ^p\d{2}$
schema.unknown-classificationWclassification value not in the enum (forward-compat)
schema.unknown-change-kindWkind value in diff not in the enum (forward-compat)
schema.unknown-looks-likeWinferred.looks_like value not in the enum (forward-compat)
schema.unknown-sensitivityWinferred.sensitivity value not in the enum (forward-compat)
schema.unknown-epoch-unitWinferred.epoch_unit value not in the enum (forward-compat)

Format version (§5)

CodeSevTrigger
version.missing-format-versionERequired artifact missing format_version field
version.invalid-format-versionEformat_version isn't a positive integer
version.unknown-format-versionEformat_version is a MAJOR the v1 validator doesn't understand (e.g., 2) — the validator's job is to validate v1; cannot do so for unknown majors

Manifest cross-checks (§2.5)

CodeSevTrigger
manifest.missing-artifactEManifest entry references file that doesn't exist on disk
manifest.orphaned-artifactWFile on disk not listed in any manifest entry (could be in-progress write; not strictly a violation). The only finding for a user-authored file in a directory no manifest entry declares
manifest.table-fqn-mismatchEManifest's table FQN doesn't match the table field in the referenced statistics.yaml / relationships.yaml
manifest.path-fqn-mismatchEA manifest key's segments differ from its entry's path segments (§1.3)
manifest.path-outside-printEA table entry's path is absolute or has a .. segment, or its directory, a file its artifacts declares, or a connection-root file resolves outside the connection root through a symbolic link (§2.5); no consumer follows it
manifest.selectors-mismatch-diffEselectors disagrees with diff.yaml's target.selectors when both are present (§2.5, §2.6.3)
manifest.columns-count-mismatchEA table entry's columns count disagrees with the size of its statistics.yaml columns map (§2.5)
manifest.profile-disabled-mismatchEA table's effective profile switches a measurement off and its statistics.yaml's disabled does not name it, or the reverse (§2.5, §2.2.21). Checked only where the manifest carries profile

Statistics invariants (§2.2)

CodeSevTrigger
stats.missing-required-field-for-classificationEColumn missing a field marked R in the §2.2.3 matrix for its classification
stats.forbidden-field-for-classificationEColumn has a field marked — in the §2.2.3 matrix
stats.cardinality-exceeds-row-countEcardinality > row_count (invariant violation)
stats.null-count-exceeds-row-countEnull_count > row_count (invariant violation)
stats.nullable-contradicts-null-countEnullable: false alongside a nonzero null_count — nullable is a DDL fact (§2.2.2), never a data measurement, so it cannot disagree with a NULL the column's own scan found
stats.degenerate-count-exceeds-row-countEzero_count, negative_count, empty_count, quantized_count or non_finite_count exceeds the non-null scanned count (invariant violation)
stats.null-rate-mismatchEnull_rate disagrees with null_count / rows_scanned, rounded per §2.2.6
stats.cardinality-ratio-mismatchEcardinality_ratio disagrees with cardinality / rows_scanned, rounded per §2.2.6
stats.values-sum-mismatchWAn exhaustive values list (values_coverage of 1.0) whose counts do not sum to rows_scanned - null_count, OR a truncated list whose counts exceed it. WARNING because phase A and phase B are measured in separate statements against a table taking writes — on a live database the two counts can disagree by design, not by producer defect
stats.values-coverage-mismatchWvalues_coverage disagrees with the listed counts over rows_scanned - null_count. WARNING for the same cross-phase reason as stats.values-sum-mismatch: the two ratios share the same listed/non_null pair, and a drift small enough to still disagree at six decimals is the same drift, not a distinct producer defect
stats.values-list-short-of-cardinalityEAn exhaustive values list (values_coverage of 1.0, values_coverage_method absent or measured) carries fewer entries than cardinality, cardinality_method: exact (§2.2.4). ERROR because values_coverage and cardinality are both required exact fields whenever this fires — a live table taking writes between the two phases can widen cardinality past a list that was already complete when it was read, which is the phase A / phase B disagreement stats.values-sum-mismatch's neighbors tolerate at warning; this code's own severity is unchanged. values_coverage_method: bounded reports stats.values-list-short-of-cardinality-bounded instead — see below
stats.values-list-short-of-cardinality-boundedWThe same disagreement as stats.values-list-short-of-cardinality, where values_coverage_method: bounded (§2.2.4) already discloses that values_coverage is a clamp, not a measurement read at the same instant as cardinality. WARNING because the producer has already named the cause this error exists to catch
stats.values-list-exceeds-cardinalityWAn exhaustive values list carries more entries than cardinality, cardinality_method: exact. WARNING for the same cross-phase reason as stats.values-sum-mismatch
stats.values-not-orderedEvalues entries are not ordered by count DESC with a tie-break on the published text of value (§2.2.4)
stats.spelling-of-target-unlistedEA values entry's spelling_of names a value the same list does not carry as a canonical member — an entry with no spelling_of of its own (§2.2.4)
stats.spelling-of-key-mismatchEA values entry's value and its spelling_of do not fold to one key — trimmed and case-folded, the key normalized_cardinality is measured with — so the group was formed under a key §2.2.4 forbids
stats.distribution-mismatchWWhen verifiable from an exhaustive values list, distribution value doesn't match §2.2.5 rules. WARNING because heuristic-prone for truncated lists and exact-match boundaries.
stats.distribution-contradicts-frequenciesEOn a numeric/temporal column with cardinality_method: exact, distribution disagrees with the verdict recomputed from frequencies (§2.2.5). ERROR because the two come from the same fetch and are exact arithmetic, not a heuristic over a possibly-truncated list
stats.scope-not-a-mappingEscope is present but is not a mapping (§2.2.8)
stats.scope-missing-rows-scannedEscope is present without an integer rows_scanned (§2.2.8)
stats.scope-rows-scanned-exceeds-row-countEscope.rows_scanned > row_count while row_count_method is exact — a subset cannot be larger than the set it was drawn from. Permitted under approximate, which is the estimate-undershot case §2.2.8 sanctions
stats.scope-sample-out-of-rangeEscope.sample outside the interval (0, 1]
stats.scope-sample-and-filterEscope carries both sample and filter — a table is narrowed one way or the other (§2.2.8)
stats.scope-asserts-nothingWscope covers the whole table and records neither sample nor filter; omit it
stats.excess-precisionEA numeric statistic is not the value §2.2.6's rounding rule emits for it: more than six decimals, or — for a measured field, where six decimals would round a nonzero value to zero — more than six significant figures (§2.2.6)
stats.redacted-without-markerEA values entry carries no value but the column declares no redacted primitive (§2.2.9)
stats.redacted-literal-publishedEA column or part carrying a redacted marker publishes a literal its primitive withholds: under drop, a values entry carrying value; under mask, a values entry's value, range.min, range.max or percentiles entry other than [redacted]; under hash, one that is not 16 lowercase hexadecimal characters; under any marker, a spelling_of (§2.2.9, §2.2.4)
stats.unrepresentable-names-unemitted-fieldEunrepresentable names a field (min, max, or a percentiles key) the column did not emit (§2.2.4)
stats.unmeasured-names-emitted-fieldEunmeasured names a field the same column also emits — a measurement and its own absence cannot both be claimed (§2.2.4)
stats.unmeasured-names-unrequired-fieldEunmeasured names a field the §2.2.3 matrix does not mark R for this column's classification; that absence is structural and needs no marker (§2.2.4). inferred.looks_like and inferred.epoch_unit on a classification §4.1.5 samples are the exception
stats.disabled-names-emitted-fieldEdisabled names a measurement whose block, or whose field on any column or part, the same file emits (§2.2.21)
stats.disabled-and-unmeasuredEA block or field the file's or a column's unmeasured names belongs to a measurement the file's disabled names (§2.2.21)
stats.unmeasured-names-emitted-blockEthe file's unmeasured list names a table-level block the same file also emits (§2.2.1)
stats.unrepresentable-emptyEunrepresentable is present as an empty list; the key MUST be omitted instead (§2.2.4)
stats.span-days-mismatchErange.span_days disagrees with day_count(range.min, range.max) (§2.2.4). Skipped when either bound is redacted (mask/hash), named in unrepresentable, or not a parseable ISO date/instant
stats.percentiles-not-orderedEpercentiles do not ascend with their keys (§2.2.4). Compares parsed values, not key spelling; a temporal value that does not parse as an instant (unrepresentable) compares by its leading year, and two values of one year that cannot both be parsed are not compared
stats.percentile-outside-rangeEA percentile lies outside [range.min, range.max] (§2.2.4). range and percentiles are read by one statement per column, so this does not carry the cross-phase tolerance stats.values-sum-mismatch does. A temporal value that does not parse as an instant compares by its leading year, so a year-10000 percentile against a year-9999 maximum is reported. Skipped when the column carries any redacted marker
stats.non-finite-statisticErange.min, range.max, a percentiles entry, mean or sum on a numeric column is NaN or infinite; every numeric statistic reads the finite values alone (§2.2.4)
stats.mean-outside-rangeEmean lies outside [range.min, range.max] (§2.2.4). An average outside its own bounds is impossible over any nonempty column, so it reports a producer that lost the magnitude of one of the three
stats.length-order-violatedElength.min, length.avg and length.max disagree with min <= avg <= max (§2.2.4)
stats.normalized-cardinality-exceeds-cardinalityWnormalized_cardinality exceeds cardinality; folding case and trimming whitespace cannot increase distinctness (§2.2.4). Not checked when cardinality_method is approximate — the comparison is approximate on both sides there. WARNING because the two counts are measured in separate statements against a table taking writes — on a live database the later count can legitimately be the larger, not by producer defect
stats.max-age-days-mismatchEfreshness.max_age_days disagrees with max(0, day_count(range.max, profiled_at)) (§2.2.4). Skipped when the column carries any redacted marker, max is named in unrepresentable, or range.max is not a parseable instant
stats.uncoarsened-redacted-day-countEA temporal column carrying a redacted marker emits freshness.max_age_days or range.span_days not floored to a multiple of 90 (§2.2.9)
stats.candidate-key-mismatchEinferred.candidate_key disagrees with whether cardinality_ratio clears the SPEC 4.2 threshold — present where it should be absent, or the reverse. Independent of classification
stats.candidate-key-exception-mismatchEinferred.candidate_key_exception disagrees with the value recomputed from cardinality, cardinality_ratio, cardinality_method and null_count (§4.2)
stats.looks-like-candidate-with-verdictEinferred.looks_like_candidate is present alongside inferred.looks_like; the near-miss and the verdict are mutually exclusive (§4.1.3)
stats.looks-like-candidate-at-verdict-thresholdEinferred.looks_like_candidate_share is at or above the 95% verdict threshold — a share that high would have been inferred.looks_like instead (§4.1.3)
stats.population-marker-mismatchEA column's rows_scanned disagrees with what the file's scope requires — absent or wrong when scoped, present when not (§2.2.8)
stats.null-patterns-absent-with-nullsEnull_patterns is omitted although some column reports a non-zero null_count and neither the file's own unmeasured nor its disabled list names it, or present although no column reports one (§2.2.10)
stats.null-patterns-unknown-columnEA null_patterns entry names a column absent from the file's columns map (§2.2.10)
stats.null-patterns-sum-exceeds-rows-scannedEThe null_patterns counts sum to more rows than were scanned (§2.2.10), coverage_method measured, or absent at coverage: 1.0. coverage_method: bounded, and a truncated list (coverage below 1.0, coverage_method absent), report stats.null-patterns-sum-exceeds-rows-scanned-bounded instead — see below
stats.null-patterns-sum-exceeds-rows-scanned-boundedWThe same disagreement as stats.null-patterns-sum-exceeds-rows-scanned, where coverage_method: bounded (§2.2.10) already discloses that the pattern counts and rows_scanned were not read at the same instant, or the list is truncated (coverage below 1.0, coverage_method absent) and so can overrun only through writes between the two reads. WARNING because the cause this error exists to catch is already named or the only one possible
stats.null-patterns-coverage-mismatchEnull_patterns.coverage disagrees with the listed counts over rows_scanned (§2.2.10)
stats.null-patterns-not-orderedEnull_patterns.patterns is not ordered by count descending, ties by ascending columns (§2.2.10)
stats.null-patterns-duplicate-combinationEThe same combination of columns appears in more than one null_patterns entry (§2.2.10)
stats.null-patterns-reconciliation-mismatchEThe null_patterns counts naming a column exceed its null_count, or fall short of it at coverage: 1.0 (§2.2.10), coverage_method measured, or absent at coverage: 1.0. coverage_method: bounded, and a truncated list (coverage below 1.0, coverage_method absent), report stats.null-patterns-reconciliation-mismatch-bounded instead — see below
stats.null-patterns-reconciliation-mismatch-boundedWThe same disagreement as stats.null-patterns-reconciliation-mismatch, where coverage_method: bounded (§2.2.10) already discloses the cause, or the list is truncated (coverage below 1.0, coverage_method absent) and so can overcount a column only through writes between the reads. WARNING for the same reason
stats.physical-name-matches-keyWphysical_name is present and equals the column's own map key; the field asserts nothing and MUST be omitted (§2.2.4)
stats.physical-layout-unknown-columnEphysical_layout.keys names a column absent from the file's columns map (§2.2.11)
stats.merging-unknown-columnEmerging.key names a column absent from the file's columns map (§2.2.19)
stats.physical-layout-key-not-declaredEA column carries physical_layout_key: true but is not named in physical_layout.keys (§2.2.11)
stats.physical-layout-key-missing-markerEphysical_layout.keys names a column that does not carry physical_layout_key: true (§2.2.11)
stats.grain-unknown-columnEA grain.keys entry names a column absent from the file's columns map (§2.2.12)
stats.grain-duplicate-keyEThe same column combination appears in more than one grain.keys entry (§2.2.12)
stats.grain-measured-under-scopeEgrain.keys carries a detection: measured entry on a file that also carries scope — uniqueness over a sample is not uniqueness (§2.2.12)
stats.grain-measured-on-empty-tableEgrain.keys carries a detection: measured entry on a table with row_count: 0 — every combination is trivially unique there (§2.2.12)
stats.dependencies-unknown-columnEA dependencies entry names a determinant or dependent absent from the file's columns map (§2.2.13)
stats.dependencies-self-referentialEA dependencies entry's determinant and dependent name the same column (§2.2.13)
stats.dependencies-strength-out-of-rangeEA dependencies entry's strength is not in (0, 1] (§2.2.13)
stats.dependencies-direction-impossibleEA dependencies entry's determinant has lower cardinality than its dependent — a function's image cannot exceed its domain, so that direction cannot hold (§2.2.13)
stats.dependencies-measured-under-scopeEdependencies is non-empty on a file that also carries scope — a dependency measured over a sample is not a dependency (§2.2.13)
stats.dependencies-measured-on-empty-tableEdependencies is non-empty on a table with row_count: 0 — every combination is trivially functional there (§2.2.13)
stats.timeline-unknown-columnEtimeline.column names a column absent from the file's columns map (§2.2.16)
stats.timeline-anchor-not-temporalEtimeline.column names a column whose classification is not temporal (§2.2.16)
stats.timeline-anchor-redactedEtimeline.column names a column carrying a redacted marker — the anchor rule MUST NOT choose a redacted column (§2.2.16)
stats.timeline-under-scopeEtimeline is present on a file that also carries scope — a bucketed count over a sample is not a timeline (§2.2.16)
stats.timeline-on-empty-tableEtimeline is present on a table with row_count: 0 — there is nothing to bucket (§2.2.16)
stats.timeline-buckets-unorderedEtimeline.buckets is not ascending by start (§2.2.16)
stats.timeline-coverage-mismatchEtimeline.coverage disagrees with the listed bucket counts over rows_scanned, recomputed through the same bounded share the producer writes it with (§2.2.16)
stats.timeline-buckets-exceed-rows-scannedWThe listed bucket counts sum above rows_scanned (§2.2.16). WARNING because the buckets and the row count are measured in separate statements against a table taking writes — on a live database the two can disagree by design, not by producer defect
stats.populated-without-timelineEA column carries populated on a file with no timeline block — there is nowhere to read the anchor its instants are dated against (§2.2.4, §2.2.16)
stats.populated-out-of-anchor-rangeWpopulated.from or populated.to falls outside the anchor column's own measured range (§2.2.4). WARNING because the window and the anchor's bounds are measured in separate statements against a table taking writes — an anchor advancing between them carries to past a bound recorded earlier, not by producer defect
stats.geometry-count-mismatchEA geometry list's counts do not sum to the column's non-null count, or empty_count/invalid_count exceeds it (§2.2.4)
stats.extent-invertedEAn extent carries min_x > max_x or min_y > max_y (§2.2.4)
stats.vector-bounds-invertedEA dimension or norm carries min > max (§2.2.4)
stats.types-sum-mismatchWA types map does not sum to its population: the scanned rows less null_count on a column, occurrences - null_count on a part (§2.2.4). WARNING because types is counted in its own statement, apart from the ones reading rows_scanned and null_count, against a table taking writes — the cross-statement disagreement stats.values-sum-mismatch tolerates for the same reason
stats.part-path-not-canonicalEA parts key does not parse under the §2.2.18 grammar, or is not in its one canonical spelling
stats.part-parent-unlistedEA part's parent part is not listed in the same parts map (§2.2.18)
stats.parts-found-below-listedEparts_found is below the number of parts entries (§2.2.18)
stats.sketch-unknown-methodEsketch.method is not a value this MAJOR defines (§2.2.14)
stats.sketch-invalid-encodingEsketch.values is not valid base64 of a length that is a multiple of 8 bytes (§2.2.14)
stats.sketch-oversizedEA decoded sketch.values carries more entries than method's own k (§2.2.14)
stats.sketch-not-ascendingEA decoded sketch.values is not sorted ascending (§2.2.14)
stats.sketch-on-redacted-columnEA column declaring a redacted primitive carries a sketch — the digests enumerate the cell values that primitive withheld (§2.2.14)
stats.measurement-under-catalog-onlyEA column carries a field beyond sql_type, nullable, classification, physical_name, collation, physical_layout_key, derived_from on a file that also carries catalog_only — a measurement published where none was queried (§2.2.15)
stats.depends-on-on-tableEdepends_on is present but type is table — the field names what a view/matview reads and MUST NOT appear on a plain table (§2.2.17)

Privacy (§4.4)

CodeSevTrigger
privacy.unredacted-sensitiveWColumn carries inferred.sensitivity, or a part sits inside a column or part that does, and publishes at least one of values, range, percentiles, extent, with no redacted primitive covering it (§4.4.2)
privacy.redacted-value-comparedEdiff.yaml carries a statistic_changed event for values, range, percentiles or any field §2.2.9 withholds from a marked column, on a column whose statistics.yaml declares a redacted marker (§2.6.6). ERROR, not WARNING: the marker is a producer's own declaration, so no inference is being second-guessed

Relationships invariants (§2.3)

CodeSevTrigger
relationships.column-array-length-mismatchEcolumn and target_column (in refers_to) or column and referencer_column (in referenced_by) have different array lengths
relationships.broken-reciprocityEA referenced_by entry points to a source table that's in the manifest BUT lacks the matching refers_to entry
relationships.unmirrored-refers-toEA refers_to entry's target_table is in the manifest and declares a relationships.yaml, but that file's referenced_by has no entry naming this table as referencer_table, this column as referencer_column and this target_column as column (§2.3.6). The reverse of relationships.broken-reciprocity; a target outside the manifest, or one declaring no relationships.yaml, is not checked
relationships.mirror-mismatchEA refers_to entry and its referenced_by mirror address the same edge but differ on a field both entry schemas define other than the address (detection, on_delete, on_update, constraint_name, observed), including a field present on one side only (§2.3.3, §2.3.10). Where several entries share one address, both sides must carry the same set
relationships.inferred-edge-under-disabledErefers_to carries a detection: inferred entry while the same object's statistics.yaml names relationships.infer in disabled (§2.2.21)
relationships.measured-edge-under-disabledEA detection: measured entry has an endpoint whose statistics.yaml names sketches in disabled (§2.2.21)
relationships.ineligible-target-is-referencedEeligible_target: false but referenced_by carries an inferred entry — naming inference cannot resolve an edge to an ineligible object (§2.3.8). A declared entry there is unaffected; a composite key inference never reaches can still be a real FK target
relationships.path-on-composite-endpointEpath (or target_path) is present but its partner array (column or target_column) names more than one column (§2.3.9) — a path endpoint is representable only on a single-column endpoint
relationships.observed-fanout-mismatchEobserved.fanout_avg disagrees with the referencing column's own (row_count - null_count) / cardinality, numerator floored at cardinality, rounded per §2.2.6 (§2.3.10)
relationships.observed-coverage-mismatchEobserved.target_coverage disagrees with the two endpoints' own cardinality / cardinality, rounded per §2.2.6 (§2.3.10)
relationships.observed-coherent-mismatchEobserved.coherent disagrees with whether the referencing column's cardinality exceeds the referenced column's (§2.3.10)
relationships.observed-containment-mismatchEobserved.containment disagrees with the bottom-k intersection estimate recomputed from the two endpoints' own sketch (§2.3.10, §2.2.14)
relationships.observed-containment-forbiddenEobserved.containment is published where the two endpoints' sketches carry no evidence to support it — the answerable subset between them is empty (§2.3.10)
relationships.observed-answerable-count-mismatchEobserved.answerable_count disagrees with the count of the referencing column's own retained hashes below the shared threshold (§2.3.10, §2.2.14), including when the answerable subset is empty and no count should be published at all

Diff invariants (§2.6)

CodeSevTrigger
diff.summary-count-mismatchWSummary count doesn't match actual events count for that kind (informational redundancy; producer bug but not format break)
diff.summary-total-mismatchWtables_modified + unchanged_tables + unevaluated_tables + tables_added doesn't equal target.tables_scanned (§2.6.4). WARNING on the same grounds as diff.summary-count-mismatch: the events remain readable, and every operand is redundant with them
diff.relationship-modified-no-changeErelationship_modified event without on_delete OR on_update sub-objects (§2.6.6 rule)
diff.comment-target-column-mismatchEcomment_changed with target=column missing column field, OR target=table with column field present
diff.statistic-changed-delta-on-non-numericEstatistic_changed with delta or delta_pct on stats where they MUST be omitted (distribution, classification, values, cardinality_method)
diff.statistic-changed-delta-pct-sign-mismatchEstatistic_changed where delta and delta_pct disagree in sign
diff.row-count-changed-delta-mismatchEtable_row_count_changed where delta doesn't equal after - before
diff.grain-changed-no-changeEgrain_changed event's before and after are identical (§2.6.6)
diff.physical-layout-changed-no-changeEphysical_layout_changed event's before and after are identical (§2.6.6)
diff.merging-changed-no-changeEmerging_changed event's before and after are identical (§2.6.6)
diff.depends-on-changed-no-changeEdepends_on_changed event's before and after are identical (§2.6.6)
diff.table-type-changed-no-changeEtable_type_changed event's before and after are identical (§2.6.6)
diff.external-changed-no-changeEexternal_changed event's before and after are identical (§2.6.6)
diff.column-physical-name-changed-no-changeEcolumn_physical_name_changed event's before and after are identical (§2.6.6)
diff.column-collation-changed-no-changeEcolumn_collation_changed event's before and after are identical (§2.6.6)
diff.statistic-changed-not-a-measurementEA statistic_changed event's stat names a field that is not a measured statistic — a catalog fact with its own kind (physical_name, collation), a restatement (physical_layout_key), or a field drift never compares (§2.6.6)

DDL (§2.1)

CodeSevTrigger
ddl.empty-fileEddl.sql exists but is empty
ddl.missing-trailing-newlineWFile doesn't end with newline (POSIX convention; trivially auto-fixable)

Annotations (§2.7)

CodeSevTrigger
annotations.unknown-columnWA key in statistics.annotations.yaml's columns map names a column not present in the table's statistics.yaml (stale key; not fatal)
annotations.grain-unknown-columnWA grain.keys[] entry in statistics.annotations.yaml names a column not present in the table's statistics.yaml columns map (§2.7.1; stale key, not fatal)
annotations.claim-contradicts-statisticWA claims predicate (§2.7.1) evaluates against the column's own statistics.yaml and fails. WARNING — the axis is advisory (§2.4); a contradicted claim is never a gating defect
annotations.claim-unassertableWA claims predicate cannot be evaluated: the stat is not in the assertion DSL's vocabulary, the predicate is malformed, the stat is not emitted for the column's classification, or the column is redacted
annotations.unknown-edgeWA verdict in relationships.annotations.yaml addresses an edge absent from the table's own relationships.yaml refers_to (§2.7.2; stale, not fatal)
annotations.verdict-on-declared-edgeWA verdict addresses an edge whose matching refers_to entry carries detection: declared (§2.7.2) — an annotation may correct an inference, never contradict a measurement (§2.4)
annotations.unknown-sourceWA derived_from entry in statistics.annotations.yaml names a table absent from manifest.yaml's tables, or a column that table's statistics.yaml does not have (§2.7.1; stale, not fatal)
annotations.unknown-valueWA values entry in statistics.annotations.yaml names a value the column's own exhaustive values list does not have (§2.7.1; stale, not fatal — a truncated list is not checked)
annotations.value-note-unassertableWA values entry addresses a column carrying a redacted marker (§2.2.9) — there is no literal to check the note against

6.4 Catalog totals

  • 165 codes across 10 groups
  • 132 error codes (gate conformance)
  • 33 warning codes (recoverable anomalies)

The catalog MAY grow in MINOR releases (additive only). Existing codes' semantics MUST NOT change.

6.5 Run-all-then-report

The conformance suite MUST emit ALL issues found, not stop at first. Matches the run-all-then-report failure-isolation principle from multi-table generate runs.

6.6 Issue ordering

Issues are ordered for diff stability: first by path (lexicographic), then by code (lexicographic). Deterministic across runs.

6.7 Conformance test suite

The reference conformance suite SHALL be a pytest module at tests/conformance/. Any producer SHOULD run this suite against its output as part of CI.

API:

from dbprint.conformance import validate_print, Issue


issues: list[Issue] = validate_print("/path/to/prints/connection_name/")
errors = [i for i in issues if i.severity == "error"]
warnings = [i for i in issues if i.severity == "warning"]


conforms = len(errors) == 0

6.8 Forward compatibility

  • Consumers MUST tolerate unknown code values
  • Consumers MUST tolerate unknown severity values (treat unknown as warning by default)
  • This validator cannot validate an artifact whose format_version it does not know — it emits a version.unknown-format-version error

6.9 Validator edge cases

CaseResolution
Empty diff.yaml (no changes, changes: [])CONFORMS, not an issue
Missing description.mdCONFORMS — file is optional (§2.4)
Missing relationships.yaml on plain viewCONFORMS per §1.4 (MAY be absent for plain views)
Unknown change_kind in diff.yamlWARNING (schema.unknown-change-kind); forward-compat
Mixed format_version values within a printEach file validated separately; a file whose version this validator does not know emits version.unknown-format-version, the rest validate normally

7. Reading an absence

A consumer meets an absence far more often than it meets a value, and the rules governing what one means are stated where each field is defined — across §2.2, §2.3, §3 and §4, all written from the producer's side. This section collects them from the reader's.

Normative where it restates a MUST, informative where it collects. No rule originates here; every row cites the section that governs it, and that section governs on any disagreement.

7.1 Partial reads and failed reads

A statistic measured over part of the table is present, not missing. Under a scope block (§2.2.8) every count in the file is a count over rows_scanned rather than over row_count, so a number an order of magnitude below what a reader expects is a narrower population, not a partial artifact. The file says which: scope.rows_scanned at the head, echoed on every column. A consumer MUST read the two together before concluding anything about a small number.

A producer failure is representable, and only where a producer said so. unmeasured (§2.2.4 per column, §2.2.1 per file) names the fields and blocks a run tried and failed to measure, and the manifest's failed_tables (§2.5) names a whole table it tried and failed to profile, so that cause is distinguishable from every structural cause in the tables below — but only on an artifact that carries the marker. Where it is absent, a measurement that failed and a measurement the rules forbade are still the same bytes, and the cause columns below are what a reader has.

A switched-off measurement is representable too. disabled (§2.2.21) names a measurement the producer's configuration switched off for the object, distinguishable from a failure and from every structural cause below. A field or block that measurement produces reads as not measured, never as absent.

7.2 Absent per-column fields

Every field the §2.2.3 matrix marks anything but R on at least one classification. The five it marks R everywhere — sql_type, nullable, null_count, null_rate, classification — are absent from no conforming column and so from this table.

One cause runs across every row and is deliberately not repeated in them: the column's own unmeasured list (§2.2.4) names it, so the run attempted the measurement and did not get one. That cause is distinguishable from every structural cause below by the marker alone, and it is the only one under which a field the matrix marks R may be absent at all. Where unmeasured is absent or does not name the field, read the causes in its row.

Absent fieldCandidate causesDistinguishable by
cardinality, cardinality_ratio, cardinality_methodThe producer declined to measure a cardinality, which is what makes a column unsupported in a queried file (§3.1) — or the column is spatial, which is never counted (§3.3) — or the file carries catalog_only (§2.2.15), absent from every column regardless of classificationclassification, then the file's own catalog_only marker
valuesForbidden for this classification — json, binary and unsupported only (§2.2.3) — or the column is text reporting looks_like: prose, exempted from the grouped scan (§2.2.3 ‡)classification, then inferred.looks_like
values_coverageForbidden for this classification — json, binary, unsupported, numeric and temporal, the last two despite values itself being required there (§2.2.3) — or the column is text reporting looks_like: prose (§2.2.3 ‡)classification, then inferred.looks_like
values_coverage_methodvalues_coverage is itself absent (§2.2.4) — or the list is truncated, a condition this field does not cover (its sibling null_patterns.coverage_method, §2.2.10, excludes truncation the same way)values/values_coverage; values_coverage itself against 1.0, for the second cause
distributionThe same two causes as values, which it is derived from wherever a value list existsclassification, then inferred.looks_like
frequenciesForbidden outside numeric and temporal (§2.2.3)classification
range, range.span_days, percentilesForbidden for this classification (§2.2.3) — or withheld, since redacted: drop emits no literal and a bound is nothing but a literal (§2.2.3 †) — or the numeric column has no finite value (§2.2.4)classification, then redacted
mean, sumForbidden outside numeric (§2.2.3) — or withheld, since the column carries a redacted marker (§2.2.3 ¶, §2.2.9) — or the column has no finite value (§2.2.4)classification, then redacted
zero_count, negative_countzero_count is forbidden outside numeric and vector, negative_count outside numeric (§2.2.3) — or withheld, since the column carries a redacted marker (§2.2.3 ¶)classification, then redacted
non_finite_countNo non-finite value was scanned (§2.2.4) — or the classification is not numeric (§2.2.3) — or withheld, since the column carries a redacted marker (§2.2.3 ¶)classification, then redacted
empty_countForbidden outside text and binary (§2.2.3) — or withheld, since the column carries a redacted marker (§2.2.3 ¶)classification, then redacted
quantized_countForbidden outside numeric and temporal (§2.2.3) — or the column is temporal and its sql_type has no day to truncate to: DATE/DATE32 (already their own truncation), TIME/TIME WITH TIME ZONE, or MySQL's YEAR (§2.2.3 ※) — or withheld, since the column carries a redacted marker (§2.2.3 ¶)classification, then sql_type for the second cause, redacted for the third
lengthForbidden outside text, binary, categorical and foreign_key_candidate (§2.2.3) — or the column is categorical/foreign_key_candidate and either its sql_type carries neither a string value nor bytes or the column is all-null (§2.2.3 ‖) — or withheld, since the column carries a redacted marker (§2.2.3 ¶)classification, then sql_type and null_rate for the second cause, redacted for the third
normalized_cardinalityForbidden outside text, categorical and foreign_key_candidate (§2.2.3) — the column's sql_type carries no string value (§2.2.4) — or it is not a member of the join-key population §2.2.4 restricts this field to (edge, declared single-column unique key, inferred.candidate_key) — or withheld, since the column carries a redacted marker (§2.2.9)classification, then sql_type, then relationships.yaml's refers_to/referenced_by, this table's own declared unique keys, and inferred.candidate_key, then redacted
freshnessForbidden outside temporal (§2.2.3). Never withheld: it survives every redaction primitive, coarsened rather than omitted (§2.2.9)classification
unmeasuredThe run measured every field this column should carry — the ordinary case (§2.2.4)Nothing to distinguish: the marker has exactly one absent meaning
unrepresentableNo emitted bound lies outside years 0001–9999 — or the column emitted no bound the marker could name (§2.2.4) — or withheld, since the column carries a redacted marker (§2.2.9)range and percentiles; absent bounds leave nothing to mark; then redacted
rows_scannedThe file carries no scope, so row_count is the population of every count in it (§2.2.8)the file's own scope block; the two agree or the artifact is inconsistent
physical_nameThe catalog's own spelling is the map key already (§2.2.4)nothing further — the absence is the statement
collationThe column sets no explicit collation, or sets one identical to the connection default (§2.2.4)not distinguishable, and the answer is the same either way: default_collation (§2.5)
physical_layout_keyThe column is not named in this file's physical_layout.keys (§2.2.11)that list; the two surfaces agree or the file contradicts itself
derived_fromNo source column was traced for it: no transformation tool configured, the object absent from its manifest, a definition or column the parser could not resolve, or a column computed from no column (§2.2.4). Never that it has no sourcenot distinguishable, and not intended to be
populatednull_count is 0 (fully populated — range already says this) or the column is all-null — or the file carries scope, an empty table, or no eligible anchor, all of which also leave timeline absent, which populated MUST NOT be published without (§2.2.4, §2.2.16) — or the file's disabled names timeline (§2.2.21)null_count against rows_scanned; timeline's own presence for the second cause; the file's disabled
inferred.looks_likeDetection ran and no pattern reached the threshold (§4.1.3) — or never runs on this classification (§4.1.5) — or the winning pattern was numeric_string and the column's own SQL type is already numeric (§4.1.5) — or the sample draw failed and the column names it unmeasured (§2.2.4) — or the file's disabled names looks_like (§2.2.21)classification, then sql_type for the third cause, the column's unmeasured for the fourth, the file's disabled
inferred.sampled, inferred.matchedlooks_like is absent; the pair describes that verdict alone and is emitted only beside it (§2.2.4, §4.1.3)inferred.looks_like
inferred.looks_like_candidate, inferred.looks_like_candidate_sharelooks_like is present, since the two are mutually exclusive — or no pattern's share reached the 50% floor — or detection never ran on this classification (§4.1.5), or was withheld as numeric_string on a numeric SQL type (§4.1.5) (§2.2.4, §4.1.3) — or the file's disabled names looks_like (§2.2.21)inferred.looks_like, then sql_type, then the file's disabled
inferred.sensitivityNothing was detected — on name evidence alone where the file's disabled names looks_like (§4.4.3). Never that the column is safe to publish (§4.4.2)not distinguishable, and not intended to be
inferred.epoch_unitNeither bounds nor sampled values fell inside one window (§4.5.1) — or no evidence rule reaches this classification (§4.5.2) — or the sample draw failed and the column names it unmeasured (§2.2.4) — or the file's disabled names looks_like, on a classification §4.1.5 samples (§2.2.21)classification, then the column's unmeasured, then the file's disabled
inferred.candidate_keycardinality_ratio does not clear the §4.2 thresholdcardinality_ratio; recomputable, and a validator recomputes it
inferred.candidate_key_exceptioncardinality_ratio is exactly 1.0, or candidate_key is itself absent (§4.2)cardinality_ratio and inferred.candidate_key
inferred.fk_candidateReserved; a v1 producer MUST NOT emit it (§4.3)nothing — it is absent from every v1 artifact
redactedNo redact rule matched this column, or the connection configures none (§2.2.9)redaction_rules_configured in the manifest (§2.5)
geometryForbidden outside spatial (§2.2.3) — or the file carries catalog_only (§2.2.15)classification, then the file's own catalog_only marker
dimensionForbidden outside vector (§2.2.3) — or the column has no non-null value (§2.2.3 ◊) — or the file carries catalog_onlyclassification, then null_count against the scanned rows, then catalog_only
normForbidden outside vector (§2.2.3) — or every non-null value is a zero vector, or there is none (§2.2.3 ◊) — or the file carries catalog_onlyclassification, then zero_count against the non-null count, then catalog_only
parts, parts_foundForbidden outside composite and json, and OPTIONAL on json (§2.2.3) — the producer did not descend into the column: the descent failed, or its engine has no descent for the column's kind, which the file alone cannot tell apart (§2.2.18) — or the file's disabled names parts (§2.2.21) — or the file carries catalog_onlyclassification, then the file's disabled
sizeForbidden outside composite and json (§2.2.3) — or the column is a record, whose member count is fixed (§2.2.18) — or the producer did not descend into it — or the file's disabled names parts (§2.2.21)classification, then sql_type, then the file's disabled
occurrencesEvery column: it is a part's field alone (§2.2.18)—
typesForbidden outside json (§2.2.3) — or the file carries catalog_only (§2.2.15) — or the producer's engine names no per-value typeclassification, then catalog_only, then the manifest's adapter
extentForbidden outside spatial (§2.2.3) — or every non-null value is empty, or there is none (§2.2.3 ◊) — or withheld, since the column carries a redacted marker (§2.2.9) — or the file carries catalog_onlyclassification, then geometry.empty_count against the non-null count, then redacted, then catalog_only
sketchThe column is not a join-key participant on either side of an edge in this table's own relationships.yaml; its type has no canonical encoding (§2.2.14); it carries a redacted marker; its file carries a top-level scope block; or its file's disabled names sketches (§2.2.21)relationships.yaml's refers_to/referenced_by, sql_type, redacted, the file's own scope, and the file's disabled — all five, since any one of them alone can explain the absence

7.3 Absent blocks and files

Shapes above column grain, which the §2.2.3 matrix does not reach.

The file's own unmeasured list (§2.2.1) runs across four of these rows and is not repeated in them: naming physical_layout, merging, null_patterns or dependencies says this run attempted the read and did not get one, and it is the only cause under which those four may be absent where the rules below would otherwise require them. Where the marker is absent or does not name the block, read the causes in its row.

Absent shapeCandidate causesDistinguishable by
catalog_onlyThe producer issued a query for this object, the ordinary case (§2.2.15)nothing further — the absence is the statement
externalThe object's rows are stored by the database the connection reads (§2.2.20)nothing further — the absence is the statement
row_count, row_count_methodThe file carries catalog_only (§2.2.15) — no query was issued to obtain eitherthe file's own catalog_only marker
scopeThe producer read every row. The absence is an assertion, not a gap (§2.2.8)nothing further — the absence is the statement
null_patternsNo column in the file carries a null, or the pattern count failed and the file names it unmeasured (§2.2.10, §2.2.1), or the file's disabled names it (§2.2.21)every column's null_count and the file's own unmeasured and disabled lists; the block MUST be present when any count is non-zero and neither marker names it
null_patterns.coverage_methodThe pattern list was truncated by the producer's own cap, a different condition the field does not cover (§2.2.10)patterns's own length against the cap
physical_layoutThe table declares no clustering or partitioning key, the adapter cannot express the concept for this engine, or the file carries catalog_only (§2.2.15), whose absence here needs no further explanation. Never "not checked" unless the file names it unmeasured (§2.2.11)the file's own catalog_only and unmeasured markers rule in the third and fourth causes; the first two remain not distinguishable from each other
mergingThe table's engine combines no rows, its adapter has no such engine, or the file carries catalog_only (§2.2.15). Never "not checked" unless the file names it unmeasured (§2.2.19)the manifest's adapter, the file's catalog_only and unmeasured markers, and ddl.sql's ENGINE clause
grainA conforming producer always emits the block, keys empty rather than the block omitted (§2.2.12) — its absence is a producer that did notnothing further — a conforming producer never leaves it absent
grain.searchThe measured probe never ran: the file carries scope, the file carries catalog_only (§2.2.15), a count is zero, or a column already carries inferred.candidate_key (§2.2.12)the file's own catalog_only marker for the second cause — row_count does not exist to read under it; otherwise scope, row_count, and every column's inferred.candidate_key distinguish the rest
dependenciesThe file carries catalog_only (§2.2.15), which forbids the block outright, or the probe failed and the file names it unmeasured (§2.2.1). A conforming, queried producer always emits the block otherwisethe file's own catalog_only and unmeasured markers
unmeasured (the file's own)The run measured every block this file should carry — the ordinary case (§2.2.1)Nothing to distinguish: the marker has exactly one absent meaning
disabled (the file's own)Nothing was switched off (§2.2.21)Nothing to distinguish
timelineNo temporal, non-redacted, calendar-typed column exists on the table; the file carries scope; row_count is 0; or the file's disabled names it (§2.2.16, §2.2.21)scope, row_count and disabled rule in three causes directly; the remaining one, no eligible anchor, is then the cause
depends_onThe object is a table, which never carries the field — or the object is a view/matview whose dependency read did not happen: no grant, no such catalog table on this engine version, or the read failed (§2.2.17)type rules in the first cause directly; the second is not distinguishable from the artifact alone
eligible_targetThe producer never ran the declared-keys pre-pass (§2.3.1, §2.3.7)nothing further — and its absence is what makes referenced_by: [] ambiguous
an entry in referenced_byThe referencing table lies outside the print's selectors, so the two-pass resolution never saw it (§2.3.6)the manifest's selectors (§2.5), which names what was left out
a table absent from tablesThe table lies outside the connection's selectors; the target did not list it at the last run; a CLI-narrowed run never reached it (the concession §2.5 makes); or the last run attempted it and could not profile itthe manifest's selectors for the first, failed_tables for the last; the middle two are not distinguishable from the print alone
observed on a refers_to/referenced_by entryThe edge is composite, or either endpoint carries no cardinality this run could measure (§2.3.10)the entry's own column/target_column array length; the endpoint's own cardinality
statistics.yamlA conforming producer emits one for every object, a plain view's carrying catalog_only where no query was issued for it (§2.2.15) — its absence is a producer that did notnothing further — a conforming producer never leaves it absent
relationships.yamlThe object is a plain view, for which the file is optional (§1.4)the manifest's artifacts for that table
description.md, statistics.annotations.yaml, relationships.annotations.yamlNo human wrote one. Producers never author them (§2.4, §2.7)nothing — absence carries no meaning beyond "unwritten"

7.4 Empty is not absent

The format emits some collections empty rather than omitting them, and an empty collection is a measurement with its own causes.

ShapeCandidate causesDistinguishable by
values: []The column has no non-null rows, or the table has none at all (§2.2.7). values_coverage is 1.0 in both cases — a list with nothing to carry covers all of itrow_count, null_rate, and scope.rows_scanned: an empty scanned set (§2.2.7) is a third case and keeps the table's own row_count
referenced_by: []Nothing references the object and something could have; or nothing could, because it supplies no declared-unique column (§2.3.8); or the pre-pass never raneligible_target: true, false, and absent are the three answers (§2.3.7)
refers_to: []The object declares no outgoing foreign key and the naming rule resolved no inferred edge (§2.3.8) — or the producer ran no inference at all — or this object's statistics.yaml names relationships.infer in disabled (§2.2.21)eligible_target's absence says inference never ran (§2.3.7); the object's disabled says it was switched off for this object. Where neither applies, the remaining causes are not distinguishable
grain.keys: []Nothing declared identifies a row, and the measured probe either found nothing, was cut short, or never ran (§2.2.12)grain.search: absent means the search did not run, exhausted: false means it stopped before testing every candidate, exhausted: true means it found nothing
dependencies: []No candidate pair cleared the 95% threshold, or the search never ran at all — the file carries scope, or row_count is 0 (§2.2.13)scope and row_count: both absent/nonzero means the search ran and found nothing; either present means it never ran
timeline.buckets: []The chosen anchor column has no non-null value over the scanned rows — every row that could carry one is null there (§2.2.16)the anchor's own null_rate on timeline.column: 1.0 confirms it, since the anchor rule would not have chosen it otherwise unless every remaining candidate was equally null
depends_on: []The catalog was read and this view/matview reads no other object in the print — a view over a literal or a constant expression, for instance (§2.2.17)nothing further — the catalog was read, and this is the result

Appendix A — Example print

examples/ contains a complete reference print: realistic per-table directories plus the manifest and diff files.

Cross-references

The artifacts below are normatively defined by this spec:

  • examples/ — reference per-table directories
  • spec/v1/*.schema.json — JSON Schemas
  • tests/conformance/ — pytest validation suite