ClickHouseSinkSchema

class ClickHouseSinkSchema(val type: AggregateType, val database: String, val aggregate: AggregateSinkOptions, val flatColumns: List<FlatColumn>, deduplicationWindow: Int = ClickHouseSinkOptions.DEFAULT_DEDUPLICATION_WINDOW)

The tables and views of one aggregate type, in database, named after the type (<ctx>_<agg>, prefixOf):

  • <ctx>_<agg>_state_events, the base table the Sink inserts into: a ReplacingMergeTree partitioned by the month of commit_time, ordered by (tenant_id, aggregate_id, version), one row per commit (ClickHouseRows); with a TTL on commit_time when AggregateSinkOptions.historyTtl is set.

  • <ctx>_<agg>_state, a ReplacingMergeTree(version) ordered by (tenant_id, aggregate_id), the current state: filled by the materialized view <ctx>_<agg>_state_mv from the base table.

  • <ctx>_<agg>_events, a ReplacingMergeTree ordered by (tenant_id, aggregate_id, version, sequence), one row per event: filled by <ctx>_<agg>_events_mv, which ARRAY JOINs the base table's events; under the same TTL.

  • <ctx>_<agg>_state_flat, a view of _state FINAL: the system columns and the state schema's top-level scalars as typed columns (FlatColumns); its definition's flatFingerprint and those of the writers before it are kept in its comment (FlatLineage), so that during a rolling upgrade an older writer leaves a newer writer's view as it is.

Without the history (AggregateSinkOptions.history false) only _state and _state_flat exist, and the Sink inserts into _state directly (target).

Every table remembers the last deduplicationWindow insert blocks (non_replicated_deduplication_window), so an insert retried with the same token is dropped from the base table and, the inserts setting deduplicate_blocks_in_dependent_materialized_views, from the views' tables too. Beyond that window (a replay from an earlier checkpoint, an old owner beside the new one, a batch made up differently), the history tables' ReplacingMergeTree engine, keyed by ref@version (and the event's sequence), merges a replayed row away: a read with FINAL counts every commit and event once.

Constructors

Link copied to clipboard
constructor(type: AggregateType, database: String, aggregate: AggregateSinkOptions, flatColumns: List<FlatColumn>, deduplicationWindow: Int = ClickHouseSinkOptions.DEFAULT_DEDUPLICATION_WINDOW)

Types

Link copied to clipboard

Properties

Link copied to clipboard
Link copied to clipboard
Link copied to clipboard

The statements that create what is missing, in order: the tables, then the materialized views that fill them. The flat view is not among them: it is replaced only when its writers' lineage allows (flatView).

Link copied to clipboard
Link copied to clipboard
Link copied to clipboard

What Ddl.VALIDATE checks: every table and view with its columns, the materialized views by name.

Link copied to clipboard
Link copied to clipboard

The fingerprint of the flat view's definition: the first 16 hex digits of the SHA-256 of its SELECT, its database aside, so that every writer of the same state schema has the same.

Link copied to clipboard
Link copied to clipboard
Link copied to clipboard
Link copied to clipboard
Link copied to clipboard

The table the Sink inserts into: the base table, or _state without the history.

Link copied to clipboard

Functions

Link copied to clipboard
fun flatView(lineage: FlatLineage): String

The statement that creates or replaces the flat view, its comment the writers' lineage, this one's last.

Link copied to clipboard
fun qualified(name: String): String

name in database, quoted.

Link copied to clipboard
open override fun toString(): String