ClickHouseSink

The Sink of one aggregate type into ClickHouse: each record one row of the schema's target table (ClickHouseSinkSchema.target, ClickHouseRows), from which ClickHouse's materialized views fill the current state and the events.

• Batches: one INSERT at a time per sink, which carries every write queued while the one before ran, in order, at most batchSize records per INSERT; write returns once ClickHouse has acknowledged its records. The sink subscription writes one record per call from many aggregates at once, so a sustained load is written in a few large inserts instead of one small part per commit, and an idle sink adds no delay.

• Idempotent by ref@version: each insert carries a deduplication token, the record's id for a batch of one, else a digest of the batch's ids, so a retry of the same batch within the tables' deduplication window is dropped at once, the views' included. Beyond it, and for a batch made up differently, the history tables are ReplacingMergeTrees keyed by ref@version (ClickHouseSinkSchema): a replayed record is merged away, and a read with FINAL never sees it twice.

• DDL: before its first write, and again once recheckEvery has passed since, the Sink creates or validates its tables as ddl says; a failure fails the write, which the subscription retries. The check repeated finds what changed since: a table dropped or altered fails the writes until it is as the Sink expects.

• The flat view across versions: during a rolling upgrade, writers of two state schemas run at once. The view's comment keeps the fingerprints of the definitions it was created with (FlatLineage): a writer replaces it only when its own definition is newer than every one before it, so an older instance that restarts leaves a newer one's view as it is, and validates it by existence only.

• Catalog: once its tables are created or validated (at once with ddl=none), the Sink appends the catalog entries of its tables to the database's wow_catalog (ClickHouseCatalog), each with the table's current uuid; auto creates wow_catalog when it is missing and adds the column an earlier version's lacks (ClickHouseCatalog.migration), and validate refuses a wow_catalog without it. A table whose uuid the account cannot read gets no row, and a failed write is logged and never fails the Sink's write.