clickhouse/me.ahoo.wow.clickhouse.sink

Package-level declarations

Types

class AggregateSinkOptions @InternalWowApi constructor(val history: Boolean = true, val historyTtl: Duration? = null)

How the Sink keeps one aggregate type's commits.

data class CatalogEntry(val table: String, val role: String, val service: String, val context: String, val aggregate: String, val ownership: String, val tenantColumn: String, val ownerColumn: String, val spaceColumn: String, val sensitive: List<CatalogSensitive>, val references: List<CatalogReference>, val eventRevisions: Map<String, Int>, val columnsHash: String, val access: String = "")

What the Sink adds to the warehouse's own metadata of one of its tables, one row of wow_catalog (ClickHouseCatalog): who the table is of, by the Manifest and never by its name.

CatalogReference

@InternalWowApi

data class CatalogReference(val path: String, val context: String, val aggregate: String)

A value of a Wow table that refers to an aggregate: its path, and the aggregate's context and aggregate.

CatalogSensitive

@InternalWowApi

data class CatalogSensitive(val path: String, val level: String, val mask: String = "", val event: String = "", val revision: Int = 0)

A sensitive value of a Wow table: its path, the event and revision it is of (empty and 0 for none), its level and mask.

ClickHouseCatalog

@InternalWowApi

object ClickHouseCatalog

The wow_catalog table of a database: the Sink appends one row per table it keeps once it created or validated them, the warehouse's metadata (system.tables, system.columns) staying the base of the analytics catalog. A ReplacingMergeTree(written_at) ordered by (database, table, fingerprint): a row of the same content replaces the one before it, a changed one is appended beside it, so several rows of one table stand side by side during a rolling upgrade, and the catalog takes the one that matches the table's columns.

ClickHouseColumn

@InternalWowApi

data class ClickHouseColumn(val name: String, val type: String)

A column as system.columns reads it: its name and its type.

ClickHouseObject

@InternalWowApi

data class ClickHouseObject(val name: String, val columns: List<ClickHouseColumn>?)

A table or view of the Sink: its name, and the columns it has, or null when only its existence is checked.

ClickHouseSink

@InternalWowApi

class ClickHouseSink : Sink

The Sink of one aggregate type into ClickHouse: each record one row of the schema's target table (ClickHouseSinkSchema.target, ClickHouseRows), from which ClickHouse's materialized views fill the current state and the events.

ClickHouseSinkOptions

class ClickHouseSinkOptions @InternalWowApi constructor(val database: String = DEFAULT_DATABASE, val ddl: Ddl = Ddl.AUTO, val batchSize: Int = DEFAULT_BATCH_SIZE, val deduplicationWindow: Int = DEFAULT_DEDUPLICATION_WINDOW, val aggregates: Map<AggregateType, AggregateSinkOptions> = emptyMap())

The ClickHouse Sink's options.

ClickHouseSinkProvider

class ClickHouseSinkProvider : SinkProvider

The ClickHouse Sink, type clickhouse: switched on by sink.enabled: true alone, since writing every event to a warehouse is a cost a service chooses; neither a client nor an endpoint switches it on. It writes every stream the service's Manifest (a resource of the application) publishes, through the application's Client when it has one, which it never closes, else through one of endpoint and the credentials, which it closes with the sinks (ClickHouseClients.clientOf); with neither, it fails the start naming both. Its settings, relative to its configuration (wow.clickhouse.*), are its options: the connection's, ddl (Ddl, auto by default; the host gives wow.ddl when it is not set), sink.batch-size, sink.deduplication-window, and each aggregate type's history.enabled and history.ttl under aggregates.<context>.<aggregate>, which the host gives from wow.aggregates.[<context>.<aggregate>].clickhouse.history.* (ClickHouseSinkOptions).

ClickHouseSinks

@InternalWowApi

class ClickHouseSinks

The ClickHouse Sinks of a service: sinkOf gives the ClickHouseSink of an aggregate the manifest declares, its tables in options' database, its flat view's columns from the aggregate's state schema in the Manifest's $defs (FlatColumns).

ClickHouseSinkSchema

@InternalWowApi

class ClickHouseSinkSchema(val type: AggregateType, val database: String, val aggregate: AggregateSinkOptions, val flatColumns: List<FlatColumn>, deduplicationWindow: Int = ClickHouseSinkOptions.DEFAULT_DEDUPLICATION_WINDOW)

The tables and views of one aggregate type, in database, named after the type (<ctx>_<agg>, prefixOf):

FlatColumn

@InternalWowApi

data class FlatColumn(val name: String, val type: String, val expression: String, val property: String = name)

A column of a view: its name, its ClickHouse type as system.columns reads it, the expression it is, and the state's property it is read from.

FlatLineage

@InternalWowApi

class FlatLineage(val fingerprints: List<String>)

The fingerprints of the flat view's definitions that writers created it with, oldest first, as its comment keeps them (wow-sink-flat:<fingerprint>,<fingerprint>): the last is the view's definition.