analytics-catalog-domain/me.ahoo.wow.analytics.catalog.domain.dataset/DatasetCompiler

DatasetCompiler

Compiles a checked definition (BoundDataset) into one ClickHouse view of one version (-2), a pure function: the view viewOf the dataset and the version, of database (wow_datasets by default), the DDL that creates it, and what its queries are served by (DatasetArtifact): the dataset's scope, its fields and their protection, its lineage.

The view reads each source as its rows are now: a source whose engine merges rows (ReplacingMergeTree and the others that collapse, aggregate or sum) through FINAL; a left join keeps empty right sides NULL (join_use_nulls); a value inside a JSON text column through JSONExtractString, inside a JSON column through its subcolumn. Every name is quoted and every text literal escaped, so nothing of a definition is spliced in.

Properties

DEFAULT_DATABASE

const val DEFAULT_DATABASE: String

The database the views are created in unless the service names another.

Functions

compile

fun compile(definition: DatasetDefinition, sources: Map<String, SourceState?>, context: String, name: String, version: Int, database: String = DEFAULT_DATABASE, title: String? = null, approvedOnly: Boolean = false): DatasetCompilation

version of the dataset name of context from definition, checked against the catalog's sources as they are now (DatasetBinder), its view in database; with approvedOnly, only of sources an approver approved.

viewOf

fun viewOf(context: String, name: String, version: Int): String

The name of version's view of the dataset name of context: ds_<context>_<name>_<digest>_v<version>, the context and the name lower-cased with every other character than a letter, a digit or _ written _, so an operator recognizes the dataset, and the digest the first 10 hex digits of the SHA-256 of its exact id (AnalyticsCatalogContext.datasetIdOf), so ids that differ only by case, -, _ or where the context ends get views of their own.