Package-level declarations

Types

Link copied to clipboard
@AggregateRoot(lifecycle = [], ownership = Ownership.TENANT)
@Description(value = "A wide table over the catalog's sources: its definition, its versions and where each is published")
class Dataset(state: DatasetState)

A wide table over the catalog's sources (-2), named as its queries read it, <context>.<name>: an analyst drafts and revises its definition, which the catalog checks against its sources as they are each time (DatasetBinder), and publishes it, each version compiled into its own view (DatasetCompiler) which the publication creates in the warehouse; queries read the current version until the dataset is switched back to another or retired, whose views are dropped once the retention ends.

Link copied to clipboard
sealed interface DatasetCompilation

What compiling a dataset's definition gives: its Compiled artifact, or why it does not compile.

Link copied to clipboard

Compiles a checked definition (BoundDataset) into one ClickHouse view of one version (-2), a pure function: the view viewOf the dataset and the version, of database (wow_datasets by default), the DDL that creates it, and what its queries are served by (DatasetArtifact): the dataset's scope, its fields and their protection, its lineage.

Link copied to clipboard
data class DatasetPolicy(val database: String = DatasetCompiler.DEFAULT_DATABASE, val retention: Duration = 7.days, val approvePublications: Boolean = false, val approveSources: Boolean = false)

How the catalog publishes datasets: the warehouse database their views are created in (wow_datasets), how long a retired dataset's views are kept (retention, 7 days, -2), and the approval switches (③b; off by default): whether every publication takes an approver (approvePublications; one that computes from sensitive values or lowers a protection always does) and whether a dataset reads only the sources an approver approved (approveSources).

Link copied to clipboard

Recompiles the published datasets a source's change may break (-2): every commit of a source sends RecompileDataset to each published dataset whose published definition reads it (or that needs revision already), read from the datasets' snapshots; each dataset then compiles its published definition against the source as it is now.

Link copied to clipboard

The codes the dataset aggregate rejects a command with (-2).

Link copied to clipboard
fun interface DatasetSources

The catalog's sources as a dataset's definition reads them: the state of each source by its id, as the catalog has it now. The service reads the sources' snapshots.

Link copied to clipboard

A dataset of the catalog: its name, its working definition and the sources it reads, and its published versions, of which queries read the current one (published).

Link copied to clipboard

Where a dataset's versions are created: the warehouse's DDL, which only a publication, a rollback and the end of a retirement run, each recorded by its event. The service binds the ClickHouse one.

Link copied to clipboard

Why the warehouse did not run a dataset's DDL, as the warehouse said.