analytics-catalog-api/me.ahoo.wow.analytics.catalog.api.dataset

Package-level declarations

Types

@Summary(value = "Approve the requested publication of a dataset")

Approves, as an approver, the publication of the dataset's definition that was requested: it is published.

data class DatasetArtifact(val context: String, val name: String, val version: Int, val database: String, val view: String, val ddl: String, val scope: DatasetScope, val fields: List<DatasetField>, val sources: List<String>, val title: String? = null, val identity: String? = null, val eventTime: String? = null, val approvals: List<String> = emptyList(), val columns: List<SourceColumn> = emptyList(), val select: String = "")

One compiled version of a dataset (-2): the ClickHouse view of database that serves it, the ddl that creates it (audited with the publication), and what a query node needs to serve it without asking the catalog: its scope, its fields with their protection and lineage, its identity and eventTime columns. sources are the catalog's sources it reads; approvals say why publishing it takes an approver, empty when it does not. columns are the view's columns as the warehouse created them, their warehouse types: what a query node declares each field as; empty until the publication creates the view. select is the query the view is, which the catalog asks the warehouse to describe before it publishes a recompilation.

DatasetColumn

data class DatasetColumn(val name: String, val from: String? = null, val expression: String? = null, val title: String? = null, val sensitivity: DatasetSensitivity? = null)

A column of the dataset, named as queries read it: the value from a source (<alias>.<column>[.<path>]), or the expression computed from such values. Its protection follows its lineage: the highest of the values it reads, with their mask when it reads one; sensitivity, when given, decides it instead, and only an approver lets it be lower than the lineage says.

DatasetDefinition

data class DatasetDefinition(val sources: List<DatasetSource>, val subject: String, val columns: List<DatasetColumn>, val joins: List<DatasetJoin> = emptyList(), val filters: List<String> = emptyList(), val identity: String? = null, val eventTime: String? = null)

What a wide table is made of (-2): catalog sources, each under an alias, the joins that bring every source but the subject in, the columns it shows, and the filters its rows pass.

DatasetDrafted

@Summary(value = "A dataset was drafted")

data class DatasetDrafted(val context: String, val name: String, val definition: DatasetDefinition, val sources: List<String>, val title: String? = null)

The dataset name of context was drafted from definition, which reads the catalog's sources.

DatasetField

data class DatasetField(val name: String, val title: String? = null, val sensitivity: DatasetSensitivity? = null, val inputs: List<String> = emptyList(), val computed: Boolean = false, val reference: DatasetReference? = null)

A column of a dataset's view: its name and title, the sensitivity its lineage gives it (null when it is not sensitive), the values it reads (inputs, each <source id>/<column>[.<path>]), whether it is computed, and the aggregate its value is the id of (reference), as the source it is read from says: a row's action on that aggregate goes to the service that owns it.

DatasetJoin

data class DatasetJoin(val alias: String, val on: List<DatasetJoinKey>, val kind: DatasetJoinKind = DatasetJoinKind.INNER)

Brings the source alias in, matched by on: each key pairs a column of the sources before it (DatasetJoinKey.left) with one of alias (DatasetJoinKey.right).

DatasetJoinKey

data class DatasetJoinKey(val left: String, val right: String)

One key of a join: the left column (<alias>.<column> of a source joined before) equals the right one.

DatasetJoinKind

enum class DatasetJoinKind : Enum<DatasetJoinKind>

How a join keeps the rows of the left side that have no match on the right.

DatasetNeedsRevision

@Summary(value = "A dataset needs revision")

data class DatasetNeedsRevision(val problems: List<String>, val serving: Boolean = false)

The dataset's published definition needs revision, for problems: it no longer compiles against the catalog, or it compiles into other columns than its published version has, or it now takes an approver. While the published version's view still gives the columns it was published with, it goes on serving; otherwise its queries are answered wow.query.dataset-unavailable until a revision is published or its sources come back.

DatasetPublicationRefused

@Summary(value = "A dataset's publication was refused")

data class DatasetPublicationRefused(val reason: String)

The requested publication was refused, for reason.

DatasetPublicationRequested

@Summary(value = "A dataset's publication was requested")

data class DatasetPublicationRequested(val definition: DatasetDefinition, val reasons: List<String>)

Publishing definition takes an approver, for reasons: it is published once one approves.

DatasetPublished

@Summary(value = "A dataset's version was published")

data class DatasetPublished(val artifact: DatasetArtifact, val definition: DatasetDefinition, val recompiled: Boolean = false)

definition was published as the dataset's version DatasetArtifact.version, its view created: artifact; recompiled when the catalog published it after a source changed.

DatasetReference

data class DatasetReference(val context: String, val aggregate: String)

The aggregate a dataset's value is the id of: aggregate of the bounded context context.

DatasetRetired

@Summary(value = "A dataset was retired")

data class DatasetRetired(val version: Int)

The dataset is retired: its queries stop; version was the one they read.

DatasetRevised

@Summary(value = "A dataset was revised")

data class DatasetRevised(val definition: DatasetDefinition, val sources: List<String>, val title: String? = null)

The dataset's definition is now definition, which reads the catalog's sources.

DatasetRevisionCleared

@Summary(value = "A dataset's version compiles again")

data class DatasetRevisionCleared(val version: Int)

The dataset's published version version compiles again: its queries read it again.

DatasetRolledBack

@Summary(value = "A dataset was switched back to an earlier version")

data class DatasetRolledBack(val version: Int, val from: Int)

The dataset's queries read its version version again, in place of from.

DatasetScope

data class DatasetScope(val tenant: String? = null, val owner: String? = null, val space: String? = null, val platform: Boolean = false)

The columns of a dataset's view that hold its rows' scope: tenant, owner and space, from its subject; none for a platform dataset, whose rows every tenant shares.

DatasetSensitivity

data class DatasetSensitivity(val level: SensitivityLevel? = null, val keepPrefix: Int = 0, val keepSuffix: Int = 0)

The protection a column is given: at level, or none when level is null, its mask keeping keepPrefix and keepSuffix characters.

DatasetSource

data class DatasetSource(val alias: String, val sourceId: String)

The catalog source sourceId (<connection>:<database>:<table>), referred to as alias in the definition.

DatasetStatus

enum class DatasetStatus : Enum<DatasetStatus>

Where a dataset is in its life.

DatasetVersion

data class DatasetVersion(val artifact: DatasetArtifact, val definition: DatasetDefinition)

A published version of a dataset: its compiled artifact and the definition it was compiled from.

DatasetViewsDropped

@Summary(value = "A retired dataset's views were dropped")

data class DatasetViewsDropped(val views: List<String>)

The retired dataset's views were dropped from the warehouse.

DraftDataset

@CreateAggregate

@Summary(value = "Draft a dataset over the catalog's sources")

data class DraftDataset(val context: String, val name: String, val definition: DatasetDefinition, val title: String? = null)

Drafts the dataset name of context from its definition (-2): the dataset's id is me.ahoo.wow.analytics.catalog.api.AnalyticsCatalogContext.datasetIdOf of context and name, the name its queries read it by. The definition is checked against the catalog as it is now: its sources are there and usable, its references name their columns, its joins key on the tenant, its expressions use only the functions allowed.

DropDatasetViews

@InternalCommand

@Summary(value = "Drop a retired dataset's views")

data object DropDatasetViews

Drops the views of a dataset retired for the retention; the catalog schedules it when it retires one, and only that schedule sends it: internal.

PublishDataset

@Summary(value = "Publish the dataset's definition as its next version")

data object PublishDataset

Publishes the dataset's definition as its next version (-2): compiled against the catalog as it is now into the version's own view, ds_<context>_<name>_<digest>_v<N> (the digest that of the exact id, so ids that read the same once lower-cased and made safe have views of their own), which the publication creates in the warehouse (the only way the catalog changes the warehouse, audited by the event); queries then read that version. A definition that compiles to what is published already changes nothing.

RecompileDataset

@InternalCommand

@Summary(value = "Compile a published dataset again after a source changed")

data object RecompileDataset

Compiles the dataset's published definition again against the catalog as it is now, because a source it reads changed (-2): what compiles the same changes nothing; what compiles otherwise (a source's values now more protected, say) is published as the next version; what no longer compiles marks the dataset NEEDS_REVISION. The catalog sends it when a source changes, and nothing else does: internal.

RefuseDatasetPublication

@Summary(value = "Refuse the requested publication of a dataset")

data class RefuseDatasetPublication(val reason: String)

Refuses the requested publication of a dataset, for reason.

RetireDataset

@Summary(value = "Retire a dataset")

data object RetireDataset

Retires the dataset: its queries stop; its views stay in the warehouse for the retention (7 days by default), then the catalog drops them. Publishing it again before then brings it back.

ReviseDataset

@Summary(value = "Revise a dataset's definition")

data class ReviseDataset(val definition: DatasetDefinition, val title: String? = null)

Replaces the dataset's definition and title, checked against the catalog as a draft's is.

RollbackDataset

@Summary(value = "Switch a dataset back to an earlier version")

data class RollbackDataset(val version: Int)

Switches the dataset's queries back to its published version, whose view is created again if it is gone.