Package-level declarations

Types

Link copied to clipboard

Who sends what to the analytics catalog: every command changes what the platform's datasets are built on, so only the catalog's roles send them (AnalyticsCatalogRoles):

Link copied to clipboard
@Configuration(proxyBeanMethods = false)
@EnableConfigurationProperties(value = [AnalyticsCatalogProperties::class])
class AnalyticsCatalogConfiguration

The analytics catalog's beans: who calls it and what each caller may send or read, and the catalog's ports.

Link copied to clipboard
@ConfigurationProperties(prefix = "wow.analytics.catalog")
class AnalyticsCatalogProperties(var connection: String = DEFAULT_CONNECTION, var databases: MutableList<String> = mutableListOf(), var trustedWriters: MutableMap<String, MutableList<String>> = linkedMapOf(), var refreshInterval: Duration = DEFAULT_REFRESH_INTERVAL, var watch: AnalyticsCatalogProperties.Watch = Watch(), var datasets: AnalyticsCatalogProperties.Datasets = Datasets(), var requireSourceApproval: Boolean = false, var requirePublicationApproval: Boolean = false)

The analytics catalog service's settings, wow.analytics.catalog.*: the warehouse whose tables and views are the sources, the accounts trusted to describe them, how the catalog watches them, and its datasets.

Link copied to clipboard

Keeps the catalog to the catalog's roles: a read of the catalog's own aggregates (its sources and datasets, their columns, sensitivity and lineage, the control plane), a query of any of their models, a state read or a trace, by a caller with none of AnalyticsCatalogRoles.READERS is refused as wow.auth.forbidden before anything is read: the read is right, the caller may not make it. Trusted in-process code, which reads with no caller, is not asked.

Link copied to clipboard

The analytics catalog's roles, which a caller lists in its attribute ROLES_ATTRIBUTE: from the identity the CoSec gateway verified and forwarded (the cosec profile), or, in development, from the Wow-Roles header (the host's development callers). Its callers are of the platform's tenant, (platform), and have no owner.

Link copied to clipboard
@SpringBootApplication(proxyBeanMethods = false)
class AnalyticsCatalogServer

The analytics catalog service on the Wow host: the control plane of the analytics platform, the warehouse's data-source catalog, centrally managed. The host finds the source model through analytics-catalog-domain's binding index and serves the analytics catalog context's routes:

Link copied to clipboard
data class CatalogSettings(val connection: String, val requireSourceApproval: Boolean, val requirePublicationApproval: Boolean, val rolesDecide: Boolean)

What a console asks of the catalog it works with, so it offers what this catalog does: the warehouse connection a source's id starts with, whether a source needs an approver before datasets read it (requireSourceApproval) and a publication one before queries read it (requirePublicationApproval), and whether the catalog's roles decide who sends what (rolesDecide, AnalyticsCatalogAuthorizer) or CoSec's policies do, which a console cannot read ahead.

Link copied to clipboard

The catalog's CatalogSettings, a service route of the context: GET /analytics-catalog/settings.

Link copied to clipboard
fun interface CatalogSources

The sources the catalog holds: the tables of connection's database it holds a source of that it has not marked missing, which the watcher compares with what the warehouse lists when it starts.

Link copied to clipboard
class CatalogWatcher(catalog: WarehouseCatalog, commands: () -> CommandClient, interval: Duration, sources: CatalogSources) : SmartLifecycle

The change detection of the catalog: every interval, it lists each database's tables and views (their uuid, metadata modification time and a fingerprint of their columns, system.tables and system.columns: the time is in seconds, so an ALTER in the second of the last look shows only in the columns) and when each last got a wow_catalog row, and sends SyncSource for each table whose listing changed since it last looked, or that is gone: the source's aggregate reads the table again and applies the trust rules. Its first look of a database, at startup, syncs every table, which changes nothing where nothing changed, and every table the catalog holds a source of there (sources) that the warehouse no longer lists: one dropped while the catalog was down is marked missing too.

Link copied to clipboard

The datasets' views on a ClickHouse server, over the host's client-v2 client: a publication creates its database if it is not there, runs its DDL and reads the view's columns back (a view whose column a dataset cannot serve, such as a map, is dropped again and refused); the end of a retirement drops the views. The account needs CREATE DATABASE, CREATE VIEW and DROP VIEW on the datasets' database, and SELECT on the sources the views read.

Link copied to clipboard
class ClickHouseWarehouseCatalog(client: Client, val connection: String, val databases: List<String>) : WarehouseCatalog

The WarehouseCatalog of a ClickHouse server, over the host's client-v2 client:

Link copied to clipboard
data class CommandInput(val name: String, val type: String, val required: Boolean, val nullable: Boolean = false, val options: List<String>? = null, val description: String? = null)

One input of a ReferenceCommand: a property of the command's body, its JSON type (string, number, integer or boolean), whether it is required and may be nullable, and the options of an enumeration.

Link copied to clipboard
class DatasetPublicationWatcher(queries: () -> NodeQueryApi, datasets: QueryDatasets, interval: Duration, reconcileInterval: Duration, clock: Clock = Clock.systemUTC()) : SmartLifecycle

The data plane's view of the datasets' publications (-2): every node of the service runs one, and keeps its datasets (the datasets its queries serve) in step with the published versions, read from the datasets' snapshots as trusted code, never on a query's path.

Link copied to clipboard
class DatasetReferenceActions(datasets: suspend (String) -> DatasetState?, manifests: suspend (context: String, caller: Caller) -> Manifest, described: Set<String>)

The actions of a dataset's rows on the aggregates they refer to: a console offers, on a row, the commands of the aggregate a REFERENCE field holds the id of, and sends them here; the catalog sends each to the owning service as the caller, so that service authorizes it as it would a direct call. The browser talks to the catalog alone: no other service's URL, no CORS.

Link copied to clipboard

The sources the catalog holds, from the sources' snapshots, read as a job that reads the whole model, as trusted code, the catalog's own, in the catalog's tenant. A node whose backend serves no snapshot query holds none it can read, and says so in its log: the watcher then knows only the tables it sees.

Link copied to clipboard

The catalog's sources as a dataset's definition reads them: each source's latest committed state, read as trusted code, the catalog's own, in the catalog's tenant.

Link copied to clipboard

The aggregates the catalog knows, from the sources' snapshots: an aggregate some Wow source of the catalog is of, read as trusted code, the catalog's own, in the catalog's tenant. A node whose backend serves no snapshot query knows of none, and says so in its log.

Link copied to clipboard

The codes the reference actions refuse with, besides the framework's.

Link copied to clipboard
data class ReferenceActions(val dataset: String, val fields: List<ReferenceField>)

The reference actions of a served dataset, by field.

Link copied to clipboard
data class ReferenceCommand(val name: String, val description: String?, val inputs: List<CommandInput>)

A command a dataset's row offers on the aggregate a field refers to: its name, what it says it does, its inputs.

Link copied to clipboard
data class ReferenceField(val field: String, val context: String, val aggregate: String, val commands: List<ReferenceCommand>)

A REFERENCE field of a dataset, the aggregate it refers to, and the commands its rows offer on it.

Functions

Link copied to clipboard
fun main(args: Array<String>)