DatasetPublicationWatcher

class DatasetPublicationWatcher(queries: () -> NodeQueryApi, datasets: QueryDatasets, interval: Duration, reconcileInterval: Duration, clock: Clock = Clock.systemUTC()) : SmartLifecycle

The data plane's view of the datasets' publications (-2): every node of the service runs one, and keeps its datasets (the datasets its queries serve) in step with the published versions, read from the datasets' snapshots as trusted code, never on a query's path.

  • Every interval it reads the datasets whose snapshot changed since the newest change it saw (less a margin for the clocks of the nodes that commit them), and every reconcileInterval all of them, which also catches a dataset that is gone. A version published is queryable on every node within interval and one read.

  • A dataset that serves a version is put under its name, its version's view as its table; one retired is removed; one that needs revision is put unavailable, its queries answered wow.query.dataset-unavailable, and why is logged. A dataset of the same name from a dataset file takes precedence: it is left in place.

  • A failed read is logged and tried again at the next one; the datasets keep the versions they have, so queries go on while the catalog's store is down. synced says whether one read ever succeeded (the node's readiness), lastSuccess when the last one did, lag how long the newest applied publication took to be applied.

A node-local subscription of the datasets' state stream would do this without reading the snapshots: 10.x.

Constructors

Link copied to clipboard
constructor(queries: () -> NodeQueryApi, datasets: QueryDatasets, interval: Duration, reconcileInterval: Duration, clock: Clock = Clock.systemUTC())

Types

Link copied to clipboard
object Companion

Properties

Link copied to clipboard
open override val isRunning: Boolean
Link copied to clipboard
Link copied to clipboard
Link copied to clipboard

Functions

Link copied to clipboard
open override fun isRunning(): Boolean
Link copied to clipboard
open override fun start()
Link copied to clipboard
open override fun stop()
Link copied to clipboard
suspend fun sync(): Int

Reads the datasets changed since the last read, or all of them when a reconcile is due; the changes applied.

Link copied to clipboard

Seconds since the last read succeeded, NaN before the first.