NodeHealth

The health of a node, host-agnostic: a host maps it to its probes (Spring Boot's Actuator, say).

Components (health), each UP, DEGRADED or DOWN by the options:

• node: the node's lifecycle state; DOWN unless it is RUNNING, and DOWN for good once the node is fatal (NodeFatals), with the fatal error under fatal.

• ownership: each lease group the node takes part in (each context's; each topic's it reads): the partitions it owns against its share by HRW, the members, whether a hand-over is in progress, and the context's lease store. DOWN when the lease store is down, when the ownership does not run, or when the node has owned none of a share that is not empty for HealthOptions.handoverGrace; DEGRADED when it owns another set than its share for that long, or the lease store failed lately. On a node on standby (ServiceNode.standby), which joins no lease group until it is activated, a group that does not run is UP, marked standby: the node serves reads.

• stores: each event log (one per log group), from the node's own calls and a probe of its head every HealthOptions.sampleInterval, and each store instance the write path needs (one holding a log, a type's primary snapshot store) from its probe (me.ahoo.wow.spi.store.Store.probe): the last success, the failures lately and since when they fail. DOWN when its calls have failed (or its probe has not answered) for HealthOptions.storeDownAfter with no success since.

• secondaryStores: each other placed store instance (an additional snapshot store, a query or search store), from its probe, as stores shows one; never part of readiness: its outage degrades the reads it serves, not the node.

• topics: each topic the node reads, by where it is in joining its lease group: DOWN while it is joining when HealthOptions.topicsRequired, DEGRADED otherwise; UP, marked standby, while a node on standby has not joined it.

• subscriptions: each subscription of each runtime, by its group and name, from the node's last sample (NodeSampler): its lag behind the log's head in the partitions it runs here, its parked entries (the service's, from the ledgers: each context's snapshot its own), and the partitions it has paused (SubscriptionPauses, each runtime's systemic pauses). Never DOWN: a subscription that falls behind does not stop the node from serving.

• executor: the commands in flight against wow.executor.max-in-flight, and per context the executor's commands and the appends queued; DEGRADED at HealthOptions.executorSaturation.

• liveness: see liveness.

Readiness (readiness, whether the node can serve traffic): node, stores, ownership and topics; DOWN when one of them is, UP otherwise (a degraded node still serves).

Liveness (liveness, whether the process is stuck, so a restart helps): DOWN from the node's first fatal error (ServiceNode.fatals: an error of the JVM its own work met, out of memory say; NodeFatals.isFatal) until the process ends, whatever the node's state, with that error under fatal (the part of the node, what it ran, the partition, the error's class, when) and the count of them under fatalErrors: nothing in the process recovers from it. Otherwise DOWN only when the node runs and, for HealthOptions.stallTimeout, one of these holds:

• dispatcher: its dispatcher has not ticked;

• drivers: the dispatcher of the backends' blocking driver calls (drivers) has not run a tick of its own. Every store and topic call of the node runs there, so a wedged one (every carrier of the virtual threads pinned, say) stops the node's leases, heartbeats and reads, with no call failing;

• reconcile: a reconcile of one of its lease groups has not ended. The node left the group's members after the group's reconcileTimeout, which is shorter, so the others run its partitions meanwhile;

• kernel: commands are in flight, every log answered within that time, and none finished in it.

Each is measured on its own (the drivers tick on the drivers alone), so more than one may hold at once: the details list every one under stalls, in this order, and name the first as stalled.

A storage outage never fails it: a store call that fails or times out returns, and a reconcile waiting on a slow store ends once it answers, while a restart would not bring the stores back, and a restart storm would only add load to them. So the bound is minutes, well above what a slow store under load takes. A node that is not running (starting, stopping) is live: the startup probe covers the start, and a draining node is not restarted.

A probe reads what was recorded and sampled: it never calls a store. sampler, the node's one sampler, which its owner starts and closes, samples the node; each of its samples probes the stores the node has not called and checks the hand-overs. start starts the dispatcher's tick, once the node has started; close stops it.