runtime/me.ahoo.wow.runtime.health

Package-level declarations

Types

class ComponentHealth(val name: String, val status: HealthStatus, val details: Map<String, Any?> = emptyMap(), val components: List<ComponentHealth> = emptyList())

The health of one component of a node, named: its status, what it is made of (components, each with its own health), and details an operator reads, plain values only (numbers, strings, booleans, lists and maps of them), never a message that may name a host or hold a secret.

HealthOptions

@InternalWowApi

class HealthOptions(val sampleInterval: Duration = DEFAULT_SAMPLE_INTERVAL, val storeDownAfter: Duration = DEFAULT_STORE_DOWN_AFTER, val storeWindow: Duration = DEFAULT_STORE_WINDOW, val handoverGrace: Duration = DEFAULT_HANDOVER_GRACE, val topicsRequired: Boolean = true, val executorSaturation: Double = DEFAULT_EXECUTOR_SATURATION, val lagDegradedAt: Long = DEFAULT_LAG_DEGRADED_AT, val parkedDegradedAt: Int = DEFAULT_PARKED_DEGRADED_AT, val stallTimeout: Duration = DEFAULT_STALL_TIMEOUT)

The thresholds of a node's health, conservative by default: a probe that fails too soon takes nodes out of service, or restarts them, for what a retry would have absorbed.

HealthStatus

@InternalWowApi

enum class HealthStatus : Enum<HealthStatus>

How healthy a component of a node is, from the best to the worst.

NodeFatal

@InternalWowApi

data class NodeFatal(val component: ObservedEvent.Fatal.Component, val name: String, val partition: Int?, val error: String, val at: Long)

A fatal error a node met (NodeFatals): in its component's work on name, in partition when that work runs one, an error of that class (its full name), at at (epoch milliseconds).

NodeFatals

@InternalWowApi

class NodeFatals

The fatal errors of one node's own work (isFatal): an OutOfMemoryError, an InternalError or an UnknownError of the virtual machine that one of its loops met. After one, the JVM's state is not to be trusted (the heap is the whole process's), and a retry could loop on the same error, so nothing retries that work in the process: the first fatal error makes the node fatal for good (first). NodeHealth then reports its liveness and readiness DOWN until the process ends, so its orchestrator restarts it, and the node hands off what it holds (listen), so the other nodes take the work over from its checkpoints even where nothing restarts it.

NodeHealth

@InternalWowApi

class NodeHealth(val node: ServiceNode, sampler: NodeSampler, val options: HealthOptions = HealthOptions(), context: CoroutineContext = Dispatchers.Default, drivers: CoroutineContext = VirtualThreads.dispatcher, nanos: () -> Long = System::nanoTime, pausedOf: (SubscriptionRuntime) -> List<PausedPartition> = SubscriptionRuntime::paused, clock: Clock = node.clock) : AutoCloseable

The health of a node, host-agnostic: a host maps it to its probes (Spring Boot's Actuator, say).

StoreActivity

@InternalWowApi

class StoreActivity

What a node saw of one store it calls: when a call last succeeded, since when the calls fail, and how many failed lately. The node records it on every call of its event logs and lease stores (trackedLog, trackedLeases); NodeHealth reads it, so a probe never calls a store itself.

StoreWritePressure

@InternalWowApi

class StoreWritePressure : WritePressure

The append latency of one store, as the node's logs on it measure each call of append and appendAll (the backend's own time, the wait for a connection included; not the partition append queue's): a sample is the 90th percentile of the period's latencies against a baseline of the earlier periods.