Health, Metrics and Tracing
A node tells the platform whether to send it traffic and whether to restart it, through Spring Boot Actuator's health groups. With wow-opentelemetry, it also records a span for each stage of its work, in the trace of the request, entry or message that started it, and metrics of commits, subscriptions, timers and queries. This page describes both and lists what to alert on.
Health
With Spring Boot Actuator's health support on the class path, the host contributes the node's health:
| Endpoint | Reports |
|---|---|
/actuator/health/readiness | Whether the node should receive traffic: Actuator's own readiness and the node's |
/actuator/health/liveness | Whether the node is alive: Actuator's own liveness and the node's |
/actuator/health/wow | Every component of the node, each UP, DEGRADED or DOWN |
The components:
| Component | DOWN (not ready) when | DEGRADED when |
|---|---|---|
node | The node is not running: starting, draining or stopped | |
stores | A log has kept failing for wow.health.stores.down-after (15 s) with no success since | A log failed within wow.health.stores.window (1 min) |
ownership | Its lease store is down, or the node owns none of its share of a lease group's partitions past wow.health.ownership.handover-grace (1 min) | It owns another set than its share for that long, or the lease store failed lately |
topics | A topic of another context it reads is still joining (unless wow.health.topics.required: false) | |
subscriptions | Never | A subscription is lag-degraded-at (10,000) entries behind, or has parked-degraded-at (100) parked entries |
executor | The executor runs at saturation (90 %) of its in-flight bound | |
liveness | The node is stuck: its dispatcher does not tick, or commands in flight do not finish while every log answers, for wow.health.liveness.stall-timeout (3 min). This alone fails liveness |
A degraded component does not change a group's status: it is there to be seen and alerted on. Liveness never fails for a storage outage or while the node drains, so an outage does not restart every node at once. The node samples what a probe must not wait for (the logs' heads, the subscriptions' lag, the parked entries) every wow.health.sample-interval (10 s), so a probe answers at once.
What a response shows beyond the status follows Actuator's own settings (management.endpoint.health.show-details, never by default). The details hold counts, states and exception class names, never messages or addresses. Kubernetes shows the probes.
Metrics and traces
Put wow-opentelemetry on the class path. The spans go through the application's OpenTelemetry bean, such as Spring Boot's with an SDK and its exporters, else the global one, such as the Java agent's. The metrics go to the application's Micrometer MeterRegistry, such as Actuator's with a Prometheus or OTLP registry, else Micrometer's global registry. wow.opentelemetry.enabled: false turns both off.
Spans. Each stage of a command and of a subscription is a span, in the trace its request, log entry or Kafka message carries:
| Stage | Span |
|---|---|
| Admission | admit <context>.<aggregate>, a server span |
| Execution, decision, append | execute, decide, append <context>.<aggregate> |
| A subscription handling an entry of the log | handle <subscription> |
| A subscription handling a message of a topic | <topic> process, a consumer span |
| A timer firing | fire <command type> |
| A query | A span per query |
| A Kafka send | A producer span per send |
The spans carry the context, aggregate, tenant, command and subscription as attributes (wow.context, wow.aggregate, wow.tenant.id, wow.command, wow.subscription, …), so a trace follows a command through its sagas across services.
Metrics. The main ones, tagged by context, aggregate, subscription or partition:
| Metric | Type | Measures |
|---|---|---|
wow.command.commit | Timer | A command's execution, from taking it to its outcome, by outcome |
wow.command.refused | Counter | Commands that ended before the kernel took them: refusals, busy lanes or nodes, unavailable stores |
wow.subscription.lag | Timer | From a commit to a subscription handling it |
wow.subscription.backlog | Gauge | The entries of a partition after a subscription's position |
wow.subscription.progress | Gauge | A subscription's position over the log's head, 0 to 1: a rebuild's progress |
wow.subscription.parked | Gauge | The aggregates a subscription has parked |
wow.subscription.paused | Gauge | 1 while a subscription has paused a partition on a systemic failure |
wow.subscription.throttled | Gauge | 1 while a subscription throttles a partition on backpressure |
wow.subscription.undecodable | Counter | Messages of a topic this version cannot decode |
wow.timer.delay | Timer | How late a timer fires after its due time |
wow.query | Timer | A query, from its admission to its end |
wow.query.limit | Gauge | A store's current bound on concurrent queries |
wow.kafka.offset.gaps | Counter | Offset gaps on another context's topic (Kafka) |
wow.opentelemetry.sample-interval (15 s) is how often the sampled gauges, the parked entries and the subscriptions' positions, are read.
What to alert on
| Signal | Means | Look at |
|---|---|---|
| Readiness down on a node for minutes | A store down, or a node that lost its partitions | /actuator/health/wow |
wow.subscription.parked above 0 | An aggregate waits for a fix and an operator | The Ops Service and Console |
wow.subscription.paused at 1 for long | A downstream store or service is down | The subscription's logs, the store |
wow.subscription.backlog growing | A subscription falls behind | Its lag, its handler's latency, the node's executor |
wow.timer.delay growing | Timers fire late | The node's load, the timers' store |
wow.kafka.offset.gaps above 0 | A subscription skipped or stopped at retained data | The topic's retention |
wow.command.refused with wow.command.busy | Nodes refuse commands under load | The number of nodes, the store's latency |