Troubleshooting

Each section starts from what you see, names the usual causes, and says what to do. Every answer a node gives carries a code; Error Codes lists them all.

A node will not start

The host refuses to start rather than run with a setting that would quietly do the wrong thing. The log names the cause:

The log says	Cause	Do
No durable backend	No DataSource or MongoClient bean, outside a development profile and with an internal port that binds more than loopback	Give the application its store, or set wow.backend: memory on purpose
No cluster token	wow.internal.token is missing outside a development profile	Set WOW_INTERNAL_TOKEN to a secret of at least 32 characters, the same on every node
Unknown security configuration keys	A mistyped key under wow.cosec., wow.http.services., wow.calls. or wow.internal.	Fix the key (Configure a Service)
A subscription, aggregate or context the node does not serve	A name under wow.subscriptions, wow.aggregates or wow.contexts that matches nothing	Fix the name, or remove the entry
No command transport	A saga sends commands to another context and the node has neither Kafka nor in-memory topics	Set wow.kafka.bootstrap-servers
A missing topic, or a command topic with the wrong number of partitions	wow.kafka.topics.ddl: validate, and the operator has not created the topic as the log needs it	Create the topic: a command topic has exactly wow.log.partitions partitions (Kafka)
A structure of another format	A table created by an incompatible build of Wow	Run the migration of that major version; never edit wow_format

Commands time out

wow.wait.timeout (504) means the wait ran out, not that the command failed: the command may have committed. The problem names the last stage reached.

• Send it again with the same request id: a command that did commit is answered as a duplicate, with its original result.
• A wait for a later stage (HANDLED, a saga's hop, SNAPSHOT) waits for subscriptions too. A parked or lagging subscription on the chain makes it time out (Waiting for Outcomes).
• wow.command.unknown (503) means a forward to the owner got no answer in time: whether it committed is unknown, and a retry with the same request id is safe.

Commands are refused

Code	Cause	Do
wow.command.busy (429)	The node or the aggregate has too many commands in flight	Retry after Retry-After; add nodes if it persists (Scaling and Capacity)
wow.aggregate.version-mismatch (409, 412 with If-Match)	The aggregate is not at the version the client expected	Read it again and decide whether to retry
wow.command.contended (409)	Concurrent writers kept winning the version	Retry with the same request id
wow.command.request-id-reused (409)	The request id was used by another command	Give each command its own request id
wow.aggregate.forbidden (403)	The aggregate belongs to another owner or space than the caller's	Check the caller's scope (Secure a Deployment)
wow.auth.forbidden (403)	The authorizer refused the command for the caller	Check the caller's roles or policies
wow.call.command-forbidden (403)	A service called a command its entry under wow.http.services does not list	List the command for that service, on the target
wow.storage.unavailable (503)	A store could not be reached	Check the database; retry with the same request id
wow.node.standby (503)	The node is on standby for a migration's cut-over	Resume it at the cut-over (Deployment Model)

A subscription parks aggregates

wow.subscription.parked is above zero, and the ops console lists the aggregates. Each entry carries the error and the version the subscription stopped at.

• Fix the cause first: deploy the corrected handler, or bring back the outside system it calls.
• Then resume the aggregates: the subscription reads each one's stream again from where it stopped (The Ops Service and Console).
• A rejection of a command a saga sent parks the aggregate only when the saga has no @OnRejected handler for it (Failures and Retries).

A subscription stops for every aggregate

wow.subscription.paused is 1 for a partition: the subscription took a failure for an outage of something every aggregate needs, a store or a downstream service, and pauses the partition until it is back. Nothing is parked meanwhile. Find what is down from the subscription's log; the partition resumes by itself.

A subscription falls behind

wow.subscription.backlog grows and the subscriptions health component is degraded. The handler is slower than the log grows: a slow external call, writes to a store that needs an index, or a downstream that pushes back (wow.subscription.throttled). Speed up the handler, then add nodes to spread its partitions (Scaling and Capacity).

A partition has no owner

The ownership health component is DOWN, or commands for some aggregates are slow while a node is gone. Leases of a node that died expire after wow.cluster.lease-ttl (10 s), then the partitions move. When they do not:

• The nodes cannot reach each other's internal port: check the network policy and wow.node.address, the address each node advertises.
• The lease store is down: the stores health component says so.

A topic of another context

Symptom	Cause	Do
A node stays not ready after it starts	A topic it reads is still joining its lease group	Wait; or set wow.health.topics.required: false
wow.kafka.offset.gaps is above zero	Retention deleted offsets a subscription had not read	Raise the topic's retention; decide on wow.kafka.offset-gap (Kafka)
wow.subscription.undecodable is above zero	Messages this version cannot decode	Upgrade the reading service's api, or the writer sends an unknown format

Queries are refused or slow

wow.query.busy (429) means the store's query bound is reached; it adapts down while the store's appends are under pressure. wow.query.timeout (504) means the query ran longer than its budget: narrow it rather than run it again. Give the queries a pool or a replica of their own (PostgreSQL).