NodeFatals

The fatal errors of one node's own work (isFatal): an OutOfMemoryError, an InternalError or an UnknownError of the virtual machine that one of its loops met. After one, the JVM's state is not to be trusted (the heap is the whole process's), and a retry could loop on the same error, so nothing retries that work in the process: the first fatal error makes the node fatal for good (first). NodeHealth then reports its liveness and readiness DOWN until the process ends, so its orchestrator restarts it, and the node hands off what it holds (listen), so the other nodes take the work over from its checkpoints even where nothing restarts it.

A StackOverflowError is not fatal: it happens on one thread alone, whose stack unwinds as for any exception, so the JVM is unharmed, and it happens again on the same input. A handler that overflows its stack fails as one that throws NotImplementedError does, retried and then parked in the failure ledger: were it fatal, one poison entry would make every node that takes the partition over step out in turn, until the whole cluster stood still.

Each error is recorded once (record), by whoever meets it first: the loop that met it, which knows what it was doing, or the handler of the uncaught failures (handler) of the node's coroutine scopes, each of which carries the part of the node it runs (contextOf). The first is logged at ERROR with its trace, a later one at WARN without it; each is observed (ObservedEvent.Fatal), which the observability module counts as wow.node.fatal.errors.

NONE, the default of a part built outside a node, records nothing: it logs each error at ERROR, and gives its scopes no handler of their own.