The Ops Service and Console

When a subscription keeps failing on an aggregate, it parks that aggregate in its service's failure ledger (Subscriptions, failures and replay). The ops service collects the ledgers of all services in one place, and its console is where operators see what is parked and decide what to do about it. This page explains how it fits together, how to deploy it, and how to work with it.

How it fits together

• The ledger, and its retries, stay with the service. Parking must be as reliable as the checkpoint beside it, so each service keeps its failure ledger in its own store and retries its parked aggregates itself, whether or not the ops service is deployed. The node that owns an entry's partition looks for due retries every 10 seconds (wow.subscription.parked.scan-interval). An entry gets 10 retries by default (max-retries), the first 3 minutes after the failure (initial-backoff), each next one twice as long after the one before. A retry claims its entry for 2 minutes (claim-ttl), so that no other node, such as the partition's next owner, starts it again meanwhile; the claim cancels nothing. A subscription's own values go under wow.subscriptions.[<context>.<processor.function>].parked.* (Parked aggregates). The ops service retries nothing: it sums up the parked entries of every service and is where operators act on them, not where failures live.
• Services publish their ledgers. A service sends each entry to the ops context as a command when it parks, when it spends its retries and after an operator's action on it, and again on a scan every 30 seconds (wow.ops.publish.interval), so the ops service catches up after an outage. It does so by itself once the ops service is deployed: with wow.ops.publish.enabled at auto, the default, it publishes while its topic transport (Kafka, or the in-memory topics in development) has the ops command topic, which the ops service creates, and checks at start and every wow.kafka.topics.partition-refresh (30 seconds). It logs ops takes the failure ledger now: <service> publishes it (the ops publication). when it starts, and ops does not take the failure ledger: <service> stops publishing it. when the topic goes away. Without a transport, auto publishes nothing; true publishes whatever it finds, and refuses to start without a transport; false never publishes.
• Ops never acts itself. An operator's action is recorded on the entry, and the ops saga sends it to the owning service as one of the ledger's system commands, wow.ledger.*, on that service's command topic. The owning service runs it on its own ledger and records what came of it. A service that publishes its ledger takes these commands from its command topic by itself, even when it takes no other command there (wow.kafka.command-intake.enabled off): its subscription wow.ledger.commands runs the ledger's commands alone, from the end of the topic. It accepts them only from the saga of the context wow.ops.publish.context names, ops by default, and warns of any other.

Each entry in the ops service is one parked aggregate of one subscription, with its status:

Status	Meaning
scheduled	Parked; the owning service will retry it automatically, with backoff.
exhausted	Its automatic retries are spent; it waits for an operator.
unrecoverable	Marked as not worth retrying automatically, or its source no longer holds what it is parked at, so it can only be skipped.
resolved	No longer parked: retried successfully, or skipped. Only the ops service shows it; the owning service has no entry left. Its failedAt is the latest parking resolved, which may be one whose record never reached the ops service; a record of that parking or an earlier one arriving late does not reopen it, a later parking does.

Deploy it

The ops service is a Wow service on MongoDB and Kafka (or PostgreSQL), behind CoSec, which serves its console as static files. deploy/ops holds its values for Wow's chart, two nodes on the same choices as the eShop services (Kubernetes):

1. Build the image with the console in it, or use a released image.
2. Provide MongoDB and Kafka; for PostgreSQL instead, pass values-postgres.yaml after values.yaml.
3. Create the topics the ops context owns: wow.ops.commands, wow.ops.receipts and wow.ops.parked.state. The command topic has exactly as many partitions as the ops context's log.
4. Put the CoSec gateway in front, and route the console and the ops routes through it: the gateway verifies each operator's access token, and the service reads the identity it forwards (CoSec-Identity), with no secret of its own (Secure a Deployment).

Then each service on the same Kafka starts publishing its ledger by itself within wow.kafka.topics.partition-refresh of the ops command topic appearing, and sends the entries it parked meanwhile; a service that should not appear in the console sets wow.ops.publish.enabled: false. The operators' actions arrive on each owning service's own command topic, which a service that publishes its ledger reads for them by itself.

Who may do what

The operator signs in to the console with a CoSec access token. The service reads the operator's tenant, subject and roles from the identity the gateway verified:

wow:
  cosec:
    mode: gateway
    # The operators and viewers, tenant-level callers (their scope has no owner): an operator acts on the entries, a
    # viewer reads them, and a principal with neither role reads nothing (403 wow.auth.forbidden). An entry is its
    # tenant's, so no caller's owner restricts its reads: the roles do. The services' ledgers publish to Wow's
    # default tenant, (0), which a principal of CoSec's default tenant is in.
    tenant-level:
      roles: ops.operator, ops.viewer

• ops.operator reads the entries and acts on them.
• ops.viewer only reads them.
• A principal with neither reads nothing: its queries of the entries and its state reads are refused with 403 wow.auth.forbidden. A request without a valid token is answered 401, and the console signs out.

The services' ledgers publish to Wow's default tenant, (0). Where CoSec decides the ops service's requests itself, with wow.cosec.authorizer.enabled: true (not behind the gateway, where the service has no CoSec decision), CoSec's documents decide who reads the entries and who is an operator instead of the roles, a read by its query route's path and each action by its command's action: ops:Retry, ops:Skip, ops:MarkRecoverable and ops:ApplyRetrySpec on parked/<id> (Action names). A service that serves its own operator routes (/wow/ops/parked, below) judges them as its wow.ledger.* commands. Write a policy for each route you expose. Whoever decides, no caller may send the records (ops.parked.RecordParked, RecordResolved, RecordActionRefused): only the services' ledgers and the ops saga produce them. They are internal commands (Keep a command internal): the public port answers their routes 404, as any unknown path, and on the internal port the ops service refuses them to every caller, another service too.

Work with the console

• Overview. The home page: how many entries are parked in a time window and overall, how many need attention, have a retry scheduled, are unrecoverable or resolved; parked entries over time; the top failure clusters, contexts and failing subscriptions. Each number opens the queue it counts. It refreshes every 15 seconds.
• Queues. The parked entries as record views: Needs attention (exhausted or unrecoverable), Scheduled (a retry is due), Unrecoverable, Resolved, Parked and All, and analyses of every parked entry: Parked by context, Failure clusters (grouped by error type, context and subscription), Top failing subscriptions and Parked over time. Every field can be filtered, sorted and grouped. The ops service keeps no views, so the views an operator saves stay in this browser: nobody else sees them, and the console offers no link to one.
• Detail. One entry: the error type and message as the owning service recorded them, the aggregate and the version it parked at, its attempts against its retry spec, the last operator request and how it was resolved, then its history as a timeline.
• Actions. Retry with a reason, skip with a reason, mark recoverable, apply a retry spec (the entry's most retries, initial backoff and claim TTL, its durations as PT3M, 3m or 1h 30m). The ops service checks a retry spec as the command's input: a negative count or a duration Kotlin's Duration.parse does not read as finite and not negative is refused with wow.command.invalid (400), each under its field in violations, and a count past a 32-bit integer as wow.command.malformed. The console checks the same before sending, and when the service refuses what was typed anyway, the form comes back with it and the reason under the field. Each asks for confirmation and works on one entry or a selection. Once the ops service has recorded the request, the console says it was sent to the owning service; what the owning service made of it arrives as the entry's next record, a resolved entry or a new status, or as a refusal shown with its error code. A resolved entry takes no action. A retry or a skip names the parking the console showed: when the aggregate was resolved and parked again meanwhile, the ops service refuses it as ops.parked.stale, and the owning service as wow.ledger.stale, so an action never skips events nobody looked at. Read the entry again and act on what it shows now.

Retrying re-reads the aggregate's events from the version it stopped at and hands them to the subscription again, including events it had not reached yet. Skipping moves past them. Choose skip only when you know the subscription's effect for those events is not needed, or you have made up for it some other way.

The console does not show stack traces: a service records a failure as its exception type and message, which is what an operator groups and searches by.

Without the ops service

A single service can expose its own parked entries: wow.http.ops.enabled: true serves /wow/ops/parked on its public port, with the same four actions, retry, skip, mark-recoverable and apply-retry-spec. It is off by default. An entry's path names its subscription by its key, <context>.<processor.function>, then its aggregate: /wow/ops/parked/{subscription}/{context}/{aggregate}/{tenant}/{id}. Reading is authorized as the system command wow.ledger.Read, and each action as its wow.ledger.* command, by the application's CommandAuthorizer: without one, the reads and the actions are all refused with 403, and the host warns at startup.

• A skip names the parking it skips: its body is {"failedAt": <epoch ms>}, the entry's failedAt as the operator read it, and a skip without it is refused with 400. A skip discards what was parked, so it must be what someone looked at.
• A retry may leave failedAt out and act on the entry as it is parked now, since it only hands the parked events to the subscription again.
• Either is refused as 409 wow.ledger.stale when the entry it names was resolved and parked again since.
• Where it runs. A retry or a skip answers 200 when this node ran it, with the outcome resolved or parked. On a node that does not own the entry's partition, it leaves the request on the entry and answers 202 with the outcome requested: the owner runs it on its next scan of the parked entries, every 10 seconds by default. Marking an entry and applying a retry spec write the ledger at once; the owner retries by them from its next scan.

Values on the wire are lowercase, as everywhere in Wow: the outcome, and an entry's status, request and recoverable.

The same switch serves GET /wow/ops/subscriptions, a read-only view of the node's subscriptions: each by its key, with its group, its source (log or topic), its parked count, and per partition its position and, where this node runs it, its head, lag, and whether it is paused or throttled. It lists too the checkpoints its groups hold under names no subscription has any more (orphans), what a rename leaves behind (Rename and rewind). The view is the whole service's, so its caller must reach every tenant (403 otherwise); a route authorizer then decides it as the route ops.subscriptions of the context wow (wow:ops.subscriptions in CoSec), and without one every such caller reads it.

Rename and rewind

A subscription's name is its identity: its context, its processor's name and its handler's, <context>.<processor.function>, the processor's @Name or class name and the handler's @Name or function name. Its checkpoint, its parked entries and its settings are kept under that name.

• A rename is a new subscription. Renaming the class or the function, or moving the processor to another context, gives the handler a new name with no checkpoint: it starts where a new subscription of its kind starts, a saga or a reactor at the end of what is already there (it reads nothing it missed), a projection at the beginning (it rebuilds its read model). The old name's entries stay parked under it, and nothing reads them again.
• To keep the position, pin the name: give the renamed class or function the old name with @Name. There is no alias.
• Each node warns at its first start of the checkpoints it finds under names it has no subscription of, with each name's positions and how many entries are parked under it: Subscription group <group> has checkpoints of subscriptions it no longer has: …. GET /wow/ops/subscriptions lists them as orphans. Pin the name if the rename was not meant to restart the subscription; otherwise leave them.
• A new name is the only rewind. A checkpoint only moves forward: Wow has no operation that sets it back. To handle a stream again from an earlier point, deploy the handler under a new name and set where it starts, wow.subscriptions.[<context>.<processor.function>].start: beginning, end or an instant (Rebuild and Replay). Mind its effects: a saga or a reactor under a new name sends its commands and effects again for what it reads.

Webhooks

Each service can also notify a webhook when an aggregate parks or spends its retries: set wow.ops.webhook.url, with wow.ops.webhook.format generic for Wow's JSON or wecom for a WeCom (WeChat Work) robot, and wow.ops.webhook.events to choose between parked and exhausted.