FailureLedger
The failure ledger of one service (documentation/designs/04-subscriptions.md §2).
When a subscription has retried an aggregate's entry past its limit, it parks the aggregate: the ledger records (subscription, aggregate, from version, reason) and the subscription moves on with every other aggregate, its checkpoint passing over the parked aggregate's later entries. Nothing of those entries is copied: to retry, the subscription reads the aggregate's streams from the log from ParkedAggregate.fromVersion on, so the ledger grows with the number of parked aggregates, not with what was written while they were parked.
The ledger keys an entry by its subscription's identity, its context and name (SubscriptionId), and its aggregate: names are unique within a context only, and the contexts of a service whose stores are one store share one ledger, so two subscriptions of one name in two contexts never see, count, amend or resolve each other's entries.
The ledger lives in the failing service's own storage; wow-ops only aggregates it. Its writes are idempotent: parking an aggregate that is already parked keeps the earliest from version, so a replayed or duplicated park never moves the retry point past an entry that was not handled.
An entry also carries its long-horizon retry state: the automatic retries made so far and when the last one started, the retry spec an operator applied to it, whether it is recoverable, the request an operator left for the owner of its partition, and when that owner next looks at it (ParkedAggregate.nextAttemptAt). park and repark write only where the aggregate is parked and why (a new entry's retry state comes with it); the retry state changes only through amend, where the import of 9.x compensations also restates why (ParkedAmendment.Restate).
Every stored change of an entry gives it the next ParkedAggregate.revision, the ledger's concurrency token: a new entry is at 0, and a repark, a park that replaces an entry, and every amendment add one. An amendment that acts on what its caller read (ParkedAmendment.ClaimRetry, ParkedAmendment.Schedule, ParkedAmendment.TakeRequest, ParkedAmendment.Restate) names the revision read and applies only at it, so nothing written since, an operator's change or a repark among them, is ever covered by a decision made before it. Every change but a schedule and a take also clears ParkedAggregate.nextAttemptAt, so an owner's write that loses to one finds the entry due on its next scan.
The owner of a partition reads only what is due there (due), and the gauges count the entries without reading them (count): a ledger answers both from an index, on (subscription, partition, next attempt), so neither grows with the entries that are not due, nor with the nodes of the service. A listing of one tenant's entries (list with a tenant) is answered from an index too.
Functions
Changes the retry state of the entry parking ref for subscription as amendment says, in one write, giving it the next revision; a restate (ParkedAmendment.Restate) also changes why it is parked. A claim of an automatic retry (ParkedAmendment.ClaimRetry), a schedule (ParkedAmendment.Schedule), the take of a request (ParkedAmendment.TakeRequest) or a restate applies only at the revision its caller read, so of two nodes that both read it (an owner handing its partition off and the next one) only one claims or takes it, and nothing written since the read is covered.
How many aggregates subscription has parked, counted up to max: max when there are more.
Up to limit of subscription's entries in partitions that the owner of their partition looks at by now (epoch milliseconds): those whose ParkedAggregate.nextAttemptAt is null (new, retried, or changed by an operator since the owner last looked) or not after now; at ParkedAggregate.NEVER, every entry of partitions, as a node that acquires a partition reads them. Ordered as list orders them, after after (exclusive; null from the start): pass the last entry's ref as after to read the next page.
Whether subscription has parked ref: its entries are not handled until the entry is resolved.
Up to limit aggregates subscription has parked, ordered by aggregate (context, aggregate name, tenant, id, each compared by code point), starting after after (exclusive; null from the start), only tenantId's when given. Pass the last entry's ref as after to read the next page.
Parks entry's aggregate for its subscription. When the aggregate is already parked, the entry with the earlier from version is kept (on a tie, the one already there), its partition and from position with it; an entry replaced this way keeps its retry state, takes the next revision and is no longer scheduled; a new one takes entry's retry state, at revision 0.
The entry parking ref for subscription, if any.
Replaces where the entry parking entry's aggregate stands (its partition, from version and position), why (its reason) and when (its failure time) with entry's, while the aggregate is parked in partition from; it never parks one that is not. It keeps the entry's retry state, gives it the next revision and clears its schedule. Its callers:
Resolves the entry parking ref for subscription, as resolveAnywhere does, while the aggregate is parked in partition: the worker of that partition resolves it after its retry handled every parked stream it read there, or as an operator's skip. An entry re-homed to another partition meanwhile (repark) stays: its streams there are that partition's worker's.
Resolves the entry parking ref for subscription by removing it, after the aggregate was retried successfully or the operator chose to skip it: either way the subscription handles the aggregate's entries again. Which of the two happened is the caller's to publish; the ledger only holds what is still parked.