PartitionOwnership

This node's share of the partitions of one lease group (–), without a leader: a context's log, or another context's published topic that the service's subscriptions read, whose subscriptions runtime then reads the topic instead of the log:

• Membership is the node's in its service (ServiceMembership), shared by all its groups: one record listing the groups it takes part in, heartbeated every ClusterOptions.heartbeatInterval, and one read of the members per interval, which this group's reconcile and routes reuse. The group's members are those listing it. A reconcile that runs longer than ClusterOptions.reconcileTimeout, stuck stopping or starting a partition, takes the group off the node's record until the reconcile ends, so the others take the partitions it cannot run instead of handing them back to it. One still running after wow.health.liveness.stall-timeout fails the node's liveness, so its orchestrator restarts it: such a reconcile waits on a call that never returns, which only a restart ends.

• Reconciling: every ClusterOptions.reconcileInterval, and at once when it notices the members changed, the node computes its target, the partitions HRW (Hrw) gives it among the live members, hands off the partitions it holds outside it and acquires those inside it. Members that briefly disagree are arbitrated by the leases and converge on the next round; a partition the old owner still holds is acquired once it is released or expires.

• Leases are renewed every ClusterOptions.renewInterval, counted from the start of the previous round, all held leases of a round with one LeaseStore.renew, so a round does not grow with the number held times the store's latency and asks the store's pool for one connection, not one per lease. Renewing starts before the first acquisition, and covers a partition from the moment its lease is taken, while its runtimes, intake and timers start: a start slower than the margin keeps its lease. Each round is reported to renewalRounds and logged, as a warning when it took more than half the margin leaseTtl − renewInterval. A lease lost to expiry stops the partition here, and so does one left unrenewed for leaseTtl − renewInterval (the store unreachable, say), without a release; the next reconcile acquires it anew.

• Hand-off of a partition goes in this order: stop accepting new commands for it, stop its command intake, which settles the commands it took and saves its checkpoint, wait for its other commands in flight for at most ClusterOptions.drainTimeout, stop its subscriptions, which saves their checkpoints (SubscriptionRuntime.stopPartition), and release the lease, so the new owner acquires it on its next round. State caches are not handed over: the new owner loads from the snapshot and the log's tail, and each node drops the partition's cached states whenever it acquires, hands off or loses it (partitionChanged).

• Routes: every ClusterOptions.routeRefreshInterval, and at once on refreshSoon, the node reads the owners into its routes, beside the members its membership read last. Each other member they show at an address not seen before is handed to peers, apart from the refresh: the node connects to it there (ClusterChannel.connect) before forwarding to it.

Acquiring a partition starts its subscriptions in subscriptions and its other log groups (SubscriptionRuntime.startPartition), then the same partition of the context's command topic in intake, then its timers, before the node accepts commands for it: while they start, the partition is still pending, its lease renewed with the others. A hand-off stops the timers first, once the node stops accepting, so it sends no command it would not run. The partitions the first reconcile of a start acquires read nothing, their entries or their parked retries, until it has started all of them, or until the job start is given completes: a backlog the first of them would read at once (a history migrate-data imported, a long stop) would slow down the starts of the others. Without a job, the hold lasts at most ClusterOptions.reconcileTimeout, so a reconcile stuck starting one partition does not hold the others back. The partitions later reconciles acquire read at once.

stop is a graceful leave: the node leaves the members at once, so the others take its partitions on their next round, then hands off every partition it holds.

stepOut is the leave of a node that met a fatal error (me.ahoo.wow.runtime.health.NodeFatals): for good, within a bound, and whatever the hand-off comes to, renewing no lease from then on, so a partition it could not hand off is the others' once its lease expires. It keeps reading the routes, so a command the node is still given goes to the partition's new owner.