runtime/me.ahoo.wow.runtime.cluster

Package-level declarations

Types

class CallerWire @InternalWowApi constructor(val tenantId: String = Caller.DEFAULT_TENANT, val scope: Scope = Scope.NONE, val subject: String? = null, val crossScope: Boolean = false, val attributes: Map<String, List<String>> = emptyMap(), val service: Boolean = false)

Who sent a command, in its wire form (CommandWire.caller): the fields of a Caller, {tenantId, scope, subject, crossScope, attributes, service}, and nothing else: no credential, token or request header. Its toString leaves the attributes out, the caller's claims.

@InternalWowApi

fun interface ClusterChannel

The cluster channel: the one internal channel between the nodes of a service, listening on their internal port only and trusted without authentication. This port is what the runtime needs of it; a transport (HTTP/2 with JSON, in the host) implements it, reaching a node by its internal address.

ClusterChannelException

@InternalWowApi

class ClusterChannelException(message: String, cause: Throwable? = null, val unsent: Boolean = false) : RuntimeException

A peer could not be reached over the cluster channel. unsent says the request surely did not reach the peer (no connection to it, refused or not opened in time, or not sent by its deadline), so the sender may run the command itself while its deadline is ahead; any other failure of a forward may have come after the peer received it.

ClusterChannelHandler

@InternalWowApi

fun interface ClusterChannelHandler

What a node serves on the cluster channel: the forwarded commands it executes.

@InternalWowApi

object ClusterCodec

Reads and writes the messages between the nodes of a service as JSON: the cluster channel's ForwardCommand and ForwardReply, and the WaitSignal a node sends to the origin of a wait, reading leniently: unknown fields are ignored at every level, and an unknown status, stage or function kind of a result is read as UNKNOWN (WowJson.readerBuilder), so a node of the minor version before reads a newer one's.

class ClusterOptions @InternalWowApi constructor(val leaseTtl: Duration = 10.seconds, val renewInterval: Duration = 3.seconds, val memberTtl: Duration = 10.seconds, val heartbeatInterval: Duration = 3.seconds, val reconcileInterval: Duration = 1.seconds, val routeRefreshInterval: Duration = 1.seconds, val forwardTimeout: Duration = WaitPlan.DEFAULT_TIMEOUT, val maxInFlightPerPeer: Int, val drainTimeout: Duration = 10.seconds, val leaseParallelism: Int = 16, val reconcileTimeout: Duration = 1.minutes)

The tuning of partition ownership and the cluster channel on one node (–).

@InternalWowApi

fun interface ClusterStream

One stream to a peer, taken by ClusterChannel.reserve for one forward: forward sends it, after which the channel frees the stream once the reply ended it; release frees a stream no forward was sent on.

@InternalWowApi

fun interface CommandRouter

Decides where a command runs: here, through local, or on another node. Whatever runs it, the answer is the Settlement of its execution.

class CommandWire @InternalWowApi constructor(val type: String, val id: String, val requestId: String, val ref: AggregateRef, val body: JsonNode, val expectedVersion: Int? = null, val scope: Scope = Scope.NONE, val headers: Map<String, String> = emptyMap(), val causation: Causation? = null, val operator: Operator? = null, val replyTo: ReplyTo? = null, val caller: CallerWire? = null)

A command in its wire form: { type, id, requestId, ref, body, expectedVersion, scope, headers, causation, operator, replyTo }. type is the command type's model name, which the receiver reads body by; the wait plan does not travel, only a wait context does.

@InternalWowApi

data class ForwardCommand(val v: Int = VERSION, val command: CommandWire, val transport: Transport? = null, val deadline: Long)

The cluster channel's one message kind for now: a node forwards command to the owner of its partition and waits for the reply until deadline (epoch milliseconds). transport carries the command's wait context and the trace context it executes in, in the one form of a transport (TransportCodec); null when it has neither. The wait's origin stays the node the command arrived at, so later hops signal that node directly, never through the channel.

ForwardEndpoint

@InternalWowApi

class ForwardEndpoint(coordinator: WaitCoordinator, models: Collection<AggregateModel<*>>, groups: Collection<PartitionOwnership>, clock: Clock = Clock.systemUTC(), observer: Observer = Observer.NONE) : ClusterChannelHandler

What a node serves on the cluster channel: it executes the commands other nodes forward to it through coordinator (so its own wait context goes on with the command and later hops signal the origin), for the aggregates of models.

@InternalWowApi

data class ForwardReply(val v: Int = VERSION, val result: CommandResult? = null, val reached: List<CommandResult> = emptyList(), val refused: String? = null)

The owner's reply to a ForwardCommand: the result its execution reached at COMMITTED (a commit, a duplicate, no change, a rejection or a failure), followed by the later hops it reached at once (the next HANDLED hop of a command that changed nothing, or of a duplicate whose subscription is past it). Hops reached later are signalled to the wait's origin.

@InternalWowApi

object Hrw

Highest random weight (rendezvous) assignment of partitions to nodes: owner(p) = argmax weight(node, p). Every node computes the same assignment from the same member list without a leader, and a change of members moves only the partitions the joining node wins or the leaving node held, about P/N of them.

class LeaseGroup @InternalWowApi constructor(val service: String, val context: String)

The lease group of one bounded context in one service: the nodes of service share the partitions of context's log, and command routing and every subscription of the context share their owners. A service that hosts several contexts has one group, with its own partitions and leases, per context.

@InternalWowApi

object NodeAddress

A node's internal address as its members read it (me.ahoo.wow.spi.subscription.Member.address) and the internal port's clients join it into a URI: host:port, an IPv6 literal in brackets (RFC 3986), [::1]:9090, so that a colon of the host is never read as the port's. The one place an address is built from a host and a port, or its host read back; everywhere else an address is compared as it is.

PartitionedRouter

@InternalWowApi

class PartitionedRouter(groups: Collection<PartitionOwnership>, channel: ClusterChannel, options: ClusterOptions = ClusterOptions(), observer: Observer = Observer.NONE, clock: Clock = Clock.systemUTC()) : CommandRouter

The Partitioned router: a command runs on the owner of its log partition, whose state cache is warm and who appends without contention.

PartitionOwnership

@InternalWowApi

class PartitionOwnership

This node's share of the partitions of one lease group (–), without a leader: a context's log, or another context's published topic that the service's subscriptions read, whose subscriptions runtime then reads the topic instead of the log:

@InternalWowApi

object Refusal

Why a node refused a forwarded command.

@InternalWowApi

data class RenewalRound(val group: LeaseGroup, val node: String, val leases: Int, val lost: Int, val failed: Int, val duration: Duration)

One renewal round of node in group: the leases it held, of which lost were found already expired or taken, or went unrenewed for leaseTtl − renewInterval, and failed threw, and how long the round took. A round longer than leaseTtl − renewInterval lets the node's own leases expire between renewals.

class ReplyTo @InternalWowApi constructor(val context: String, val subscription: String, val ref: AggregateRef, val version: Int, val committed: Boolean = false)

Where the receipt of a cross-context command goes when the target rejects it: the receipts topic of context, the sender's, and the saga handler subscription (processor.function) whose execution of the commit ref@version derived it, the executionId the saga's @OnRejected handler gets. The design's {context, subscription} carries the commit too, so the receipt names the execution exactly.

@InternalWowApi

data class RouteTable(val owners: Map<Int, Member> = emptyMap(), val members: List<Member> = emptyList())

Who owns which partition of a lease group, as one node last read it: the live owners by partition, each a live member with its address, and the live members. A lease whose holder is no longer a member is left out: its owner is unknown, so a command for it runs locally.

ServiceMembership

@InternalWowApi

class ServiceMembership(val service: String, val node: String, addressOf: () -> String, leases: LeaseStore, options: ClusterOptions = ClusterOptions(), context: CoroutineContext = Dispatchers.Default)

One node's membership in its service on one lease store: a single record, heartbeated every ClusterOptions.heartbeatInterval, listing the lease groups the node takes part in, and a single read of the service's live members every ClusterOptions.reconcileInterval (or ClusterOptions.routeRefreshInterval, the shorter), which every PartitionOwnership of the node reuses for its reconcile and its routes instead of reading its own. The store operations of membership so stay constant in the number of lease groups: a heartbeat and a read per node and interval, where each group used to heartbeat, and read its members twice, on its own.

@InternalWowApi

class TopicRouter(val types: Set<AggregateType>, val topic: String, partitions: Int, producer: TopicProducer)

The topic router of one context: the commands of its types are admitted by writing them to the context's command topic, whose partitions are the context's log partitions (partitions), each in the one the log's Partitioner gives its aggregate, so the owner of the aggregate's log partition reads and executes it (me.ahoo.wow.runtime.command.CommandIntake) with no forwarding, and one aggregate's commands keep their order.