HttpSignalSender
Sends wait signals to the internal signal entry (InternalHttp.SIGNALS_PATH) of the node a wait started on, at its Origin.address: the remote of a me.ahoo.wow.runtime.wait.LocalSignalSender. A signal is written by ClusterCodec, as every message between the nodes of a service.
send never waits: it puts the signal in its origin's queue of queueCapacity and returns. Each origin's queue is sent from by at most concurrency requests at a time. A request carries what the queue holds then, up to maxBatch signals: under a burst, one request per batch instead of one per signal; a lone signal goes as before. An origin that refuses a batch as malformed (HTTP 400: a node of an earlier version reads only single signals) is sent one signal a request from then on.
A request is given until the latest deadline of the waits its signals end, as a forward is: a signal is worth delivering while its waiter waits, so an origin slow to take it, its event loops lagging under load, still ends the wait. A fixed limit shorter than the wait dropped such signals, and their waiters answered a committed command as timed out (item 1).
Each origin's concurrency is its share of the streams of the one connection to the origin, which the node's HttpClusterChannel shares (InternalHttp.MAX_CONCURRENT_STREAMS): the channel leaves the default to the signals.
A request that fails for a while only, its origin's address not resolving or refusing connections yet, its connect timing out, the connection's streams all taken (too many concurrent streams: the request was not sent), or the origin answering 503 or 429 (a partition owner being replaced, a node starting), is sent again after retryBackoff, doubled each time up to MAX_RETRY_BACKOFF, at most maxAttempts times in all, and never past the latest deadline of its waits: the signal of a committed command then still ends its wait, where one failure answered it as timed out. The origin deduplicates a signal it took twice, as one sent by two nodes, so a request that reached it and failed after is safe to send again. An origin's definite answer, any other status, is never retried.
A signal is dropped when its queue is full, when its wait is over before it is sent (ObservedEvent.SignalDropped.Reason.EXPIRED), when its request fails otherwise, or after its last attempt, or outlives its waits, or when the origin answers other than 2xx: a lost signal costs its waiter a timeout, never a fact. Every drop is counted (dropped, and observer's ObservedEvent.SignalDropped by reason) and logged, at WARN the first time for an origin and every WARN_EVERYth time after, at DEBUG otherwise: never silent, never a log line per signal of a burst.
An origin's queue goes away after idleTimeout without signals, so the origins of nodes long gone do not accumulate. close stops sending; what is queued then is dropped.
Every signal names this node's service in Wow-Service, whether its origin is a node of this service or of another one, the origin of a wait that crossed into this one: the signal entry takes a signal from a service that names itself, the internal port being reachable by the platform's own pods alone.