HttpClusterChannel

The ClusterChannel over HTTP: a forward is a POST of its JSON to InternalHttp.FORWARD_PATH on the peer's internal port, through the JDK client (HTTP/2 over h2c).

The client's first request to a peer opens the connection and upgrades it to h2c, which on a node that started a moment ago takes longer than a forward's timeout. connect opens it with a request of its own, a HEAD of InternalHttp.FORWARD_PATH that the peer answers without running anything (with any status), bounded by connectTimeout: concurrent callers share one, and a peer once connected is not asked again until a forward to it fails for want of a connection. A peer that cannot be reached in time is a ClusterChannelException.

A peer's connection carries at most InternalHttp.MAX_CONCURRENT_STREAMS requests at a time, which the JDK client does not queue past. The channel keeps at most streams forwards open to a peer, leaving the rest of the peer's streams to the node's HttpSignalSender. A forward reserves one of them before it is sent, waiting for one, in order, with no bound of its own: the router bounds the wait by the forward's deadline, and a forward that got none by then was not sent, which the router answers as such (withdrawn or busy), never by running it locally, which near the deadline would put a second writer beside the owner. One the JDK still refuses for the peer's limit, a stream just ended that it has not released yet, is sent again on its stream after a short pause, until the deadline, and is then unsent too.

The caller bounds a forward (me.ahoo.wow.runtime.cluster.PartitionedRouter), and the channel stops waiting for the reply answerGrace after the forward's deadline too, so a forward never outlives that. Its stream stays open, counted among streams, until the owner's answer ends it, which comes about the deadline (the owner withdraws a command still queued at its wait's deadline), and the router waits for that answer as long. Only an owner that has not answered answerGrace after the deadline has the stream reset: resetting each stream at the very deadline it is answered at made the peer's server close the whole connection under load, failing every forward on it (InternalHttp.ANSWER_GRACE).

Whatever keeps a reply from arriving (no connection, no reply by the deadline and its grace, a reset stream or a closed connection, a status other than 200, a body that is not a reply) is a ClusterChannelException. Only one raised before the request was sent (no connection to the peer, refused or not opened in time, or refused by the JDK for the peer's limit until the deadline) is unsent, which the router degrades to running the command locally while the forward's deadline is ahead; nothing after the send is, whose outcome is unknown.

Each forward names the node's service in Wow-Service. A peer that answers 401 or 403 did not take the forward, being a node of another service: a refusal, which is unsent too, and logged as a warning, as the nodes are configured apart.