Kubernetes

A Wow service runs on Kubernetes as a Deployment: its pods are the service's nodes, they find each other through the service's database, and other services reach them through a headless Service. Nothing else is needed: no operator, no ZooKeeper, no Kafka consumer group. Wow's Helm chart, wow-service, renders those objects for one service, and is released with Wow at Wow's version. A platform installs it once per service. This page installs the eShop's ordering service with it; Deployment Model explains the ideas behind it.

Install a service

A release takes two values files, the second over the first:

1. The values the build knows. ./gradlew :<server>:generateWowHelmValues writes them from the service's contract into build/wow/helm/values.yaml: the service's name, after which the chart names every object; the peers whose contracts are on its class path, each with the service that hosts it; and the route its nodes serve the Manifest at. They follow the code, so never copy them by hand. The basket's:

   service:
     name: "eshop-basket-service"
     peers:
       - context: "eshop-ordering"
         service: "eshop-ordering-service"
     manifest:
       route: "/wow/manifest"

2. The deployment's own. The image, the ports, the replicas, the platform, the profiles, the stores and their secrets, the resources. The ordering service's, in deploy/eshop-ordering:

# The eshop-ordering service's own values for the wow-service chart (deploy/helm/wow-service), on MongoDB and Kafka:
# what the build does not know. Pass them after the values its build generates, so they override them:
#
#   ./gradlew :eshop-ordering-server:generateWowHelmValues
#   helm install eshop-ordering deploy/helm/wow-service \
#     -f examples/eshop/ordering/ordering-server/build/wow/helm/values.yaml -f deploy/eshop-ordering/values.yaml
#
# values-postgres.yaml after these puts the service on PostgreSQL instead.
image:
  # The image built from this repository (README.md).
  repository: wow-eshop-ordering-server
  tag: local
# The ports of the image's own configuration (application.yaml), which the chart sets on the pod.
ports:
  http: 8083
  internal: 9093
# The nodes share the partitions through leases in the service's database: no ZooKeeper, no consumer group.
replicas: 2
# The internal port admits the pods of the platform, labelled app.kubernetes.io/part-of: eshop.
partOf: eshop
config:
  # The mongo profile keeps the service's data on MongoDB, the replica set at secret.yaml's URI (deploy/mongodb), in a
  # database of its own; the kafka profile connects it to the other contexts through Kafka; the cosec profile takes the
  # callers from the CoSec gateway in front of the service.
  SPRING_PROFILES_ACTIVE: mongo,kafka,cosec
  WOW_KAFKA_BOOTSTRAPSERVERS: kafka:9092
  # The topics the service owns are created by the operator; the node only validates them. The kafka profile's auto
  # is for trying the service out.
  WOW_KAFKA_DDL: validate
  # Fixed once the context's log holds entries; the command topic wow.eshop-ordering.commands has as many partitions.
  WOW_LOG_PARTITIONS: "64"
  # Traces and metrics: the node's spans and Micrometer meters go through the OpenTelemetry Java agent, which the
  # OpenTelemetry Operator injects (podAnnotations, with an Instrumentation resource in the namespace). Without the
  # agent they are no-ops. It exports to the collector below over OTLP, and bridges Micrometer's global registry.
  OTEL_EXPORTER_OTLP_ENDPOINT: http://otel-collector:4318
  OTEL_EXPORTER_OTLP_PROTOCOL: http/protobuf
  OTEL_INSTRUMENTATION_MICROMETER_ENABLED: "true"
# The database's connection: secret.yaml, an example to replace.
envFrom:
  - secretRef:
      name: eshop-ordering-service-mongo
podAnnotations:
  instrumentation.opentelemetry.io/inject-java: "true"
resources:
  requests:
    cpu: 500m
    memory: 768Mi
  limits:
    memory: 1Gi

Then install it from the OCI registry, at the Wow version the service is built with, or from a checkout of the repository (deploy/helm/wow-service):

helm upgrade --install eshop-ordering oci://ghcr.io/ahoo-wang/wow-service --version <wow-version> \
  -f examples/eshop/ordering/ordering-server/build/wow/helm/values.yaml \
  -f deploy/eshop-ordering/values.yaml --wait

The chart is also on oci://registry-1.docker.io/ahoowang/wow-service and oci://registry.cn-shanghai.aliyuncs.com/ahoo/wow-service. Its values.schema.json holds every value to the keys the chart declares: a misspelt key fails the install, as a node refuses a wow.* key it does not know. The chart's README lists every value.

Before you install

1. Build the image and push it, or load it into a local cluster.
2. Provide the database and, when the service uses it, Kafka. deploy/mongodb holds a MongoDB replica set to start from; for PostgreSQL, pass values-postgres.yaml after values.yaml, and set the server as PostgreSQL says.
3. Create the Kafka topics the service owns, since values.yaml sets WOW_KAFKA_DDL=validate: a command topic needs exactly as many partitions as the log (Kafka).
4. Create the secrets envFrom names, from the examples beside the values (secret.yaml, postgres-secret.yaml) or your own store.

deploy/eshop installs the whole eShop, a release per service.

What it renders

Every object is named after service.name, the service's spring.application.name, never after the release:

Object	Name	Holds
Deployment	<name>	The nodes
ConfigMap	<name>-config	config as environment variables, settings as SPRING_APPLICATION_JSON, javaOpts, the shutdown's bounds
Service	<name>	The public HTTP port
Service	<name>-cluster	The internal port, headless
NetworkPolicy	<name>-public, <name>-internal	Who may reach each port
PodDisruptionBudget	<name>	How many nodes a voluntary disruption may take
ServiceAccount	<name>	No token mounted: the nodes call no Kubernetes API
HorizontalPodAutoscaler or ScaledObject	<name>	Optional: Scale
ServiceMonitor or PodMonitor	<name>	Optional: Monitor; a ServiceMonitor comes with a headless Service <name>-metrics

The image

The image runs the service's Gradle distribution (./gradlew :eshop-ordering-server:installDist) on a JDK 25 base, pinned by digest, as a numeric non-root user, and exposes the HTTP port and the internal port. The chart sets both ports on the pod (SERVER_PORT, WOW_INTERNAL_PORT, from ports), so the Services, the probes and the image's configuration cannot disagree.

The JDK

Wow is built for JDK 21 and runs on it, but run it on JDK 25 in production. The backends make their blocking driver calls (JDBC, the MongoDB and Kafka clients) on virtual threads. Before JDK 24, a virtual thread that blocks inside synchronized code, as drivers and Java agents often do, pins the carrier thread it runs on. Enough of them at once leave no carrier for the others: every store and topic call of the node stops returning, its leases lapse, and the other nodes take its partitions. The node's liveness then fails after wow.health.liveness.stall-timeout, and Kubernetes restarts it (Health). JDK 24 and later let those threads unmount. A node on an earlier JDK warns of this once when it starts.

Memory

The chart's default javaOpts is -XX:MaxRAMPercentage=75.0 -XX:+ExitOnOutOfMemoryError: the heap is three quarters of the container's memory limit, and an OutOfMemoryError ends the process, so Kubernetes starts the pod again and the other nodes take its partitions over meanwhile. The images Wow builds set -XX:+ExitOnOutOfMemoryError too. A javaOpts of your own replaces the default: keep -XX:+ExitOnOutOfMemoryError in it. Without it the node still steps out of its lease groups on an OutOfMemoryError its own work meets, and its liveness fails until the process ends (Fatal errors), but an error elsewhere in the process, a request's say, leaves it to the JVM.

Size the heap for what a node holds at its peak, not for its average: the state cache, a tenth of the heap unless wow.state-cache.capacity bounds it (What Wow derives); the commands in flight, up to wow.executor.max-in-flight, each with the state its aggregate loads, which for an aggregate with a long history and no recent snapshot means its events replayed; and the entries each subscription holds in each partition it runs, up to its in-flight bound. A memory limit the heap cannot fit those in under load turns overload into restarts.

Shared settings

Every node of a service gets the same configuration from the ConfigMap, as environment variables that Spring Boot binds relaxed: config in the values, as the ordering's above shows, in the canonical form (wow.log.partitions is WOW_LOG_PARTITIONS). A key whose map key an environment variable cannot carry, one with dashes or dots, goes under settings as a tree, which the chart renders as SPRING_APPLICATION_JSON. The database's connection comes from a Secret, through envFrom. The internal port needs no secret: the network is its boundary.

The nodes

# A new node joins before an old one leaves: the partitions move to it through the leases, and none is left without
# an owner while the rollout runs.
strategy:
  type: RollingUpdate
  rollingUpdate:
    maxSurge: {{ .Values.rollout.maxSurge }}
    maxUnavailable: 0

Two replicas are two nodes, which split the log's partitions through leases in the database. A rolling update starts a new pod before it stops an old one (maxUnavailable: 0), so the partitions always have an owner. Each node needs an identity and an address its peers can reach, which the chart gives it from the pod:

env:
  - name: POD_NAME
    valueFrom:
      fieldRef:
        fieldPath: metadata.name
  - name: POD_IP
    valueFrom:
      fieldRef:
        fieldPath: status.podIP
  # The ports the Services and the probes name: the chart's, whatever the image's configuration says.
  - name: SERVER_PORT
    value: {{ .Values.ports.http | quote }}
  - name: WOW_INTERNAL_PORT
    value: {{ .Values.ports.internal | quote }}
  # The node id is the pod's name: a replaced pod is a new member.
  - name: WOW_NODE_ID
    value: $(POD_NAME)
  # The internal port binds the pod's private address only, and the node advertises that address: a
  # Deployment's pods have no DNS name of their own.
  - name: WOW_INTERNAL_HOST
    value: $(POD_IP)
  # The namespace a peer's SRV name is in, unless wow.peer.dns.namespaces places the peer elsewhere.
  - name: WOW_PEER_DNS_NAMESPACE
    valueFrom:
      fieldRef:
        fieldPath: metadata.namespace
  {{- with .Values.env }}
  {{- toYaml . | nindent 12 }}
  {{- end }}

• The node id is the pod's name. A replaced pod joins as a new member; the old one's leases expire.
• The internal port binds the pod's own address, and the node advertises that address, <pod IP>:<internal port>: the host's default, so the chart sets no wow.node.address. A Deployment's pods have no DNS name of their own: a headless service publishes <pod>.<service> records only for pods with spec.hostname, which a Deployment cannot give each pod. Set wow.node.address only where the pod's own address is not the one its peers reach, such as a StatefulSet whose pods have stable names (serviceName and hostname) or a host with several interfaces.
• The namespace comes from the pod (WOW_PEER_DNS_NAMESPACE): the namespace the node looks its peers up in.

Two services

# The internal port's headless Service, named after the service with -cluster: its SRV records,
# _internal._tcp.<service>-cluster.<namespace>.svc, name every node's address, the address each node advertises (its pod
# IP), and the other services find the nodes through them. It publishes the nodes that are not ready, so a node that is
# starting or draining is still found. Never expose it outside the cluster: it takes no token, each node and service
# names itself on each request (Wow-Service), and the internal NetworkPolicy admits the platform's own pods alone.
apiVersion: v1
kind: Service
metadata:
  name: {{ $name }}-cluster
  labels:
    {{- include "wow-service.labels" . | nindent 4 }}
spec:
  clusterIP: None
  publishNotReadyAddresses: true
  selector:
    {{- include "wow-service.selectorLabels" . | nindent 4 }}
  ports:
    - name: internal
      port: {{ .Values.ports.internal }}
      targetPort: internal
      protocol: TCP

Both are named after the service. eshop-ordering-service serves the public port to the gateway, through which clients reach it. eshop-ordering-service-cluster is headless: its SRV records, _internal._tcp.eshop-ordering-service-cluster.<namespace>.svc, name every node's address, and other services find the nodes through them by the service's name alone (Calling Other Services). It publishes the nodes that are not ready, so a node that is starting or draining is still found.

Network policies

Never expose the internal port outside the cluster. It takes no token: each request names its service (Wow-Service), and the network policies are the boundary (Secure a Deployment):

networkPolicy:
  enabled: true
  public:
    # Who reaches the public port: the gateway's pods by default, in any namespace.
    from:
      - namespaceSelector: {}
        podSelector:
          matchLabels:
            app.kubernetes.io/component: gateway
  internal:
    # More pods that reach the internal port, besides the service's own, its platform's and its peers'.
    from: []
  metrics:
    # Who scrapes the metrics port, when it is one of its own: Prometheus' pods, as kube-prometheus-stack labels them.
    from:
      - namespaceSelector: {}
        podSelector:
          matchLabels:
            app.kubernetes.io/name: prometheus

• <name>-public admits the gateway's pods, labelled app.kubernetes.io/component: gateway in any namespace, to the HTTP port. Label your gateway's pods so, or set networkPolicy.public.from to match them; an empty list is refused, since it would admit every pod.
• <name>-internal admits to the internal port the service's own nodes, the pods of its platform (partOf, which labels every object app.kubernetes.io/part-of), the pods of the peers it calls, which send the wait signals back, and networkPolicy.internal.from.

A network plugin that does not enforce NetworkPolicy ignores them; then keep the internal port off every network but the platform's by other means.

Peers in other namespaces

A peer in another namespace takes its namespace in service.peers, in your values file:

service:
  peers:
    - context: eshop-payment
      service: eshop-payment-service
      namespace: payments

The chart then gives the node wow.peer.dns.namespaces.eshop-payment-service: payments, so it looks the peer up at _internal._tcp.eshop-payment-service-cluster.payments.svc, and the internal policy admits the peer's pods from that namespace. The peer's own internal policy must admit this service's pods in turn: its networkPolicy.internal.from, with a namespaceSelector.

Probes

The host gives the node's health to Spring Boot Actuator's readiness and liveness groups, on the HTTP port:

startupProbe:
  httpGet:
    path: /actuator/health/readiness
    port: http
  periodSeconds: {{ .Values.probes.startup.periodSeconds }}
  failureThreshold: {{ .Values.probes.startup.failureThreshold }}
readinessProbe:
  httpGet:
    path: /actuator/health/readiness
    port: http
  periodSeconds: {{ .Values.probes.readiness.periodSeconds }}
  failureThreshold: {{ .Values.probes.readiness.failureThreshold }}
livenessProbe:
  httpGet:
    path: /actuator/health/liveness
    port: http
  periodSeconds: {{ .Values.probes.liveness.periodSeconds }}
  failureThreshold: {{ .Values.probes.liveness.failureThreshold }}

Readiness fails while the node cannot serve: before it runs, while its log, a store its writes need (a type's primary snapshot store) or its lease store is down, while it owns none of its share of partitions past the hand-over grace, or while a topic it reads is still joining. Liveness fails only when the node is stuck or has met a fatal error (Fatal errors), never for a storage outage and not while it drains, so an outage does not restart every pod at once. The startup probe gives a node three minutes to start (probes.startup).

Graceful shutdown

When Kubernetes stops a pod, the node first finishes the waits still open, then hands its partitions off after the commands in flight, saves its checkpoints, and closes its internal port last. The grace period must cover all of it, so the chart derives it from the same values it gives the node:

# The node's shutdown: the grace period is the preStop pause, the public server's longest wait, the node's stop bound
# from these values and the margin, one after the other; the chart sets wow.wait.max-timeout, wow.cluster.drain-timeout
# and wow.subscription.stop-timeout from them.
shutdown:
  # Time for the endpoints to drop the pod before Spring Boot stops taking requests.
  preStopSeconds: 5
  # The longest wait the public server finishes before it stops (wow.wait.max-timeout); at least
  # wow.wait.default-timeout (10 s unless set), or the node does not start.
  waitMaxTimeoutSeconds: 60
  # How long a hand-off waits for its partition's commands in flight (wow.cluster.drain-timeout).
  drainTimeoutSeconds: 10
  # How long a subscription's stop waits for its entries in flight (wow.subscription.stop-timeout).
  subscriptionStopTimeoutSeconds: 10
  # Room for the checkpoints to be saved and the JVM to exit.
  marginSeconds: 15

The grace period counts the shutdown's steps one after the other: the preStop pause; the public server's graceful shutdown, which lasts as long as its longest request, a wait (waitMaxTimeout); the node's stop, up to its stop bound; and the margin. That is 5 + 60 + 90 + 15 = 170 seconds by default. The stop bound is max(waitMaxTimeout, 2 × subscriptionStopTimeout + drainTimeout) + 3 × drainTimeout, 90 seconds by default, the formula of ServiceNodeOptions.stopBound and of the host's shutdown phase (Starting and stopping). Its waits are measured from the node's own stop, so they come after the public server's. The chart sets wow.wait.max-timeout, wow.cluster.drain-timeout and wow.subscription.stop-timeout from these values; setting any of them in config is refused. A short preStop pause gives the endpoints time to drop the pod before Spring Boot stops taking requests.

Scale

Add replicas to spread the partitions over more nodes; remove them to gather them again. The leases move partitions between nodes as members join and leave, about one partition in N each time. A move is not free:

• The hand-off. The leaving owner finishes the partition's commands in flight, within wow.cluster.drain-timeout, and saves its subscriptions' checkpoints. Meanwhile the commands of its aggregates wait, or run on the node that received them (Deployment Model).
• The subscriptions go on from their checkpoints on the new owner, so a handler may see again what it handled after its last checkpoint.
• The state cache of the new owner is cold: each aggregate's first command there loads its snapshot and the log after it.

So scale on a signal that lasts, and slowly. A node beyond the number of partitions, 64 by default, owns none, which is one reason the count is chosen up front: keep the maximum replicas below it. The signals worth scaling on:

Signal	Means
CPU	Command and query work, for a service whose handlers compute
wow.subscription.backlog	Subscriptions falling behind the log: projections, sagas, Sinks
wow.command.duration	The commands' latency, when the stores answer in time

autoscaling.hpa makes a HorizontalPodAutoscaler on CPU, and metrics of your own:

# A HorizontalPodAutoscaler on CPU, and metrics of your own. Exclusive with keda.
hpa:
  enabled: false
  minReplicas: 2
  maxReplicas: 8
  cpuUtilization: 70
  metrics: []
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 60
      policies:
        - type: Pods
          value: 1
          periodSeconds: 60
    scaleDown:
      stabilizationWindowSeconds: 600
      policies:
        - type: Pods
          value: 1
          periodSeconds: 300

Its behavior adds a node a minute at most, and removes one every five minutes after ten quiet ones, so a burst does not move the partitions back and forth. With KEDA installed, autoscaling.keda scales on the backlog through Prometheus instead; the two are exclusive:

# A KEDA ScaledObject on the subscriptions' backlog, through Prometheus; needs KEDA installed. Exclusive with hpa.
keda:
  enabled: false
  minReplicas: 2
  maxReplicas: 8
  pollingInterval: 30
  cooldownPeriod: 300
  prometheus:
    serverAddress: ""
    # sum(wow_subscription_backlog{namespace="<namespace>", pod=~"<name>-.*"}) when empty.
    query: ""
    # The backlog per node above which a node is added.
    threshold: 1000
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 60
      policies:
        - type: Pods
          value: 1
          periodSeconds: 60
    scaleDown:
      stabilizationWindowSeconds: 600
      policies:
        - type: Pods
          value: 1
          periodSeconds: 300

The chart requires neither: both are off by default, and the replicas are then replicas.

Monitor

The nodes' spans and meters go through the OpenTelemetry Java agent (Observability), which pushes them over OTLP, as the eShop's values set it, or serves the meters on a Prometheus endpoint (OTEL_METRICS_EXPORTER=prometheus, port 9464). For the Prometheus Operator, the chart adds a ServiceMonitor or a PodMonitor on that endpoint:

# Where Prometheus scrapes a node: the OpenTelemetry Java agent's Prometheus exporter (OTEL_METRICS_EXPORTER=prometheus,
# port 9464, /metrics), or Actuator's endpoint on the HTTP port (port: the HTTP port, path: /actuator/prometheus).
metrics:
  port: 9464
  path: /metrics
  serviceMonitor:
    enabled: false
    interval: 30s
    labels: {}
  podMonitor:
    enabled: false
    interval: 30s
    labels: {}

A metrics port of its own is opened on the pod and admitted by <name>-public from networkPolicy.metrics.from, Prometheus' pods by default. Give the monitor the labels your Prometheus selects (labels). What to alert on is in Observability.

Disruption budgets

A voluntary disruption, a node drained or the cluster upgraded, takes the nodes one at a time: each node that leaves hands its partitions off, and the others take them through the leases before the next one goes. The chart's PodDisruptionBudget says so, and spreads the nodes over zones and hosts:

podDisruptionBudget:
  enabled: true
  # Or minAvailable, which wins when given.
  maxUnavailable: 1
  # AlwaysAllow lets a drain evict a node that is not ready, which a budget would otherwise hold.
  unhealthyPodEvictionPolicy: AlwaysAllow

topologySpread:
  enabled: true
  maxSkew: 1
  whenUnsatisfiable: ScheduleAnyway

AlwaysAllow lets a drain evict a node that is not ready, which a budget would otherwise hold for ever. Prefer maxUnavailable to minAvailable: with two replicas, minAvailable: 2 blocks every drain.

Upgrade

The chart's version is the Wow version, and its appVersion the default tag of the image. Upgrade the service's Wow dependency and the chart together, then:

helm upgrade eshop-ordering oci://ghcr.io/ahoo-wang/wow-service --version <new-wow-version> \
  -f examples/eshop/ordering/ordering-server/build/wow/helm/values.yaml \
  -f deploy/eshop-ordering/values.yaml --wait

The rollout replaces the nodes one at a time, a new one ready before an old one goes, so for a while two versions run side by side: Upgrades says what may, and which settings must never change, the log's partitions among them. A change of the shared settings alone rolls the nodes too: the pods carry a checksum of the ConfigMap. Regenerate the build's values with every build: a peer added in the code is then admitted by the policies and found by the node.

With Kustomize

Render the chart and lay your patches over the output, instead of keeping a second set of manifests:

helm template eshop-ordering oci://ghcr.io/ahoo-wang/wow-service --version <wow-version> --namespace shop \
  -f examples/eshop/ordering/ordering-server/build/wow/helm/values.yaml \
  -f deploy/eshop-ordering/values.yaml > base/eshop-ordering.yaml

List eshop-ordering.yaml among the resources of base/kustomization.yaml, and render it again on each upgrade and with each build's values.

In an umbrella chart

A platform chart holds the chart once per service, under an alias, each service's values under its alias:

dependencies:
  - name: wow-service
    version: <wow-version>
    repository: oci://ghcr.io/ahoo-wang
    alias: ordering
  - name: wow-service
    version: <wow-version>
    repository: oci://ghcr.io/ahoo-wang
    alias: basket

The schema accepts the umbrella's global values, which the chart does not read.