---
url: /guide/performance.md
description: >-
  What one Wow node does with a command, how fast on stated hardware, how it
  compares with Axon Framework 4 under one set of rules, and where it stops;
  every figure generated from a committed benchmark report.
---

# Performance

What one Wow node does with a command, how fast it does it on stated hardware, how it compares with Axon Framework 4 under one set of rules, and where it stops. Every number on this page is written from a committed benchmark report when the documentation is built, and each names the build, the CPU and the JDK it was measured on.

::: tip In one screen

* **On a 4-core cloud runner**, one node sharing 3 cores with the database and Kafka (the load generator on 1 more), sustains writes at 1,957.8/s on PostgreSQL within a p99 of 139.3 ms (runner EPYC 7763), and 2,975.1/s on MongoDB (runner Xeon 6975P-C); each store ran on a runner of its own, so the two are not compared ([On a 4-core cloud runner](#on-a-4-core-cloud-runner)).
* **On an 18-core workstation**, one node sustains writes at 41,706.1/s on PostgreSQL within a p99 of 167.9 ms, at 4,096 clients ([How far one node goes](#how-far-one-node-goes-on-a-workstation)).
* **Against Axon Framework 4** on one PostgreSQL, under the same rules, Wow's peak on the saga chain is 1.9–2.3 times Axon's with the node and PostgreSQL on 2 cores, and 5.2–5.4 times on 18 cores ([Against Axon Framework 4](#against-axon-framework-4)).

Sizing a deployment instead? See [Capacity planning](../operate/performance).
:::

## What Wow does with a command, and why that is cheap

A command goes to the node that owns its aggregate's partition. An aggregate in use is in its owner's cache, so its command is decided against memory and reads nothing from the database on its way; one the node meets for the first time is loaded once, from its snapshot and the events after it ([A cold cache](#a-cold-cache) measures that case).

* **One aggregate's commands run in order in its own lane, with no lock.** Commands queued behind one commit with it, and the appends of several partitions merge into one statement, so a write takes at most one round trip to the database and one commit, and commands share both as the rate rises.
* **Off the command's path:** snapshots, sagas and sinks follow the log after the commit.
* **The answer comes when the caller asks for it** (`Wow-Wait`): `committed` for a write, `snapshot` to read your write, `queryable` when a list must show it, or a saga's next step.

**What it costs in return.** Every command keeps its request id under a unique key, so a command sent twice is handled once. That is why an event takes 615 bytes in Wow's PostgreSQL log, against 462 in Axon's, indexes included. A node on the workstation is ready 1,790 ms after launch, at 461 MB resident (build `cc11b16aa8`, Apple M5 Max, JDK 25.0.5+8).

## How it was measured {#what-was-measured}

The domain is the benchmark's own: accounts, and a transfer between two of them run by a saga ([`test/benchmarks/server`](https://github.com/Ahoo-Wang/Wow/tree/main/test/benchmarks/server)).

| Scenario | What one operation is | Answered when |
| --- | --- | --- |
| E1 write | A deposit to one account | Its commit is stored |
| E2 read your write | A deposit, then a read of the account | The read shows the deposit |
| E3 saga chain | A transfer prepared on one account | The saga's entry on the other account is stored |
| E4 read | One account by id, or a page of ten | The answer |

* **Peak:** the best closed-loop step with at most 1% of its commands failed. **≥** marks a peak where the measurement itself could go no further.
* **Sustainable:** the highest rate whose p99 stays within 200 ms: with no error on the workstation, with at most 1% errors on the cloud runner, whose rate is found by bisection in open loop. This is the figure to plan on.
* **Latencies are of the answered commands.** Past the peak a refused command returns at once, so the answered rate is what counts.
* **Each figure names its build, CPU and JDK, and nothing is computed across builds.** A comparison is made only within one batch on one machine, its arms alternating (W A A W against Axon), so that both arms meet the same machine at the same time. Two figures from two runs are never set side by side as a difference.
* The method in full and the raw data are in the [benchmark reports](https://github.com/Ahoo-Wang/Wow/tree/main/test/benchmarks/results/reports).

## On a 4-core cloud runner

One node on a GitHub Actions runner of 4 cores and 15–16 GB, 100,000 accounts. The harness pins the node, the database and Kafka to 3 of the cores, and the load generator to the other 1; the cliff, cold-cache and three-node runs below are pinned the same way. Each suite runs on its own runner, whose CPU model its row names.

Each figure is from the build, the JDK and the runner its row names. The scenarios are those of [What was measured](#what-was-measured).

**GitHub Actions, one 4-core runner a suite**: [actions-snapshot](https://github.com/Ahoo-Wang/Wow/blob/main/test/benchmarks/results/reports/actions-snapshot.md), build `e3a4e937aa`.

These rows ran on 2 CPU models, which each row names: two rows on different models do not compare the stores, nor the scenarios.

| Store | Nodes | Scenario | Keys | Peak | Sustainable | CPU, JDK, build |
| --- | ---: | --- | --- | --- | --- | --- |
| MongoDB | one | E1 | uniform | ≥ 5,010.8/s, closed loop | 2,975.1/s, open loop, p99 198.7 ms | Xeon 6975P-C, JDK 25.0.5+8, `e3a4e937aa` |
| MongoDB | one | E2 | uniform | 2,261.6/s, closed loop | 1,272.2/s, open loop, p99 190.5 ms | Xeon 6975P-C, JDK 25.0.5+8, `e3a4e937aa` |
| MongoDB | one | E3 | uniform | 1,466.6/s, closed loop | 1,008.3/s, open loop, p99 173.1 ms | Xeon 6975P-C, JDK 25.0.5+8, `e3a4e937aa` |
| PostgreSQL | one | E1 | uniform | 4,474.9/s, closed loop | 1,957.8/s, open loop, p99 139.3 ms | EPYC 7763, JDK 25.0.5+8, `e3a4e937aa` |
| PostgreSQL | one | E3 | uniform | 1,493.1/s, closed loop | 839.9/s, open loop, p99 190.5 ms | EPYC 7763, JDK 25.0.5+8, `e3a4e937aa` |

Not measured on the runner: the sustainable rates of E2 on PostgreSQL and of E4.

## At light load

Most of the time a service runs far below its peak. There, a write's latency is mostly the database's sync to disk, not the node's code.

* **A saga chain on the cloud runner, PostgreSQL**: transfers at 50/s, p50 2.7 ms and p99 7.6 ms; at 500/s, p50 1.7 ms and p99 14.2 ms (Xeon 6975P-C, build `e3a4e937aa`).
* **The same on MongoDB**: at 50/s, p50 5 ms and p99 14.8 ms; at 500/s, p50 8.4 ms and p99 62.5 ms (Xeon 6975P-C).
* **What this means for a reader:** storage with fast, steady syncs is what lowers light-load latency.

## How far one node goes on a workstation

Closed-loop sweeps from 1 to 4,096 clients on an Apple M5 Max (18 cores, 64 GB), with the database, Kafka and the load generator on the same machine, 100,000 accounts. The answered rate against p99, the 200 ms line drawn; a hollow point is a step where the node refused more than 1% of its commands.

Each figure is from the build, the JDK and the runner its row names. The scenarios are those of [What was measured](#what-was-measured).

**Apple M5 Max, 18 cores, 64 GB, one host**: [m5max-snapshot](https://github.com/Ahoo-Wang/Wow/blob/main/test/benchmarks/results/reports/m5max-snapshot.md), build `103f9b6746`.

| Store | Nodes | Scenario | Keys | Peak | Sustainable | CPU, JDK, build |
| --- | ---: | --- | --- | --- | --- | --- |
| MongoDB | one | E1 | one hot account | 9,948.1/s, at 256 clients | 9,948.1/s, at 256 clients, p99 78.8 ms | Apple M5 Max, JDK 25.0.5+8, `103f9b6746` |
| MongoDB | one | E1 | uniform | ≥ 28,609.4/s, at 4,096 clients | 28,196.1/s, at 1,024 clients, p99 66.6 ms | Apple M5 Max, JDK 25.0.5+8, `103f9b6746` |
| MongoDB | one | E1 | Zipf | 15,957.5/s, at 1,024 clients | 14,914.4/s, at 256 clients, p99 42.2 ms | Apple M5 Max, JDK 25.0.5+8, `103f9b6746` |
| MongoDB | one | E3 | uniform | 7,738.2/s, at 1,024 clients | 5,898.6/s, at 256 clients, p99 72.2 ms | Apple M5 Max, JDK 25.0.5+8, `103f9b6746` |
| PostgreSQL | one | E1 | one hot account | 11,853.7/s, at 256 clients | 11,853.7/s, at 256 clients, p99 76.8 ms | Apple M5 Max, JDK 25.0.5+8, `103f9b6746` |
| PostgreSQL | one | E1 | uniform | ≥ 41,706.1/s, at 4,096 clients | 41,706.1/s, at 4,096 clients, p99 167.9 ms | Apple M5 Max, JDK 25.0.5+8, `103f9b6746` |
| PostgreSQL | one | E1 | Zipf | 19,746.8/s, at 256 clients | 19,746.8/s, at 256 clients, p99 29.6 ms | Apple M5 Max, JDK 25.0.5+8, `103f9b6746` |
| PostgreSQL | one | E3 | uniform | 8,042.7/s, at 1,024 clients | 3,637.9/s, at 64 clients, p99 185.3 ms | Apple M5 Max, JDK 25.0.5+8, `103f9b6746` |

Three nodes on this one host: the nodes, the stores and the load generator divide the same cores, so these rows show what a cluster costs (forwarding between the nodes), not what it gains. Scaling across separate machines has not been measured.

| Store | Nodes | Scenario | Keys | Peak | Sustainable | CPU, JDK, build |
| --- | ---: | --- | --- | --- | --- | --- |
| MongoDB | 3 | E1 | uniform | 19,052.3/s, at 1,024 clients | 19,052.3/s, at 1,024 clients, p99 131.1 ms | Apple M5 Max, JDK 25.0.5+8, `103f9b6746` |
| MongoDB | 3 | E3 | uniform | 6,060.2/s, at 1,024 clients | 5,462.9/s, at 256 clients, p99 78.8 ms | Apple M5 Max, JDK 25.0.5+8, `103f9b6746` |
| PostgreSQL | 3 | E1 | uniform | 25,104.3/s, at 1,024 clients | 25,104.3/s, at 1,024 clients, p99 90.6 ms | Apple M5 Max, JDK 25.0.5+8, `103f9b6746` |
| PostgreSQL | 3 | E3 | uniform | 8,623/s, at 1,024 clients | 7,279/s, at 256 clients, p99 59.6 ms | Apple M5 Max, JDK 25.0.5+8, `103f9b6746` |

## A hot aggregate

One aggregate's commands run one after another on its owner, so one hot account bounds what any single aggregate takes, whatever the cluster's size. On the workstation one hot account peaks at 11,853.7/s on PostgreSQL and 9,948.1/s on MongoDB; keys drawn from a Zipf distribution peak at 19,746.8/s and 15,957.5/s. Spread load over aggregates where the model allows.

## Past the peak

A node refuses what it cannot take, with `429 wow.command.busy` and `Retry-After`. It does not queue without end: past the peak it goes on answering while it refuses the rest at once, and the hollow points of the figure above show the rate it still answered at each such step.

The limit that remains: a saga's command waits on a subscription, and admission does not see a subscription's queue. When a downstream subscription lags, overload can still show as a `504` after 10 s rather than a `429`.

## A cold cache

With a million accounts and nothing cached, on the cloud runner (build `e3a4e937aa`; runners Xeon 6975P-C for PostgreSQL and EPYC 9V74 for MongoDB):

* a write at 200/s answers within a p99 of 5.1 ms on PostgreSQL and 154.6 ms on MongoDB;
* a point read at 500/s within 3 ms and 4.9 ms.

## Three nodes, and a node lost

Three nodes on one cloud runner, sharing 3 cores with the database and Kafka (build `e3a4e937aa`; runners EPYC 7763 for PostgreSQL and Xeon 6975P-C for MongoDB):

* **In steady state**, writes at 150/s answer within a p99 of 14.1 ms on PostgreSQL and 13.7 ms on MongoDB.
* **A node stopped mid-run:** its partitions move to the others in seconds. The worst second's p99 of writes was 3,129.3 ms on PostgreSQL and 1,671.2 ms on MongoDB; 0 and 0 writes failed. Through the same hand-over, 39 saga chains failed on MongoDB and 1 on PostgreSQL, one run each, at build `e3a4e937aa`. MongoDB's come after the stopped node's restart, while its partitions caught up a few at a time: that is [#6407](https://github.com/Ahoo-Wang/Wow/issues/6407), fixed after this build ([the report](https://github.com/Ahoo-Wang/Wow/blob/main/test/benchmarks/results/reports/actions-snapshot.md#l4-cluster-and-hand-over-e5) has the timing, and [How a cluster recovers](#recovery) measures the fix).
* **Not measured: scaling across separate machines.** These nodes share one machine with the database, so they show what a cluster costs, not what it gains. [Capacity planning](../operate/performance) gives the bound, and the run to make on your own machines.

### How a cluster recovers {#recovery}

* **A node that loses every lease** has all its partitions back within 3.5 s of the loss; before the fix, not all were back when the round ended.
* **A planned hand-over on MongoDB** times out fewer saga waits, at a lower p99, than before the fix. A timed-out wait remains in each round and is being worked on; PostgreSQL had none.

Each table's builds ran in one batch on one machine, alternating; the [recovery report](https://github.com/Ahoo-Wang/Wow/blob/main/test/benchmarks/results/reports/m5max-recovery.md) has the method and every run.

**A node that loses all its leases at once** ([#6379](https://github.com/Ahoo-Wang/Wow/issues/6379)). MongoDB is paused and then served slowly, and the node loses every lease. Before the fix, it stopped the lost partitions one after another, each stop waiting on the slow store, and took them back only after the last.

Apple M5 Max, 18 cores, 64 GB, JDK 25.0.5+8, MongoDB, one node that loses all 64 of its leases; 3 rounds a side, alternating in one batch. Each time runs from a partition's loss to its acquisition.

| Build | First back | Last back | Back at each round's end |
| --- | ---: | ---: | ---: |
| fix `93f53932fa` | 2.4–2.7 s | 2.4–3.5 s | 64, 64, 64 of 64 |
| before the fix `7f46c8dfb4` | 47.2–47.8 s | not all back | 16, 28, 16 of 64 |

**A planned hand-over** ([#6407](https://github.com/Ahoo-Wang/Wow/issues/6407)). A node is stopped gracefully and restarted while sagas run. Before the fix, the moved partitions queued two at a time for the node's catch-up permits; now a partition a page or less behind its head reads at once.

Apple M5 Max, 18 cores, 64 GB, JDK 25.0.5+8, 3 nodes, 5,250 commands a round; a failed one is a saga's timed-out wait. Each batch alternated its two MongoDB builds, then ran one PostgreSQL round.

**Batch 1:**

| Build | Store | Failed waits per round | p99, lowest to highest round |
| --- | --- | ---: | ---: |
| before the fix `e3a4e937aa` | MongoDB | 15, 2, 2 | 2,851–5,636 ms |
| first fix `55937a4b2c` | MongoDB | 1, 1, 2 | 1,581–1,983 ms |
| first fix `55937a4b2c` | PostgreSQL | 0 | 22 ms |

**Batch 2:**

| Build | Store | Failed waits per round | p99, lowest to highest round |
| --- | --- | ---: | ---: |
| first fix `f43a7a75bf` | MongoDB | 1, 1, 1 | 1,778–1,974 ms |
| the code that ships `0249db185b` | MongoDB | 1, 1, 1 | 1,737–1,901 ms |
| the code that ships `0249db185b` | PostgreSQL | 0 | 20 ms |

Batch 1's builds carried log probes, which add log lines only.

**Still open:** after both fixes, a wait still timed out in every MongoDB round. A moved partition's catch-up waits out MongoDB's stall grace at each sequence block the old owner left open. This is being worked on, and its result has not been measured.

**Catching up on a large backlog.** A node catching up on a large backlog, such as an imported history or a long stop, reads a bounded number of partitions at a time, so that live commands keep the store. The bound is `wow.subscription.max-catch-ups`, two permits by default: a partition more than a page behind its head waits for a permit before it reads; one a page or less behind reads at once. With the default two, no live order failed; with no bound, orders failed.

GitHub Actions ubuntu-26.04, CPU model not recorded, JDK 25.0.5, live orders while a V10 host catches up on an imported log; one run each (44137290807 and 44137289607), on separate runners.

| Catch-up permits | Job | Failed of 6,000 | Worst second's p99 | Catch-up |
| --- | --- | ---: | ---: | ---: |
| 2 (the default), `0853cbf27b` | runner | 0 | 10,260 ms | 87.5 s |
| 2 (the default), `0853cbf27b` | kind | 0 | 4,614 ms | 59.4 s |
| no bound, `7c0896effd` | runner | 7 | 22,566 ms | 66.3 s |
| no bound, `7c0896effd` | kind | 2,587 | 30,475 ms | 107.2 s |

Both runs were of `f33284fa69` with the rehearsal's instrumentation, at the earlier rule, which treats a large backlog the same after its first page.

The [recovery report](https://github.com/Ahoo-Wang/Wow/blob/main/test/benchmarks/results/reports/m5max-recovery.md#a-nodes-first-activation-on-a-migrated-history-6412-then-6424) also has a node's first activation on a migrated history, the code that ships against the code before these fixes.

## Against Axon Framework 4

**What "fair" means here:**

* **The same setup on both sides:** the same domain, load generator, harness and PostgreSQL, and the same connection pool (Spring Boot's 10), with a run at 32 apart.
* **Axon 4 as its reference guide configures production without Axon Server:** a JDBC event store and token store, a tracking processor of 16 segments, a cache, and a snapshot every 100 events.
* **Wow with publishing off,** since Axon publishes nothing; the same 10 s timeout on both.
* **Noise:** the arms alternate in one batch, W A A W, and a difference within an arm's own A/A spread is called level.
* **Correctness:** each session's database is checked afterwards; an arm whose data is wrong is not compared.

On a GitHub Actions runner of 4 cores, with the node (Wow's or Axon's) and PostgreSQL pinned to 2 of them and the load generator to the other 2, each range Wow's rate divided by Axon's inside the band of the arms' own spread; a hollow range is level:

On 18 cores (the workstation), in a batch of its own at its own build:

Not compared: a tuned Axon (a Disruptor command bus, batching), Axon 5, Axon Server, and Wow with Kafka.

## Run it yourself

On your own machine, with Docker and JDK 25:

```bash
test/benchmarks/harness/bench.sh build
test/benchmarks/harness/bench.sh up --backend=postgres --accounts=100000 --prime
test/benchmarks/harness/bench.sh saturate --reuse --backend=postgres --accounts=100000 --scenario=E1 --sweep
test/benchmarks/harness/bench.sh down
```

On GitHub's runners: `gh workflow run bench.yml -f tier=L4 -f head_ref=<tag>`. The [benchmark README](https://github.com/Ahoo-Wang/Wow/tree/main/test/benchmarks) has every option.

## Where the limits are

| Limit | What to do |
| --- | --- |
| One database carries every node | A faster or less shared database; MongoDB sharding; aggregate types on separate stores |
| A hot aggregate ([above](#a-hot-aggregate)) | Model so that the load spreads over aggregates |
| Light-load latency is the disk's sync | Storage with fast, steady syncs |
| One aggregate type's log on one database | Planned for 10.x |
| More bytes per event than Axon | The price of handling a command sent twice once |
| Admission does not see subscription queues | Plan on the sustainable rate, not the peak |

And the measurement's own limits: the database, the load generator and the node share a machine; the workstation's figures are single runs; no cluster was measured on separate machines; and a cloud runner's figures move with its CPU model, which each row names.
