Framework Tests and Benchmarks

This page applies only to the Wow framework repository. Use these tasks when changing framework source, TCKs, adapters, build logic, or benchmarks. For a business-application release, use Testing Wow Applications instead of copying this repository's root tasks, Codecov flags, or JMH conclusions.

Completion signal

A framework change is complete when the affected modules' check and relevant test layers pass. If benchmark entry points changed, smoke also passes. Any performance claim additionally requires baseline/confirmation results with the same workload, comparable environment, and complete provenance. The next layer is code review and CI, not turning historical figures into product promises.

Choose the Test Layer by Dependency

Layer	Source set	Root task	Runtime condition	Evidence scope
Local	src/test	allLocalTest	No containers	Local-safe unit, domain, and component behavior
Contract	src/contractTest	allContractTest	No containers	Registered TCK implementations satisfy shared contracts
Integration	src/integrationTest	allIntegrationTest	Docker/Testcontainers required	Middleware adapters and end-to-end integration

The root build currently registers contractTest only for :wow-core, :wow-opentelemetry, and :wow-mock. It registers integrationTest only for :wow-bi, :wow-mongo, :wow-redis, :wow-kafka, :wow-elasticsearch, and :wow-it. Do not guess task names for modules that do not register them.

check runs standard test tasks and includes contractTest in configured modules; it does not automatically run container-backed integrationTest. Therefore, a green check does not prove every store and broker integration was verified.

Narrowest Local Feedback

Run directly affected modules first:

./gradlew :wow-core:check
./gradlew :wow-test:check :example-domain:check

Expand only when the entire local-safe layer is needed:

./gradlew allLocalTest
./gradlew allContractTest
./gradlew check

Domain specifications still use AggregateSpec and SagaSpec from the Domain Test Suite. They live in the owning module's src/test and belong to the Local layer; they are not a separate application-release proof.

Architecture Guardrails

Two kinds of test guard the module boundaries and the decorators. Both run in the Local layer:

• DependencyRulesTest (:wow-core:test) scans production sources. wow-core and wow-query reference no HTTP, Spring or storage-driver types. No backend module (wow-mongo, wow-redis, wow-kafka, wow-elasticsearch) references another backend or its driver. Outside their query packages, wow-mongo and wow-elasticsearch reference no query packages. me.ahoo.wow.api.query protocol types are allowed. The test task declares the scanned source trees as inputs, so a change in any of those modules re-runs it.
• MetricDecoratorContractTest (:wow-core:test) and TracingDecoratorContractTest (:wow-opentelemetry:test) require every Metric* and Tracing* decorator to override each SPI member that has a default body. Otherwise a call falls back to the interface default and skips the delegate, for example EventStore.existsRequestId, which loads the whole stream. ElasticsearchEventStoreContractTest applies the same rule to existsRequestId on the Elasticsearch event store. Each test keeps a KNOWN_GAPS list, empty since 9.2.3. A new gap fails the test, and so does a fixed gap that is still listed; an entry may only be added with the work item that removes it.

Wire-Format Golden Samples

The v9 wire is frozen: during a rolling upgrade, 9.2.x and later 9.x nodes share topics, streams, stores and the command wait endpoint, so each must read what the other writes. Golden tests lock those bytes. Each one serializes a fixed sample (me.ahoo.wow.tck.wire.WireSamples) through the production code and compares the result byte for byte with a committed file under the module's src/test/resources/wire/v9/, then decodes the committed file through the production reader.

Module	Test	What it freezes
wow-core	WireFormatGoldenTest	Command message JSON (stage and chain wait headers), domain event stream JSON, state event JSON
wow-kafka	KafkaWireFormatGoldenTest	Kafka record topic, partition, timestamp, key, record headers and value for the command, event and state buses
wow-redis	RedisWireFormatGoldenTest	Redis stream key and entry fields for the command, event and state buses
wow-spring-boot-starter	WaitSignalWireFormatGoldenTest	The WaitSignal body posted to a remote command wait endpoint

The samples carry the headers the runtime stamps: command_operator, user_agent, remote_ip, local_first, trace_id, upstream_* and the command_wait_* keys from a real wait plan. The goldens were generated from 9.2.2. A failing golden test means the wire changed; do not re-generate the file to make it pass. Changing a golden needs a design decision that covers mixed-version clusters; once that is decided, rerun the test with WOW_GOLDEN_UPDATE=true and review the diff.

Container-Backed Integration Tests

Run all registered integration tasks with:

./gradlew allIntegrationTest --stacktrace

Or run only an affected adapter:

./gradlew :wow-mongo:integrationTest --stacktrace
./gradlew :wow-redis:integrationTest --stacktrace
./gradlew :wow-kafka:integrationTest --stacktrace
./gradlew :wow-elasticsearch:integrationTest --stacktrace
./gradlew :wow-it:integrationTest --stacktrace

These tasks require Docker/Testcontainers and intentionally are not attached to check. :wow-it validates integration combinations inside the Wow repository; it cannot replace a business application's configuration, protocol, recovery, and security gates.

Mixed-Version Cluster Test

MixedVersionClusterTest in :wow-it checks a rolling upgrade: the released example server image and the example server built from the current source share one Kafka and one MongoDB, join the same consumer groups, and each must process what the other sends. Local-first routing is off, and the test chooses aggregate ids by partition owner, so in both directions:

• a command sent to one version is processed by the other, and its PROCESSED wait returns to the sender;
• a state event appended by one version is snapshotted by the other, and the SNAPSHOT wait returns across versions;
• a saga on one version (CartSaga) handles the domain events the other version appends;
• both versions read the same aggregate state and event stream.

The test runs only when its environment names both nodes, so allIntegrationTest skips it. The Mixed-Version workflow runs it on pull requests that touch the wire modules (wow-api, wow-core, wow-kafka, wow-redis, wow-mongo, wow-webflux, wow-spring, wow-spring-boot-starter), the example server, dependency versions or the harness, pulling the released image (PREVIOUS_IMAGE in the workflow). To check the harness locally without pulling that image, run it against two copies of the current build:

./gradlew :example-server:installDist
home=$PWD/example/example-server/build/install/example-server
WOW_MIXED_CURRENT_HOME=$home WOW_MIXED_PREVIOUS_HOME=$home \
  ./gradlew :wow-it:integrationTest --tests 'me.ahoo.wow.it.mixed.MixedVersionClusterTest'

Set WOW_MIXED_PREVIOUS_IMAGE instead of WOW_MIXED_PREVIOUS_HOME to run the released image, as CI does. The example server has no Redis bus, so Redis is covered by wire-format tests, not by this cluster.

Coverage Is Layered Evidence

The current aggregate and layer report tasks are:

./gradlew codeCoverageReport
./gradlew :code-coverage-report:localCoverageReport
./gradlew :code-coverage-report:contractCoverageReport
./gradlew :code-coverage-report:integrationCoverageReport
./gradlew :example-domain:jacocoTestCoverageVerification

The aggregate XML is written to:

test/code-coverage-report/build/reports/jacoco/codeCoverageReport/codeCoverageReport.xml

Layer reports are written under the matching localCoverageReport, contractCoverageReport, and integrationCoverageReport directories. Pull requests upload the local, contract and integration flags: changed main code must be 90% covered (Codecov patch status), and each published Kotlin module is a Codecov component, measured from those three flags, whose coverage may not drop by more than 1%. The Codecov workflow on main or manual dispatch uses codeCoverageReport to upload the full flag and uploads the contract and integration layers too, so they have data on main.

:example-domain, :example-transfer-domain, and :wow-compensation-domain currently configure a 0.8 Jacoco verification minimum. The threshold runs only when the corresponding jacocoTestCoverageVerification task is invoked explicitly; these modules' check tasks and the current CI workflows do not attach a verification task automatically. It is an optional repository gate, not a Wow coverage guarantee for business applications. Coverage shows executed code and cannot replace assertions about events, state, rejection, and recovery.

Benchmarks Have Three Uses

Use	Entry point	Supported conclusion
Smoke	benchmarkSmoke	Selected JMH jar and paths compile, start, and finish
Quick	benchmarkQuick*	Bounded regression clues on the current machine
Baseline / confirmation	benchmarkBaseline*, benchmarkConfirm*	Comparable evidence under matching methods, parameters, forks, and environment

Smoke is not a performance report, Quick is not a production-capacity model, and isolated component results are not framework end-to-end throughput promises.

Pull-Request Safety

./gradlew :wow-benchmarks:test :wow-benchmarks:benchmarkSmoke --stacktrace

This matches the current Benchmark Smoke CI workflow. The root alias is also available:

./gradlew benchmarkSmoke

The completion signal is that selected paths execute successfully, not that a performance baseline was produced or updated.

Quick Regression and Diagnosis

Generate a quick Framework E2E report:

./gradlew :wow-benchmarks:benchmarkQuickE2E \
  :wow-benchmarks:generateBenchmarkReport

Run the paired batch-command-write workload with:

./gradlew :wow-benchmarks:benchmarkQuickBatchE2E \
  :wow-benchmarks:generateBatchBenchmarkReport

When locating a bottleneck, select a layer instead of running the complete catalog:

./gradlew :wow-benchmarks:benchmarkQuickComponent
./gradlew :wow-benchmarks:benchmarkQuickWebFlux -PbenchmarkQuickWebFluxThreads=1
./gradlew :wow-benchmarks:benchmarkQuickInfrastructureE2E

The WebFlux suite does not start a real Netty server. The current benchmarkQuickInfrastructureE2E includes both Redis and Mongo workloads, so local Redis and MongoDB are both required services; either one missing leaves the suite's runtime requirements unmet. Reports must retain workload, thread, JVM, service, and source provenance; do not interpret numbers across layers as directly comparable.

Work that changes a framework hot path compares two labelled runs of the refactor catalog on the same machine: one before the change and one after it. benchmarkQuickRefactorHotPaths needs no services and covers event dispatch (1 and 8 processors, metrics on and off, in-memory and local-first buses), local-first command send, the command HTTP edge (header appenders, full router dispatch), per-command aggregate processing and event-stream query delivery. benchmarkQuickRefactorTransport measures Redis Streams receive and acknowledge and needs the benchmark Redis. Kafka receive and acknowledge (KafkaEventReceiveAckBenchmark) is not in a Gradle suite because the benchmarks have no Kafka service; run it from the JMH jar against a broker.

./gradlew :wow-benchmarks:benchmarkQuickRefactorHotPaths \
  :wow-benchmarks:generateQuickRefactorHotPathsBenchmarkReport \
  -PbenchmarkRefactorReportLabel=9.2.2-baseline

Formal Regression Evidence

Collect comparable evidence for exact Framework E2E workloads with:

./gradlew :wow-benchmarks:benchmarkBaselineE2E --no-parallel
./gradlew :wow-benchmarks:benchmarkCompare

A threshold crossing from benchmarkCompare is only a regression or improvement candidate. Run benchmarkConfirmE2E for the affected method with the same JVM, threads, parameters, forks, warmup, measurement, and profiler before forming a confirmed conclusion.

updateBenchmarkBaseline accepts only a clean manifest produced from the current clean HEAD. Do not update a baseline from a dirty worktree, different service configuration, or missing manifest.

A/B in CI

Benchmark A/B (.github/workflows/benchmark-ab.yml) compares two refs on GitHub runners instead of a local machine. Dispatch it with a base ref (default main), a head ref, an include regex or a list of them, and a profile:

gh workflow run benchmark-ab.yml -f base=main -f head=my-branch \
  -f include=CommandIdComponentBenchmark -f profile=quick

It builds the JMH jar of each ref once and caches it by commit and by the hash of the benchmark sources and build files, so a repeated A/B of the same commits skips the Gradle build. It then fans the selected benchmarks out into matrix jobs. Inside a job, base and head forks alternate on the same runner (base-head, head-base, …), so runner and drift variance cancel. Every fork runs with -prof gc. Optional inputs override @Param values (name=v1,v2;…), thread counts, the noise threshold (default 3%) and a pull request number to comment on. Benchmarks under benchmark.infrastructure need Redis, MongoDB, Elasticsearch or Kafka and are skipped.

Profile	Forks per side	Warmup, measurement	Default split	Typical wall time
quick	up to 3	2 × 2 s, 3 × 2 s	class	about 10–15 minutes for one class
gate	up to 8	3 × 3 s, 5 × 3 s	params	about 10–25 minutes, at most 16 jobs at a time

split chooses the job granularity: class, method, or params, one job per method and @Param combination (the combination is pinned with JMH -p). The default, auto, uses class for quick and params for gate. A gate run of a class with several methods and params used to run every combination serially in one job (EventDispatchComponentBenchmark: about 130 minutes); split by params, its 8 combinations run in parallel and an A/A gate run takes about 11 minutes. The plan step lists the jobs in the run summary, runs at most 16 at a time (GitHub Free allows 20 concurrent jobs per account), warns above 48 jobs, and fails above GitHub's matrix limit of 256; narrow the include patterns or pin params then.

A job stops early once every row it measures is decided. After each round from the second until the last but one, it computes each side's interval over the iterations so far, at a stricter confidence than the report: the 0.1% that JMH's 99.9% interval leaves is split evenly over the early looks a job can take (Bonferroni; six looks for gate, one for quick). A row is decided when:

• separated: the intervals do not overlap and |Δ| reaches the threshold. The report's 99.9% intervals are narrower, so it flags the row the same way;
• noise: even the far ends of the two intervals differ by less than the threshold, so more forks could not show a change of that size. A 0% threshold never stops on noise.

Anything else runs another round, up to the profile's forks. The verdict rule itself is unchanged: the report pools whatever forks each row ran.

The report in the run summary lists base and head score ± error (pooled over every fork of that side, JMH's 99.9% interval), Δ%, the verdict, the forks per side the row used and why its job stopped there (separated, noise or max forks), B/op and GC ms per iteration, and the run's wall time. A row is flagged faster or slower only when the two intervals do not overlap and |Δ| reaches the threshold; overlapping intervals are noise. The benchmark-ab label on a pull request runs the quick profile for the JMH classes that pull request changed, head against its merge base, and comments the report.

Read Historical Reports Correctly

Reports under wow-benchmarks/results/reports/ are bound to the source, run specification, machine, JVM, and service configuration that produced them. They are qualified historical evidence or investigation starting points, not universal promises across versions, machines, or stores.

Follow three rules:

1. do not hand-edit report rows or frontier JSON; use the corresponding generation task;
2. do not use Quick point estimates to claim a formal throughput change;
3. do not use component or simulated-I/O results to claim production end-to-end capacity.

CI-to-Local Evidence Map

Workflow	Current command
Local Test	allLocalTest + localCoverageReport
Contract Test	allContractTest + contractCoverageReport
Integration Test	allIntegrationTest + integrationCoverageReport
Mixed-Version	:example-server:installDist + :wow-it:integrationTest --tests 'me.ahoo.wow.it.mixed.MixedVersionClusterTest' with the released image
Benchmark Smoke	:wow-benchmarks:test + :wow-benchmarks:benchmarkSmoke
Benchmark A/B (dispatch or benchmark-ab label)	:wow-benchmarks:jmhJar per ref, then the JMH jar with interleaved forks
Codecov	codeCoverageReport

Choose these layers locally according to change risk. CI is fresh evidence in another environment; local validation, CI validation, application release, and production verification remain separate completion conditions.