Elasticsearch

wow-elasticsearch implements Elasticsearch-backed EventStore, SnapshotStore, and event-stream/snapshot query backends. Use it when Elasticsearch is already operated and the read side needs full text, aggregations, or large cursor scans. Do not add a search cluster only for event persistence.

Architecture Overview

Wow owns index names, templates, document shape, version guards, query schema, and storage bindings. Elasticsearch owns mappings, analyzers, shards, replicas, refresh, PIT, search_after, and bulk execution. Classpath presence is not wiring; event or snapshot storage must select elasticsearch.

Installation

Direct dependencies:

implementation("me.ahoo.wow:wow-elasticsearch")
implementation("org.springframework.boot:spring-boot-starter-data-elasticsearch")

Starter capability:

implementation("me.ahoo.wow:wow-spring-boot-starter") {
    capabilities { requireCapability("me.ahoo.wow:elasticsearch-support") }
}

Configuration

spring:
  elasticsearch:
    uris: http://localhost:9200

wow:
  eventsourcing:
    store:
      storage: elasticsearch
    snapshot:
      storage: elasticsearch

Defaults are wow.elasticsearch.enabled=true, auto-init-template=true, query.batch-size=10000, and query.keep-alive=1m; compatibility-version and index-prefix are unset. Event/snapshot batching is disabled. Enabled defaults are max-size=128, max-delay=1ms, max-pending-*=4096, and lane-count=1.

Spring Data Elasticsearch Configuration

spring.elasticsearch.* owns connection, authentication, TLS, and client timeouts. Wow uses Spring's reactive client and operations rather than duplicating those properties.

Wow Configuration

query.batch-size must be in 1..10000, and keep-alive must be at least 1ms. Set compatibility-version only when the deployment requires REST compatibility headers, then verify the value against the target cluster.

Write Batching

EventStore batching uses Bulk create. Both direct and batched SnapshotStore writes atomically guard on _source.version, so an older snapshot cannot overwrite a newer one. Enable batching only from throughput evidence; queue bounds, shutdown draining, and partial bulk failures become new runtime boundaries.

Index Naming Rules

Default event indexes are wow.${contextAlias}.${aggregateName}.es; snapshot indexes are wow.${contextAlias}.${aggregateName}.snapshot. Names participate in storage and query routing, so renaming is a data migration.

Several deployments on one cluster

The names are fixed per aggregate, so two deployments of the same service on one cluster (dev and staging, say) would read and write each other's indices. Give each one a prefix with wow.elasticsearch.index-prefix:

wow:
  elasticsearch:
    index-prefix: staging.

The prefix is put verbatim before every name Wow derives, on the write path, the event store and snapshot reads, the query backends and their schema:

What	Unset (default)	index-prefix: staging.
Event stream index	wow.sales.order.es	staging.wow.sales.order.es
Snapshot index	wow.sales.order.snapshot	staging.wow.sales.order.snapshot
Event stream template	wow-event-stream-template (wow.*.es)	staging.wow-event-stream-template (staging.wow.*.es)
Snapshot template	wow-snapshot-template (wow.*.snapshot)	staging.wow-snapshot-template (staging.wow.*.snapshot)

Unset or blank, every name is exactly as before. An index definition keeps its unprefixed file name (wow.sales.order.snapshot.json) and creates the prefixed index; aliases a definition declares are created as written, so give them per-deployment names yourself. The view store's indices follow the same prefix.

The prefix must be lowercase, contain none of \ / * ? " < > | , # : or whitespace, and not start with -, _, + or .; an invalid prefix fails startup. It may not start with wow. either: its template patterns would overlap the unprefixed ones at the same priority, which Elasticsearch refuses. For the same reason, two prefixes on one cluster must not be one another plus wow. (dev. and dev.wow.).

Setting or changing the prefix of a running deployment moves it to new, empty indices; reindex the old ones first (POST _reindex from wow.sales.order.snapshot into staging.wow.sales.order.snapshot, after the prefixed templates and definitions exist). Index privileges granted by name or pattern must cover the prefixed names, and the template privileges the prefixed templates.

Snapshot Query Field Resolution

The storage adapter (ElasticsearchQuerySchemaAdapter, a QueryStorageAdapter) reads the target mappings and reports them as storage facts: the physical paths bound for exact match, range, sorting, presence, and projection. QuerySchemaCatalog compiles those facts with the logical model into the QueryModelSchema; admission resolves each field reference against those bindings, and the compilers consume the resulting ResolvedFields. Multi-fields, runtime fields, and disabled objects follow Elasticsearch mappings; do not guess .keyword in the HTTP layer.

Keywords with ignore_above

A keyword whose ignore_above is 8191 or more counts as indexing every value and has the operators of an uncapped keyword: exact match, in/notIn, prefix and contains, range, sorting and terms aggregation. Wow's default templates give tags.* (the ABAC tags), id, *Id and the event stream's dynamic strings exactly this cap, the most characters that fit Lucene's 32766-byte term limit; without it, a longer value would fail the whole document write. The limit: a value longer than ignore_above is kept in _source but not indexed, so no filter matches it (eq, in, prefix or contains with that value find nothing; ne and notIn include the document), it sorts as missing and it falls into no terms bucket. Presence filters (exists, isNull, isEmpty) still see the value through Elasticsearch's _ignored field, so a resource whose ABAC tag is over-long never reads as untagged, that is public; inside a nested element such a value reads as missing. _ignored also names a field whose value ignore_malformed dropped (a number or date that did not parse), so isNull, notExists and isEmpty treat that value as present as well. A flattened field records nothing in _ignored, so one with ignore_above offers no presence filter.

A smaller ignore_above, such as the keyword sub-field (ignore_above: 256) Elasticsearch's own dynamic mapping puts under an unmapped string's text, can drop values a query would need, so the field has no query operators; a string enum whose every declared value fits the cap is the exception. Map such fields explicitly (see String field aggregability). Wow 9.1.x refuses every keyword with ignore_above apart from such enums.

From 9.2, Wow's snapshot template maps a dynamic string under state as text with a keyword sub-field capped at 8191 (the string_as_text_with_keyword rule): its .keyword has the operators above, and full-text search on the text is unchanged for 9.1 nodes. A dynamic floating-point value is mapped as double (floating_as_double) instead of Elasticsearch's 32-bit float. Templates apply when an index is created: an index created before the upgrade keeps its 256 cap and its float fields until it is reindexed (see Reindex an existing index).

Decimals on integral fields

A value the logical model declares as a decimal (a JSON Schema number, such as a BigDecimal amount) is filtered, sorted and aggregated only on a floating field: double, float, half_float or scaled_float. On an integral field (long, integer, …) Elasticsearch truncates the decimals it indexes, so a sum of 10 and 12.75 reads 22; dynamic mapping makes such a field when the first value it sees is integral. That path offers presence only: any other operator on it is refused as UNSUPPORTED_CAPABILITY, and the server logs one warning per index and path naming the field. Map decimal amounts as scaled_float (with a scaling_factor, such as 100 for cents) or double in an index definition.

Before the First Write

Wow creates an aggregate's index on its first write, so a new aggregate has no index until then. Queries of a missing index answer nothing: lists, pages and cursors are empty, counts are 0, and an aggregation without groups answers its empty summary, as a snapshot load of a missing index does. The query schema of a missing index is compiled from the mapping the index will be created with: the matching index templates, as Elasticsearch simulates them for the index name (POST _index_template/_simulate_index/<index>), or no field when no template matches. That needs the manage_index_templates cluster privilege, which auto-init-template needs as well; without it the schema stays unavailable (QuerySchemaUnavailable, HTTP 503) until the index exists.

A field the simulated mapping does not map yet (one dynamic mapping will add, or an ABAC tag key) is admitted with every operator its logical type allows, so a filter, sort or group-by on it answers nothing rather than being refused; a field the templates map keeps what its mapping proves. This provisional schema is never published: it is kept for 2 seconds (DefaultQueryModelSchemaProvider.DEFAULT_PROVISIONAL_TTL), so an aggregate that is never written does not simulate its templates on every query, and the first query after that compiles it again. Within 2 seconds of the first write the index's own mapping is compiled and published, with the fields its first documents added, without waiting for a revalidation. An aggregate with an index definition (below) has its index from startup and never reaches this path.

Revalidate the Runtime Query Schema

After mappings change, the runtime schema must be resolved again. Each instance revalidates its query schemas every wow.query.schema.revalidate-interval (5 minutes by default), so a field that dynamic mapping adds to an existing index, or a new index behind an alias, is not queryable for up to that long; to pick up a change now, call the wowQuerySchema actuator endpoint on every instance. Revalidation updates the in-memory schema only; it does not backfill documents or change mappings.

Configure Event Stream Index Template

With auto-init-template=true, IndexTemplateInitializer verifies the event template. Request failure, empty response, or missing acknowledgment fails storage wiring. If the platform owns templates, disable initialization only with versioned template and deployment evidence.

The event template stores each event's body unindexed (enabled: false). An application that queries the fields of its events (body.body.*) ships a concrete index definition for that aggregate's event stream, META-INF/wow/elasticsearch/wow.sales.order.es.json or config/wow/elasticsearch/wow.sales.order.es.json, with the same rules as the snapshot definitions below. Map body.body as an object with dynamic: false and only the fields queried, so the bodies of the other events add no field.

Configure Snapshot Index Template

The snapshot template defines system fields and the dynamic-state baseline. A template affects new indexes or later mapping behavior; it does not repair an existing index.

The generic snapshot template is the fallback for storage-only snapshots. Queryable snapshots should provide a concrete index definition with business mappings at META-INF/wow/elasticsearch/wow.sales.order.snapshot.json or config/wow/elasticsearch/wow.sales.order.snapshot.json:

{
  "mappings": {
    "properties": {
      "state": {
        "properties": {
          "status": { "type": "keyword" }
        }
      }
    }
  }
}

The resource key is the index name Wow computes without index-prefix; with a prefix, the index created is the prefixed one. The working-directory file replaces classpath files; without a working file, duplicate classpath files fail startup. Missing resources keep the generic-template behavior. Existing indexes are skipped, so mapping changes require explicit reindex or migration: when an existing index maps a path of its definition otherwise (an index created from the template alone, before the definition shipped), startup logs a warning naming those paths and carries on. Resource JSON follows Elasticsearch client and cluster validation semantics. Resource presence requests creation regardless of storage-routing configuration.

The drift check compares properties only, path by path with each field's own parameters (its type, ignore_above, scaling_factor, dynamic of an object, …); root parameters of the mapping, such as a root dynamic, and the index settings are not compared.

Privileges. Startup reads and creates the indices that have a definition, so the host's Elasticsearch user needs, on those indices, view_index_metadata (whether the index exists, and its _mapping for the drift check) and create_index. With auto-init-template=true it needs the cluster privilege manage_index_templates as well, which also lets the query schema of a missing index be simulated from the templates (see Before the First Write). Reading and writing documents need their usual privileges.

Reindex an existing index

Wow does not change an existing index's mapping. To give an index the mapping of a new definition or template, reindex it through a temporary index with the application stopped:

1. PUT <index>-new with the body {"mappings":{"enabled":false}}, and POST _reindex from <index> into it. The temporary name matches no Wow template (wow.*.es, wow.*.snapshot), so only its body maps it: a body that maps part of the documents (the definition alone, say) lets Elasticsearch map the rest dynamically, and the reindex fails on a document whose fields have another shape. With the mapping disabled, it keeps _source only, which is all the way back needs.
2. Delete <index>, PUT it again with the body of its definition (the template supplies the rest; an index without a definition can be left to the first write, or created with an empty body), and POST _reindex back from <index>-new.
3. Delete <index>-new and start the application: the drift warning is gone.

Checklist for a host on Elasticsearch

• Ship an index definition under META-INF/wow/elasticsearch/ for each aggregate whose state (or events) the application queries; startup creates it before the first write.
• Map an array of objects that is filtered or expanded element by element as nested: Elasticsearch flattens a plain object array, so conditions on two fields could match two different elements.
• Map decimals as scaled_float or double (Decimals on integral fields).
• Existing indices keep their mapping: a new definition or Wow's newer template rules reach an index only through a reindex; the startup warning names the paths that differ.
• Grant the privileges the startup needs.

Full-Text Search

Full text comes from a target field's text mapping and analyzer. wow-elasticsearch does not promise it for every string.

Add Full-Text Index for State Fields

Declare analyzers and text/multi-fields in platform-owned templates without replacing required Wow system fields. Verify mappings for both old and new indexes.

Execute Full-Text Search

Use a field through the Wow query API only when runtime schema publishes the corresponding capability. Native Elasticsearch DSL is not automatically part of the public Wow request model.

Aggregation Queries

The Wow aggregation AST compiles to Elasticsearch aggregations. Nested elements, numeric/time types, and missing-value semantics depend on both the public contract and mappings. Verify them with real backend TCK/integration tests.

Most groups and metrics run on the mapped fields themselves. These run a Painless script instead:

• per document, as a request-local runtime field: DATE_PART groups; groups, metrics and metric filters over a DATE_DIFF or arithmetic expression; and a DATE_HISTOGRAM on an epoch field unless the field is a single-valued long (or another integral type) in milliseconds grouped in UTC. A numeric field takes no time_zone, so an epoch-millis histogram in any other zone also runs the script;
• per bucket: bucket_script for derived metrics, which costs far less than a per-document script.

A cluster that restricts inline scripts (script.allowed_types, script.allowed_contexts) refuses these aggregations, and with search.allow_expensive_queries: false it refuses the filters on runtime fields. Map a field natively, for example epoch milliseconds as long, to keep its aggregations script-free.

A distinct count (cardinality) is exact up to 40000 distinct values per bucket (precision_threshold, Elasticsearch's maximum) and an estimate above, which the schema states in approximateMetrics. Each bucket keeps about 8 bytes per distinct value up to that threshold.

Index Design Recommendations

Design indexes from query, write, retention, and recovery objectives. Do not add text/keyword multi-fields to every state field by default.

Sharding Strategy

Elasticsearch owns shards, replicas, and routing. Validate actual shard size, write concurrency, and query fan-out; Wow does not select a topology.

Index Lifecycle Management (ILM)

If EventStore is authoritative, deleting events through ILM breaks replay. Configure rollover/delete only when data ownership and recovery explicitly permit it. Snapshot lifecycle must match its rebuild path.

Performance Optimization

Observe bulk latency/errors, refresh, segments, heap, PIT count, and query latency before changing batch, mappings, or topology.

Bulk Indexing

Batch options require max-size>1, positive max-delay, pending capacity no smaller than batch size, and lane-count>0. One aggregate remains in one lane. Increase lanes only for a measured concurrency bottleneck.

Query Optimization

Full scans use PIT plus search_after, with configured batch size and keep-alive. Batch size also cannot exceed target index.max_result_window. Fix mappings and query shape before blindly increasing it.

An offset page reads from + size records, which Elasticsearch caps at the index's index.max_result_window (10000 unless the index sets it). A paged query whose page × size passes it is rejected before the search as SIZE_OUT_OF_RANGE (HTTP 400) on pagination; read deeper with a cursor query.

The query schema reports this bound as limits.maxPageWindow: the smaller of the query budget's window and the index's index.max_result_window. A budget window of 0 sets no bound of its own, so the index's still applies.

Troubleshooting

Verified failures include template request/empty/unacknowledged responses, invalid query or batch bounds, bulk item errors, stale-snapshot guards, and mapping/schema conflicts.

Common Issues

Retain index/alias, resolved mapping, request, item-level response error, and runtime schema as evidence.

1. Query reports an unmapped, incompatible, or ambiguous multi-field

Inspect the actual mapping and runtime schema. Do not hard-code .keyword for every field. Correct templates/mappings or the explicit public field contract, then revalidate the schema.

2. The revalidation endpoint is unavailable or revalidation fails

Verify that Spring Boot Actuator is on the classpath and exposes the wowQuerySchema endpoint on the management plane, and that the query factory is wired so QuerySchemaCatalog has the model to revalidate; the catalog's wow.query.schema.refresh metric reports each outcome. Mapping-read failure must remain a failure rather than degrading to “all fields are queryable.”

3. An alias or data stream cannot be resolved

The current converter emits concrete index names, and writes go to them. When a name resolves to several indices (an alias over rollover indices, a data stream), queries see only the fields every index behind it maps the same way; a field one index lacks or maps differently stays unqueryable until every index maps it alike. Indices whose _source settings differ fail closed. Moving writes to aliases or data streams still requires a migration consistent across reads, writes, and mapping resolution.

4. Old data is still unqueryable after updating a template and revalidating

Templates do not rewrite historical mappings or data. Reindex or migrate explicitly; schema revalidation only rereads current backend capability.

5. A runtime-field query is rejected

Runtime-field projection and some capabilities are deliberately limited by mapping resolution. Follow the runtime schema instead of bypassing public query validation.

Complete Configuration Example

spring:
  elasticsearch:
    uris: ${ELASTICSEARCH_URIS}

wow:
  elasticsearch:
    auto-init-template: true
    query:
      batch-size: 10000
      keep-alive: 1m
    event-store-batch:
      enabled: false
    snapshot-store-batch:
      enabled: false
  eventsourcing:
    store:
      storage: elasticsearch
    snapshot:
      storage: elasticsearch

Best Practices

• Select event/snapshot storage explicitly and inspect resulting bindings.
• Let the platform own mappings, templates, ILM, backups, and reindexing.
• Preserve snapshot version guards and item-level bulk failures.
• Verify mappings, PIT, aggregations, and upgrades on a real cluster.

Focused check:

./gradlew :wow-elasticsearch:check

Next, read Query and Infrastructure configuration.