PartitionedSource

Reads the partitions of a topic from an offset: the port a Kafka-sourced subscription reads through. The runtime assigns the partitions to nodes by its own leases and keeps the offsets in its checkpoint store, so a source has no consumer group, commits nothing and keeps no position: each read starts where it is told.

  • partitions are the topic's partitions, in order.

  • partitionOf is the partition a message keyed key lands in, as the TopicProducer of the same broker picks it: every message of a key is in that partition, so the records of one aggregate are found by re-scanning it.

  • offsets are what a partition holds now: where a subscription that starts from the end starts, and whether an offset is still retained.

  • read emits the messages of one partition with an offset at or after from, in offset order. A message that is no longer retained is skipped, so the first one emitted may be later than from; offsets may also have gaps within a partition. The flow does not complete: after the messages written so far it waits for new ones, until it is cancelled. Bounded by until, it emits those before until only and completes once it has passed them.

Functions

Link copied to clipboard
abstract suspend fun offsets(topic: String, partition: Int): PartitionOffsets
Link copied to clipboard
abstract suspend fun partitionOf(topic: String, key: String): Int
Link copied to clipboard
abstract suspend fun partitions(topic: String): List<Int>
Link copied to clipboard
abstract fun read(topic: String, partition: Int, from: Long): Flow<SourceMessage>

open fun read(topic: String, partition: Int, from: Long, until: Long): Flow<SourceMessage>

The messages of partition from from to before until, then completes. By default the unbounded read up to the message at until - 1: a source whose partitions may end in a gap (such as transaction markers) overrides it.