Persistence
Persistence modes
kafka-flow keeps the state of each key in memory while it processes a partition. To survive a restart or a partition rebalance without replaying the whole input topic from the beginning, that state is persisted and recovered when the partition is next assigned. Persistence is optional — it is only needed when reprocessing the full journal on every recovery is too expensive — and two backends are provided:
- Cassandra (
kafka-flow-persistence-cassandra-v4) — stores per-key journals (the folded events) and/or snapshots (the latest state) in Cassandra tables. SeeCassandraPersistenceand Cassandra below. - Kafka (
kafka-flow-persistence-kafka) — stores per-key snapshots in a dedicated Kafka compacted topic, recovered by reading that topic to the end on assignment. SeeKafkaPersistenceModuleOf.
Both backends recover state per key during partition assignment, relying on Kafka's guarantee that a partition is owned by a single consumer in the group. The stale-writer protections below cover the one case where that ownership guarantee is not enough for the Kafka backend. The Cassandra backend writes snapshots unconditionally (last write wins, no offset check), so it stays exposed to the stale-writer overwrite, like a custom store (see Custom snapshot storage).
Cassandra
kafka-flow-persistence-cassandra-v4 is built on the Apache Cassandra Java driver 4, via
scassandra (scassandra-v4) and
cassandra-sync. It supersedes
kafka-flow-persistence-cassandra, built on the DataStax Java driver 3, which is kept for the existing
users only.
CassandraModule connects to the cluster, creates the keyspace if configured to, and provides the
session, the lock used to serialize schema changes and a health check:
import cats.effect.{IO, Resource}
import com.evolution.kafka.flow.cassandra.{CassandraConfig, CassandraModule, CassandraPersistence}
import com.evolution.kafka.flow.key.KeySegments
import com.evolutiongaming.catshelper.LogOf
import com.evolutiongaming.kafka.flow.persistence.PersistenceModule
import pureconfig.ConfigSource
def persistence(implicit logOf: LogOf[IO]): Resource[IO, PersistenceModule[IO, String]] = {
val config = ConfigSource.default.at("cassandra").loadOrThrow[CassandraConfig]
for {
cassandra <- CassandraModule.make[IO](config)
persistence <- CassandraPersistence
.withSchema[IO, String](cassandra.session, cassandra.sync, config.consistencyOverrides, KeySegments.default)
.toResource
} yield persistence
}
The client section is the configuration of the Java driver 4, i.e. the contents of its
datastax-java-driver section, see its
reference.conf:
cassandra {
schema.keyspace.name = "kafka_flow"
consistency-overrides {
read = "LOCAL_QUORUM"
write = "LOCAL_QUORUM"
}
client {
basic.contact-points = ["127.0.0.1:9042"]
# required by the driver 4 once the contact points are set
basic.load-balancing-policy.local-datacenter = "datacenter1"
# the default of the driver 4 is 2 seconds, while scassandra for the driver 3 defaulted to 12 seconds
basic.request.timeout = 12 seconds
}
}
Migrating from kafka-flow-persistence-cassandra
The tables are the same, so the data written by kafka-flow-persistence-cassandra is read as is,
there is nothing to migrate in Cassandra. The code and the configuration need the following changes.
-
Replace the dependency with
kafka-flow-persistence-cassandra-v4. Do not keep both modules on the classpath, i.e. via a library depending on the old one: both definecom.evolution.kafka.flow.cassandra.CassandraModule, so one of them silently shadows the other. -
Change the imports, all the classes of the module are in
com.evolution.kafka.flow:kafka-flow-persistence-cassandrakafka-flow-persistence-cassandra-v4com.evolutiongaming.kafka.flow.cassandra.*com.evolution.kafka.flow.cassandra.*com.evolutiongaming.kafka.flow.key.{CassandraKeys, KeySchema, KeySegments}com.evolution.kafka.flow.key.*com.evolutiongaming.kafka.flow.journal.{CassandraJournals, JournalSchema}com.evolution.kafka.flow.journal.*com.evolutiongaming.kafka.flow.snapshot.{CassandraSnapshots, SnapshotSchema}com.evolution.kafka.flow.snapshot.*com.evolutiongaming.scassandra.CassandraSessioncom.evolution.scassandra.CassandraSessioncom.evolutiongaming.cassandra.sync.CassandraSynccom.evolution.cassandra.sync.CassandraSynccom.datastax.driver.core.ConsistencyLevelcom.datastax.oss.driver.api.core.ConsistencyLevelThe interfaces of the core module, such as
KeyDatabase,SnapshotDatabaseorPersistenceModule, stay incom.evolutiongaming.kafka.flow. -
CassandraModule.ofisCassandraModule.make, and needs noParallelanymore. -
Move the
clientconfiguration to the format of the driver 4, see Migrating theclientconfiguration. -
retriesapplies to the health check queries only, as before. The retries of the other queries are configured in the driver, i.e. withadvanced.retry-policyof theclientsection. -
basic.request.timeoutof the driver 4 limits a query including all its retries, while the timeout of the driver 3 limited each attempt. So a health check query is failed once the timeout runs out, even if fewer thanretriesattempts were made.
Migrating the client configuration
The old configuration, read as is, gives the defaults of the driver. CassandraConfig reads
schema, retries and consistency-overrides as before, but takes client as the configuration of
the driver 4. The driver 4 only reads its own options, under basic, advanced and profiles, so it
silently ignores every key of the old format:
cassandra.client {
contact-points = ["cassandra-1", "cassandra-2"] # ignored, the driver connects to 127.0.0.1:9042
query.consistency = "LOCAL_QUORUM" # ignored, the default LOCAL_ONE is used
authentication { username = "user", password = "..." } # ignored, no authentication
}
If Cassandra is not on the local host, the application fails to start, as 127.0.0.1:9042 is not
reachable. If it is, the application runs with the defaults of the driver.
There are two ways to migrate.
Rewrite client at once. See the
translation of the keys
for the option of the driver 4 matching each key. The options that are not set get the defaults of
the driver 4, which differ from the ones of scassandra for the driver 3, so set the ones to keep:
cassandra.client {
basic.contact-points = ["cassandra-1:9042", "cassandra-2:9042"]
# required by the driver 4 once the contact points are set; the driver 3 took it from the contact
# points unless `load-balancing.local-dc` was set
basic.load-balancing-policy.local-datacenter = "dc1"
basic.request.consistency = LOCAL_QUORUM
# the default of the driver 4 is 2 seconds, while scassandra for the driver 3 defaulted to 12 seconds
basic.request.timeout = 12 seconds
advanced.auth-provider { class = PlainTextAuthProvider, username = "user", password = "..." }
}
It is read by CassandraConfig as is, as in the example above.
Migrate gradually with scassandra-v4-compat. It is a separate dependency:
libraryDependencies += "com.evolution" %% "scassandra-v4-compat" % "5.9.0"
It translates the keys of the old format, keeping the defaults of scassandra for the driver 3, and
applies the options of the driver 4 under client.driver on top. So the keys can be moved to
client.driver one by one:
cassandra.client {
# the old format, translated
contact-points = ["cassandra-1", "cassandra-2"]
query.consistency = "LOCAL_QUORUM"
# the format of the driver 4, wins over the translated keys
driver {
basic.request.timeout = 3 seconds
}
}
CassandraConfig doesn't read client.driver either, so client is replaced with the translated
configuration. The keys having no equivalent in the driver 4, such as jmx-reporting, are dropped
with a warning:
import cats.MonadThrow
import cats.syntax.all._
import com.evolution.kafka.flow.cassandra.CassandraConfig
import com.evolution.scassandra.compat.CassandraConfigCompat
import com.evolutiongaming.catshelper.Log
import com.typesafe.config.Config
import pureconfig.ConfigSource
import pureconfig.error.ConfigReaderFailures
def cassandraConfig[F[_]: MonadThrow](config: Config, log: Log[F]): F[CassandraConfig] =
for {
loaded <- liftConfig[F, CassandraConfig](ConfigSource.fromConfig(config).at("cassandra").load[CassandraConfig])
translated <- liftConfig(CassandraConfigCompat.fromLegacyConfig(config, "cassandra.client"))
_ <- translated.warnings.traverse_(warning => log.warn(s"Legacy Cassandra configuration: $warning"))
} yield loaded.copy(client = translated.config)
def liftConfig[F[_]: MonadThrow, A](result: Either[ConfigReaderFailures, A]): F[A] =
result.leftMap(failures => new IllegalArgumentException(failures.prettyPrint())).liftTo[F]
Once all the keys are under client.driver, move them up to client, drop scassandra-v4-compat
and read the configuration with CassandraConfig as is:
# before
cassandra.client.driver {
basic.contact-points = ["cassandra-1:9042", "cassandra-2:9042"]
basic.load-balancing-policy.local-datacenter = "dc1"
basic.request.timeout = 12 seconds
}
# after
cassandra.client {
basic.contact-points = ["cassandra-1:9042", "cassandra-2:9042"]
basic.load-balancing-policy.local-datacenter = "dc1"
basic.request.timeout = 12 seconds
}
Before dropping scassandra-v4-compat, set the options which still rely on it, as without it they get
the defaults of the driver 4: e.g. basic.request.timeout = 12 seconds to keep the previous timeout,
and basic.load-balancing-policy.local-datacenter, which it takes from the contact points if
load-balancing.local-dc is not set.
scassandra-v4-compat depends on scassandra for the driver 3, so the driver 3 stays on the
classpath until then.
Protecting against stale snapshot writes
Consumer-group ownership of the input topic does not extend to the snapshot store. During a rebalance a previous owner that has not yet observed the revocation (a network issue, a GC pause, a slow poll loop — typical during broker maintenance or high-load restarts) keeps folding events and flushing snapshots alongside the new owner; with the default last-write-wins persistence a stale snapshot overwrites a newer one, and the next recovery loads stale state — losing the events between the two snapshots even though their offsets were committed. See kafka-flow#732; overlaps of tens of seconds have been seen in production.
This page is about turning the protection on and running it; for how it fences a stale writer, see the Kafka single-writer design.
Timer settings change how often the window is hit:
TimerFlowOf.persistPeriodically(flushOnRevoke = true) makes it more likely (revoked partitions
flush while the new owner starts up); a higher persistEvery makes it less likely, at the cost of
more events to replay on recovery.
For the Kafka snapshot backend the protection is transactional snapshot writes — opt-in, off by
default, enabled with KafkaPersistenceModuleOf.cachingTransactional. (A custom SnapshotDatabase
can implement its own protection — see Custom snapshot storage.)
What a rejected write looks like
You do not catch the rejection yourself; it is handled for you:
- Periodic flush — the conflict fails the stale instance's flow. That is safe (it no longer owns
the partition), unless you set
persistPeriodically(ignorePersistErrors = true), in which case it is logged and swallowed. - Flush-on-revoke — the conflict surfaces as a cache-entry release error that scache prints to
System.err— not via the logging framework — and swallows (scache: failed to release cache entry: ...), so the partition hands off cleanly.
Either way the rejected write does not land and no offset is committed for it, so the new owner replays the affected events.
Transactional snapshot writes (Kafka)
EXPERIMENTAL — use at your own risk: the mechanism is design-verified but not yet proven in production operation, and unknown defects may remain. No compatibility guarantee: configuration, API, and behavior may change in any release, without deprecation.
Enable with KafkaPersistenceModuleOf.cachingTransactional. The flow supplies the driving consumer's
group metadata (generation) to the module, which uses it to fence stale writers — so you build the
module like any other and wire it into the flow as usual:
val moduleOf = KafkaPersistenceModuleOf.cachingTransactional[F, State](
consumerOf = consumerOf,
producerOf = producerOf,
config = KafkaPersistenceModule.TransactionalConfig(
consumerConfig = snapshotConsumerConfig,
producerConfig = snapshotProducerConfig,
transactionalIdPrefix = applicationId,
snapshotTopic = stateTopic,
// also tunable: maxWritesPerTransaction, recoveryStallTimeout (both below)
),
)
// wire it into the flow as usual:
// KafkaFlow.resource(consumerResource, ConsumerFlowOf(inputTopic, TopicFlowOf(kafkaEagerRecovery(moduleOf, /* ... */))))
idempotence and the per-partition transactional.id are set for you — don't configure them in
producerConfig — and the snapshot consumerConfig is forced to read_committed, with groupId
cleared and autoCommit off: the recovery readers never join a group or commit offsets.
The id is stable per partition ("<prefix>-snapshot-<partition>"): every owner of a partition shares it, so a
takeover's initTransactions fences the previous owner's producer and aborts any transaction it left
open. The input topic whose offsets are committed transactionally, and the consumer generation used
to fence stale writers, are both supplied
by the flow (from the assigned partition and the driving consumer), so neither is part of
TransactionalConfig. One module serves one flow: snapshots are keyed by partition number alone, so
each input topic needs its own module with its own snapshotTopic — sharing a snapshot topic between
flows would mix their state on recovery.
transactionalIdPrefix does not affect fencing of stale writers (that is by consumer generation) —
it is a readable label and, on an ACL-secured cluster, the transactional.id prefix your producer
principal must be authorized for. Because the id is stable per partition, the prefix must be unique
per flow: use your applicationId, and an application running several flows must append a per-flow
discriminator (e.g. the input topic) or the flows share ids and fence each other — an
"<applicationId>*" prefixed ACL still covers it.
Snapshot writes and the input-offset commit run in one Kafka transaction per assigned partition; a
write from a stale consumer generation is fenced by the broker
(KIP-447,
brokers 2.5+) and surfaces as
CommitFailedException — or as a producer-epoch error (ProducerFencedException or
InvalidProducerEpochException, by transaction protocol version) when a new owner's initTransactions has
already fenced the stale producer; rejected either way. Recovery reads read_committed, so a
fenced writer's aborted records are
never recovered. After a hard crash the new owner takes over immediately (aborting the crashed
owner's unfinished transaction) and recovers everything that was committed. If an unfinished
transaction belongs to some other transactional.id (see the limitations for when that happens),
recovery waits until the broker aborts it instead — slower, but nothing committed is ever missed.
- Cost — snapshot writes commit in Kafka transactions (a few ms each on real brokers), and cost
tracks the number of transactions more than their size. Concurrent key flushes are group-committed,
so a burst of N dirty keys is ≈ N /
maxWritesPerTransactiontransactions (default 256) — at the default cap the overhead is small (see the design doc's Measurements). Each partition also holds its own producer and transaction-coordinator state on the brokers. - Tuning for transaction time — a transaction must commit within
transaction.timeout.ms(a producer config, default 1 min, ≤ the broker'stransaction.max.timeout.ms). Large snapshots lengthen it with the batch — lowermaxWritesPerTransaction(at a throughput cost) or raise the timeout. Raising it does not slow normal recovery (a takeover aborts this id's unfinished transactions immediately); it only lengthens the prefix-change wait (below). - Output is at-least-once — output produces stay outside the snapshot transaction, so a replayed batch re-emits them; the consuming side must tolerate duplicates. Only the snapshot store and the input-offset commit are kept consistent (corruption prevention, not exactly-once).
- Rollout — no migration (recovery under
read_committedstill reads existing non-transactional records). A rolling deploy is safe; while the two modes coexist a non-transactional instance is not fenced — the same exposure you already have without this mode, gone once every instance is transactional. - Recovery fails loudly rather than hangs — a recovery read that makes no progress for
recoveryStallTimeout(default 3 min) fails withRecoveryReadStalledErrorinstead of hanging the rebalance until the member is silently evicted atmax.poll.interval.ms. The error names its diagnosed cause. Truncation: the snapshot log lost acknowledged records under the read (an unclean leader election or an equivalent disaster) — the records are gone, an offset-reset or restore decision. An outlived transaction: one whose producer'stransaction.timeout.msmerely exceeds the deadline heals on its own once the broker aborts it; a hanging transaction, whose last-stable-offset pin never clears, is detected and aborted with thekafka-transactions.shtool (KIP-664); brokers 3.6+ prevent it from arising by default (KIP-890). If the diagnosis comes back undetermined (the high-watermark re-read itself failed), fall back to the broker alerts that follow. Cluster-side, the matching broker alerts areUncleanLeaderElectionsPerSec > 0(truncation risk) andPartitionsWithLateTransactionsCount > 0(hanging transactions). Consumer lag metrics read zero during the wait or stall (lag is measured to the last-stable-offset, where the read parks), so alert on this mode's log signals, not on lag. Keep the value well belowmax.poll.interval.msand above the legitimate wait for an unfinished transaction (transaction.timeout.msplus the broker's abort scan). - Reducing truncation risk — the deadline only flags lost records; it cannot recover them, and it
catches truncation only while a recovery read is in flight. Reads run only at partition assignment,
so a truncation usually lands between them and is adopted silently by the next recovery. So guard
against it at the broker: keep the snapshot topic durable with
unclean.leader.election.enable=false(the default),min.insync.replicas≥ 2, and a replication factor ≥ 3 — the transactional producer already forcesacks=all. An acknowledged snapshot then survives any single broker failure; truncation requires an opted-in unclean election or a disaster beyond the replication factor.
Limitations:
- A batch shares its transaction's outcome: if the transaction fails, every write in it fails.
- An old owner can be fenced while flushing on revoke; its last state delta is then neither persisted
nor committed, so the new owner replays those events — noise, not loss. Under the classic
cooperative assignor this is every revocation: the revoke-time flush is always fenced, so
flushOnRevokedoes not shrink the replay window there. - A stale owner's late
initTransactionscan fence the current owner's producer: the current owner's flow fails once and recovers (rebalance and replay); no wrong write can land. Rare, and a different fence — the producer epoch (its errors above), not the group generation (CommitFailedException). - A
transactionalIdPrefixchange can cost recovery a wait: an old-prefix instance that dies mid-transaction during the rollout (any unclean death — a crash, an OOM kill, a forced pod delete) leaves that transaction under an id no new instance will ever init, so recovery waits until the broker times the transaction out — up to ~70 s at the defaults (transaction.timeout.msplus the broker's abort scan), never a wrong read. A transaction is open only during a synchronous flush or offset commit, so a graceful rollout leaves nothing open. (A foreign producer's transaction on the snapshot topic is waited out the same way — but the topic must be exclusive to the flow regardless.) - The mode always uses the identity
KafkaPersistencePartitionMapper(fencing is per input partition); a non-identity mapper is not supported here. - The fence works under both the classic and the consumer group protocols
(
group.protocol=classic|consumer). Withconsumer, use brokers 4.3.0+ (KIP-1251) — below that a still-valid owner can be spuriously fenced during a rebalance and crash; the restart converges, but any later rebalance can fence again (safe, never corruption, but not stable). Under classic,CooperativeStickyAssignorkeeps retained partitions writing through the generation bump, so the spurious fence can occur on any rebalance and each one crashes the flow; underretryOnErroreach restart fences the peers in turn, and the loop can last minutes. An eager assignor (StickyAssignor,RangeAssignor) avoids it by tearing every flow down before the bump.
Custom snapshot storage
You can plug in your own snapshot store: implement SnapshotDatabase and wire it through
SnapshotsOf.backedBy into PersistenceOf.snapshotsOnly/restoreEvents. A custom store is
last-write-wins, so it is exposed to the same stale-writer overwrite
(#732) unless its persist rejects a
write when the store already holds a newer offset (taken from the snapshot) — that conditional write
is the fence (the buffer wiring does not provide it). Note that the delete(key) method carries no
offset, so a delete cannot be offset-gated through this interface; a custom store's delete stays
unconditional.
Compression
Kafka-flow has a built-in support for compressing application's state
when it's being persisted. This can be achieved by creating an instance of Compressor
and enhancing a user-defined instance of ToBytes[F, State] with it
via a syntax extension. Additionally, you need to provide instances of
ToBytes and FromBytes to encode/decode a Header which contains
meta-information about compressed data.
The example below illustrates the approach. Note that it's using a simplified approach towards encoding both state and headers, and you may want to encode them differently (as JSON for example).
import cats.effect.IO
import com.evolutiongaming.kafka.flow.persistence.compression.{Compressor, Header}
import com.evolutiongaming.skafka.{FromBytes, ToBytes}
import com.evolutiongaming.kafka.flow.persistence.compression.CompressorSyntax._
import scodec.bits.BitVector
import scodec.codecs.{bool, int32}
// Application's state
final case class State(int: Int)
// Encoder of the application's state
val toBytes: ToBytes[IO, State] = (state, _) =>
IO.fromTry(int32.encode(state.int).map(_.toByteArray).toTry)
// Encoder/decoder of metainformation header
implicit val headerToBytes: ToBytes[IO, Header] =
(header, _) => IO.fromTry(bool.encode(header.compressed).map(_.toByteArray).toTry)
implicit val headerFromBytes: FromBytes[IO, Header] =
(bytes, _) => IO.fromTry(bool.decode(BitVector(bytes)).map(result => Header(result.value)).toTry)
// Resulting instance can be passed to other parts of kafka-flow's API
for {
compressor <- Compressor.of[IO](compressionThreshold = 10000)
toBytesWithCompression = toBytes.withCompression(compressor)
} yield ()
Compression metrics
Compressor can report metrics of a size of data before and after compression. Metrics support is available as a part
of FlowMetrics API from kafka-flow-metrics module in form of FlowMetrics#compressorMetrics(component)
where component is the name of the label that will be used for metrics of this compressor.
The following metrics are reported:
compressor_raw_bytes- the size of state before compressingcompressor_compressed_bytes- the size of compressed state (including library-added meta-information)
Note: these metrics had a _total suffix in earlier versions.
Starting with prometheus-metrics v1.0.0 this suffix is no longer allowed and has therefore been removed.
Users of simpleclient forked version 0.9.999-evo1 will see a change in the metric name, since the _total suffix is not automatically added in that version.
import cats.effect.syntax.resource._
import com.evolutiongaming.kafka.flow.FlowMetrics
import com.evolutiongaming.kafka.flow.metrics.syntax._
import com.evolutiongaming.smetrics.CollectorRegistry
val registry: CollectorRegistry[IO] = CollectorRegistry.empty[IO]
for {
flowMetrics <- FlowMetrics.of(registry)
compressor <-
Compressor
.of[IO](compressionThreshold = 10000)
.map(_.withMetrics(flowMetrics.compressorMetrics("settlement")))
.toResource
toBytesWithCompression = toBytes.withCompression(compressor)
} yield ()
Backward compatibility
To support smooth transition from raw state to using compression, the library implementation of Compressor tries
to detect if the byte array it tries to decompress starts with an opening curly bracket({). In this case it makes
an assumption that you keep the state in JSON and the particular byte array is in a raw format (without compression)
and doesn't attempt to decompress the byte array, returning it as-is.
Please note that it's going to work only if the application's state was encoded as JSON before!