Exactly-once is a contract, not a checkbox

Turning on processing.guarantee=exactly_once_v2 feels like solving the problem. It solves one link in the chain: the read-process-write cycle inside Kafka. The moment your consumer touches a database, an HTTP API, or a file, you are back to at-least-once with extra steps.

What the transaction actually covers

The producer's transaction spans the records it writes and the offsets it commits. Both land or neither does.

producer.initTransactions();
try {
    producer.beginTransaction();
    for (ConsumerRecord<String, Order> record : records) {
        producer.send(toShipment(record));
    }
    producer.sendOffsetsToTransaction(offsets, consumer.groupMetadata());
    producer.commitTransaction();
} catch (ProducerFencedException e) {
    producer.close();
} catch (KafkaException e) {
    producer.abortTransaction();
}

Note what is missing: any call to a system that is not Kafka. Put a repository.save() in that loop and the guarantee is gone — the database commit and the Kafka commit are two separate decisions, and no amount of configuration makes them one.

The part people skip

Consumers must be told not to read uncommitted data. The default is not what you want.

A transactional.id that changes on restart defeats fencing: the broker cannot tell your new instance from a zombie old one.

When to stop trying

If the write target is not transactional with Kafka, aim for idempotency instead. Give each message a deterministic key, make the downstream write an upsert, and accept duplicates as a fact of life. That is a smaller promise, and unlike exactly-once across systems, it is one you can actually keep.