I'm interested in setting up different semantics when retrieving data from Kafka with PDI: at most once, at least once, exactly once. I read [PDI-17272] Implement explicit commit in Kafka Consumer - Pentaho Platform Tracking and since there are no comments I'm a bit puzzled.
What am I doing?
I have one transformation with Kafka Consumer step (Kafka Consumer - Pentaho Documentation ) and on subtrasformation just logging this data to Elasticsearch. I can have the sub trasform to fail and so I'd like to test different reprocessing of data.
I'm on PDI CE 184.108.40.206-365 and Kafka 2.11
So, is this semantic configuration possible? And where should I start?
An idea I came up with is about using "enable.auto.commit" and "isolation.level" (Apache Kafka ) but I don't know if those are supported / overwritten...
Has anybody some experience to share?