AnsweredAssumed Answered

Kafka Consumer semantic in PDI 8.1

Question asked by Virgilio Pierini on Apr 17, 2019
Latest reply on Jun 12, 2019 by Virgilio Pierini


Hi all

I'm interested in setting up different semantics when retrieving data from Kafka with PDI: at most once, at least once, exactly once. I read [PDI-17272] Implement explicit commit in Kafka Consumer - Pentaho Platform Tracking and since there are no comments I'm a bit puzzled.

 

What am I doing?

I have one transformation with Kafka Consumer step (Kafka Consumer - Pentaho Documentation ) and on subtrasformation just logging this data to Elasticsearch. I can have the sub trasform to fail and so I'd like to test different reprocessing of data.

I'm on PDI CE 8.1.0.0-365 and Kafka 2.11

 

So, is this semantic configuration possible? And where should I start?

An idea I came up with is about using  "enable.auto.commit" and "isolation.level" (Apache Kafka ) but I don't know if those are supported / overwritten...

 

Has anybody some experience to share?

Thank you

Virgilio

Outcomes