AnsweredAssumed Answered

How to abort streaming Kafka Consumer

Question asked by Brad Gatewood on May 7, 2019
Latest reply on Jun 14, 2019 by Ramon Ankersmit

Pentaho Data Integration CE

I'm using the Kafka Consumer step in PDI v8.2. Since this step runs continuously, documentation suggests using an Abort step to stop the process periodically. I'm looking for some suggestions on how/where to implement the abort.

For scheduling purposes, I would prefer to use job running time as the abort condition. This would allow me to easily schedule the job at regular intervals and know that it would have already hit the abort condition before the next one kicks off. I tried doing this within the parent job. I added a loop and a condition to abort the job after 10 minutes. But as the job flow is sequential, after entering the Kafka Consumer step, it never returns from that continuously running step to complete the loop and evaluate the running time.

I can possibly abort the process in the sub-transformation if I can figure out how to periodically test for the time condition. However, I can't figure out how to engineer the conditional logic with the available transformation steps.

Does anyone have an example of how they were able to implement an abort for the Kafka Consumer?