Pentaho

Hi Community members,

We are exploring PDI for Kafka streaming / ETL purpose. As part of which , I tried using Kafka Consumer step to consume the events stream from Kafka server. The same Kafka server is working fine with other consumer applications (written in Java). But when used with Pentaho (Version 9), the streaming is not working. Given below is the screenshot, also attached (PDI Consumer.png) where you could see that the consumer gets stopped on its own without throwing any errors. Tried launching the transformation with Log level set to Row Level mode, still not much information on the Spoon / Pentaho logs.

From the Kafka server log, I could see that Consumer group "n4snapshotcg1" which I have configured in Kafka Consumer step gets registered in Kafka. The broker then tries to rebalance the group and assigns the work to the consumer group. But following that, the Pentaho consumer leaves the group by itself and terminates its work. Attached the Kafka server log snippet (PDI Consumer with Kafka.png) showing this detail.

I can see that the network connectivity exists between Pentaho to Kafka and back from Kafka to Pentaho. But still not sure, why Pentaho transformation gets completed and not continuing to stream the messages.

Consumer Option "auto.offset.reset" is set to earliest. Sub Transformation exists with "Get Records from Stream" step. Batch configuration option in kafka consumer is as given below and attached as well (PDI Consumer Batch Options.PNG):

 I am stuck with this stage and not able to proceed further. Any help on this is highly appreciated.

Thanks!!!