AnsweredAssumed Answered

"Kafka Consumer" step run abnormally when using AEL-spark

Question asked by Chun Peng Hao Employee on Feb 13, 2018
Latest reply on Feb 14, 2018 by Chun Peng Hao

Dear experts


     I'm trying to use PDI to process streaming data from kafka, I have created two transform.

     In the parent transform, there is only one step "Kafka Consumer", it's used to receive the stream data from "topic1" on kafka cluster.

     In the sub-transform, there are two steps: "Get records from stream" link to "Kafka Producer 2", the former one get steam data from parent transform and the next step write the data to another topic "topic2" on the same kafka cluster.



The transform ran well in local mode, but it got a problem when using AEL to run in spark on yarn mode.

It seems that the "Get records from stream" can't get any data, and the "Kafka Producer 2" step only output "NULL" to "topic2" on kafka.



My softwares version are:PDI-CE8.0 spark2.1 kafka0.10 scala2.11 jdk8 centos7