I'm trying to use PDI to process streaming data from kafka, I have created two transform.
In the parent transform, there is only one step "Kafka Consumer", it's used to receive the stream data from "topic1" on kafka cluster.
In the sub-transform, there are two steps: "Get records from stream" link to "Kafka Producer 2", the former one get steam data from parent transform and the next step write the data to another topic "topic2" on the same kafka cluster.
The transform ran well in local mode, but it got a problem when using AEL to run in spark on yarn mode.
It seems that the "Get records from stream" can't get any data, and the "Kafka Producer 2" step only output "NULL" to "topic2" on kafka.
My softwares version are:PDI-CE8.0 spark2.1 kafka0.10 scala2.11 jdk8 centos7