Pentaho

 View Only

 Unable to execute transformation through Java program

  • Pentaho
  • Kettle
  • Pentaho
  • Pentaho Data Integration PDI
Zia Uddin's profile image
Zia Uddin posted 08-26-2018 17:30

Hi All,

I created a Pentaho kafka consumer transformation (2.ktr) in pentaho kettle 8.1 to consume kafka json messages with "get records from stream" sub transformation (3.ktr)

This "get records from stream"  sub transformation parse messages with "input json" and save it in "mongo db output".

 

But when i am running below code to execute above transformation through eclipse i am getting exception. Please suggest so that i can run below trasnformation program successfully.

 

i have attached ktr files,  screenshots and code below please refer

================================= Java code=============================

public static void main(String[] args) {

String file="c:/pdi/2.ktr";

try {

          KettleEnvironment.init();

          TransMeta metadata=new TransMeta(file);

          Trans trans=new Trans(metadata);

          trans.execute(null); // GETTING EXCEPTION HERE

          trans.waitUntilFinished();

          if(trans.getErrors()>0){

               System.out.println("Error Executing Transformation");

          }

     } catch (KettleException e) {

          e.printStackTrace();

     }

============Exception============\

2018/08/26 15:07:49 - Kafka Consumer.0 - ERROR (version 8.1.0.0-365, build 8.1.0.0-365 from 2018-04-30 09.42.24 by buildguy) : Can't run transformation due to plugin missing

2018/08/26 15:07:49 - Kafka Consumer.0 - ERROR (version 8.1.0.0-365, build 8.1.0.0-365 from 2018-04-30 09.42.24 by buildguy) : Error initializing step [Kafka Consumer]

2018/08/26 15:07:49 - 2 - ERROR (version 8.1.0.0-365, build 8.1.0.0-365 from 2018-04-30 09.42.24 by buildguy) : Step [Kafka Consumer.0] failed to initialize!

org.pentaho.di.core.exception.KettleException:

We failed to initialize at least one step.  Execution can not begin!

at org.pentaho.di.trans.Trans.prepareExecution(Trans.java:1163)

at org.pentaho.di.trans.Trans.execute(Trans.java:627)

at RunningTransformations.runTransformationFromFileSystem(RunningTransformations.java:153)


#Kettle
#PentahoDataIntegrationPDI
#Pentaho
Johan Hammink's profile image
Johan Hammink

The error message said that a plugin is missing. Is the Apache Kafka Consumer loaded in eclipse?

Zia Uddin's profile image
Zia Uddin

Hi Johan,

Thanks for reply. I have already imported kafka-clients-0.10.2.1.jar in the project class path.

Earlier it was giving plugin error while executing statement Trans trans=new Trans(metadata); but after importing above plugin it started giving error in next line below:-

trans.execute(null); // Error while executing this statement

Please see screenshots and ktr files i have attached Actually my Kafka consumer transformation takes another sub transformation :"Get record for stream" which writes json message to mongo in this transformation. it seems due to nested/sub trasnformation it is giving this error. but log is not very descriptive so i am unable to find out which further plugin is missed. Kindly refer attachments

Regards,

Zia

Dan Keeley's profile image
Dan Keeley

It's the *Pentaho* kafka plugin that is missing in your environment.   This is more than just the kafka jars.   I don't know but I think you'll have to find a way to initialize pdi with it's plugins as well, is the kafka plugin an osgi one?

Have a look in the plugins/steps folder and read up on that side of things.

Data Conversion's profile image
Data Conversion
Attachment  View in library
3.ktr.zip 2 KB
Data Conversion's profile image
Data Conversion
Attachment  View in library
2.ktr.zip 1 KB
Data Conversion's profile image
Data Conversion
Attachment  View in library
3rd.jpg 34 KB
Data Conversion's profile image
Data Conversion
Attachment  View in library
2nd.jpg 82 KB
Data Conversion's profile image
Data Conversion
Attachment  View in library
1st.jpg 21 KB