Pentaho

 View Only

  • 1.  Kafka consumer step will not read messages

    Posted 05-06-2022 13:15
    I can run Kafka producer step and load messages but Kafka consumer step will not work.
    It is just Triggering heartbeat signal at every 10 seconds and there is no error message.
    I guess that this is an environmental issue and there may be some hidden setting, in the configuration that needs turning on or
    sum permission that needs setting for the step to consume the messages.
    This is a new Linux Box for PDI 8.3

    ------------------------------
    Vince Popplewell
    Systems Engineer
    Capgemini
    ------------------------------


  • 2.  RE: Kafka consumer step will not read messages

    Posted 05-08-2022 20:02
    Hi Vince

    It's a bit hard to tell without seeing what you've done in the transformation, but I presume you have set it up according to the reference Kafka Consumer and have a 'get records from Stream' step going ?

    Also, try this video Kafka components in Pentaho .   Does a very nice job in covering all the pieces of the puzzle.

    Regards

    ------------------------------
    Andrew Cave
    Systems Engineer
    BizCubed Pty Ltd
    Australia
    ------------------------------

    Original Message


  • 3.  RE: Kafka consumer step will not read messages

    Posted 05-09-2022 08:17
    Thanks Andrew,
         I am going through the links to Documentation and i have a new error:

    ERROR (version 8.3.0.23-1298, build 8.3.0.23-1298 from 2021-06-16 04.08.02 by buildguy) : java.lang.IllegalAccessError: tried to access field org.pentaho.di.trans.streaming.common.BaseStreamStep.subtransExecutor from class org.pentaho.big.data.kettle.plugins.kafka.KafkaConsumerInput

         Do you have any thoughts please?

    Regards Vince

    ------------------------------
    Vince Popplewell
    Systems Engineer
    Capgemini
    ------------------------------

    Original Message


  • 4.  RE: Kafka consumer step will not read messages

    Posted 05-09-2022 20:49

    hmmmm...that doesn't tell us much.  It looks like it is trying to access a private method in a class.

    https://github.com/pentaho/pentaho-kettle/blob/master/engine/src/main/java/org/pentaho/di/trans/streaming/common/BaseStreamStep.java   at line 235 looks like it should be trying to use getSubTransExecutor() instead.

    The current version of Kafka input https://github.com/pentaho/big-data-plugin/blob/master/kettle-plugins/kafka/src/main/java/org/pentaho/big/data/kettle/plugins/kafka/KafkaConsumerInput.java (line 87) is calling getSubTransExecutor

    and i can see that the previous commit on that step https://github.com/pentaho/big-data-plugin/blob/9fe165456988119264354d2cdf4e19b172b41fe3/kettle-plugins/kafka/src/main/java/org/pentaho/big/data/kettle/plugins/kafka/KafkaConsumerInput.java   was using subTransExecutor (which is the point where your step is failing)

    So the answer is update you EE install.   There should be version 27 available which will have all the fixes you need.

    Regards



    ------------------------------
    Andrew Cave
    Systems Engineer
    BizCubed Pty Ltd
    Australia
    ------------------------------

    Original Message


  • 5.  RE: Kafka consumer step will not read messages

    Posted 05-10-2022 09:38
    Thanks for the Reply Andrew,

         We do not want to change the version of Pentaho, as we have it installed in Pre-Prod and Prod and our code was all working last year. This is a new Dev Box for a reinstated project, that all worked in the past. We think the problem is with the Pentaho to Kafka connection. We can run Producer but no Consumer and we think it is a setting or a permission, on one of the Brokers, in the video or a key/trust store certificate error.

        How is the the broker, linked to that certificate please?

    Regards Vince

    ------------------------------
    Vince Popplewell
    Systems Engineer
    Capgemini
    ------------------------------

    Original Message


  • 6.  RE: Kafka consumer step will not read messages
    Best Answer

    Posted 05-10-2022 19:03
    Hi Vince

    The problem is that the version of the plugin you are using is calling subTransExecutor() in this class https://github.com/pentaho/pentaho-kettle/blob/master/engine/src/main/java/org/pentaho/di/trans/streaming/common/BaseStreamStep.java

    In the previous commit on kettle, the function subTransExecutor() was set to private and getsubTransExecutor() was implemented instead.

    In the current commit on the KafkaInput step, the coder has changed the call to getsubTransExecutor()  https://github.com/pentaho/big-data-plugin/blob/master/kettle-plugins/kafka/src/main/java/org/pentaho/big/data/kettle/plugins/kafka/KafkaConsumerInput.java

    But your step code is still calling the old version of the kettle class.  You are going to have to either update the step, or unpatch the kettle install to a lower version.  Currently, your step version and kettle versions are incompatible at that line.

    If it is a key/trust error, you will be getting an error from one of the internet related classes.    The line of error output you posted only shows the error for the incorrect/superseded call.


    If there are other errors, then please post them so we can get further into the problem.

    Regards

    ------------------------------
    Andrew Cave
    Systems Engineer
    BizCubed Pty Ltd
    Australia
    ------------------------------

    Original Message


  • 7.  RE: Kafka consumer step will not read messages

    Posted 05-16-2022 03:49
    Thank you Andrew,
         This was fixed by reverting back to the old code in GitLab and Re-committing in Dev. Thank You.
    Regards Vince

    ------------------------------
    Vince Popplewell
    Systems Engineer
    Capgemini
    ------------------------------

    Original Message


Related Content