Pentaho

 View Only

  • 1.  Error from Concat fields in PDI CE 9.3.0.0-428

    Posted 05-23-2022 11:40
    Hello everyone,
      I have noticed that in PDI CE 9.3 (on Open jdk 8) the Concat Fields component throws an error if in the Advanced tab the option Fast data dump (no formatting) is unchecked. With such option unchecked it goes into error even if the Length of Target Field is provided (is not 0 but a large number).

    Did anyone experience the same or is it just working fine for you?
    Thanks,
     Antonio

    2022/05/20 10:46:00 - Concat fields.0 - ERROR (version 9.3.0.0-428, build 9.3.0.0-428 from 2022-04-12 04.56.25 by buildguy) : Unexpected error
    2022/05/20 10:46:00 - Concat fields.0 - ERROR (version 9.3.0.0-428, build 9.3.0.0-428 from 2022-04-12 04.56.25 by buildguy) : org.pentaho.di.core.exception.KettleStepException:
    2022/05/20 10:46:00 - Concat fields.0 - Error writing line
    2022/05/20 10:46:00 - Concat fields.0 - 0
    2022/05/20 10:46:00 - Concat fields.0 -
    2022/05/20 10:46:00 - Concat fields.0 - at org.pentaho.di.trans.steps.textfileoutput.TextFileOutput.writeRow(TextFileOutput.java:511)
    2022/05/20 10:46:00 - Concat fields.0 - at org.pentaho.di.trans.steps.concatfields.ConcatFields.processRow(ConcatFields.java:147)
    2022/05/20 10:46:00 - Concat fields.0 - at org.pentaho.di.trans.step.RunThread.run(RunThread.java:62)
    2022/05/20 10:46:00 - Concat fields.0 - at java.lang.Thread.run(Thread.java:748)
    2022/05/20 10:46:00 - Concat fields.0 - Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
    2022/05/20 10:46:00 - Concat fields.0 - at org.pentaho.di.trans.steps.textfileoutput.TextFileOutput.writeRow(TextFileOutput.java:500)
    2022/05/20 10:46:00 - Concat fields.0 - ... 3 more

    ------------------------------
    Antonio Petrella
    Data Service Manager
    UNOG
    ------------------------------


  • 2.  RE: Error from Concat fields in PDI CE 9.3.0.0-428

    Posted 05-24-2022 01:37
    Hi Antonio

    I tried it on 93EE without a problem.

    Try running it using Azul OpenJDK11 instead of 8 ( add
    export JAVA_HOME=/Path/to/java11
    to the spoon.bat/command/sh file

    If that doesn't work, can you come back with the OS, and how long the fields you're concatenating are as well.

    ------------------------------
    Andrew Cave
    Systems Engineer
    BizCubed Pty Ltd
    Australia
    ------------------------------

    Original Message


  • 3.  RE: Error from Concat fields in PDI CE 9.3.0.0-428

    Posted 05-24-2022 04:47
    Hi Andrew,
      Thank you for your reply. I tested a very simple transformation ( Generate rows -> Concatenate Fields -> Log ) on two test environments that I have and the results are:

    - on Linux, the test transformation works without problems on PDI CE 9.3 (even with AdoptOpenJDK 8)
    - on Windows 10 20H2 the test transformation fails on PDI CE 9.3 (with either AdoptOpenJDK 8 or Azul open Jdk 11)
    - on Windows 10 20H2 the test transformation works on PDI CE 9.2 (with AdoptOpenJDK 8).

    The total length of the strings that I'm trying to concatenate is short, 3 strings of less than 15 chars each.

    Below the pan output at Rowlevel log level from the failed attempt on Windows.



    ------------------------------
    Antonio Petrella
    Data Service Manager
    UNOG
    ------------------------------

    Original Message


  • 4.  RE: Error from Concat fields in PDI CE 9.3.0.0-428

    Posted 05-24-2022 12:45
    Can you include your test transform?  I can surmize what you are doing from the log, but cannot reproduce it.   The fact that concat fields is giving a textFileOutput error is likley the clue.  No length specified in the field for output, but you are not using any textoutput steps yourself.

    I am not seeing any text output in my concat and even added a textfileoutput to the end of the stream without issues.
    Similar test worked on Windows 10 with OpenJdk 8_133.

    ------------------------------
    Stephen Donovan
    Digital Solutions Architect
    Hitachi Vantara
    ------------------------------

    Original Message


  • 5.  RE: Error from Concat fields in PDI CE 9.3.0.0-428

    Posted 05-26-2022 03:12
      |   view attached
    Hi Stephen,
     Thank you for taking the time and looking into this strange behavior.

    Please find attached the test transformation.

    I tried to run it from Spoon and from the command line using pan on Windows 10 with OpenJDK 11.0.15 and got the failure.

    As said, if I configure the Concat fields step by checking the "Fast data dump (no formatting)" checkbox, then it runs.

    ------------------------------
    Antonio Petrella
    Data Service Manager
    UNOG
    ------------------------------

    Attachment(s)

    ktr
    testConcatenation.ktr   16 KB 1 version
    Original Message


  • 6.  RE: Error from Concat fields in PDI CE 9.3.0.0-428

    Posted 05-26-2022 13:35

    Good new bad News.  Works just fine for me in 9.3 CE and EE.  Logging with row level detail is below.

    I believe that the call out to a TextFileOutput from within the concat rows step is still the clue.  Some change to the overall logging level, debug or somethign environmental? 

    -----------------------------

    2022/05/26 13:30:51 - Spoon - Running transformation using the Kettle execution engine
    2022/05/26 13:30:51 - General - We have 0 connections...
    2022/05/26 13:30:51 - General - Reading 3 steps...
    2022/05/26 13:30:51 - General - Looking at step #0
    2022/05/26 13:30:51 - General - Looking at step #1
    2022/05/26 13:30:51 - General - Looking at step #2
    2022/05/26 13:30:51 - General - We have 2 hops...
    2022/05/26 13:30:51 - General - Looking at hop #0
    2022/05/26 13:30:51 - General - Looking at hop #1
    2022/05/26 13:30:51 - General - nr of steps read : 3
    2022/05/26 13:30:51 - General - nr of hops read : 2
    2022/05/26 13:30:51 - testConcatenation - Transformation is pre-loaded.
    2022/05/26 13:30:51 - testConcatenation - nr of steps to run : 3 , nr of hops : 2
    2022/05/26 13:30:51 - Spoon - Transformation opened.
    2022/05/26 13:30:51 - Spoon - Launching transformation [testConcatenation]...
    2022/05/26 13:30:51 - Spoon - Started the transformation execution.
    2022/05/26 13:30:51 - testConcatenation - Dispatching started for transformation [testConcatenation]
    2022/05/26 13:30:51 - testConcatenation - Nr of arguments detected:0
    2022/05/26 13:30:51 - testConcatenation - This is not a replay transformation
    2022/05/26 13:30:51 - testConcatenation - I found 3 different steps to launch.
    2022/05/26 13:30:51 - testConcatenation - Allocating rowsets...
    2022/05/26 13:30:51 - testConcatenation - Allocating rowsets for step 0 --> Generate rows
    2022/05/26 13:30:51 - testConcatenation - prevcopies = 1, nextcopies=1
    2022/05/26 13:30:51 - testConcatenation - Transformation allocated new rowset [Generate rows.0 - Concat fields.0]
    2022/05/26 13:30:51 - testConcatenation - Allocated 1 rowsets for step 0 --> Generate rows
    2022/05/26 13:30:51 - testConcatenation - Allocating rowsets for step 1 --> Concat fields
    2022/05/26 13:30:51 - testConcatenation - prevcopies = 1, nextcopies=1
    2022/05/26 13:30:51 - testConcatenation - Transformation allocated new rowset [Concat fields.0 - Write to log.0]
    2022/05/26 13:30:51 - testConcatenation - Allocated 2 rowsets for step 1 --> Concat fields
    2022/05/26 13:30:51 - testConcatenation - Allocating rowsets for step 2 --> Write to log
    2022/05/26 13:30:51 - testConcatenation - Allocated 2 rowsets for step 2 --> Write to log
    2022/05/26 13:30:51 - testConcatenation - Allocating Steps & StepData...
    2022/05/26 13:30:51 - testConcatenation - Transformation is about to allocate step [Generate rows] of type [RowGenerator]
    2022/05/26 13:30:51 - testConcatenation - Step has nrcopies=1
    2022/05/26 13:30:51 - Generate rows.0 - distribution activated
    2022/05/26 13:30:51 - Generate rows.0 - Starting allocation of buffers & new threads...
    2022/05/26 13:30:51 - Generate rows.0 - Step info: nrinput=0 nroutput=1
    2022/05/26 13:30:51 - Generate rows.0 - output rel. is 1:1
    2022/05/26 13:30:51 - Generate rows.0 - Found output rowset [Generate rows.0 - Concat fields.0]
    2022/05/26 13:30:51 - Generate rows.0 - Finished dispatching
    2022/05/26 13:30:51 - testConcatenation - Transformation has allocated a new step: [Generate rows].0
    2022/05/26 13:30:51 - testConcatenation - Transformation is about to allocate step [Concat fields] of type [ConcatFields]
    2022/05/26 13:30:51 - testConcatenation - Step has nrcopies=1
    2022/05/26 13:30:51 - Concat fields.0 - distribution activated
    2022/05/26 13:30:51 - Concat fields.0 - Starting allocation of buffers & new threads...
    2022/05/26 13:30:51 - Concat fields.0 - Step info: nrinput=1 nroutput=1
    2022/05/26 13:30:51 - Concat fields.0 - Got previous step from [Concat fields] #0 --> Generate rows
    2022/05/26 13:30:51 - Concat fields.0 - input rel is 1:1
    2022/05/26 13:30:51 - Concat fields.0 - Found input rowset [Generate rows.0 - Concat fields.0]
    2022/05/26 13:30:51 - Concat fields.0 - output rel. is 1:1
    2022/05/26 13:30:51 - Concat fields.0 - Found output rowset [Concat fields.0 - Write to log.0]
    2022/05/26 13:30:51 - Concat fields.0 - Finished dispatching
    2022/05/26 13:30:51 - testConcatenation - Transformation has allocated a new step: [Concat fields].0
    2022/05/26 13:30:51 - testConcatenation - Transformation is about to allocate step [Write to log] of type [WriteToLog]
    2022/05/26 13:30:51 - testConcatenation - Step has nrcopies=1
    2022/05/26 13:30:51 - Write to log.0 - distribution activated
    2022/05/26 13:30:51 - Write to log.0 - Starting allocation of buffers & new threads...
    2022/05/26 13:30:51 - Write to log.0 - Step info: nrinput=1 nroutput=0
    2022/05/26 13:30:51 - Write to log.0 - Got previous step from [Write to log] #0 --> Concat fields
    2022/05/26 13:30:51 - Write to log.0 - input rel is 1:1
    2022/05/26 13:30:51 - Write to log.0 - Found input rowset [Concat fields.0 - Write to log.0]
    2022/05/26 13:30:51 - Write to log.0 - Finished dispatching
    2022/05/26 13:30:51 - testConcatenation - Transformation has allocated a new step: [Write to log].0
    2022/05/26 13:30:51 - testConcatenation - This transformation can be replayed with replay date: 2022/05/26 13:30:51
    2022/05/26 13:30:51 - testConcatenation - Initialising 3 steps...
    2022/05/26 13:30:51 - Generate rows.0 - Released server socket on port 0
    2022/05/26 13:30:51 - General - Size of ServiceMap: 2
    2022/05/26 13:30:51 - Write to log.0 - Released server socket on port 0
    2022/05/26 13:30:51 - Concat fields.0 - Released server socket on port 0
    2022/05/26 13:30:51 - testConcatenation - Step [Generate rows.0] initialized flawlessly.
    2022/05/26 13:30:51 - testConcatenation - Step [Concat fields.0] initialized flawlessly.
    2022/05/26 13:30:51 - testConcatenation - Step [Write to log.0] initialized flawlessly.
    2022/05/26 13:30:51 - General - From step: Concat fields, looking at 1 prev. steps.
    2022/05/26 13:30:51 - General - From step: Generate rows, looking at 0 prev. steps.
    2022/05/26 13:30:51 - General - Getting fields from step: Generate rows, type=RowGenerator
    2022/05/26 13:30:51 - General - Found fields to add: [t1 String], [t2 String], [t3 String]
    2022/05/26 13:30:51 - General - Getting fields from step: Concat fields, type=ConcatFields
    2022/05/26 13:30:51 - General - From step: Generate rows, looking at 0 prev. steps.
    2022/05/26 13:30:51 - General - Getting fields from step: Generate rows, type=RowGenerator
    2022/05/26 13:30:51 - General - From step: Write to log, looking at 1 prev. steps.
    2022/05/26 13:30:51 - General - From step: Concat fields, looking at 1 prev. steps.
    2022/05/26 13:30:51 - General - Found fields to add: [t1 String], [t2 String], [t3 String]
    2022/05/26 13:30:51 - General - Getting fields from step: Concat fields, type=ConcatFields
    2022/05/26 13:30:51 - General - Found fields to add: [t1 String], [t2 String], [t3 String], [output String]
    2022/05/26 13:30:51 - General - Getting fields from step: Write to log, type=WriteToLog
    2022/05/26 13:30:51 - Generate rows.0 - Starting to run...
    2022/05/26 13:30:51 - Concat fields.0 - Starting to run...
    2022/05/26 13:30:51 - Write to log.0 - Starting to run...
    2022/05/26 13:30:51 - testConcatenation - Transformation has allocated 3 threads and 2 rowsets.
    2022/05/26 13:30:51 - Write to log.0 -
    2022/05/26 13:30:51 - Write to log.0 - ------------> Linenr 1------------------------------
    2022/05/26 13:30:51 - Write to log.0 - TEST CONCATENATION
    2022/05/26 13:30:51 - Write to log.0 -
    2022/05/26 13:30:51 - Write to log.0 - t1 = this is
    2022/05/26 13:30:51 - Write to log.0 - t2 = a
    2022/05/26 13:30:51 - Write to log.0 - t3 = simple test
    2022/05/26 13:30:51 - Write to log.0 - output = this isasimple test
    2022/05/26 13:30:51 - Write to log.0 -
    2022/05/26 13:30:51 - Write to log.0 - ====================
    2022/05/26 13:30:51 - Generate rows.0 - Wrote row 1 : [this is], [a], [simple test]
    2022/05/26 13:30:51 - Concat fields.0 - Wrote row #1 : [this is], [a], [simple test], [this isasimple test]
    2022/05/26 13:30:51 - Generate rows.0 - Signaling 'output done' to 1 output rowsets.
    2022/05/26 13:30:51 - Generate rows.0 - Finished processing (I=0, O=0, R=0, W=1, U=0, E=0)
    2022/05/26 13:30:51 - Concat fields.0 - Signaling 'output done' to 1 output rowsets.
    2022/05/26 13:30:51 - Concat fields.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
    2022/05/26 13:30:51 - Write to log.0 - Signaling 'output done' to 0 output rowsets.
    2022/05/26 13:30:51 - Write to log.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
    2022/05/26 13:30:51 - testConcatenation - searching for annotations
    2022/05/26 13:30:51 - testConcatenation - no annotations found
    2022/05/26 13:30:51 - General - Size of ServiceMap: 3
    2022/05/26 13:30:51 - General - Size of ServiceMap: 3
    2022/05/26 13:30:51 - General - Size of ServiceMap: 3
    2022/05/26 13:30:51 - Spoon - The transformation has finished!!



    ------------------------------
    Stephen Donovan
    Digital Solutions Architect
    Hitachi Vantara
    ------------------------------

    Original Message


  • 7.  RE: Error from Concat fields in PDI CE 9.3.0.0-428

    Posted 05-30-2022 05:34
    Thank you again for these tests Stephen.

    I'll try to better examine my environment.

    I need also to understand how the TextFileOutput call happens from the Concat Fields.

    ------------------------------
    Antonio Petrella
    Data Service Manager
    UNOG
    ------------------------------

    Original Message


  • 8.  RE: Error from Concat fields in PDI CE 9.3.0.0-428

    Posted 07-30-2022 16:04
    Antonio:
    I have tested your sample transformation using PDI 8.3 EE with the following versions of Java by modifying Spoon.bat respectively for each version:
    REM set PENTAHO_JAVA_HOME=C:\Program Files\Java\jdk1.8.0_333\
    REM set PENTAHO_JAVA_HOME=C:\Program Files\Java\jdk-11.0.14\
    REM set PENTAHO_JAVA_HOME=C:\Zulu\jdk8.0.302\
    REM set PENTAHO_JAVA_HOME=C:\Zulu\jdk11.0.16\
    REM set PENTAHO_JAVA_HOME=C:\AdoptJDK\jdk8u342-b07\
    REM set PENTAHO_JAVA_HOME=C:\AdoptJDK\jdk-11.0.16+8\

    Tested with the Advanced > Fast Data Dump checked and unchecked and all 12 iterations of my tests worked.
    Do let us know if you continue to experience the same issue

    ------------------------------
    Carlos Lopez
    Application Architecture Engineering - Expert
    Hitachi Vantara
    ------------------------------

    Original Message


  • 9.  RE: Error from Concat fields in PDI CE 9.3.0.0-428

    Posted 08-02-2022 09:54

    Hello Carlos,
     thank you for taking the time and trying different configurations. I believe that you meant PDI 9.3 EE. I have observed this problem on PDI 9.3 CE.

    In any case, I was unable to solve the problem. The tests performed by others suggest a local system (mine) configuration. But it's difficult to trace the root cause. Would you have any troubleshooting suggestion?

    Finally I have solved the problem.
    In my $HOME/.kettle/plugins there were steps (and many other) plugins from PDI version 9.2.

    When running PDI these plugins (ver. 9.2) were loaded along with the ones from version 9.3 causing therefore unpredictable behavior. I have cleared up this plugin directory, restarted PDI 9.3, and successfully ran the test transformation.

    Thanks,  

    Antonio



    ------------------------------
    Antonio Petrella
    Data Service Manager
    UNOG
    ------------------------------

    Original Message


Related Content