Pentaho

 View Only

 PDI Transformation stuck with a large amount data

  • Pentaho
  • Kettle
  • Pentaho
  • Pentaho Data Integration PDI
Digital Brain's profile image
Digital Brain posted 10-22-2020 12:23

Hello Community,

 

I created one transformation, it take rows from table and each row pass to sub job of transformation. Sub job create different sheets and files. Here Sub job work with Number of copies. (In my case is 24X)

 

This transformation work perfectly with partial data like 1000 / 1500 rows but when I try to run it with my full data sets with 12000+ rows. It stop progressing after few thousands of row. (No error thrown).It just never ending job.

 

When I break 12000+ rows in parts and run transformation it works perfectly no error in any row

 I am using PDI 7.1

 


#Pentaho
#Kettle
#PentahoDataIntegrationPDI
Ana Gonzalez's profile image
Ana Gonzalez

It seems like a memory problem, the transformation is consuming resources on your machine until it can grow more, and it gets stuck. Maybe increasing the JVM capacity (if your machine has resources to allow it) would work.