Pentaho

 View Only

 Can not write on large excel file PDI

  • Pentaho
  • Kettle
  • Pentaho
  • Pentaho Data Integration PDI
Samuel Flores's profile image
Samuel Flores posted 08-09-2019 22:02

Good afternoon.

I have a problem with Pentaho Data Integration. I am using CE (tested in PDI 8.0, 8.1, 8.2 and 8.3 without success).

I am trying to write on Excel with 75 MB of size. But the step Excel output writer simple does not work. If the excel is minor of 75 MB it works.

So how can I write in a large excel file with pentaho data integration.

I have tried almost everything, increase memory of spoon, change the transformation rowset size, enable streaming. 

I have -xmx in 2048 MB in spoon.bat file.

pastedimage_1


#Kettle
#PentahoDataIntegrationPDI
#Pentaho
Paulo Pires's profile image
Paulo Pires posted 08-19-2019 11:55

Hi Samuel,

 

This thread is old but probably your issue is related, see if it helps

https://forums.pentaho.com/threads/129264-Excel-Writer-stops-at-10000-records/

 

Regards

Sergio Ribeiro's profile image
Sergio Ribeiro posted 08-26-2019 14:14

Hello @Samuel Flores​,

 

Is the image you shared the actual transformation you're using?

If so, check if the "Block this step until steps finish" step is correctly waiting for the finish of the "InformeGeneralCSC" step: assuming both "InformeGeneralCSC" and "InformeGeneralCSC 2" write to the same Excel file, it can be a problem of the first step still having the file locked for writing when the second tries to open it!

You can also have a look in the log files to see if there's more information on the actual error!

 

Regards,

 

Sérgio Ribeiro

Porto - Portugal

 

Related Content