Pentaho

 View Only

 PDI update step is too slow ant never ends

  • Pentaho
  • Kettle
  • Pentaho
  • Pentaho Data Integration PDI
Sirajus Sirajus's profile image
Sirajus Sirajus posted 08-06-2018 13:39

I need to update Data Based on Last Days updated Data. To do this I have tried both Update and Upsert step in Data Integration. But both steps too much slow. In my case I need more than 2 lac update per day. Is there any faster way to do this update ?


#PentahoDataIntegrationPDI
#Pentaho
#Kettle
John Craig's profile image
John Craig posted 08-09-2018 17:09

With a bit more information, we maybe can be more help...

What is the source of the data?

Approximately how many rows are you processing during a run?

What settings are you using on the Update step?

What is the target database (PostgreSQL, SQL Server, mySQL, whatever)?

Do all rows you are processing require updating (or are some entries unchanged)?

If some/many entries are unchanged, you could try the Merge/Synchronize after merge approach (that is sometimes vastly faster for us, but it depends heavily on the nature of the data that's being updated).

Related Content