Pentaho

 View Only

 Table Output on MySQL writing at 4 r/s even after parameter tuning on Windows machine

  • Pentaho
  • Kettle
  • Pentaho
  • Pentaho Data Integration PDI
Bala vishal's profile image
Bala vishal posted 06-17-2020 13:32

Windows 10 Machine, so Bulk loader won't work.

 

Even after adding parameters:

useServerPrepStmts=false

rewriteBatchedStatements=true

useCompression=true

 

and setting batch size as 10,000.

 

Still the writing process is 5 rows per second. I'm reading from MongoDB at ~1000 rows/second.

 

what can I do to increase the rate?

25th step is bottleneck.


#PentahoDataIntegrationPDI
#Kettle
#Pentaho
Brandon Jackson's profile image
Brandon Jackson

Bulk loading on MySQL using the Bulk Load Step is better under Linux because it supports piping data across to Mysql as text streams.

 

Your best performance option in Windows would be to drop indexes on the destination tablet and use batch inserts.