Pentaho

 View Only

 Large Table Sync between Database

Chan Patel's profile image
Chan Patel posted 07-02-2024 12:33

Hi All,

I have been trying to sync few tables between Oracle (Primary) to MySQL (DataWarehouse) using Synchronize after Merge Step and it has been working fine, however I have few tables which have too many rows and are wide, takes like a few hours to update. If I truncate the table and re-run it, it even takes longer, below is a sample of the setup I have created.

Any suggestions to increase the performance ?

Frank Tucker's profile image
Frank Tucker

Hello,

Batch Processing: Instead of updating all rows at once, break the updates into smaller batches. This can help reduce the load on the database and improve performance.
Index Optimization: Ensure that your tables are properly indexed. Indexes can significantly speed up the synchronization process by allowing the database to quickly locate the rows that need to be updated.
Parallel Processing: If possible, use parallel processing to handle multiple synchronization tasks simultaneously. This can help distribute the load and speed up the overall process.
Incremental Updates: Instead of truncating and reloading the entire table, consider using incremental updates. This involves only updating the rows that have changed since the last synchronization.
Database Configuration: Optimize your database configuration settings. For example, increasing the buffer pool size in MySQL can help improve performance for large tables.
Network Optimization: Ensure that your network connection between the Oracle and MySQL servers is optimized. A slow network can significantly impact the synchronization speed.    MyCenturaHealth Org

Best Regards

franktucker907