Pentaho

 View Only

 Is it possible to extract data from multiple table and load different tables in one transformation.

  • Pentaho
  • Kettle
  • Pentaho
  • Pentaho Data Integration PDI
Data Conversion's profile image
Data Conversion posted 05-03-2019 06:03

Hi,

Is it possible to extract data from multiple table and load different tables in one transformation. i.e Table A , Table B, Table C from Source A (Oracle DB) and load into Table X, Table Y & Table Z into Source B(Oracle DB).

Please share your solution in detail.


#Kettle
#PentahoDataIntegrationPDI
#Pentaho
Ana Gonzalez's profile image
Ana Gonzalez

It's possible, but I wouldn't consider it a good practice, if one of the inserts in one of the tables goes wrong, it stops all of the inserts going on all the tables. If the stream of data inserted in the three tables is not related, I consider it a better practice to set up a transformation to load each table, so if something goes wrong and the data doesn't get inserted, you have more control on what has to be reloaded or corrected before loading.

Regards

Derek Wilson's profile image
Derek Wilson

Following on from Ana's comment, I think this is a great example of where metadata injected ETL (MDI) comes in very useful. The idea is you establish the metadata for each source to target external to your transformation then use the ETL Metadata Injection step to inject the metadata into a transformation template at run-time.

Here are a few references you can review to get a better idea.

https://support.pentaho.com/hc/en-us/categories/200888603-Best-Practices

https://www.hitachivantara.com/en-us/pdf/datasheet/metadata-injection-datasheet.pdf?source=pentaho-redirect

https://wiki.pentaho.com/display/EAI/ETL+Metadata+Injection

http://diethardsteiner.blogspot.com/2011/07/metadata-driven-etl-and-reporting.html

https://dankeeley.wordpress.com/2015/01/08/non-native-metadata-injection/

http://kettle.bleuel.com/2016/04/14/pentaho-6-1-part-2/

http://kettle.bleuel.com/2017/02/16/metadata-injection-examples-for-special-scenarios/