Pentaho

 View Only

 How to use metadata injection with Update step

  • Pentaho
  • Kettle
  • Pentaho
  • Pentaho Data Integration PDI
Ana Gonzalez's profile image
Ana Gonzalez posted 04-12-2019 11:31

I'm using Pentaho Data Integration 8.2 CE and trying to use metadata injection to update data in table.

For that I have created a transformation to get the metadata to inject, and a second transformation that contains a dummy Data Grid input step just generating one row and one column (that I don't use, but it seems it's needed for the transformation to work), and the Update step for which I'll inject the metadata, so the step is empty, it only has the connection informed:

20190412_001

The metadata for that step (target table, keys for lookup and update fields) is injected from the first transformation. When I have tried to use it, it fails saying that the field [name of the column to update] is required and couldn't be found!.

I'm attaching two transformations with a simple example, to make them work you'll have to update the connection information in the update step with something that will work for you (I have it with an Oracle connection, but I don't think that's the problem), and in that database create a table (again, this script works for Oracle, adapt to your type of database):

CREATE TABLE my_dummy_table AS (SELECT '1' dummy_column FROM DUAL UNION SELECT '2' dummy_column FROM DUAL);

I have tried with uppercase and lowercase for columns and table, (for some steps it doesn't matter, but for others it must be the exact syntax).

 

It's probably something stupid I haven't configured, but I can't find it.

Thanks a lot for your help


#PentahoDataIntegrationPDI
#Pentaho
#Kettle
Ana Gonzalez's profile image
Ana Gonzalez

Just adding a comment to up the question in the queue of updates, in case someone who could help me was out on Easter holidays and didn't see it.

Regards

Derek Wilson's profile image
Derek Wilson

Ana, apologies for my delay. I hope this answer works for you.

Note, the Update step is seeking a stream of data for the following:

  • values to compare against key table field
  • values to update if result is true

Therefore, one does not provide the values above via MDI. One provides the name of the fields that hold these values within the connected stream.

I am including a zip containing 3 KTRs & 1 SQL.

  • TR_TEST_DMW_Injector.ktr – injector KTR to prepare the 3 sets of injected metadata
  • TR_TEST_DMW_Template.ktr – template transformation
  • TR_TEST_DMW_Template_Functional.ktr – the same template transformation but completed and functional
  • my_dummy_table.sql – sql to create and populate the dummy table

The tricky part is the Data Grid. Note, the data portion of the Data Grid step must be injected with a single vector of values equal in number to the n-rows x n-columns required. Further, one needs to provide the Data Grid step a listing of the fields in the Meta tab so that the injection knows the number of columns. Otherwise, you get an overflow error.

Ana Gonzalez's profile image
Ana Gonzalez

Thanks a lot, Derek, I'm going to test it and learn how it works.

Regards

Ana Gonzalez's profile image
Ana Gonzalez

Thanks a lot Derek, I did a quick check and it works, now I have to finish my current task so I have time to play with it and adapt it to my use case.

One question though, you use a step to create a hash with the updates cases that I haven't seen you use later in the transformation. In this simple example the hash is unique, but I'm my real case is going to have different values. Will I need to use it later to inject the metadata?

Regards

Data Conversion's profile image
Data Conversion
Data Conversion's profile image
Data Conversion
Data Conversion's profile image
Data Conversion