Pentaho

 View Only

 PDI Kitchen/ Spoon logging of metrics to monitor if job/ transformation ran successfully

  • Pentaho
  • Kettle
  • Pentaho
  • Pentaho Data Integration PDI
Saagar Khimasia's profile image
Saagar Khimasia posted 07-12-2021 06:45

Hi there,

 

What is the most effective way to monitor if jobs are running to completion?

 

I am trying to setup logging to monitor if my jobs are running successfully. This is because jobs can fail at times but there is no way for us to know without manually checking on a daily basis.

 

My environment is as follows:

• Windows Server 2016

• Jobs & transformations setup in Spoon

• Windows Task Scheduler setup to run Kitchen that executes the jobs

• The transformations are picking data from predominantly Oracle DB's, manipulating the data, and inserting it into a MySQL DB.

• The MySQL DB is connected to my SMS gateway and sends out SMS's to customers

• The jobs are run once a day at a set time

• It is not necessary that data be available for SMS's to be sent as the SMS's are things like customer birthday & I may not have a customer with a birthday on any given day. This means the job will run but not insert any data into MySQL

 

Task Scheduler can run but the job can fail.

 

I have tried to setup job logging in Spoon. The logs write to a MySQL DB. I have setup Job Log, Job Entry Log and Job Channel Log. But the first two are only outputting 0 for LINES_READ, LINES_WRITTEN etc.

 

Any insights would be greatly appreciated.

 

 

Regards,

 

 

Saagar K.

 


#PentahoDataIntegrationPDI
#Kettle
#Pentaho
Ana Gonzalez's profile image
Ana Gonzalez posted 07-12-2021 09:39

Looking to the ERRORS column in Job Log table you should be able to see if the Job has completed successfully or if it has failed.

Something that also works for me, is having a Send Mail step in the job and all the steps have a failed step transition that send the email, something like this:

20210712_001

Saagar Khimasia's profile image
Saagar Khimasia posted 07-13-2021 06:17

Thank you very much for your response Ana.

 

My ERRORS column is showing 0. Does that mean currently there are no ERRORS?

 

So essentially anything that is not 0 means there was a failure somewhere?

Ana Gonzalez's profile image
Ana Gonzalez posted 07-13-2021 08:40

It's going to depend on the logic of your workflow, but in most cases, yes. I have some workflows where I don't mind if certain steps fail, or I "force" an error to stop the flow of data due to the internal logic, so in those workflows the ERRORS column is not going to help, but in most cases that column is going to work.

Related Content