AnsweredAssumed Answered

PDI AEL Daemon throws Unexpected error running Spark job

Question asked by Cristopher Z on Nov 28, 2017
Latest reply on Nov 29, 2017 by Christopher Caspanello

Hey all,

Last week I got to try the Pentaho 8 Data Integration with Spark. I did a fresh install of Hadoop/Yarn and Spark with Ubuntu running in a VM and started trying to run in Local mode. Everything went fine, I ran a simple transformation over some CSV data and a math operation and got the results. Next step: Run it in Yarn mode, on the pseudo-distributed cluster.

 

I configured the entire environment, pdi-spark-executor inside the hdfs, got the daemon running properly and proceeded to run the same transformation using this new Run Configuration I set up. The daemon receives the request but throws a Unexpected Error.

 

Attached here are two images: My Spoon execution log and the daemon log from the console.

Following the stack trace I opened the SparkRunner class file to check this "execute" method, and there's a bunch of code inside a try/catch block and any exceptions are rethrowed as this "UnexpectedError" exception, so I have no clue about what really happened.

 

Could someone shed me a light on this matter, please?

Outcomes