new niu

hadoopfileoutput will always write to the local path

Discussion created by new niu on Apr 14, 2019

I use the pan or kitchen command to call hadoop file output will always write to the local disk

I use “pdi-ce-8.2.0.0-342\data-integration ” this version

and “\pdi-ce-8.2.0.0-342\data-integration\plugins\pentaho-big-data-plugin\hadoop-configurations\cdh514”

this is plugin path

 

my ktr file use 2 step . first random a num then save it to hdfs file (just a simple test)

 

If running on a linux system (centos 6.5)

will create path like this

/home/kettle/data-integration/hdfs:/root@cdh3:8020/user/root

But when I use the spoon UI to run it, everything is correct.

the file will be uploaded correctly to hdfs

The same error in windows10 system

 

 

here is log:

2019/04/15 13:34:38 - Hadoop file output.0 - Released server socket on port 0

2019/04/15 13:34:38 - Hadoop file output.0 - We can not find parent folder [file:///D:/pdi-ce-8.2.0.0-342/data-integration/hdfs:/hdfs:oracle@cdh1:8020/user/root]!

2019/04/15 13:34:38 - Hadoop file output.0 - ERROR (version 8.2.0.0-342, build 8.2.0.0-342 from 2018-11-14 10.30.55 by buildguy) : Couldn't open file file:///D:/pdi-ce-8.2.0.0-342/data-integration/hdfs:/hdfs:oracle@cdh1:8020/user/root/asdzzxassxxxxxx.txt

2019/04/15 13:34:38 - Hadoop file output.0 - ERROR (version 8.2.0.0-342, build 8.2.0.0-342 from 2018-11-14 10.30.55 by buildguy) : org.pentaho.di.core.exception.KettleException:

2019/04/15 13:34:38 - Hadoop file output.0 - Error opening new file : org.apache.commons.vfs2.FileSystemException: Could not create folder "file:///D:/pdi-ce-8.2.0.0-342/data-integration/hdfs:".

2019/04/15 13:34:38 - Hadoop file output.0 -

2019/04/15 13:34:38 - Hadoop file output.0 - at org.pentaho.di.trans.steps.textfileoutput.TextFileOutput.initFileStreamWriter(TextFileOutput.java:235)

2019/04/15 13:34:38 - Hadoop file output.0 - at org.pentaho.di.trans.steps.textfileoutput.TextFileOutput.initOutput(TextFileOutput.java:866)

2019/04/15 13:34:38 - Hadoop file output.0 - at org.pentaho.di.trans.steps.textfileoutput.TextFileOutput.init(TextFileOutput.java:837)

2019/04/15 13:34:38 - Hadoop file output.0 - at org.pentaho.di.trans.step.StepInitThread.run(StepInitThread.java:69)

2019/04/15 13:34:38 - Hadoop file output.0 - at java.lang.Thread.run(Thread.java:748)

 

on Linux :

 

2019/04/15 13:37:36 - 生成随机数.0 - Released server socket on port 0

2019/04/15 13:37:36 - Hadoop file output.0 - Released server socket on port 0

2019/04/15 13:37:36 - Hadoop file output.0 - We can not find parent folder [file:///home/kettle/data-integration/hdfs:/hdfs:oracle@cdh1:8020/user/root]!

2019/04/15 13:37:36 - Hadoop file output.0 - Folder [file:///home/kettle/data-integration/hdfs:/hdfs:oracle@cdh1:8020/user/root] was created.

2019/04/15 13:37:36 - Hadoop file output.0 - Opening output stream using provider: None

2019/04/15 13:37:36 - Hadoop file output.0 - Opening output stream in default encoding

2019/04/15 13:37:36 - Hadoop file output.0 - Opened new file with name [file:///home/kettle/data-integration/hdfs:/hdfs:oracle@cdh1:8020/user/root/asdzzxassxxxxxx.txt]

 

 

Mike Bates  Can you help me with that?

Outcomes