Pentaho

 View Only
Back to discussions
Expand all | Collapse all

Map Reduce step not writing to HIVE in readable format

This thread has been viewed 7 times

  • 1.  Map Reduce step not writing to HIVE in readable format

    Posted 10-27-2022 05:31
    We are upgrading Pentaho from 7.1 to 8.3.
    v7.1 connects to Hadoop 2.6
    whereas v8.3 connects to Hadoop 3.1

    and we are finding that the map-reduce step in 7.1 writes to Hadoop 2.6 in readable format. Code is exactly the same in both cases.
    whereas the same step in 8.3 writes to Hadoop 3.1 in non-readable format.

    As an example, below is what we see in 7.1 when we read the data in HDFS

    hdfs dfs -text msgs_hr21_d221025_gz/part-00000 |more

    1``214348`T                                FF: 4DDX300                                                    :

    whereas the same in 8.3 yields

    hdfs dfs -text msgs_hr21_d221025_gz/part-00000 |more

    6`��`��5`�`�r�+��)P��H���7��`h�`�`=`w�      `    `  `          ` ` ` ` `   `      ` `    ` `    `    ` `   `    `    ` `   `   ` `     `          ` `


    Has anyone else faced this issue before? If you have, please can you provide any pointers or solutions?

    thank you.

    ------------------------------
    Satyan Iyer
    Director of Services
    Travelport, LP
    ------------------------------


  • 2.  RE: Map Reduce step not writing to HIVE in readable format

    Posted 10-27-2022 23:04

    Hi Satyan,

    This could be something to do with the encoding. Have you tried a simple PMR job to write/parse a few records into HDFS to check if the works ? See here - https://pentaho-community.atlassian.net/wiki/spaces/BAD/pages/114659647/Using+Pentaho+MapReduce+to+Parse+Weblog+Data to build a sample and test.



    ------------------------------
    Sandeep Chinaga Kemparaju
    Application Architecture Engineering - Manager
    Hitachi Vantara
    ------------------------------

    Original Message


Related Content