Pentaho

 View Only

  • 1.  Pentaho 9.4 BigQuery input/output?

    Posted 06-08-2023 13:00

    Hi,

    we have tried to use BigQuery as source and target. Reading is fine with step 'table input'. Writing into BigQuery with step 'table output' fails.

    Also tested this with steps 'Beam BigQuery Input' and 'Beam BigQuery Output'.. No luck :/

    Has anyone got this to work?

    Br, Mika




    ------------------------------
    Mika Stykki
    Systems Engineer
    Veikkaus Oy
    ------------------------------


  • 2.  RE: Pentaho 9.4 BigQuery input/output?

    Posted 06-09-2023 10:49

    Did you follow these steps?
    https://help.hitachivantara.com/Documentation/Pentaho/9.4/Setup/Google_BigQuery

    Can you include the error for "it fails"?



    ------------------------------
    Stephen Donovan
    General IT Consulting & Implementation Pre-eminent Professional
    Hitachi Vantara
    ------------------------------

    Original Message


  • 3.  RE: Pentaho 9.4 BigQuery input/output?

    Posted 06-09-2023 12:03

    More details:

    We use a Simba BigQuery driver for TableInput from BIgQuery.  The Simba driver will not work well with TableOutput because there are quotas on Insert statements. It is why we have a  job entry plugin in the Enterprise Endition to bulk load BIgQuery.

    https://help.hitachivantara.com/Documentation/Pentaho/8.1/Products/Data_Integration/Job_Entry_Reference/Google_BigQuery_Loader

    imagehttps://cloud.google.com/bigquery/docs/reference/odbc-jdbc-drivers
    https://cloud.google.com/bigquery/docs/reference/standard-sql/data-manipulation-language#quotas




    ------------------------------
    Stephen Donovan
    General IT Consulting & Implementation Pre-eminent Professional
    Hitachi Vantara
    ------------------------------

    Original Message


  • 4.  RE: Pentaho 9.4 BigQuery input/output?

    Posted 06-12-2023 14:49

    Hi Stephen,

    And thank you for quick reply. Error we are getting ->

    BigQuery table output ->"The Simba driver for a Google Big Query database connection does not support regular DDL statements.
    Please use the GBQ Bulk Loader step to create your table"

    Beam BigQuery Output. Logging says  -> 
    2023/06/12 14:49:58 - BigQuery_input.0 - Finished reading query, closing connection
    2023/06/12 14:49:58 - BigQuery_input.0 - Finished processing (I=3, O=0, R=0, W=3, U=0, E=0)
    2023/06/12 14:49:58 - Beam BigQuery Output.0 - Finished processing (I=0, O=0, R=3, W=3, U=0, E=0)
    2023/06/12 14:49:58 - Spoon - The transformation has finished!!

    But there is 0 rows in target table :/

    And yes, we followed documentation. We can read from BigQuery so connection etc is ok.

    -Mika

     



    ------------------------------
    Mika Stykki
    Systems Engineer
    Veikkaus Oy
    ------------------------------

    Original Message


Related Content