Pentaho

 View Only

 BigQuery plugin disappeared

  • Pentaho
  • Kettle
  • Pentaho
  • Pentaho Data Integration PDI
Virgilio Pierini's profile image
Virgilio Pierini posted 06-12-2019 09:33

Hi guys,

I was giving a try at Matt's kettle beam plugin following Diethard superb blog post: http://diethardsteiner.github.io/pdi/streaming/2018/12/01/Kettle-Beam.html but something strange happened. Problem is: I managed to run some tests but, suddenly, the BigQuery plugin disappeared.

By disappeared I mean that if I try to load the attached transformation it reports BeamBQInput as missing.

If I try to build a new one from scratch... no step in the palette.

But -you see- actually I had managed to create the ktr, so it once worked!

What might have happened? How to debug with some more detail?

I previously had PDI CE 8.1 on the machine. Then installed 8.2 on same machine (of course same user) but different folder. I see no errors at startup, karaf is reading from the correct (new) location.

Is there any cache/caveat I'm missing?

Thanks

Virgilio


#Pentaho
#Kettle
#PentahoDataIntegrationPDI
Virgilio Pierini's profile image
Virgilio Pierini

I got an answer by myself. Blog post points to a zip containing version 0.0.5 while  https://github.com/mattcasters/kettle-beam/releases/ provides a link for downloading the one with bigquery and other stuff.

Voilà. back to work.

Some seconds and a new error came:

"Required parameter projectId must be specified"

 

 

org.apache.beam.sdk.Pipeline$PipelineExecutionException: java.lang.NullPointerException: Required parameter projectId must be specified.

at org.apache.beam.sdk.Pipeline.run(Pipeline.java:319)

at org.apache.beam.sdk.Pipeline.run(Pipeline.java:299)

at org.kettle.beam.pipeline.KettleBeamPipelineExecutor.executePipeline(KettleBeamPipelineExecutor.java:229)

at org.kettle.beam.pipeline.KettleBeamPipelineExecutor.execute(KettleBeamPipelineExecutor.java:100)

at org.kettle.beam.pipeline.KettleBeamPipelineExecutor.execute(KettleBeamPipelineExecutor.java:90)

at org.kettle.beam.xp.RunBeamTransExecutionPoint.lambda$executeInSpoon$2(RunBeamTransExecutionPoint.java:156)

at java.lang.Thread.run(Thread.java:748)

Caused by: java.lang.NullPointerException: Required parameter projectId must be specified.

at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:895)

at com.google.api.client.util.Preconditions.checkNotNull(Preconditions.java:140)

at com.google.api.services.bigquery.Bigquery$Jobs$Insert.<init>(Bigquery.java:1559)

at com.google.api.services.bigquery.Bigquery$Jobs.insert(Bigquery.java:1509)

at org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl.startJob(BigQueryServicesImpl.java:231)

at org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl.startJob(BigQueryServicesImpl.java:216)

at org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl.startExtractJob(BigQueryServicesImpl.java:175)

at org.apache.beam.sdk.io.gcp.bigquery.BigQuerySourceBase.executeExtract(BigQuerySourceBase.java:186)

at org.apache.beam.sdk.io.gcp.bigquery.BigQuerySourceBase.extractFiles(BigQuerySourceBase.java:121)

at org.apache.beam.sdk.io.gcp.bigquery.BigQuerySourceBase.split(BigQuerySourceBase.java:139)

at org.apache.beam.runners.direct.BoundedReadEvaluatorFactory$InputProvider.getInitialInputs(BoundedReadEvaluatorFactory.java:214)

at org.apache.beam.runners.direct.ReadEvaluatorFactory$InputProvider.getInitialInputs(ReadEvaluatorFactory.java:89)

at org.apache.beam.runners.direct.RootProviderRegistry.getInitialInputs(RootProviderRegistry.java:76)

 

 

 

but, indeed, I have specified it (in the BigQuery step and in the run configuration, in all tabs asking for the project id).

Also the big query step is able to fetch column names and types, so connection happens, to the correct project.

What am I overlooking?

cheers

Virgilio

Ana Gonzalez's profile image
Ana Gonzalez

Hi Virgilio,

Matt isn't much around here, I would report issues in his githut repo. Also you can contact him in his Slack channel: kettle, there's a specific slack channel for beam.

Regards

Data Conversion's profile image
Data Conversion
Attachment  View in library