Pentaho

 View Only

 DI - MongoDB Aggregation broken with MongoDB 3.6

  • Pentaho
  • Kettle
  • Pentaho
  • Pentaho Data Integration PDI
Data Conversion's profile image
Data Conversion posted 02-08-2018 11:35

Any MongoDB aggregation Query using the MongoDB Input node is broken if the query runs against the (latest) 3.6 version of MongoDB.

The usage of cursors is now a requirement in aggregation pipelines and this is implemented in the latest 3.6 MongoDB Java drivers.

Here's the current stacktrace if someone tries to use the included (old, 3.2.2) Java driver:

2018/02/08 09:28:33 - MongoDB Input.0 - ERROR (version 8.0.0.0-28, build 8.0.0.0-28 from 2017-11-05 07.27.50 by buildguy) : org.pentaho.di.core.exception.KettleException:

2018/02/08 09:28:33 - MongoDB Input.0 - com.mongodb.MongoCommandException: Command failed with error 9: 'The 'cursor' option is required, except for aggregate with the explain argument' on server localhost:27017. The full response is { "ok" : 0.0, "errmsg" : "The 'cursor' option is required, except for aggregate with the explain argument", "code" : 9, "codeName" : "FailedToParse" }

2018/02/08 09:28:33 - MongoDB Input.0 - Command failed with error 9: 'The 'cursor' option is required, except for aggregate with the explain argument' on server localhost:27017. The full response is { "ok" : 0.0, "errmsg" : "The 'cursor' option is required, except for aggregate with the explain argument", "code" : 9, "codeName" : "FailedToParse" }

2018/02/08 09:28:33 - MongoDB Input.0 -

2018/02/08 09:28:33 - MongoDB Input.0 - at org.pentaho.di.trans.steps.mongodbinput.MongoDbInput.processRow(MongoDbInput.java:137)

2018/02/08 09:28:33 - MongoDB Input.0 - at org.pentaho.di.trans.step.RunThread.run(RunThread.java:62)

2018/02/08 09:28:33 - MongoDB Input.0 - at java.lang.Thread.run(Thread.java:748)

2018/02/08 09:28:33 - MongoDB Input.0 - Caused by: com.mongodb.MongoCommandException: Command failed with error 9: 'The 'cursor' option is required, except for aggregate with the explain argument' on server localhost:27017. The full response is { "ok" : 0.0, "errmsg" : "The 'cursor' option is required, except for aggregate with the explain argument", "code" : 9, "codeName" : "FailedToParse" }

2018/02/08 09:28:33 - MongoDB Input.0 - at com.mongodb.connection.ProtocolHelper.getCommandFailureException(ProtocolHelper.java:115)

2018/02/08 09:28:33 - MongoDB Input.0 - at com.mongodb.connection.CommandProtocol.execute(CommandProtocol.java:114)

2018/02/08 09:28:33 - MongoDB Input.0 - at com.mongodb.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:159)

2018/02/08 09:28:33 - MongoDB Input.0 - at com.mongodb.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:286)

2018/02/08 09:28:33 - MongoDB Input.0 - at com.mongodb.connection.DefaultServerConnection.command(DefaultServerConnection.java:173)

2018/02/08 09:28:33 - MongoDB Input.0 - at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:215)

2018/02/08 09:28:33 - MongoDB Input.0 - at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:206)

2018/02/08 09:28:33 - MongoDB Input.0 - at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:112)

2018/02/08 09:28:33 - MongoDB Input.0 - at com.mongodb.operation.AggregateOperation$1.call(AggregateOperation.java:227)

2018/02/08 09:28:33 - MongoDB Input.0 - at com.mongodb.operation.AggregateOperation$1.call(AggregateOperation.java:223)

2018/02/08 09:28:33 - MongoDB Input.0 - at com.mongodb.operation.OperationHelper.withConnectionSource(OperationHelper.java:239)

2018/02/08 09:28:33 - MongoDB Input.0 - at com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:212)

2018/02/08 09:28:33 - MongoDB Input.0 - at com.mongodb.operation.AggregateOperation.execute(AggregateOperation.java:223)

2018/02/08 09:28:33 - MongoDB Input.0 - at com.mongodb.operation.AggregateOperation.execute(AggregateOperation.java:65)

2018/02/08 09:28:33 - MongoDB Input.0 - at com.mongodb.Mongo.execute(Mongo.java:772)

2018/02/08 09:28:33 - MongoDB Input.0 - at com.mongodb.Mongo$2.execute(Mongo.java:759)

2018/02/08 09:28:33 - MongoDB Input.0 - at com.mongodb.DBCollection.aggregate(DBCollection.java:1377)

2018/02/08 09:28:33 - MongoDB Input.0 - at com.mongodb.DBCollection.aggregate(DBCollection.java:1308)

2018/02/08 09:28:33 - MongoDB Input.0 - at com.mongodb.DBCollection.aggregate(DBCollection.java:1294)

2018/02/08 09:28:33 - MongoDB Input.0 - at com.mongodb.DBCollection.aggregate(DBCollection.java:1282)

2018/02/08 09:28:33 - MongoDB Input.0 - at org.pentaho.mongo.wrapper.collection.DefaultMongoCollectionWrapper.aggregate(DefaultMongoCollectionWrapper.java:47)

2018/02/08 09:28:33 - MongoDB Input.0 - at org.pentaho.di.trans.steps.mongodbinput.MongoDbInput.initQuery(MongoDbInput.java:191)

2018/02/08 09:28:33 - MongoDB Input.0 - at org.pentaho.di.trans.steps.mongodbinput.MongoDbInput.processRow(MongoDbInput.java:79)


#PentahoDataIntegrationPDI
#Kettle
#Pentaho
Data Conversion's profile image
Data Conversion

I am also getting same error. And new to this tool. Can anyone help me to resolve this with details.

Roguen Keller's profile image
Roguen Keller

Thanks for putting this up.  I've been beating my head for days trying to figure out what was wrong with my MongoDB input transformation step. 

Antonio Petrella's profile image
Antonio Petrella

Hello,

  has anyone found a solution for this problem without downgrading to MongoDB < 3.6 (I'm using PDI 8.0)?

I've recently upgraded from 3.4 to 4.0 so downgrading MongoDB is definitely not an option for me...

Thanks,

Antonio

Roguen Keller's profile image
Roguen Keller

Hello Antonio,

Not to my knowledge.  I had the option to use a different version and so I did.  Perhaps someone else has though?

Antonio Petrella's profile image
Antonio Petrella

Did a bit of research and it seems that the support for the mandatory cursor in the aggregation framework (in mongoDB from version 3.6) starts with PDI 8.1.0 (Components Reference - Pentaho Documentation).

I have tested with 8.2.0.0 and MongoDB 4.0 and it works.