Pentaho

 View Only

 java.lang.OutOfMemoryError: GC overhead limit exceeded error

  • Pentaho
  • Pentaho
Priya A's profile image
Priya A posted 03-12-2019 03:55

We are trying to upgrade from 4.2 EE to 8.2 CE. When ran a trf where lkp table having 10000000, after reading 7000000 getting Exception in thread "qtp1982591789-180" java.lang.OutOfMemoryError: GC overhead limit exceeded.

Got this error in 8.2 CE, but ran in 4.2 EE. Any inputs on how to resolve the issue.

Have below for java options settings as PENTAHO_DI_JAVA_OPTIONS="-Xms1024m" "-Xmx4086m" "-XX:MaxPermSize=2048m"

TIA


#Pentaho
Ravikumar Kamma's profile image
Ravikumar Kamma posted 03-12-2019 08:44

A few days back I was having same problem. I was reading data(6000000) from CSV and loading into MongoDB after multiple lookups, sorting, grouping. Even after adjusting PENTAHO_DI_JAVA_OPTIONS parameters it was giving GC overhead limit exceeded error. So I decided to process in batch.

In your case If it is possible to process in batch then try that.

I don't know why it works in 4.2 and fails in 8.2.

It would be good if someone explains a different way to solve such a problem.

Jesus Antonio Santos Giraldo's profile image
Jesus Antonio Santos Giraldo posted 03-14-2019 17:26

You need *way MORE* memory !!

It is certainly very intriguing that your 10 million records table get handled successfully in 4.2 but not in 8.2.

Have in mind that 8.2 requires more memory than 4.2 but not that much and besides that, I dislike 8.2 because it is dropping SAP support for some steps.

So, I would suggest using jvisualvm (or java mission control) to check the usage of memory while extracting. You certainly will view there the amount of memory needed for your extraction. Based on this monitoring you should adjust memory parameters.

For tables, in the order of 10 million, I use 18Gb of memory! some kind of ... "-Xms18g"  "-Xmx18g".

BTW, Why are you using "XX:MaxPermSize=..."? It is deprecated in java 8 so you don't need this.

Finally is worth mentioning that you should use pan/kitchen to run transformations/jobs accordingly.

J.

Related Content