Hi everyone,
I would like to ask the help of the Pentaho community in solving this issue I have with Pentaho.
I'm using Pentaho 7.1 - Community Edition.
The problem I have is that it happens really often to me that Pentaho fills all the available memory and gets stuck when running jobs that contains jobs inside.
To better clarify what I mean, I tried to reproduce the issue I encounter using a simple example (you can find it attached to this post).
It is just a simple job (master_job.kjb) that calls two other jobs. They contain 2 transformations each inside.
The first job reads an input file (1 Milion rows, 13 fields - about 90 MB), adds a new field, concatenates two of the fields into a new one and outputs everything to file. The second job reads the previous output, does some string substitution and then outputs a second file.
The problem is that once I run this simple job in Pentaho it gets stuck after the execution of the first job, filling all my memory and not advancing anymore to the second job.
I really cannot understand what it is happening here. To me, this looks like a pretty simple task, but for some reason Pentaho is not able to perform it.
FYI: I already did some research and increased the Java Virtual Machine memory in PDI by setting the environment variable "PENTAHO_DI_JAVA_OPTIONS" to "-Xms2048m -Xmx8g -XX:MaxPermSize=1024m" without getting any benefit.
I really hope you can give me some help/advice.
Bests.
Giuseppe
Hi Giuseppe,
actually it still crashes even on 8.0 :-) and it's a matter of cardinality. 1000 rows are fine, 10.000 are good with a bit more memory, 100.000 crash and so on.
This issue seems to be out since quite a long time, you can read here something about getting Results from Jobs:
- How Do I calculate "Copy rows to result" memory limit?
- [PDI-7453] Java Heap Errors - Looping Over A Sub Job - Pentaho Platform Tracking
Can you articulate a bit more on your use case? The test you submitted needs to have the 2 transformations merged together as a standard ETL approach, but maybe I'm missing the bigger picture...
Regards
Virgilio