Hi RAKOTONDRASOA Andry
I don't have a specific answer for you but I'm working on Kafka consumers as well and I'm having some, slightly different but relevant, problems.
I had out of memory issues, I'm on CE edition so I'm using plain Carte to have the transformation running and I haven't found out yet the reason. But I noticed I was on OpenJDK and now switched to Oracle JVM. Also I'm on AWS machines which usually do not have swap space and this leads to machine stall. So I'm currently testing a more supported scenario.
As per your transformation I don't see any particular problem, afaik. I mean that, for example, if you run jobs "per every row" then PDI has to collect a lot of "result information" from every run. But you're using only transformations, am I right?
Another thing, maybe it's silly, maybe not :-) The kafka consumer gets all data and then calls the trasnformation according to the batch size (or time frame). Which configuration are you using? Maybe you're getting too much data inside of the component at once. Some things to try might be to:
- start with "latest" messages and a new consumer group to see if things go weird anyway regardless of LAG and history!
- and, if you can, to increase memory a lot and see if there is some size where it fits. I mean, so far you never managed to have a long running job, while I managed with a much more complex transform. So first would try to define a baseline.
My regards
Virgilio