I am running into a timing condition between the "Text file output" step and another step (it is a custom HCP step). The problem is that zero length files are being written by the "HCP Put" step. When observing the temporary file location, I also see that there are a lot of zero length files being created for some time, but eventually do have content.
I am aware of two kettle.properties settings that defines the behavior for Text file output step.
However, even if I set the *_STREAM_COUNT to 1 and the *_STREAM_LIFE to 50 (milliseconds), then also put in a "Delay row" stage with 100 milliseconds between the "Text file output" and the other step, there are still situations where there are empty files written to HCP. I have used larger timeouts and it does tend to lessen the number of 0 files, but does not eliminate it. As for the transformation, there actually are 5 other steps between the "Text file output" and the "HCP Put".
BTW. Even trying without HCP step, I see problems utilizing the text file in the following step in the transformation.
I really do not what to use any kind of step that delays or suspends the transformation throughput.
Any information about the Kettle engine and why this would be the case, but of course how to avoid it 100% of the time, would be greatly appreciated.