I have a transformation that uses a "streaming" input step. That is the input step is always listening for incoming messages to process, thus never finishes. Because of some complications with the transformation w.r.t. creation of temporary files, the best solution we have found to help avoid the issues was to use a Single Threader step to call a sub-transformation thus piggy backing on the batch processing to avoid timing conditions.
The Single Threader step has two Batch settings: number of records in batch and time elapsed. With the number of records, without constant receipt of new records, records can be stuck in the batch until the batch is full. So another approach was to use the time elapsed setting. The interpretation of this setting is that after every interval of time, the batch would be flushed into the single threader; however, that does not seem to be the case.
What I have seen is that if new records are not coming in, some number will still get stuck in the batch until more content comes in even if the time elapse has long expired. This seem to be broken, but perhaps I am just not really understanding what this setting is doing. Can someone explain how this setting should be utilized?
I realize that if the transformation is running outside of a Job, there is the ability to manually trigger an "end-of-input", but this has the effect of ending the transformation. And if this transformation is triggered by a Job there is no way to do this same operations.
Or maybe is there some way to "trigger" a flush of a batch especially on the Single Threader step? Any information or ideas to try would be greatly appreciated!