I’m fiddling around to convert a datetime string into a date format which the date conversion stage is understanding. After a couple of tries I think I need a bit of help.
The format of the input string is something like this: yyyymmdd_hhmmss.ddd where ddd is a part of the document id we investigate. So, essentially I need to transform the first part of the string (up to the dot) into a proper datetime format HCI understands.
I wanted to use a combination of tokenizing and appending stages. So, I have tokenized the date part into 3 different fields (yyyy, mm and dd). Then tokenized the time as well into 3 fields representing hours, minutes and seconds.
With these parts extracted I then wanted to build a new string in one field in the format yyyy-mm-dd hh:mm:ss. In the hope this can be processed by the date conversion stage.
All extracting stages run fine. But when I want to use the appender stage to put everything together again, then the first append stage always hangs. Prior to this stage I have created a new field via the tagging stage and kept it empty. Then I wanted to append the extracted yyyy field to that empty field by using the append stage. And so on for the other fields, interrupted by adding “-“ , “ “ and “:” in as well.
But the first append stage already hanging. All input fields for this append stage are available and filled up properly.
Even when I prefill the new created field with a character already and then append the year field, is not working.
Any clue why this is not working? Or do you have a better approach (except programming and own stage in Java)?
An example of my input strings:
The rest is not relevant for the date.
All I need is to get it into ISO8601 compliant format. Whereas I just found out that 19570110 is a valid ISO date. Just digging around a bit more.
In addition: Have just added one appender stage again to my pipeline and it hangs again. But always at the end of the pipeline. Regardless what the last stage is. Once I take the appender stage out from the pipeline it runs through again. What is this?