Based on some customer requirement to have logging/monitoring capabilities at document level, I have created a stage that sends logging messages.
The stage can and should be used to monitor documents at a granularity of documents/field level, getting logs for processed or failed documents at different stages of a workflow. We can use the stage to send information to the document.
The stage uses using FluentD API (https://www.fluentd.org/) to logs the messages in each of the different critical phases of a workflow.
For great performance the deployment involves a Fluentbit (https://fluentbit.io/) agent that will be installed in each node of the HCI cluster. This could even be done using containers included as part of Mesos and scheduled with Marathon, such as other components of HCI.
As per the image above, the stage will send the logs to Fluentbit, that will take care of forwarding to FluentD or other for high availability and load balancing for fail-over. Once the message arrives to FluentD servers it gets filtered and parsed to be send to other systems such as Splunk or any other logging solution or document oriented datastore.
The system is capable of handling backpressure, making a robust solution and capable of being part of the most scalable and reliable solutions.
Features, provided with the integration with FuentD and Fluentbit:
The stage was done for a POC and will require some improvements such as:
Please reach out to @Miguel Gaspar (email@example.com) for more information or if you have some user case where the stage makes sense.
Improvements can be included as per customer requirements.
If there is any interest, next post, can give more details and show how it can be used with Elasticsearch.
Nice work, @Miguel Gaspar ! Would you be interested in publishing the plugin here so others can try it out (myself included)?
Sure @Jonathan Chinitz, what's the best way to publish it? Attach it to the post? Thanks.
The plugin itself can be posted here as a jar, or if you have a gitlab repo you can publish a link to that here. If you have any supplementary docs that accompany the plugin my suggestion would be to wrap it up in a zip and post that here.
I have attached the plugin, remember you need to have fluentd up and running
A proud part of Hitachi Vantara
© Hitachi Vantara Corporation. All Rights Reserved.