This guide is the first in a series explaining how to use open source ELK to visualize the performance of a system. This post includes instructions to install the ELK software. The second guide in the series, Performance Monitoring w/ ELK - Part II: Monitoring HCP Access Logs, gives instructions to configure HCP and your newly installed ELK software to visually monitor HCP. Following the instructions in these 2 posts, you can be visualizing HCP HTTP gateway access logs in under 2 hours. All you need to begin is a Linux server or workstation and a running HCP. Let's get started.
"ELK" consists of three open source projects: Elasticsearch, Logstash, and Kibana. Elasticsearch is a search and analytics engine. Logstash is a server-side data processing pipeline that ingests data from multiple sources simultaneously, transforms it, and then sends it to a "stash" like Elasticsearch. Kibana is a visualization tool that lets users visualize data in Elasticsearch with charts and graphs.
The following chart was generated in real time by transmitting HCP access logs to Logstash over the syslog protocol, indexing the logs in Elasticsearch, and visualizing them with Kibana. This chart visualizes transaction load distribution among HCP nodes.