Ben Clifford

Performance Monitoring w/ ELK - Part I: Installing ELK

Blog Post created by Ben Clifford Employee on Jul 23, 2018

Introduction

This guide is the first in a series explaining how to use open source ELK to visualize the performance of a system. This post includes instructions to install the ELK software. The second guide in the series, Performance Monitoring w/ ELK - Part II: Monitoring HCP Access Logs, gives instructions to configure HCP and your newly installed ELK software to visually monitor HCP. Following the instructions in these 2 posts, you can be visualizing HCP HTTP gateway access logs in under 2 hours. All you need to begin is a Linux server or workstation and a running HCP. Let's get started.

 

"ELK" consists of three open source projects: Elasticsearch, Logstash, and Kibana. Elasticsearch is a search and analytics engine. Logstash is a server-side data processing pipeline that ingests data from multiple sources simultaneously, transforms it, and then sends it to a "stash" like Elasticsearch. Kibana is a visualization tool that lets users visualize data in Elasticsearch with charts and graphs.

 

The following chart was generated in real time by transmitting HCP access logs to Logstash over the syslog protocol, indexing the logs in Elasticsearch, and visualizing them with Kibana. This chart visualizes transaction load distribution among HCP nodes.

 

This first guide explains how to install the ELK stack on a Linux server that supports RPM based install (Redhat, Fedora, CentOS, SUSE, etc.). In subsequent posts, I’ll explain how to apply ELK to monitoring the performance of any Linux system as well as how to monitor the performance of several specific software systems like Pulse Secure vADC load balancer and Hitachi Content Platform.

 

For my own ELK performance monitoring, I have been using a CentOS VM system with 64G RAM and 12 vCPU. This has been adequate for monitoring a 4 node HCP system under significant load. Similarly, to use ELK, you should also have a system with significant storage capacity as the ELK indexes can get quite large. Note that this guide is not intended to give any guidance on sizing a production ELK environment or configuring your system for availability. These instructions are intended only for POC ELK environments.

 

After you have installed the Linux OS (not covered here) on your host, you’ll need to install Java, ELK, Kibana, and Logstash. The steps described below are specific to CentOS using YUM but should be easily translatable to other front-end software package managers, like DNF. Make sure that you have network connectivity between the ELK host and the systems you wish to monitor.

 

Important: This monitoring solution is not provided or supported by Hitachi Vantara. If you are looking for a supported Hitachi Vantara monitoring solution, use the Hitachi Content Monitor (HCM). For more information, refer to this announcement in the Hitachi Content Intelligence space: Announcing Hitachi Content Intelligence v1.3.

 

Installing ELK

Step 1: Disable Firewall or Open Ports

You can either disable your firewall entirely, or open the ports needed for Kibana, Elasticsearch, and any logstash listeners you configure.

 

Examples on CentOS

1a: To disable FW:

systemctl disable firewalld
systemctl stop firewalld

1b: Or to open ports:

firewall-cmd --zone=public --add-port=5601/tcp --permanent
firewall-cmd --zone=public --add-port=9200/tcp --permanent
firewall-cmd --zone=public --add-port=514/udp --permanent
firewall-cmd --zone=public --add-port=515/udp --permanent
systemctl restart firewalld
  • 5601: kibana web application
  • 9200: elastic rest API
  • 514: logstash syslog listener for HCP
  • 515: logstash syslog listener for vADC

 

Step 2: Install Java

1: Download the latest Java 8 SDK RPM from Oracle Technology Network.

 

There are multiple ways you can get the rpm to your node. For convenience here is a command to download the Java SE x64 Development Kit 8u172 directly to your ELK node via command line. This may not work as the file locations can change:

# wget --no-cookies --no-check-certificate --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2Ftechnetwork%2Fjava%2Fjavase%2Fdownloads%2Fjdk8-downloads-2133151.html; oraclelicense=accept-securebackup-cookie;" "http://download.oracle.com/otn-pub/java/jdk/8u172-b11/a58eab1ec242421181065cdc37240b08/jdk-8u172-linux-x64.rpm"

 

2: Install Java by invoking the rpm installation:

rpm -ivh jdk-8u172-linux-x64.rpm

 

Step 3: Install Elasticsearch

1: Create the file /etc/yum.repos.d/elasticsearch.repo with the following content:

[elasticsearch-6.x] 
name=Elasticsearch repository for 6.x packages
baseurl=
https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=
https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

 

2: Install elasticsearch using yum:

yum install elasticsearch

 

3: Edit the file /etc/elasticsearch/elasticsearch.yml to make elasticsearch available to external IP. Set the following property:

network.host: _<netadapter>:ipv4_,_local_

Where <netadapter> is your network adapter for the IP on which you want to expose the service. You can use the ifconfig command to find your adapter. In my case the adapter is ens32 so my setting was:

network.host: _ens32:ipv4_,_local_

 

4: Edit the file /etc/elasticsearch/jvm.options to give elastic jvm enough working memory (heap space). Set the following properties:

-Xms16g

-Xmx16g

This is one of those settings you will have to play around with in a production config. I expect on a very heavy load system with high transaction counts you will want lots of CPUs and to give elasticsearch a ton of heap.

 

5: Enable the service to start automatically on reboot, start the service, and check the status of the service is running

systemctl enable elasticsearch
systemctl start elasticsearch
systemctl status elasticsearch

 

Step 4: Install Kibana

1: Create the file /etc/yum.repos.d/kibana.repo with the following content:

[kibana-6.x]
name=Kibana repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

 

2: Install kibana using yum:

yum install kibana

 

3: Edit the file /etc/kibana/kibana.yml and set the following properties:

server.host: <ipaddress>

To make elasticsearch available on external IP. Where <ipaddress> is the IP on which you want to expose the service. You can use the ifconfig command to find your ip.

 

elasticsearch.requestTimeout: 360000

Time in milliseconds to wait for responses from the back end or Elasticsearch. 6 minutes is more than enough to avoid having your queries timeout.

 

4: Create the file /etc/kibana/jvm.options to give kibana jvm enough working memory (heap space). Use the following content:

## JVM configuration

################################################################
## IMPORTANT: JVM heap size
################################################################
##
## You should always set the min and max JVM heap
## size to the same value. For example, to set
## the heap to 4 GB, set:
##
## -Xms4g
## -Xmx4g
##
## See https://www.elastic.co/guide/en/elasticsearch/reference/current/heap-size.html
## for more information
##
################################################################

# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space

-Xms8g
-Xmx8g

If by chance the file already exists, just set the -Xms and -Xmx properties.

 

5: Enable the service to start automatically on reboot, start the service, and check the status of the service is running

systemctl enable kibana
systemctl start kibana
systemctl status kibana

 

Step 5: Install Logstash

1: Create the file /etc/yum.repos.d/logstash.repo with the following content:

[logstash-6.x]
name=Elastic repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

 

2: Install logstash using yum:

yum install logstash

 

Conclusion

Following this guide, you should be able to install and configure all of the ELK components needed to begin visualizing system performance. From beginning to end, the entire process should take under 1 hour.

 

Check out the next guide in the series, Performance Monitoring w/ ELK - Part II: Monitoring HCP Access Logs. There you will find instructions to configure HCP for monitoring, and to configure your newly installed ELK components to visually monitor HCP HTTP gateway access logs.

 

If you do choose to follow the instructions in this series I would love to hear from you. Please comment below to say you tried and it worked, or tried and failed, or to provide feedback which I can incorporate for future readers. Thanks!

Outcomes