Ben Clifford

Performance Monitoring w/ ELK - Part I: Installing ELK

Blog Post created by Ben Clifford Employee on Jul 23, 2018

Introduction

This is the first in a series of blog posts explaining how to use open source ELK to visualize the performance of a system. "ELK" consists of three open source projects: Elasticsearch, Logstash, and Kibana. Elasticsearch is a search and analytics engine. Logstash is a server‑side data processing pipeline that ingests data from multiple sources simultaneously, transforms it, and then sends it to a "stash" like Elasticsearch. Kibana is a visualization tool that lets users visualize data in Elasticsearch with charts and graphs.

 

The following chart was generated in real time by transmitting HCP access logs to Logstash over the syslog protocol, indexing the logs in Elasticsearch, and visualizing them with Kibana. This chart visualizes transaction load distribution among HCP nodes.

 

This first blog post explains how to install the ELK stack on a Linux server that supports RPM based install (Redhat, Fedora, CentOS, SUSE, etc.). In subsequent blog posts, I’ll explain how to apply ELK to monitoring the performance of any Linux system as well as how to monitor the performance of several specific software systems like Pulse Secure vADC load balancer and Hitachi Content Platform.

 

For my own ELK performance monitoring, I have been using a CentOS VM system with 64G RAM and 12 vCPU. This has been adequate for monitoring a 4 node HCP system under significant load. Similarly, to use ELK, you should also have a system with significant storage capacity as the ELK indexes can get quite large. Note that this blog is not intended to give any guidance on sizing a production ELK environment or configuring your system for availability. The instructions in this blog are intended only for POC ELK environments.

 

After you have installed the Linux OS (not covered by this post) on your host, you’ll need to install Java, ELK, Kibana, and Logstash. The steps described below are specific to CentOS using YUM but should be easily translatable to other front-end software package managers, like DNF. Make sure that you have network connectivity between the ELK host and the systems you wish to monitor.

 

Important: This monitoring solution is not provided or supported by Hitachi Vantara. If you are looking for a supported Hitachi Vantara monitoring solution, use the Hitachi Content Monitor (HCM). For more information, refer to this announcement in the Hitachi Content Intelligence space: Announcing Hitachi Content Intelligence v1.3.

 

Installing ELK

Step 1: Disable Firewall or Open Ports

You can either disable your firewall entirely, or open the ports needed for Kibana, Elasticsearch, and any logstash listeners you configure.

 

Examples on CentOS

1a: To disable FW:

# systemctl disable firewalld
# systemctl stop firewalld

1b: Or to open ports:

# firewall-cmd --zone=public --add-port=5601/tcp --permanent

# firewall-cmd --zone=public --add-port=9200/tcp --permanent

# firewall-cmd --zone=public --add-port=514/udp --permanent

# firewall-cmd --zone=public --add-port=515/udp --permanent

# systemctl restart firewalld

  • 5601: kibana web application
  • 9200: elastic rest API
  • 514: logstash syslog listener for HCP
  • 515: logstash syslog listener for vADC

 

Step 2: Install Java

1: Download the latest Java 8 SDK RPM from Oracle Technology Network.

 

Here is a command to download the Java SE x64 Development Kit 8u172 directly to your ELK node via command line:

# wget --no-cookies --no-check-certificate --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2Ftechnetwork%2Fjava%2Fjavase%2Fdownloads%2Fjdk8-downloads-2133151.html; oraclelicense=accept-securebackup-cookie;" "http://download.oracle.com/otn-pub/java/jdk/8u172-b11/a58eab1ec242421181065cdc37240b08/jdk-8u172-linux-x64.rpm

 

2: Install Java by invoking the rpm installation:

# rpm -ivh jdk-8u172-linux-x64.rpm

 

Step 3: Install Elasticsearch

1: Create the file /etc/yum.repos.d/elasticsearch.repo with the following content:

[elasticsearch-6.x] 
name=Elasticsearch repository for 6.x packages
baseurl=
https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=
https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

 

2: Install elasticsearch using yum:

# yum install elasticsearch

 

3: Edit the file /etc/elasticsearch/elasticsearch.yml to make elasticsearch available to external IP. Set the following property:

network.host: _<netadapter>:ipv4_,_local_

Where <netadapter> is your network adapter for the IP on which you want to expose the service. You can use the ifconfig command to find your adapter. In my case the adapter is ens32 so my setting was:

network.host: _ens32:ipv4_,_local_

 

4: Edit the file /etc/elasticsearch/jvm.options to give elastic jvm enough working memory (heap space). Set the following properties:

-Xms16g

-Xmx16g

This is one of those settings you will have to play around with in a production config. I expect on a very heavy load system with high transaction counts you will want lots of CPUs and to give elasticsearch a ton of heap.

 

5: Enable the service to start automatically on reboot, start the service, and check the status of the service is running

# systemctl enable elasticsearch

# systemctl start elasticsearch

# systemctl status elasticsearch

 

Step 4: Install Kibana

1: Create the file /etc/yum.repos.d/kibana.repo with the following content:

[kibana-6.x]

name=Kibana repository for 6.x packages

baseurl=https://artifacts.elastic.co/packages/6.x/yum

gpgcheck=1

gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch

enabled=1

autorefresh=1

type=rpm-md

 

2: Install kibana using yum:

# yum install kibana

 

3: Edit the file /etc/kibana/kibana.yml and set the following properties:

server.host: <ipaddress>

To make elasticsearch available on external IP. Where <ipaddress> is the IP on which you want to expose the service. You can use the ifconfig command to find your ip.

 

elasticsearch.requestTimeout: 360000

Time in milliseconds to wait for responses from the back end or Elasticsearch. 6 minutes is more than enough to avoid having your queries timeout.

 

4: Create the file /etc/kibana/jvm.options to give kibana jvm enough working memory (heap space). Use the following content:

## JVM configuration

 

################################################################

## IMPORTANT: JVM heap size

################################################################

##

## You should always set the min and max JVM heap

## size to the same value. For example, to set

## the heap to 4 GB, set:

##

## -Xms4g

## -Xmx4g

##

## See https://www.elastic.co/guide/en/elasticsearch/reference/current/heap-size.html

## for more information

##

################################################################

 

# Xms represents the initial size of total heap space

# Xmx represents the maximum size of total heap space

 

-Xms8g

-Xmx8g

If by chance the file already exists, just set the -Xms and -Xmx properties.

 

5: Enable the service to start automatically on reboot, start the service, and check the status of the service is running

# systemctl enable kibana

# systemctl start kibana

# systemctl status kibana

 

Step 5: Install Logstash

1: Create the file /etc/yum.repos.d/logstash.repo with the following content:

[logstash-6.x]

name=Elastic repository for 6.x packages

baseurl=https://artifacts.elastic.co/packages/6.x/yum

gpgcheck=1

gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch

enabled=1

autorefresh=1

type=rpm-md

 

2: Install logstash using yum:

# yum install logstash

 

Conclusion

After reading this blog, you should be able to install and configure all of the ELK components needed to begin visualizing system performance. From beginning to end, the entire process should take under 1 hour.

 

The next blog post in this series will walk you through how to monitor an HCP cluster's performance by visualizing access log data.

Outcomes