Overview
The Kubernetes monitoring system helps administrators recognize the issues in a Kubernetes cluster and implement proactive cluster management strategies. It simplifies management of a containerized infrastructure by tracking cluster resource utilization as well as CPU, memory, and storage.
In Kubernetes, cluster operators monitor the cluster and send an alarm if the necessary number of pods are not running, if resource usage is close a critical limit, or if there is a failure or configuration error that barricades a pod or node from joining the cluster.
There are number of tool available to achieve this. Some of the common tools are Grafana and Prometheus.
This blog will not cover the installation and setup of the Kubernetes cluster, Hitachi Storage Plug-in for Containers (HSPC), Hitachi Storage Plug-in for Prometheus (HSPP). Although HSPP comes with default dashboard, you can extend and create new dashboard. This guide will cover how to use Grafana/Prometheus to add additional dashboards that are related to the use case you want around persistent storage from VSP. If you want to skip and just download the new sample dashboard, you can access it here.
If you want to read more about Hitachi Integration for Kubernetes, check out recent reference architecture our team released recently “UCP and Storage Integration for RedHat OpenShift” here.
Grafana
Grafana is an open-source platform for visualization of metrics and analytics that offers built-in dashboards for Kubernetes clusters, node, pod/containers, and deployment.
Prometheus
Prometheus provides a time series-based database. It is a popular monitoring software that provides alerts with detailed metrics and analysis for Kubernetes and Docker.
Kubernetes administrators can create data-rich dashboards in Grafana leveraging the information sourced by Prometheus.
Hitachi Storage Plug-in for Prometheus
Hitachi Storage Plug-in for Prometheus (HSPP) – a Prometheus exporter for Hitachi storage systems that acutely reduces the troubleshooting challenges because this single tool empowers the Kubernetes administrator to monitor metrics of both the Kubernetes resources and Hitachi Storage system resources. Storage Plug-in for Prometheus also leverages Grafana to visualize those metrics for easy assessment by the Kubernetes administrator. Prometheus collects storage system metrics such as capacity, IOPS, and transfer rate in five-minute intervals.
Hitachi Storage Plug-in for Prometheus supports any Kubernetes cluster configured with Hitachi Storage Plug-in for Containers.
Hitachi Storage Plug-in for Containers
Hitachi Storage Plug-in for Prometheus (HSPP) – a Prometheus exporter for Hitachi storage systems that acutely reduces the troubleshooting challenges because this single tool empowers the Kubernetes administrator to monitor metrics of both the Kubernetes resources and Hitachi Storage system resources. Storage Plug-in for Prometheus also leverages Grafana to visualize those metrics for easy assessment by the Kubernetes administrator. Prometheus collects storage system metrics such as capacity, IOPS, and transfer rate in five-minute intervals.
Hitachi Storage Plug-in for Prometheus supports any Kubernetes cluster configured with Hitachi Storage Plug-in for Containers.
Requirements
A Hitachi login credential is required.
- Configure the StorageClass for Hitachi Storage Plug-in for Containers in Kubernetes or Red Hat OpenShift Container Platform.
Configuration
This example configuration is deployed and run on top of an OpenShift Cluster (version 4.9.31) configured with 3 × virtual master nodes and 3 × worker nodes. One of the worker nodes is a physical node (hh01-worker-3) connected to Hitachi Virtual Storage Platform (VSP) via an Emulex HBA on a Fibre Channel network.
Hardware configuration
The following table lists the main hardware components.
Software configuration
The following table lists the main software components.
The following table shows the OpenShift versioning as well as the cluster nodes.
Access into Grafana and Prometheus
After installing Hitachi Storage Plug-in for Containers, install and configure Hitachi Plug-in for Prometheus and Grafana.
For more information, see https://prometheus.io/ and https://grafana.com/.
1. Verify that the Prometheus and Grafana PODs are running:
Monitoring dashboard with Hitachi Storage Plug-in for Prometheus
Use the following procedure to run the monitoring dashboard.
1. Open your browser and navigate to the address output of the oc get routes command and use the following credentials to log in to Grafana.
User: admin
Password: secret
The Home Page will open.
2. Click the General icon
on the sidebar. and then select
Dashboards. The installer includes a dashboard for
HSPC Volumes.
The HSPC Volumes dashboard shows metrics for Persistent Volumes as well as Capacity, Response Time, IOPS, Read/Write Transfer Rate, and Cache Hit Rate.
These metrics can be presented by Namespace, Persistent Volume Claims VC, Storage Class, Storage Serial Number, or Storage Pool ID.
Create a customized Storage Plug-in for Prometheus dashboard on Grafana
Basically, a customized Prometheus dashboard includes multiple panels to show different monitoring resources and metrics.
Here are the steps to create a customized Storage Plug-in for Prometheus dashboard.
1. Create a new panel on a new dashboard
To create a new Storage Plug-in for Prometheus dashboard panel, navigate to the following path:
1-1. From the Home screen click “
+” to add an empty panel, or from the side bar click the General Icon
->
Manage ->
New Dashboard.
1-5. On the right side on the
New dashboard / Edit page click
Panel from the top bar.
1-6. Select a
Visualization1-7. Under the
Panel option, expand
Settings to add the panel title.
1-8. Click
Apply on the top right corner to save the configuration.
2. Update Panel
To edit a panel, do the following:
2-1. Click on the panel title.
2-2. Click on the arrow next to it and select Edit.
2-3. Click Apply to save changes.
3. Save the newly created dashboard
To save the newly created dashboard follow these steps:
3-1. From the New Dashboard/Edit Panel page, click Save on the top right.
3-2. Add a name for the new dashboard.
3-3. Select the folder to save the new dashboard in.
4. Add a new panel to the newly created dashboard
To open a newly created dashboard and add additional panels, do the following:
4-1. From the side bar, select the General icon, and then click Manage.
4-2. Select the newly created dashboard.
4-3. From the top bar on the right side, select
Add Panel 
.
4-4. Click Add an empty panel.
You can see above figure with original size from here.
The previous steps describe how to create a custom Hitachi Storage Plug-in for Prometheus (HSPP) dashboard on Grafana.
To do a deep dive, download then import the sample custom HSPP dashboard on Grafana, that has been created in Hitachi Vantara from the following link showing necessary pod/storage monitoring metrics which would be important for any Kubernetes administrator.
Download sample custom HSPP dashboard
References: