Converged & Hyperconverged Infrastructure

 View Only

Hitachi Storage Plug-in for Prometheus Custom Dashboard on Grafana

By Hossein Heidarian posted 09-06-2022 19:32

  

Overview

The Kubernetes monitoring system helps administrators recognize the issues in a Kubernetes cluster and implement proactive cluster management strategies. It simplifies management of a containerized infrastructure by tracking cluster resource utilization as well as CPU, memory, and storage.

In Kubernetes, cluster operators monitor the cluster and send an alarm if the necessary number of pods are not running, if resource usage is close a critical limit, or if there is a failure or configuration error that barricades a pod or node from joining the cluster.

There are number of tool available to achieve this. Some of the common tools are Grafana and Prometheus.

This blog will not cover the installation and setup of the Kubernetes cluster, Hitachi Storage Plug-in for Containers (HSPC), Hitachi Storage Plug-in for Prometheus (HSPP). Although HSPP comes with default dashboard, you can extend and create new dashboard. This guide will cover how to use Grafana/Prometheus to add additional dashboards that are related to the use case you want around persistent storage from VSP. If you want to skip and just download the new sample dashboard, you can access it here.

If you want to read more about Hitachi Integration for Kubernetes, check out recent reference architecture our team released recently “UCP and Storage Integration for RedHat OpenShift” here.


Grafana

Grafana is an open-source platform for visualization of metrics and analytics that offers built-in dashboards for Kubernetes clusters, node, pod/containers, and deployment.


Prometheus

Prometheus provides a time series-based database. It is a popular monitoring software that provides alerts with detailed metrics and  analysis for Kubernetes and Docker.

Kubernetes administrators can create data-rich dashboards in Grafana leveraging the information sourced by Prometheus.


Hitachi Storage Plug-in for Prometheus

Hitachi Storage Plug-in for Prometheus (HSPP) – a Prometheus exporter for Hitachi storage systems that acutely reduces the troubleshooting challenges because this single tool empowers the Kubernetes administrator to monitor metrics of both the Kubernetes resources and Hitachi Storage system resources. Storage Plug-in for Prometheus also leverages Grafana to visualize those metrics for easy assessment by the Kubernetes administrator. Prometheus collects storage system metrics such as capacity, IOPS, and transfer rate in five-minute intervals.

Hitachi Storage Plug-in for Prometheus supports any Kubernetes cluster configured with Hitachi Storage Plug-in for Containers.


Hitachi Storage Plug-in for Containers

Hitachi Storage Plug-in for Prometheus (HSPP) – a Prometheus exporter for Hitachi storage systems that acutely reduces the troubleshooting challenges because this single tool empowers the Kubernetes administrator to monitor metrics of both the Kubernetes resources and Hitachi Storage system resources. Storage Plug-in for Prometheus also leverages Grafana to visualize those metrics for easy assessment by the Kubernetes administrator. Prometheus collects storage system metrics such as capacity, IOPS, and transfer rate in five-minute intervals.

Hitachi Storage Plug-in for Prometheus supports any Kubernetes cluster configured with Hitachi Storage Plug-in for Containers.



This guide does not cover the installation and setup of the Kubernetes cluster, Hitachi Storage Plug-in for Containers, or Hitachi Storage Plug-in for Prometheus.

See the Hitachi Storage Plug-in for Prometheus Quick Reference Guide for more information.


Requirements

A Hitachi login credential is required.

  • Configure the StorageClass for Hitachi Storage Plug-in for Containers in Kubernetes or Red Hat OpenShift Container Platform.


Configuration

This example configuration is deployed and run on top of an OpenShift Cluster (version 4.9.31) configured with 3 × virtual master nodes and 3 × worker nodes. One of the worker nodes is a physical node (hh01-worker-3) connected to Hitachi Virtual Storage Platform (VSP) via an Emulex HBA on a Fibre Channel network.

Hardware configuration

The following table lists the main hardware components.

Software configuration

The following table lists the main software components.


The following table shows the OpenShift versioning as well as the cluster nodes.



Access into Grafana and Prometheus

After installing Hitachi Storage Plug-in for Containers, install and configure Hitachi Plug-in for Prometheus and Grafana.

For more information, see https://prometheus.io/ and https://grafana.com/.

1.   Verify that the Prometheus and Grafana PODs are running:


Make sure the four PODs are running.

2.   Access Grafana.

If you use NodePort, access Grafana with <Your Node IP Address>:<Grafana Port>. You can identify <Grafana Port> by using the following command.


If you expose Grafana, get endpoint as follows.

The Grafana user/password are admin/secret.



Monitoring dashboard with Hitachi Storage Plug-in for Prometheus

Use the following procedure to run the monitoring dashboard.

1.   Open your browser and navigate to the address output of the oc get routes command and use the following credentials to log in to Grafana.
User: admin
Password: secret
The Home Page will open.

2.   
Click the General icon on the sidebar. and then select Dashboards. The installer includes a dashboard for HSPC Volumes.


The HSPC Volumes dashboard shows metrics for Persistent Volumes as well as Capacity, Response Time, IOPS, Read/Write Transfer Rate, and Cache Hit Rate.

These metrics can be presented by Namespace, Persistent Volume Claims VC, Storage Class, Storage Serial Number, or Storage Pool ID.



Create a customized Storage Plug-in for Prometheus dashboard on Grafana

Basically, a customized Prometheus dashboard includes multiple panels to show different monitoring resources and metrics.

Here are the steps to create a customized Storage Plug-in for Prometheus dashboard.

1.   Create a new panel on a new dashboard

To create a new Storage Plug-in for Prometheus dashboard panel, navigate to the following path:

1-1.   From the Home screen click “+” to add an empty panel, or from the side bar click the General Icon -> Manage -> New Dashboard.

1-2. 
On the New dashboard / Edit Panel page, under the Query tab, select Prometheus from the drop-down menu.
1-3.   Select pod from the Metrics drop down menu. You can select any other desired metric.
1-4.   Select any of the pod’s child metrics from the displayed list.

1-5.   
On the right side on the New dashboard / Edit page click Panel from the top bar.
1-6.   Select a Visualization
1-7.    Under the Panel option, expand Settings to add the panel title.
1-8.   Click Apply on the top right corner to save the configuration.

2.   Update Panel

To edit a panel, do the following:

2-1.    Click on the panel title.
2-2.   Click on the arrow next to it and select Edit.
2-3.   Click Apply to save changes.

3.   Save the newly created dashboard

To save the newly created dashboard follow these steps:

3-1.    From the New Dashboard/Edit Panel page, click Save on the top right.
3-2.   Add a name for the new dashboard.
3-3.   Select the folder to save the new dashboard in.

4.   Add a new panel to the newly created dashboard

To open a newly created dashboard and add additional panels, do the following:

4-1.    From the side bar, select the General icon, and then click Manage.
4-2.   Select the newly created dashboard.
4-3.   From the top bar on the right side, select Add Panel .
4-4.   Click Add an empty panel.
4-5.   Follow all the steps in Create a New Panel on a new Dashboard.


You can see above figure with original size from here.

The previous steps describe how to create a custom Hitachi Storage Plug-in for Prometheus (HSPP) dashboard on Grafana.

To do a deep dive, download then import the sample custom HSPP dashboard on Grafana, that has been created in Hitachi Vantara from the following link showing necessary pod/storage monitoring metrics which would be important for any Kubernetes administrator.

Download sample custom HSPP dashboard



References:

0 comments
41 views

Permalink