Docker container is gaining popularity and the Docker adoption is on the rise. As demand for the Docker platform increases, Hitachi is also helping customers making better and more reliable Docker infrastructure. In this article, I would like to introduce Hitachi Storage Plug-in for Containers (HSPC) with Docker Swarm. The overview of HSPC can be also found in the link below.
The HSPC Quick Reference Guide can be found below.
Statefull Persistent Storage for Container
HSPC enables docker container applications to access the persistent storage backed by industry leading Hitachi storage, Virtual Storage Platform (VSP) series. Some of the benefits that HSPC provides are:
- Dynamically deploy reliable persistent container volumes presented by Hitachi VSP G/F storage
- Storage-based container volume cloning
- Enhance container data protection by storage-based volume snapshot and restore
The figure below shows how the HSPC volumes are presented to the containers in Docker Swarm environment.
- In the example above, 2 servers are in a Docker Swarm cluster.
- When a docker volume is created by the HSPC, a virtual volume (LUN) is created in the storage and presented dynamically to the host with the target container.
- Docker Swarm - the container orchestration layer selects which container runs on which server.
- In this case, 2 containers are running with HSPC volume #1 and #2 on Server1, and 1 HSPC volume is attached to a container on Server 2.
- The HSPC volume #1 and #2 are only visible to Server1, and other servers do not have access to them.
- Presenting the HSPC volumes to only the targeted server/container increase the data security and integrity.
- When container fail-over occurs, the container will start on the other host and the existing HSPC volume will be provisioned to that host as well.
Planned Failover + Snapshot Validation
In this article, I would like to show you an example of snapshot and restore of HSPC docker volume, and container planned/maintenance failover in docker swarm environment. This is just to show some of the useful capabilities of HSPC. The following steps outlines the high-level procedure.
- Create docker swarm 3-node cluster. Node name: manager1, worker1, and worker2
- Create a docker swarm container with persistent volume dynamically created by HSPC on worker1 node (docker swarm selected this automatically in this particular case)
- From the container, write some data to the attached HSPC volume
- Take a storage-based snapshot of the HSPC volume
- Perform docker swarm maintenance failover
- The docker swarm brings up the container with HSPC volume to worker2 node
- Perform storage-based restore from the snapshot taken in worker1 node
The table below lists the main hardware components used.
|Hitachi VSP G600||1|
2 x FC ports used
1 x HDP pool for volume creation
1 x HTI pool for snapshot
Intel Xeon CPU E5-2680 v4 - 2 Sockets
RAM 256 GB
1 x Emulex LPe12000 HBA 2 ports
The table below shows the main software components used.
|Hitachi Storage Plug-in for Container (HSPC)||1.1.0|
|Red Hat Enterprise Linux||7.5|
The figure below shows the lab setting used for this validation.
1. Create docker swarm 3-node cluster. Node name: manager1, worker1, and worker2
Follow the link here to create 3-node docker swarm cluster.
The following shows the status of swarm nodes
2. Create a docker swarm container with persistent volume dynamically created by HSPC
Use the following command to create docker service with HSPC volume. In this example, the new docker service: service3 with HSPC volume serviceVol3 was deployed to worker1 node.
# docker service create --name service3 --mount type=volume,source=serviceVol3,destination=/mnt,volume-driver=hspc,volume-opt=size=130m -d nginx
Use docker service ps service3 command to find out which node it was deployed.
Use docker volume ls command to show the newly created HSPC volume.
The newly created HSPC volume can also be found in storage management software such as Device Manager or Hitachi Storage Advisor.
3. From the container, write some data to the attached HSPC volume
For this validation purpose only, we manually go worker1 node, and go into the container: service3.1 and write some data to the HSPC volume with following command. (Normally, custom container application starts to write data into HSPC persistent volume)
# docker exec -it <containerID> /bin/bash -c 'echo before-snapshot-worker1 > /mnt/test.txt'
As shown above, a text "before-snapshot-worker1" was written into a text file on HSPC volume.
4. Take a storage-based snapshot of the HSPC volume
Use the following command to take a storage-based HSPC volume snapshot. For docker swarm environment, the snapshot can be taken in any swarm node.
# /opt/hitachi/hspc/hctl snapshot create -sourceVolName serviceVol3
You can validate snapshot creation by docker volume inspect command. Each snapshot has a unique ID called MU (Mirror Unit), and this MU will be used for restore operation later.
5. Perform docker swarm maintenance failover
In case of server maintenance, use following command to failover the running container(s) to other nodes in cluster.
# docker node update --availability drain worker1
In this example, the worker1 node went under the maintenance and the container: service3 failed-over to worker2 node.
6. The docker swarm brings up the container with HSPC volume to worker2 node
Let's validate the data consistency after the failover. For this, we manually go to the worker2 node, and read the content of the text file that we wrote in worker1 node. As shown below, the text "before-snapshot-worker1" successfully appeared.
7. Perform storage-based restore from the snapshot taken in worker1 node
Before restoring previous data from a snapshot, let's update some data on HSPC volume. The following shows that we updated the content of the text file from "before-snapshot-worker1" to "after-snapshot-by-worker2".
Use the command below to restore a HSPC volume with snapshot MU number. Restoring to previous snapshot requires no volume access from the server host, so we need to bring down the container for a short amount of time.
# /opt/hitachi/hspc/hctl snapshot restore -sourceVolName serviceVol3 -mu 3
We can validate the status of snapshot restore by checking the SplitTime update like below.
Let's bring up the container with HSPC volume after restore with the command below.
# docker service create --name service3 --mount type=volume,source=serviceVol3,destination=/mnt -d nginx
The container service3 is up again on the worker2 node.
Let's check the contents of the HSPC volume after the restore.
The content of the text file was restored successfully from "after-snapshot-by-worker2" to previous snapshot contents of "before-snapshot-worker1".
Helpful Tips and Configurations
In this section, I would like to share some of the tips and configurations that might be helpful for you to set up HSPC environment.
HSPC Configuration File
During the HSPC installation, you will need to enter the storage information into the configuration file below. If you are using multiple storage ports to setup a multi-pathing, you can enter multiple storage port information like this. For Docker Swarm environment, this file will be unique for each host since you need to create a separate storage Host Group for every HBA port.
You can find "scsiTargetNumber" from the Deivice Manager - Storage Navigator shown in the picture below. You will need to convert it to decimal value
Storage Host Group Configuration for Docker Swarm
For Docker Swarm environment, create one Host Group for every host like below.
- In this case, Host Group # is 0A and "scsiTargetNumber" is 10 (converting hex 0A to decimal) for HSPC configuration file /opt/hitachi/hspc/config.json
Find Matching LDEV ID of HSPC Volume
Normally, you will probably not need to know corresponding LDEV information for your HSPC volume, but you might need this in case of troubleshooting and validation. First, the LDEV ID can be found in the docker host by using docker volume inspect command. Note that, the LDEV ID here is shown as decimal. On the Device Manager - Storage Navigator, the LDEV ID is shown in Hex.
To convert Decimal to Hex, simply use Calculator tool on Windows shown below.
Now we know we need to look for LDEV ID of 00:6D for corresponding HSPC volume shown below.
Hitachi released version 1.1.0 of Hitachi Storage Plug-in for Containers (HSPC) in May 2018 along with support of latest VSP G/F series. In this article, we are just showing some of the capabilities that HSPC offers. Stay tuned for upcoming new releases with more features and integrations with container ecosystems.