Software Defined Storage​

 View Only

Testing MongoDB Data Replication and Failover Capability Use Cases on Hitachi Virtual Storage Software

By Subhash Sindhe posted 05-04-2023 00:20

Testing MongoDB Data Replication and Failover Capability Use Cases on Hitachi Virtual Storage Software

This Blog demonstrates a reference configuration of Red Hat as a base OS and outlines deploying the MongoDB environment on Hitachi Advanced Servers with Hitachi Virtual Storage Software (VSS).

In the test cases, we show the failover capabilities of MongoDB using the VSS platform.

What is MongoDB

MongoDB is a distributed architecture and an open-source NoSQL database management program with the essential function of creating highly accessible and adaptable web-based technologies. MongoDB is widespread among database development companies because it offers an incredibly flexible schema. In addition, MongoDB, which provides drivers for all scripting languages, enables users to create applications without spending time setting up a database. It also offers other advantages over traditional RDBMS.

Hitachi Virtual Storage Software overview

What is Virtual Storage Software?

Virtual Storage Software (VSS) is a storage software product that builds and sets up a virtual storage system from multiple general-purpose servers. The VSS storage system is two-layer software-defined storage (SDS) in which storage nodes are separated from the compute nodes (disaggregated storage).

To maximize return on investment (ROI) and deliver long-term scaling, Hitachi has redesigned Storage Virtualization Operating System (SVOS RF) to run on commodity servers. By running virtualized or bare metal on your server of choice, you can integrate your core enterprise storage platform with your existing hypervisor. This gives you the ability to run the latest distributed application alongside traditional business applications on a scale out infrastructure, allowing you to scale performance and capacity independently to consolidate even more workloads.

The following illustrations provide architectural diagrams of bare metal and virtualized VSS.


The first SDS offering from Hitachi is a true enterprise solution. With years of research and development, our patented Hitachi Polyphase Erasure Coding (HPEC) brings our legendary reliability into the SDS marketplace. By allowing data to be balanced across the entire cluster, the cost and complexity of management is reduced while offering the most resilient solution.

Benefits of VSS

Seamless data migration across Hitachi storage systems

Hitachi Polyphase Erasure Coding (HPEC) enables minimally disruptive migration from existing Hitachi storage arrays using Fibre Channel connections. Copying data is offloaded to the storage layer so compute workloads are unimpeded. Single data paths can be cutover ensuring I/O continues even during migrations.

Flexible data protection

Hitachi Polyphase Erasure (HPEC) also features a different approach to adapting software data protection. Hitachi carefully evaluated erasure coding through years of research and developed a unique patented approach (US# 10,185,624 B2 and 10,496,479 B2). This approach improved crucial elements of data and parity placement, which efficiently improved latency within Hitachi Polyphase Erasure Coding, accelerated read performance, and lowered storage resource overhead. These necessary enhancements provide a space where your data can be synthesized so you can create more value, while increasing the reliability of a Hitachi-based software defined storage solution.

HPEC can be configured in a 4D+1P RAID configuration which protects against a single node failure and 4D+2P which protects against double node failures.
HPEC(4D+2P) is highly capable in tolerant of 2-node failures and more capacity efficient than Mirror.

Fault domains

Fault domains are a group of storage nodes that share hardware such as the same power supply and network switches. Having fault domains enables storage systems to continue operations if a failure occurs in one of the fault domains.

Test case summary

The architecture benefit of VSS with MongoDB ReplicaSet can be used for data replication. These test cases show that the data replication use cases can be supported by VSS by using MongoDB ReplicaSet. They also show 2-node fault tolerance on a 6-node VSS with MongoDB.

VSS with bare metal (SUSE) was used in this solution. MongoDB was installed on VSS volumes attached to RHEL compute nodes through iSCSI. We used YCSB benchmarking to test the operations/functionality of the test cases of MongoDB.

Install MongoDB on the RHEL operating system

See the following link for more information about installing MongoDB and replica set.


Step1: After installing and configuring the MongoDB replica set, verify the MongoDB replication by logging in to the MongoDB Shell, mongosh.


Note  We used 5 × MongoDB nodes for testing.

For example, in our environment, the following was configured:

Primary Mongo DB Node 1 with IP and Secondary MongoDB Node 2 with IP and this follows with additional Secondary Mongo Nodes 113,114, and 115 as shown.

Test Scenario A

Two compute node failure testing.

1.     Power off the Primary server and the Secondary A server.

(1)   Verify that secondary B becomes a New Primary and it can write and read data.

(2)   Verify that the MongoDB Primary Node1 has no databases created using the show dbs command.

Mongo Node 1 with IP 111 and Node 2 with IP 112 are currently Primary and Secondary A.

2.     Powered off Node1, Node2, and Node 3 become Primary followed by Node 4 and Node 5 as Secondary B and Secondary C.

3.     No databases were written or loaded on Node 3 (Primary) after powering off MongoDB Node 1 and MongoDB Node 2.

We ran a YCSB benchmark to test the read/write operations.

See the following link for more information about installation of YCSB on MongoDB.

4.     Verify that the YCSB ran successfully and was able to create a database in MongoNode2 (Primary).

Summary: This test case shows that MongoDB Replicaset works and data was written to other nodes in case of failure.

Test Scenario B

Two storage node failure testing.

1.     Power off two VSS servers.

2.     Check to see if MongoDB is still working or not.

(1)   Verify the 6 × VSS nodes and volumes are showing Normal on the VSS cluster.

3.     Power off 2 × VSS nodes and verify the status on the VSS cluster.

4.     Log in to the MongoDB Primary node and check the functionality.

5.     Re-run the YCSB benchmarking and see if the databases can be written as shown.

The following shows that YCSB benchmarking ran successfully on the Mongo nodes even with 2 × VSS storage nodes down.

Summary of results

Test A

Two compute node failure testing

Test Case Details


Power off the Primary server Node 1 and the Secondary Node 2 server.


Tested and MongoDB Node 3 became Primary, and Node 4 and Node 5 became Secondary.


Check that secondary B becomes a new Primary and it can write and read data.


Tested YCSB benchmarking and it was successful. Able to write and read data.


Test B

Two storage node failure testing


Test Case Details


Power off two VSS servers.


From 6 VSS clusters (4D+2P),
2 × nodes were powered off.
The cluster recovered later. The MongoDB nodes were stable.


Check that MongoDB is still working or not.


MongoDB was still working, and we were able to run YCSB benchmarking without any issues and were able to create databases.