Timothy Palace

HCP Storage Adaptor for COSBench

Blog Post created by Timothy Palace Employee on Oct 18, 2018

Introduction

COSBench is an open source project originally developed by Intel to test the performance of cloud object stores (OBS) in a quick and scalable way. COSBench is currently the industry standard OBS benchmarking tool and potential Hitachi Content Platform (HCP) customers rely on COSBench results when choosing between object storage vendors. Unfortunately up until now, COSBench was not able to show true HCP performance capabilities. This was primarily a result of 2 factors. First, COSBench would only write to a single directory in the S3 bucket. This resulted in a bottleneck on HCP since HCP relies on a directory structure to properly distribute metadata workloads among its available resources. Second, COSBench had no built in mechanism to evenly distribute work across multiple nodes in a cluster. Consequently, if COSBench were used to test HCP without a load balancer, there would be an imbalance of work among nodes, potentially resulting in connection exhaustion on busy nodes while other nodes remained idle. The end result being that in a competitive situation, COSBench was putting HCP at a disadvantage, and not giving potential customers or an accurate representation of HCP performance capabilities.

Fortunately COSBench is designed to be extensible, allowing for plugins to be built, and to add support for different object storage products regardless of object protocol or authentication method. The Content Solutions Engineering team has delivered an engineering qualified COSBench storage adaptor for HCP. This adaptor leverages all of the strengths of COSBench while eliminating the specific deficiencies that resulted in misleading performance results for HCP. Requests will always be distributed evenly across all HCPnodes and objects evenly divided among 64,000 directories.

The COSBench adaptor for HCP also introduces a number of configurable options to better emulate customer workloads and tune performance. Our release introduces 4 options for authentication including HCP authentication, anonymous, AWS Signature Version 2, and AWS Signature Version 4. You may choose whether to use the default 64K directory distribution method, or to instead write to a single folder. And finally you may choose to disable "use_expect_continue", as an option that can result in significant improvements in small object performance.

Using the new COSBench adaptor for HCP, you can finally engage in "apples to apples" comparisons of HCP with other OBS products. You can be confident that the results will be fair, and will show HCP in an accurate light.

 

 

Installation/Configuration

The following instructions for installing COSBench are specific to Linux servers. Additional setup may be required in order to run on Windows/Mac OS X.

 

  1. Make sure the following programs are installed on each server that is going to be running in your COSBench environment: cURL, Java 1.7 (or newer), netcat. The “http_proxy” environment variable also needs to be unset.
  2. On each server download the tarball and extract the contents exactly as shown below:

    # mkdir /opt/cosbench

    # cd /opt/cosbench

    # wget \

    https://github.com/HitachiDataSystems/cosbench/releases/download/v0.4.3.0/v0.4.3.0.tar

    # tar -xvf v0.4.3.0.tar

    # rm -f v0.4.3.0.tar

    # ln -s v0.4.3.0 cos

    # cd /opt/cosbench/cos/

    These steps extract COSBench and link the COSBench home folder to the /opt/cosbench/cos directory. This is important as other tools in the ecosystem like the COSBench Wrapper depend on the COSBench files to be in this location.
  3. If COSBench is being run as a clustered environment then edit the controller.conf file found within the conf subdirectory. The number of drivers must be increased to the number of drivers being used and an entry must be added for each driver:


[root@COSBench 0.4.2.3]# cat conf/controller.conf
[controller]

drivers = 2

log_level = INFO

log_file = log/system.log

archive_dir = archive

[driver1]

name = driver1

url = http://127.0.0.1:18088/driver

[driver2]

name = driver2

url = http://172.18.225.12:18088/driver

  1. On each driver run the start-driver.sh script and then start the controller with the start-controller.sh script on the server designated to be the controller
  2. At this point you should be able to view the GUI for COSBench through <controller_ip_addres>:19088/controller
    In the GUI, you may see a red indicator next to each driver that is not on the controller node, this is okay and expected:

 

  1. To stop each running java process, run stop-driver.sh or stop-controller.sh depending which kind of process is running on each server.

 

Defining Workloads

COSBench is operated by submitting a series of XML configuration files called ‘workloads’ that specify how to interact with the cloud object store. Each workload file defines which specific storage to use, and various configuration values that are often specific to this particular storage adaptor. Following the storage definition, a ‘workflow’ is defined that is made up of one or many ‘workstages’. Each workstage defines the object level operations that are to be done such as the type (write, read, delete), the container(s) to write to, how many files to write, and how many workers to use.

To specify to use of the HCP storage adaptor, the storage type is simply set to “hcp” and the config values are specified. The required configuration parameters are endpoints as well as tenant level username and password (unless anonymous authentication is being used). The following table displays all configurable HCP storage adaptor configuration values:

 

 

 

Parameter

Type

Default

Comment

endpoint

String

The tenant endpoint connection

username

String

The tenant username with appropriate permissions

password

String

The password corresponding with the tenant username

protocol

String

https

Either http or https to specify whether or not to use SSL

max_error_retry

Integer

0

Maximum number of retries a client will do on error

connection_timeout

Integer

60000

Milliseconds allowed before timing out on a request

socket_timeout

Integer

30000

Milliseconds allowed before a socket is closed

signer_version

String

v4

The type of authentication to use. Valid values are: v4, v2, hcp, and anon

optimize_dir

Boolean

true

Whether or not to optimize the directory structure. Setting this to false will result in a flat directory structure

disable_cert_check

Boolean

true

Whether or not to disable certificate checking for SSL connections.

use_expect_continue

Boolean

true

Whether or not each client should be configured to wait for the 100 continue response from the server before beginning to send data

 

 

The following picture XML displays a simple workload that will write 10,000 4KB objects and then read them back to the namespace ‘ns1’. If the namespace ‘ns1’ does not already exist for the tenant endpoint then this namespace will be created. This will use V4 authentication over https but with SSL certification checks disabled. This example is also available in the git repo as well as in the conf folder of the deployed COSBench server.

 

<?xml version="1.0" encoding="UTF-8" ?>

<workload name="hcps3-sample" description="sample benchmark for s3">

<storage type="hcp" config="endpoint=<tenant>.<cluster>.<domain>.com;username=<tenantUser>;password=<password>;

signe_version=v4;protocol=https;disable_cert_check=true" />

 

<workflow>

<workstage name="create-bucket">

<work type="init" workers="1" config="cprefix=ns;containers=r(1,1)" />

</workstage>

<workstage name="create-10k-4kb-files">

<work name="main" type="normal" interval="10" division="object" rampup="0" rampdown="0" workers="50" totalOps="10000">

<operation type="write" config="cprefix=ns; containers=c(1); oprefix=cos_; objects=r(1,10000); sizes=c(4)KB" />

</work>

</workstage>

<workstage name="read-10k-4kb-files">

<work name="main" type="normal" interval="10" division="object" rampup="0" rampdown="0" workers="50" totalOps="10000">

<operation type="read" config="cprefix=ns; containers=c(1); oprefix=cos_; objects=r(1,10000)" />

</work>

</workstage>

</workflow>

</workload>

 

 

These XML configuration files may seem complicated at first, but there are additional examples as well as a more in-depth explanation in the COSBench User guide under Section 4. Workstages are run in the order defined in the workflow.

 

Submitting Workloads

Workloads are submited either through the bash script cli.sh with parameter submit:

[root@COSBench 0.4.2.3]# ./cli.sh submit conf/hcp-config-sample.xml

Accepted with ID: w4

 

or through the main GUI:

 

Once the workload has been submitted we can see "which workload COSBench is currently working on in the UI. Selecting the workload will provide more information about which specific workstage is being run, current snapshots, and driver status. If there is an active workload being run then any additional submissions will be queued up to run after the active workload completes.

Configuring HCP for COSBench

The tenant level endpoint that is supplied to the HCP storage adaptor must have MAPI enabled and the tenant level user must have Administrator permissions. This simple configuration alone will work for the default values of the HCP storage adaptor. COSBench is able to create a bucket through the tenant endpoint provided and this will use the namespace defaults specified for the tenant..

If anonymous authentication is to be used then the namespace must be pre-created and configured with appropriate protocol settings. The XML configuration for the workflow should then use the configuration values “cprefix” and “containers” to point to the namespace that was created with the appropriate protocols. If a range of only one value is specified for the “containers” configuration value then this will simply be appended to the “cprefix” value. For example, using a “cprefix” of “myNamespace” and a “containers” value of “r(2,2)” will have the operation point to the namespace “myNamespace2”.

 

Troubleshooting

Logging can be turned up for the controller in the controller.conf file and for each driver in the driver.conf file found in the conf subdirectory. The controllers and drivers must be restarted in order for these to take effect. The logs for COSBench can be found in the log subdirectory. The most useful information for overall system troubleshooting can also be found in the log subdirectory. Logs for specific runs of a workload can be found in the mission subdirectory and is useful if an individual stage of a workload is failing.

COSBench Results

To view the results for a workload, click on the “view details” link for that workload in the main controller UI. If you do not see your workload here (this happens when the COSBench controller has been restarted), click the “load archived workloads” link to load all of the previous workloads that COSBench is able to find.

Once within the specific workload view, there are various links that are able to be drilled into including the specific stage details and detailed performance statistics. Also, in this same view, there is an option to download the specific workload XML that has been run.

The raw data is stored as a series of snapshots that include Op-Count, Byte-Count, Average Response Time, Average Processing Time, Throughput, Bandwidth, and Success Ratio. The snapshot time-frame is configurable within the XML workload that is sent through the “interval” parameter for each workstage. All this data can be exported through a downloadable CSV file or can be found as a csv within the archive subdirectory of COSBench on the controller server.

Futures

The plan going forward is to maintain COSBench within the Hitachi Vantara open source github space.Also, COSBench will continue to be used for internal testing and benchmarking. Any additional fixes that are made will be pushed back upstream to this origin.. If there are any questions or feature requests, please don’t hesitate to either email me or leave a comment below.

Outcomes