Skip navigation

HDID for VMware

5 posts

Hitachi Data Instance Director (HDID) can be used to schedule backups of selections of VMs within a vCenter all without affecting performance of the VMs on that vCenter. Then in a recovery scenario, select the backup recovery point in time, what VMs to restore/clone and whether to recover to the same or different vCenter. This is great from the point of view of an admin dealing with storage, but what if you are not a HDID admin? What if you are a vCenter admin, or even a non-admin user with their own VM(s)? Someone without the HDID knowledge, but you still want to create a restore point of a VM in its current state, for whatever reason e.g: before you do something that could potentially break your VM! Well now you can have the power of HDID within vSphere, without having to know how to use HDID! Thanks to the HDID Connector for VMware vRealize Orchestrator (HDID Connector for vRO).

 

Just take a look at how easy it is to create a specific backup of any VM in HDID from within vSphere:

  • Step 1 - On the VM you wish to backup, select "Ad Hoc Backup" from the "All vRealize Orchestrator plugin Actions" menu:
  • Step 2 - On the "Ad Hoc Backup" Workflow parameters window that pops up, select "Finish":
  • Step 3 - Sit back and relax while HDID Connector for vRO does all the behind the scenes work of initiating a backup of only this VM using HDID:

 

This vRO context menu allows the vSphere user to perform a number of different HDID operations from a VM driven focus.

 

This first plugin for vRO is aimed at providing a VMware user simple HDID backup and restore functionality. This is achieved via prebuilt vRO Workflows which are user (VM) focused, that can be used from within either VMware vRealize Orchestrator, or within vSphere itself. The workflows allow the user to perform:

 

Backup

Ad Hoc Backup

   Perform an ad hoc backup of the selected VM only.

Recovery

Clone a VM

   Clones a selected VM that exists in vCenter (from a recovery point in HDID).

Clone This VM

   Clones the current VM (from a recovery point in HDID).

Restore a Deleted VM

   Recreates a VM no longer in vCenter, from a recovery point in HDID, that doesn't exist in vCenter.

Mount VMDKs to This VM

   Mounts, to the current VM, the disks from a VM backed up as a snapshot in HDID.

Unmount VMDKs from VM

   Unmounts, from the current VM, the disks from a VM backed up as a snapshot in HDID.

 

All of this without the VMware user having to create Dataflows, Policies, nor navigating backups to find the VM they want. This is a user (VM) focused approach to HDID from within VMware.

 

In the next few “HDID Connector for VMware vRO” Blog Posts we shall explain: how to setup HDID to allow these Workflows to be used by any VMware user; cover these Workflows in a little more detail where needed; and any additional setup that may be required for particular scenarios you might want to achieve.

 

Upcoming Posts:

HDID Connector for VMware vRO – Part 1: Connecting vRO to HDID – Enabling the recovery workflows

After a not-so-short delay I thought I should continue this series and continue the format of showing how the operations a "normal" user can perform can be completed via the HDID REST API.  What I will attempt to do in this article is to show how to do each of the steps shown in my previous post ( VMware protection in Hitachi Data Instance Director - Part 2: Protection ).   Due to the how many steps are involved in this article I'm only going to cover Software based protection.

 

NOTE: There are a _lot_ more options available than those I am going to show (due to all the variables around the different kinds of protection) and the simplest way to mimic what the user does is to use the developer tools in your web browser to monitor the REST requests.

 

Right, lets get started then!

 

Software based protection

 

Creating a Repository node requires a POST operation to the following URL

 

https://<HDID_MASTER_IP>/HDID/master/NodeManager/objects/RepositoryAgentlessNodes/

 

With the following JSON body:

{
"nodeType": "Repository",
"name": "MyRepoNode",
"resourceGroup": null,
"configuration":
{
"proxyName": "Fibre4@BR-URST33-WR239E-AU9NTZ-FAW24T[0-1-1]",
"import": false,
"rootPath": "c:\\mynewrepo",
"encrypted": false
}
}

 

REMINDERThe tricky bit in the above body is the proxyName.  In HDID this is the node that will actually do the communication with (in this instance) VMware.  In this simple blog post we're going to be using a single HDID master node, but the proxy could just as easily be another HDID client node.

 

From this POST we should get a response that resembles this:

 

{
"timestamp": "2019-05-08T09:11:24",
"name": "MyRepoNode",
"id": "MyRepoNode@00-91BF70-C9ED9D-4B77AC-11B3C5[1-1-35]",
"configuration":
{
"proxyName": "Fibre4@BR-URST33-WR239E-AU9NTZ-FAW24T[0-1-1]",
"rootPath": "c:\\mynewrepo",
"import": false,
"encrypted": false
}
}

 

 

Which is very similar to the body we provided, but also contains the ID of our Repository node.  I'm not going to duplicate the request required for a VMware node as it's been discussed previously.

 

 

Using VMware vStorage API for Data Protection (VADP)

 

For this step we'll need to create a Policy to use with our new Repository node.  To do this we need to make a POST request to the Policy handler:

 

 

https://192.168.45.188/HDID/master/PolicyHandler/objects/Policies/

 

 

With a body like the following

 

{
"name": "VADP_Policy",
"description": "",
"classificationChains": [{
"items": [{
"typeName": "vmwareesx",
"displayName": "VMware",
"normalizedTypeName": "vmwareesx",
"propertyName": "vmware",
"category": "Hypervisor",
"_policyItemType": "classifications",
"vmwareEsx": {
"include": [{
"name": "myvm",
"type": "eVirtualMachine",
"_selected": false
}],
"exclude": [],
"vmwareNodeId": "flynn@00-BCE17A-602E12-4C1B91-B6FDC6[0-1-5]"
},
"_selected": false
}]
}],
"operationChains": [{
"items": [{
"typeName": "Backup",
"displayName": "Backup",
"normalizedTypeName": "backup",
"propertyName": "backup",
"category": "Standard",
"_policyItemType": "operations",
"backup": {
"label": "Backup",
"run": {
"mode": "eRUN_ANYTIME",
"scheduleName": ""
},
"recoveryPointObjective": {
"period": 8,
"units": "eRPO_HOURS"
},
"retention": {
"period": 6,
"units": "eRETENTION_MONTHS"
},
"recoveryTimeObjective": {
"incrementalTapeBackup": false
},
"sourceOption": {
"quiesceApplications": true,
"preScriptEnabled": false,
"postScriptEnabled": false
},
"type": {
"mode": "eTRANSFER_ASYNCHRONOUS_JOURNALED"
}
},
"_selected": false
}]
}]
}

 

This is quite a large payload so I'll try and explain the different parts:

 

 

NOTE: Each of these sections is an array type, allowing multiple classifications and operations.  Whilst having many classifications will only make sense in a small number of cases, having multiple operations allows the use of both Hardware and Repository protection simultaneously.

 

It's already clear that these sections can get highly complex very quickly.  Rather than try to author Policies directly in the REST API I would advise building some "templates" using the HDID UI and then using some variable substitution to reduce complexity.

 

The response to this will contain the Policy ID which we will need later to build our Dataflow.

 

Once the Policy is complete we need to build the Dataflow using a POST request like so

 

https://192.168.45.188/HDID/master/DataFlowHandler/objects/DataFlows/

 

With a body like so

 

{
"data": {
"nodes": [{
"name": "flynn",
"type": "VMwareESX",
"nodeId": "flynn@00-BCE17A-602E12-4C1B91-B6FDC6[0-1-5]",
"id": "c1923",
"assignedPolicies": [{
"id": "63ba65c73b6e4648ac6dab202f64c412",
"operations": []
}]
}, {
"name": "MyRepoNode",
"type": "Repository",
"nodeId": "MyRepoNode@00-91BF70-C9ED9D-4B77AC-11B3C5[1-1-35]",
"id": "c2152",
"assignedPolicies": [{
"id": "63ba65c73b6e4648ac6dab202f64c412",
"operations": [{
"id": 1,
"propertyType": "ot_repositoryDestination",
"ot_repositoryDestination": {
"templateId": "000000000-Standard Repository Store",
"_storeTemplateName": "Standard"
}
}]
}]
}],
"name": "VADP_Repo_Backup_Dataflow",
"description": "",
"connections": [{
"label": "",
"type": "eCT_BATCH",
"netCompression": false,
"id": "c2165",
"source": "c1923",
"destination": "c2152",
"routedPolicies": ["63ba65c73b6e4648ac6dab202f64c412"]
}]
}
}

 

The above creates a Dataflow which connects our VMware node to our Repository node using a batch mover, like this:

 

VMware to Repository dataflow

 

The response contains most of the body the we sent previously but with the ID of the dataflow included too:

 

"id": "bb315688-cf93-4369-8451-d8c06774cbad",

 

The ID property for Dataflow nodes

 

If you look carefully at the above JSON you'll see a new ID property for each node, which is used in the "connections" section, and differs from the NodeID that we've seen before.

 

In order for the monitoring capabilities for HDID to work correctly, each Node on each Dataflow needs a unique identifier, as a Node could easily be used on many Dataflows.  Creating a unique identifier for users of the REST API , however, is a lot of work, as they would need to check all existing Dataflows for IDs that were in use.

 

The solution to this issue is to set the ID to a value that starts with "c" and has a 4 digit number after it.  This value must be consistently used within the create Dataflow request (i.e. on a given node and it's connections) and will be replaced by the Dataflow handler with a fully unique value on creation.

 

Compile and Distribute ( aka Activate the Dataflow )

 

In order to make this Dataflow do some work we need to compile and distribute it to HDID.  In the UI this is described as Activating the Dataflow.

 

Firstly we need to do a PUT request to the dataflow compiler to check that our flow makes sense:

 

https://192.168.45.188/HDID/master/RulesManager/services/Rules/actions/compile/invoke

 

With a body containing the Dataflow ID (or IDs) that we wish to check:

 

{
"ids": ["bb315688-cf93-4369-8451-d8c06774cbad"]
}

 

This returns a body which contains a summary, any warnings and any errors that were encountered whilst compiling the flow

 

{
"summary": [
"Compiling rules for data flow 'VADP_Repo_Backup_Dataflow'",
"Compilation of data flow 'VADP_Repo_Backup_Dataflow' completed successfully"
],
"warnings": [],
"errors": []
}

 

 

If this is successful then we can make a PUT request to distribute the Dataflow:

 

https://192.168.45.188/HDID/master/RulesManager/services/Rules/actions/distribute/invoke

 

With the same body as before:

 

 

{
"ids": ["bb315688-cf93-4369-8451-d8c06774cbad"]
}

 

 

Triggering the Policy Operation

 

In order to trigger our flow to perform it's backup we need to make a request to the Trigger handler.  We can use a simple GET to list all the triggers like so:

 

https://192.168.45.188/HDID/master/TriggerHandler/objects/Triggers

 

And we can use a query string to reduce this to just the Dataflow that we're interested in:

 

https://192.168.45.188/HDID/master/TriggerHandler/objects/Triggers?query=(dataflowId=bb315688-cf93-4369-8451-d8c06774cbad)

 

Which gets a response like so;

 

{
"trigger": [{
"id": "0",
"policyId": "f96262b31d3142acbf8ba77eed8d4cd0",
"policyName": "non vmware specific datastore backup",
"operationId": 1,
"operationName": "Snapshot",
"dataflowName": "os host mount",
"dataflowId": "bb315688-cf93-4369-8451-d8c06774cbad",
"sourceNodeId": "Chesil@00-731595-4BDE0E-4DBA9A-662118[1-1-2]",
"sourceNodeInstance": 16,
"destinationNodeId": "Chesil@00-731595-4BDE0E-4DBA9A-662118[1-1-2]",
"destinationNodeInstance": 16,
"moverType": "eMOVER_NONE",
"syncGroup": "e43fa163-b5d1-4b9b-8848-e88c92f52f8e",
"dataOriginNodeInstance": 16,
"dataOriginNodeId": "Chesil@00-731595-4BDE0E-4DBA9A-662118[1-1-2]"
}],
"pageInfo": {
"totalCount": 1,
"end": true
}
}

 

In order to Trigger our flow we need to make a PUT request to the Trigger handler with most (but not all) of the above data.

 

The endpoint is:

 

https://192.168.45.188/HDID/master/TriggerHandler/services/Triggers/actions/triggeroperation/invoke

 

And the Body looks like so:

 

{
"sourceNodeId": "Chesil@00-731595-4BDE0E-4DBA9A-662118[1-1-2]",
"dataOriginNodeId": "Chesil@00-731595-4BDE0E-4DBA9A-662118[1-1-2]",
"destinationNodeId": "Chesil@00-731595-4BDE0E-4DBA9A-662118[1-1-2]",
"dataflowId": "bb315688-cf93-4369-8451-d8c06774cbad",
"policyId": "f96262b31d3142acbf8ba77eed8d4cd0",
"operationId": 1,
"moverType": "eMOVER_NONE",
"syncGroup": "e43fa163-b5d1-4b9b-8848-e88c92f52f8e"
}

 

If successful we should be returned a job id:

 

{
"job": [
{
"id": "{37fcc02e-9ef6-4c5a-95f7-a0c75972f8c2}"
}
]
}

 

Phew, well done, you made it!  And you may even have a test script that can create a backup of your VM !  Next time I'll cover how we would do this for a Hardware backup.

In this blog post I want to talk about the kinds of VMware protection that HDID can provide, and cover the kinds of situations that you may want to use one or the other (or even both!).

 

The HDID VMware protection options broadly fall into two areas:

  • Software based protection
  • Hardware based protection

 

Software based protection

 

There are two methods of providing software protection for a Virtual Machine depending upon the preference to install a HDID client (agent) on every Virtual Machine that requires protection or not. This decision may be forced by how the Virtual Machine disks are attached, see the last section in this blog post for more details.

 

All the Software based protection solutions make use the HDID Repository. For those new to HDID, here's a quick run down of what the Repository is and what it can do, taken from the HDID User Guide:

 

The Repository is a multi-functional object secondary storage system that can simultaneously perform multiple types of storage operations.

 

Unlike legacy storage systems (which inefficiently create copies of data despite trying to de-duplicate it and require separate stores for backup, archiving and CDP), the Repository has the ability to provide backup, archiving and continuous data protection (CDP) in the same store. This enables the repository to employ several data reduction

techniques, delivering massive storage cost savings. With HDID, if a policy targets backup, archiving, and CDP to the same repository store, then the data is stored only once. The data also is only sent once, reducing the amount of data that is moved around the system

 

The TL;DR of this is that the HDID Repository is its incremental forever without having a rehydration and performance penalty during recovery.

 

Creating a Repository is very simple, it can be found in the Storage section

 

repo_node.PNG

 

For creating a new repository node, only a directory in which to store data is required

 

repo_node_dir.PNG

 

Note

It is important to consider carefully where to place the Repository in order to maximize performance and minimize network traffic. For instance if the Repository node and the VMware ESX Agentless node share the same Proxy then the VMware SAN Transport Mode could be used.

 

Using a HDID Agent

 

Installing a HDID Agent upon a Virtual Machine allows it to be attached to the HDID master node, it will be represented on the Nodes screen as an Operating System node (OS Host). This node will then behave just like a non-virtualized OS host, and can be protected using Path based classifications rather than VMware classifications. The data is still stored in the Repository, but is less VMware specific reducing the available restore options. As such I won't explicitly cover how to setup the Policy and Dataflow when using the HDID agent in this blog series.

 

Using VMware vStorage API for Data Protection (VADP)

 

In order to provide software protection of Virtual Machines without an the installation of an agent, HDID uses the VDDK (Virtual Disk Development Kit) provided by VMware to read data from VMDK files and store it in the HDID Repository. This method is capable of supporting Datastores on array based storage as well as VSAN and connects to vSphere over the network using the vCenter as the endpoint.

 

Creating a Policy for software based protection requires use of the "Backup" operation

 

vadp_operation.PNG

 

Which has the familiar set of attributes:

 

vadp_operation_options.PNG

 

Now that the nodes have been created and the Policy has been defined the Dataflow can be constructed

 

vadp_dataflow.PNG

 

Note

The HDID rules compiler will generate a warning explaining that data cannot be automatically validated for VMware nodes. This can be safely ignored.

 

Once distributed and triggered backups appear in the Repository section of the Restore screen

 

restore_screen.PNG

 

Like so

 

repository_restore_screen.PNG

 

Clicking on one of these records allows a view of what has been protected

 

repository_restore_screen_snap.PNG

 

From here various restore operations can be performed that I will cover in a future blog post.

 

 

Hardware based protection

 

Hardware based protection only applies to VMware Datastores being backed by Hitachi Block Storage. Each VMware Datastore is resolved to a Logical Unit (LUN) which in turn can either become part of a Snapshot or Replication within the storage.

 

Whilst this can be achieved manually, by using only Block Storage nodes and manually specifying LUNs. It is significantly simpler and more flexible to use the VMware ESX Application node to leverage the selection methods described in my previous post.

 

We've already covered setting the classification for the Policy, so it's just the operation that we need to select

 

vadp_operation.PNG

 

From this point we can either choose Snapshot (a local differential copy of the data) or Replication (a local or remote copy of the data). Replications can be continuous; always mirroring changes, or batch; only mirroring changes when triggered.

 

Either option leads to the familiar looking operation attributes screen

 

snapshot_operation_options.PNG

 

replication_operation_options.PNG

 

Implementing the Snapshot Policy on a Dataflow is very straightforward, it is done directly on the VMware ESX Agentless node

 

snapshot_dataflow.PNG

 

Once the Snapshot type and Pool have been defined, this Dataflow can be compiled and distributed.

 

For a replication the destination node is required. In this example we have a local and remote Replication being used to create a Dataflow that is compatible with HDID's SRM support.

 

replication_dataflow.PNG

 

Once triggered, we can once again use the restore screen to see our backups this time selecting Hitachi Block

 

restore_screen.PNG

 

The screen shows Snapshots and Replications

 

block_restore_screen.PNG

 

Clicking on one of these records shows the contents in a slightly different layout to the Repository backup

 

block_restore_screen_snap.PNG

 

Once again, from here various restore operations can be performed that will be discussed in the future blog post.

 

Using Software and Hardware protection together

 

Whilst I have shown the Software and Hardware protection options separately in this blog post, this is only done for clarity. These solutions are fully compatible and are often used together in order to make a more cost effective backup solution that meets to the protection objective.

 

What's the right protection for me?

 

This answer to this question is always "it depends" but in order to help make an informed choice I've listed the available protection methods against a list of requirements that should be considered

 

Hardware ProtectionSoftware Protection (VADP)Software Protection (Agent)
Recovery PointsFrequentMid/LowLow
Impact to SystemsLowHigh / Mid (if using SAN)Very High
RetentionDays-WeeksDays-YearsDays-Years
Media CostHigh (primary storage)Low (offline / nearline storage)Low (offline / nearline storage)
Offsite Recovery capableYesYesYes
Setup TimeLowLowRequires agent on every VM

 

What can't be protected?

 

The table below explains the current level of protection provided by each method

 

Hardware ProtectionSoftware Protection (VADP)Software Protection (Agent)
Hitachi Hardware SupportYesYesYes
Non-Hitachi Hardware SupportNoYesYes
vRDMNoYesYes
pRDMNoNoYes

 

Thanks for reading, next time we'll cover the various restore options for our backups.

Previously ( VMware protection in Hitachi Data Instance Director - Part 1: Selection ) I wrote about the options provided for selecting Virtual Machines via the HDID user interface.  This is a great introduction to show off the capabilities of what HDID makes possible, but what if you're a DevOps type and clicking buttons isn't really "your thing"?  Well, thankfully there's the HDID REST API, and even more helpfully the HDID SDK (see: HDID REST API  )

 

In this blog post I'll go through the required API calls to create the ESX node and use it to build a Policy that selects Virtual Machines by id, name and tag.  I'll also try and follow up each blog post with one explaining how to perform the same operations using the REST API.

 

Lets get started, as a minimum you'll need a HDID install that's up and running, and I'm going to use Postman to make my REST requests.

 

Note: the "Content-Type" header must be set to "application/json" for all the requests shown, otherwise you may get an error about unknown content type

 

Login

In order to use HDID we need to login, this is done by a POST to the login action at:

 

 

https://192.168.45.147/HDID/master/UIController/services/Users/actions/login/invoke

 

 

The HDID API talks exclusively JSON, and the body for this action is the following JSON document:

 

{"username":"my_username","space":"master","password":"my_password"}

 

Note: The HDID API runs only on the HDID master node, and some requests can only be dealt with by the master, such as logging in.

 

The response body to this request, if successful, should be empty:

 

null

 

The important part is in the headers, where the Session ID is kept:

 

Set-Cookie →Session_ID={b0d80d06-efe9-4839-9a9e-df41c76f3c3e}; path=/

 

If you're using Postman is takes care of holding onto this value for all other requests, but if you're not you need to make sure it's set in the headers of all subsequent requests to HDID.

 

Creating a VMware ESX Agentless Node

 

First of all we need a node that can communicate with our vCenter, in HDID we call this a VMware ESX Agentless Node.

 

All Agentless Nodes are created by sending a POST request to the Node Manager on the HDID master, like so:

 

 

https://192.168.45.147/HDID/master/NodeManager/objects/VMwareESXAgentlessNodes/

 

 

With the following JSON body:

 

 

{
     "nodeType": "VMwareESX",
     "name": "MyVMwareNode",
     "resourceGroup": null,
     "configuration": {
          "proxyName": "Fibre3@BX-UXUVZY-RT478H-CV2IZI-SUKNE8[0-1-1]",
          "enableVADP": true,
          "server": "my_vmware_server",
          "authentication": {
               "username": "my_vmware_username",
               "password": "my_vmware_password"
           }
     }
}

 

 

The tricky bit in the above body is the proxyName.  In HDID this is the node that will actually do the communication with (in this instance) VMware.  In this simple blog post we're going to be using a single HDID master node, but the proxy could just as easily be another HDID client node.

 

In order to figure out our proxy node we need to perform a GET request against the Node Manager for a list nodes, with a query to select only OS nodes, like so:

 

 

https://192.168.45.147/HDID/master/NodeManager/objects/Nodes/?query=(type+IN+(%22OSHost%22))

 

 

Note: The above shows a very simple example of filtering the response by adding a query, not all fields can be used in queries so be sure to check your responses carefully.

 

The response body to our GET request should look like so:

 

{
  "node": [
  {
  "id": "Fibre3@BX-UXUVZY-RT478H-CV2IZI-SUKNE8[0-1-1]",
  "name": "Fibre3",
  "resourceId": "4294967297",
  "agentless": false,
  "version": "6.7",
  "osDetails": {
  "os": "eOS_WINDOWS",
  "osVersion": "Enterprise",
  "osVariant": "Windows 2008 R2",
  "architecture": "x64",
  "chassis": "Virtual",
  "osLevel": "Server"
  },
  "networkDetails": {
  "iPAddresses": [
  "192.168.45.147"
  ]
  },
  "stateInfo": {
  "authorized": true,
  "hasRules": true,
  "upToDate": true,
  "hubState": "Connected",
  "activeRules": 1543933922,
  "connected": true,
  "accessible": true
  },
  "master": true,
  "type": "OSHost",
  "filterAvailable": true,
  "nodeAttributes": []
  }
  ],
  "pageInfo": {
  "totalCount": 1,
  "end": true
  }
}

 

This JSON response is a bit more complex, but most importantly contains an array of Node objects.  For more complex setups the nodes can be filtered by name and OS variants in order to make queries easier.

 

For our purposes selecting the Proxy is simple, as there is only a single OS node, which also contains the attribute

 

"master":true

 

Now we have the Node ID of the Node we wish to use as the Proxy, we can create our VMware ESX Agentless node.  The successful response body should look like this:

 

{
    "name": "MyVmwareSetup",
    "configuration": {
        "server": "my_vmware_server",
        "proxyName": "Fibre3@BX-UXUVZY-RT478H-CV2IZI-SUKNE8[0-1-1]",
        "authentication": {
            "username": "my_vmware_username"
        },
        "enableVADP": true
    },
    "timestamp": "2018-12-04T14:39:26",
    "id": "MyVmwareSetup@00-C666DE-6FF0F0-425D84-DAB6F1[0-1-6]"
}

 

Note: the password is not returned, and even if queried will only return an encrypted string

 

Browsing VMware

 

Now that we have an VMware ESX Agentless node, we can browse our VMware setup just as we can from the HDID user interface.  Except that we can take a shortcut.  In order to start browsing VMware items in the HDID user interface we first need to create a Policy, but using the REST API we can just browse immediately and make the Policy later.

 

Lets get a list of all Virtual Machines that are available in our vCenter, using a GET request to our new VMware ESX Agentless node, like so:

 

https://192.168.45.147/HDID/MyVmwareSetup%4000-C666DE-6FF0F0-425D84-DAB6F1%5B0-1-6%5D/VMwareBrowser/objects/VirtualMachines/?order-by=name+ASC

I've added an "order-by" in here just so that the response makes sense to us humans.

 

The response body we get should look like this:

 

"virtualMachine": [
        {
            "id": "my_vmware_server/VirtualMachine:vm-1048",
            "name": "2008R2-Test2",
            "v2i": false,
            "vadp": true,
            "isBrowsable": false,
            "nodeID": "MyVmwareSetup@00-C666DE-6FF0F0-425D84-DAB6F1[0-1-6]",
            "type": "eVirtualMachine"
        },
        {
            "id": "my_vmware_server/VirtualMachine:vm-1044",
            "name": "2008R2-Test3",
            "v2i": false,
            "vadp": true,
            "isBrowsable": false,
            "nodeID": "MyVmwareSetup@00-C666DE-6FF0F0-425D84-DAB6F1[0-1-6]",
            "type": "eVirtualMachine"
        },
        {
            "id": "my_vmware_server/VirtualMachine:vm-1045",
            "name": "2008R2-Test4",
            "v2i": false,
            "vadp": true,
            "isBrowsable": false,
            "nodeID": "MyVmwareSetup@00-C666DE-6FF0F0-425D84-DAB6F1[0-1-6]",
            "type": "eVirtualMachine"
        },
        {
            "id": "my_vmware_server/VirtualMachine:vm-15973",
            "name": "2008R2Test1",
            "v2i": false,
            "vadp": true,
            "isBrowsable": false,
            "nodeID": "MyVmwareSetup@00-C666DE-6FF0F0-425D84-DAB6F1[0-1-6]",
            "type": "eVirtualMachine"
        },
        {
            "id": "my_vmware_server/VirtualMachine:vm-15977",
            "name": "2008R2Thick",
            "v2i": false,
            "vadp": true,
            "isBrowsable": false,
            "nodeID": "MyVmwareSetup@00-C666DE-6FF0F0-425D84-DAB6F1[0-1-6]",
            "type": "eVirtualMachine"
        }, .....

 

And voila, a list of our Virtual Machines with their ID's (a concatenation of their vCenter server, type and MOREF)

 

Perhaps though we'd rather find some tags to use in our selection, we can make a slightly different GET request to our VMware ESX Agentless node:

 

https://192.168.45.147/HDID/MyVmwareSetup%4000-C666DE-6FF0F0-425D84-DAB6F1%5B0-1-6%5D/VMwareBrowser/objects/Tags/?order-by=name+ASC

 

Giving a response body like so:

{
    "managedObject": [
        {
            "id": "my_vmware_server/Tag:Andrew's",
            "name": "Andrew's",
            "isBrowsable": true,
            "type": "eTag",
            "nodeID": "MyVmwareSetup@00-C666DE-6FF0F0-425D84-DAB6F1[0-1-6]"
        },
        {
            "id": "my_vmware_server/Tag:Block",
            "name": "Block",
            "isBrowsable": true,
            "type": "eTag",
            "nodeID": "MyVmwareSetup@00-C666DE-6FF0F0-425D84-DAB6F1[0-1-6]"
        },
        {
            "id": "my_vmware_server/Tag:BlockTest",
            "name": "BlockTest",
            "isBrowsable": true,
            "type": "eTag",
            "nodeID": "MyVmwareSetup@00-C666DE-6FF0F0-425D84-DAB6F1[0-1-6]"
        },
        {
            "id": "my_vmware_server/Tag:Development",
            "name": "Development",
            "isBrowsable": true,
            "type": "eTag",
            "nodeID": "MyVmwareSetup@00-C666DE-6FF0F0-425D84-DAB6F1[0-1-6]"
        },....

 

This time we get a list of ManagedObjects as VMware tags can be attached to any object.

 

If you look carefully at the above response you'll also see the attribute:

 

"isBrowsable": true

 

This is because the Managed Object interface allows us to traverse the VMware node hierarchy.  Now that we have the Tag ID, we can use it to find Managed Objects that have that tag.

 

Here's how we construct a GET request that allows us to see what Managed Objects are tagged with the "Development" tag:

 

https://192.168.45.147/HDID/MyVmwareSetup%4000-C666DE-6FF0F0-425D84-DAB6F1%5B0-1-6%5D/VMwareBrowser/objects/ManagedObjects/my_vmware_setup%2FTag%3ADevelopment/collections/ManagedObjects

 

Note: We need to escape the URL characters in the Tag ID

 

This gives the following response body:

 

{
  "managedObject": [
  {
      "id": "my_vmware_server/VirtualMachine:vm-363",
      "name": "[AMC DEV] WIN-73ETL3VUO31",
      "isBrowsable": false,
      "type": "eVirtualMachine",
      "nodeID": "MyVmwareSetup@00-C666DE-6FF0F0-425D84-DAB6F1[0-1-6]"
  }
  ],
  "pageInfo": {
  "totalCount": 1,
  "end": true
  }
}

 

Which shows us all the Managed Objects with the Tag "Development", in this case a single Virtual Machine.

 

This Managed Object request can be used for any Managed Object that is marked as "isBrowsable".  Asking for the Managed Object collections of any Managed Object allows us to see it's children, or sub-nodes.

 

The generic Managed Object GET requests look like this:

 

https://{hdid_master_ip}/HDID/{vmware_esx_agentless_node_id}/VMwareBrowser/objects/ManagedObjects 
https://{hdid_master_ip}/HDID/{vmware_esx_agentless_node_id}/VMwareBrowser/objects/ManagedObjects/{managed_object_id_URL_encoded}/collections/ManagedObjects

 

The first GET request will return the available Datacenters for the specified VMware setup, and the second will allow tree traversal.

 

Finding Objects by Name

 

So far I've covered browsing a VMware setup, which is selecting existing objects by ID, but what if you wanted to select by name?

 

In order to this we need to build a small JSON document and make use of a call that's only available to the REST API.

 

The VMwareBrowser has a preview method, that will accept a small section of a HDID Policy and return the resulting list of Virtual Machines.

 

The preview method responds to a PUT and is found here:

 

https://192.168.45.147/HDID/MyVmwareSetup%4000-C666DE-6FF0F0-425D84-DAB6F1%5B0-1-6%5D/VMwareBrowser/services/ManagedObjects/actions/preview/invoke

 

And the JSON required looks like so:

 

{
    "include": [{
        "name": "Win7*",
        "type": "eVirtualMachine",
        "_selected": false
    }],
    "exclude": [],
    "vmwareNodeId": "MyVmwareSetup@00-C666DE-6FF0F0-425D84-DAB6F1[0-1-6]"
}

 

Which returns the following response:

 

{
    "virtualMachine": [
        {
            "id": "my_vmware_server/VirtualMachine:vm-2165",
            "name": "Win7-HDID55",
            "v2i": false,
            "vadp": true,
            "isBrowsable": false,
            "nodeID": "MyVmwareSetup@00-C666DE-6FF0F0-425D84-DAB6F1[0-1-6]",
            "type": "eVirtualMachine"
        },
        {
            "id": "my_vmware_server/VirtualMachine:vm-2118",
            "name": "Win7x32",
            "v2i": false,
            "vadp": true,
            "isBrowsable": false,
            "nodeID": "MyVmwareSetup@00-C666DE-6FF0F0-425D84-DAB6F1[0-1-6]",
            "type": "eVirtualMachine"
        },
        {
            "id": "my_vmware_server/VirtualMachine:vm-2135",
            "name": "Win7-03",
            "v2i": false,
            "vadp": true,
            "isBrowsable": false,
            "nodeID": "MyVmwareSetup@00-C666DE-6FF0F0-425D84-DAB6F1[0-1-6]",
            "type": "eVirtualMachine"
        }
    ],
    "pageInfo": {
        "totalCount": 3,
        "end": true
    }
}

 

You can also include Managed Objects that have been selected too by adding them to the JSON document like so:

 

 {
    "include": [{
        "name": "Win7*",
        "type": "eVirtualMachine",
        "_selected": false
    },
    {
        "id" : "my_vmware_server/Folder:group-v375",
        "type": "eFolder",
        "_selected":true
    }],
    "exclude": [],
    "vmwareNodeId": "MyVmwareSetup@00-C666DE-6FF0F0-425D84-DAB6F1[0-1-6]"
}

 

Now that we've learned about VMware selection using the REST API we can put our selection into a HDID Policy, and create it.

 

Creating a Policy

 

The PolicyHandler on the master is responsible for creating Policies, and it can be found here:

 

https://192.168.45.147/HDID/master/PolicyHandler/objects/Policies/

 

The full Policy JSON document looks like so:

 

{
     "name": "MyTestPolicy",
     "description": "",
     "classificationChains": [{
         "items": [{
               "typeName": "vmwareesx",
               "displayName": "VMware",
               "normalizedTypeName": "vmwareesx",
               "propertyName": "vmware",
               "category": "Hypervisor",
               "_policyItemType": "classifications",
               "vmwareEsx": {
                    "include": [{
                         "name": "Win7*",
                         "type": "eVirtualMachine",
                         "_selected": false
                    },
                    {
                         "id" : "my_vmware_server/Folder:group-v375",
                         "type": "eFolder",
                         "_selected":true
                    }],
                    "exclude": [],
                    "vmwareNodeId": "MyVmwareSetup@00-C666DE-6FF0F0-425D84-DAB6F1[0-1-6]"
               },
               "_selected": false
          }]
     }],
     "operationChains": [{
          "items": [{
               "typeName": "Local Snapshot",
               "displayName": "Snapshot",
               "normalizedTypeName": "snapshot",
               "propertyName": "snapshot",
               "category": "Standard",
               "_policyItemType": "operations",
               "snapshot": {
                    "label": "Snapshot",
                    "mode": {
                         "snapshotMode": "eSNAPSHOT_HARDWARE",
                         "hardwareType": "eSNAPSHOT_HITACHI_BLOCK"
                    },
                    "run": {
                         "mode": "eRUN_ANYTIME",
                         "scheduleName": ""
                    },
                    "recoveryPointObjective": {
                         "period": 8,
                         "units": "eRPO_HOURS"
                    },
                    "retention": {
                         "period": 1,
                         "units": "eRETENTION_WEEKS"
                    },
                    "recoveryTimeObjective": {
                         "incrementalTapeBackup": false
                    },
                    "sourceOption": {
                         "quiesceApplications": true,
                         "preScriptEnabled": false,
                         "postScriptEnabled": false
                    },
                    "type": {
                         "mode": "eTRANSFER_ASYNCHRONOUS_JOURNALED"
                     }
               },
               "_selected": false
          }]
     }]
}

 

You can see our VMware JSON in the above, as well as an operation to perform a Hardware Snapshot, with its associated settings.

 

If you're looking at changing these it is easiest to use HDID User Interface with your browsers developer tools enabled in order to find out the correct keys and values for your requirements.

 

If the request is successful then we get the Policy returned with it's ID:

 

{
    "id": "cc6d61935ca045c8a16be74e9357573a",
    "classificationChains": [
        {
            "items": [
                {
                    "typeName": "vmwareesx",
                    "vmwareEsx": {
                        "include": [
                            {
                                "name": "Win7*",
                                "type": "eVirtualMachine"
                            },
                            {
                                "id": "my_vmware_server/Folder:group-v375",
                                "name": "Andrew",
                                "type": "eFolder"
                            }
                        ],
                        "exclude": [],
                        "vmwareNodeId": "MyVmwareSetup@00-C666DE-6FF0F0-425D84-DAB6F1[0-1-6]"
                    },
                    "id": 1,
                    "propertyName": "vmwareEsx"
                }
            ]
        }
    ],
    "operationChains": [
        {
            "items": [
                {
                    "typeName": "Local Snapshot",
                    "snapshot": {
                        "label": "Snapshot",
                        "retention": {
                            "units": "eRETENTION_WEEKS",
                            "period": 1
                        },
                        "recoveryPointObjective": {
                            "units": "eRPO_HOURS",
                            "period": 8
                        },
                        "run": {
                            "mode": "eRUN_ANYTIME"
                        },
                        "sourceOption": {
                            "quiesceApplications": true,
                            "preScriptEnabled": false,
                            "postScriptEnabled": false
                        },
                        "mode": {
                            "snapshotMode": "eSNAPSHOT_HARDWARE",
                            "hardwareType": "eSNAPSHOT_HITACHI_BLOCK"
                        }
                    },
                    "id": 1,
                    "propertyName": "snapshot"
                }
            ]
        }
    ],
    "description": "",
    "nextClassificationId": 2,
    "nextOperationId": 2,
    "version": 2,
    "permissions": [
        {
            "propertyType": "pt_user",
            "pt_user": {
                "upn": "Administrator@master"
            },
            "access": "eA_READWRITE"
        }
    ],
    "name": "MyTestPolicy"
}

 

 

And there we have it, you can now build, test and create a HDID Vmware Policy using just the REST API. Thanks for reading and feel free to put questions into the comments section.

Hitachi Data Instance Director (HDID) made some significant improvements to it's VMware support back in May 2018 with its 6.5 release, over a few of these blog posts I'll try to go though each piece in some detail to explain what it does, how it behaves and how it can be used.

 

In this blog I'll talk about the different selection criteria available and what actually happens under the hood. I'm going to assume that there is already at least one VMware application node has been created in HDID.

 

Create the Policy

 

First off we need to create a Policy using the Hypervisor classification

 

vmware_selection_hypervisor.PNG

 

Once we've given our Policy a name, we can progress to choosing a VMware application node to select from and we're presented with this screen

 

vmware_selection_includes_excludes.PNG

Here we can add and remove resources to include and exclude from our backup Policy.

 

Clicking add leads us a choice about our selection methods

 

vmware_selection_method.PNG

 

The two options here work in very different ways when being processed by HDID.

 

Note

 

HDID will attempt to resolve all selections down to Virtual Machines contained within those selections. This means that selecting an empty Datastore, for instance, will result in a HDID error of "There are no VMs to backup".

 

Browse for Resources

 

This selection allows browsing of the VMware node chosen and allows selection of any existing objects within VMware

 

vmware_selection_browse.PNG

 

This method records the Managed Object Reference (MOR) in the Policy and will use it to find the selected objects in the given VMware node.

 

The MOR uniquely identifies a VMware object, regardless of changes to that object (rename, moving folder etc). If the object is deleted and recreated, however, it will be assigned a different MOR by VMware and may no longer be included in the Policy.

 

To this end, selecting VMware container objects such as Folders, Datastores, Hosts or even whole Datacenters is safer than selecting individual Virtual Machines, as HDID will resolve the container object and protect everything beneath it. If the container no longer exists then the resolution will fail.

 

Tags are the exception here as they are resolved by name. As long as the tag exists within VMware it can be browsed, it does not have to be assigned to any VMware resources.

 

Using tags in VMware and in HDID Policies allows for a very simple backup configurations to support various service level agreements for data protection.

 

The main issue with tags is that they do not belong to the resource, they only exist within the vCenter. This can cause issues when using tools such as SRM as the tags will not be replicated to the secondary vCenter.

 

Specify Resource by Name or Wildcard

 

This selection allows the VMware resources to be selected by name either explicitly or by a wildcard

 

vmware_selection_wildcard.PNG

 

This method records the resource type (Datacenter, Host, Folder and so on) and the given pattern and will search VMware for matching entities every time the backup is triggered. It does not incorporate hierarchies into the matching, and so allows for named resources conforming to a pattern to be distributed via different Hosts, Folders and Datastores within the VMware environment to be protected.

 

Whilst selecting by wildcard is very flexible, it is very easy to rename a resource such that it no longer conforms to the pattern, and therefore will not be included in the backup.

 

Includes and Excludes

 

Returning to the classification screen once again and using the available selections we can see that building an accurate backup Policy to protect your VMware environment can be achieved in numerous ways, regardless of how the VMware resources are organised.

 

vmware_selection_includes_excludes_with_content.PNG

 

Next time we'll talk about the differences between VADP protection and Hardware protection as well as covering what Virtual Machines can be protected and which ones cannot.