Just curious how many people use the XML input/output of HCS and what they use it for. Also if anybody has any useful perl/powershell scripts they use with the XML.
The SnapProtect feature of HDPS (aka CommVault Simpana) communicates with HCS using XML. It queries candidate volumes for replication and execute replication operations with this method.
** Moved to Scripting and added some tags/categories.
UCP-Pro's Director software is also a heavy user of the HCS-XML API for storage management related capabilities.
I've used it before a few times, mainly to automate repetitive tasks. The one that comes to mind was updating hundreds of labels for a number of modular arrays in a larg-ish VMware environment. I just adapted some other web services API code I had to work with HiCommand.
There are two approaches to using it:
1. Pure webservices based API calls (I have some sample code if you'd like, but its in c#)
2. Using the HiCommand CLI to get XML output from HCS. Essential the HICommmandCLI just makes webservices calls and returns the output in different formats including XML. While I don't have any sample code for Powershell, it would be quite simple to setup, all you'd need to do is call the HiCommandCLI when required with the correct parameters, return the output in XML and then parse it before apply any programming logic. If you've never used the HiCommand CLI before there is a KB article on how it works written by Mike Le Voi.
Let me know if you want the sample code or help.
Is there a guide for the REST API for Command Suite? I can only find a CLI guide.
Best to ask question in new thread.. however:
Tuning Manager and Command Director Manuals ...
Tuning Manager API Reference Guide (MK-92HC218-01)
Thanks for the response! I found the guides I needed there.
Thanks for the feedback. From the sounds of it, XML is used more in APIs. I've been experimenting with HCS CLI and was just trying to figure out how it would be useful. I think I'll stick to using batch files instead of writing something to generate XML.
I've been tinkering with xml output using powershell as well. It seems the XML output would be good to import into a database for customer billing.
If you write anything useful, please code share
You may have noticed the Developer Network here in the Community. Although initially focused on ISV development partners for HCP, we will be opening it up to anyone interested in developing to HDS' APIs. Keep an eye out for updates.
That's good to know. I'll have to check it out.
Just so folks know, customer/partner access to the Device Manager (HDvM) XML API and our SMI-S interfaces would require a NDA and HDvM PM approval. We have a number of partners and a few customers who are using these APIs today. One of my hats is to do interviews with potential partners to evaluate their API needs so I get to talk to them about their technical API requirements and find out what they need. Overall, though, we prefer folks use the SMI-S interface since it is a public standard vs. our proprietary APIs.
I work for a VAR, does this cover me? I haven't written anything that we've sold commercially or use in a commercial context. Mainly just things I've needed.
As for the XML API vs SMI-S... for adhoc programming/scripting I would never think to consider SMI-S, the knowledge required and the initial base code is considerably more complex (and I've programmed with both).
I would really like to see (and according to Nicholas Howe we will) platform specific APIs. It would make life so much easier....
Basically, if someone is trying to integrate our products into theirs, then we'd like to know about it and see if it makes sense to help them out. Someone using it to get their job shouldn't be an issue.
If you have a deep knowlege of SMI-S you can write something equivalent to the HDvM CLI/API, but, yes, it's more code and a large learning curve. The client-side tools are, in my personal opinion, the equivalent of stone knives and bearskins. Putting on my SNIA hat as an SMI-S author and chair of the SNIA SMI-S Technical Steering Group, how can we make SMI-S better?
SMI-S due to its complexity it is largely inaccessible to most programmers. A strong understanding of programming (its by no means simple) and a strong understanding of storage is really needed to make it a cost/resource effective API to use. This really restricts it to storage vendors and ISVs specializing on storage. Programmers that don't fall into this these organization are generally better off using a mix of native tools and going through bit more pain pasring poorly structured outputs rather than learning the required schemas and worklfows to make SMI-S viable.
I'm not sure it (SMI-S) really needs to change to be quite honest, I think it has its place in the world. We just need to make programming tools (libraries and what not) more accessible to work with storage. Microsoft realized this some time ago before introducing .NET and powershell and look how well introducing a simplified set of languages has worked for them.
Our group works on usability and design for GUIs, CLIs, and APIs, so if there are ways we can improve our products we'd like to know.
Make our life programmatically easier regardless of the language we chose to use.
Don't give us overly complex structures and workflows
Don't give us poorly structured outputs
Give us more options
Give us access to all functionality not just a subset
Making examples and programming content more accessible
GIVE US ACCESS TO DEVELOPERS! I'm sick of fighting with the GSC about a problem that is clearly beyond their capabilities and having to proxy ideas through them
You note "Just so folks know, customer/partner access to the Device Manager (HDvM) XML API and our SMI-S interfaces would require a NDA and HDvM PM approval. We have a number of partners and a few customers who are using these APIs today. One of my hats is to do interviews with potential partners..."
LogicMonitor (a SaaS monitoring software company) would love to start down the path of being an Hitachi partner....
We have customers with Hitachi storage that want us to add it in to our monitoring offering (we have integrations with storage from NetApp, EMC, IBM - but not Hitachi currently. Some of our integrations use SMI-S, some proprietary APIs (e.g. NetApp); some SNMP, etc.)
We have some SMI-S collection going from some HDS systems - but are seeing some very slow performance (e.g. several minute long responses), so I figure access to any partner information would help. (Or perhaps we should be using the tuning manager API...)
Anyway, if you could point us in the direction of how to apply for your partner program, that would be appreciated.
I've done quite a bit of HiCommand Scripting. I'll share a bit of my experience.
I personally think the XML has relatively limited use now. [I used it a lot a few years ago].
It's harder to use than flat files (-f csv). (Certainly when scripting as opposed to programmin (PERL/PYTHON)).
And of course there's a bit of an overhead in the programming to parse it (PERL+SAX and PYTHON are my favorite ways).
There are two main sorts of tasks you do with HiCommand scripting: reporting and changing.
That said a well written "change" application always includes extensive reporting before it commits to its work: so both techniques are needed in one tool.
In the past there was a brief fashion for using the XML interface for change applications: raw HiCommandCLI was relatively slow when making changes because only one action (like add lun or add ldev) was done at a time. If you work with HiCommand for a while you'll discover this pattern in all change applications:
If you can get multiple actions executed in step 2 performance makes a huge jump. About five years ago the XML interface was the only way to do multiple steps in one command. Now the CLI has many alternatives. As a consequence I don't use XML any more for the change part of applications. [Note: there is nothing to stop you mixing CLI and XML in the same application, I do it all the time. CLI just composes and runs XML for you anyway].
This now depends on the task in hand and the style of your programming.
The raw HiCommandCLI output is (with a bit of effort), parseable, but other formats are so much easier to parse so I only ever use it for a single interactive question where I'm too lazy to write a line of awk/grep (like what is size of a single LDEV). [I'd use raidcom for this now].
CSV format is easy to parse but is limited to a single "table".
XML requires a proper XML parser (and you can have fun when your parser doesn't quite handle the XML from HCS) but is the only way to get a proper nested report over several tables.
But if you are writing a real application you might end with the same conclusion as me: nearly all robust applications require internal access (dicts or maps) to most of the tables anyway: PORT, HOST STORAGE DOMAIN, WWN, LU etc. Rather than construct a really complex command to put all this in one XML file, parse it with great effort and place the individual parts into PERL or PYTHON objects, I just write one report per table and import them all, joining as you import. If you adopt this style then CSV or XML will work the same as one another and CSV normally easier (especially to debug).
There are a couple of other styles I've played with in the past.
If you work with XML you have probably asked "why isn't there a sort/awk/grep/cut" for "quick answer to a quick question" or "simple aggregation". Well I found one: xmlstarlet. If you frequently want an answer like: "sum the used capacity of all the DP-VOL on this port with this size and used for this customer", then it is a one line with HiCommand creating the raw XML piped into xmlstarlet to produce the report (admittedly a long line). Warning: xmlstarlet can be both really easy and really hard to use. I understand a skilled PowerShell user can do similar (never bothered to learn it).
Under the hood xmlstarlet is actually an XSL processor. But IE, firefox etc are XSL processors too. This leads to method two:
The XML output from HCS is RAW data, many reports just require you to select, sort and reformat it. That task is what XSL was designed to do. It is possible to write HTML code which obtains the XML, and presents it to the web page using an XSL page to format and select. If you're really clever you can do this dynamically, changing the format on user input.
BTW: I use raidcom most of the time now, not HCS, as that is the domain I'm working in. But my programming style is such I can always swap out raidcom and replace with HiCommand: a space separated file and a comma separated one aren't to hard to reconcile.
Steve Burr Thanks for contributing.
I'm not surprised, I've done a few things with the XML API in the past (not daily) and also found it to be clunky. I think I actually wrote a specific XML parser for every program I've written, which is pretty much in-line with your experience. I would personally like access to platform specific APIs (it would be MUCH quicker), but these aren't available outside of HDS and some ISVs. I would also like to see modern day scripting interfaces such as python bindings and powershell cmdlets (this is available, but it's REALLY limited IMO).
(I'm trying to respond to specific posts, but it looks like it just tacks them on at the end. Maybe it's time to RTM )
HDvM isn't perfect, but it does provide a normalization layer across all Hitachi arrays. You can use the same CLI/API to talk to every Hitachi array out there using pretty much the same parameters. It also serves to simplify the interactions with the arrays (which are much more complicated using the native APIs). Other CLIs like raidcom (VSP, HUS VM) and SNM2 (AMS/HUS) are array specific, meaning you'll need to redo your scripts if you want to cover other arrays.
Also, FYI, don't cut and paste. It sends the system out to lunch.
Can you expand on your comment on "platform specific APIs" and "modern day scripting interfaces"?
Thanks for your reply. fyi - you should be able to reply to a specific post by clicking on the reply within the post you want to respond to. Or, you can select Reply to original post.
It didn’t look like it when I did it, but it did in fact reply to the specific post. The lines indicating the reply “depth” are very faint, so it's hard to tell without squinting what level you're at.
Yep, well aware on how HiCommand (not just the CLI/API) is positioned around backward compatibility.
Platform specific = Native APIs. Yes this means we will lose out on code portability, since everything will be written specifically for the platform, but considering that a lot of code has/will be written to interact with CLIs (SNM2 and raidcom) we are already in this position. Providing access to the native CLIs will help speed up processing.
Scripting interfaces: Essential libraries and modules that can be used natively within a scripting language, instead of calling an external program and parsing its output. In PowerShell this could be a snap-in (I know there is one, but it terribly limited) or some python bindings so we can call functions natively.
I guess its really an OR, both Native APIs and scripting interfaces (if they are rich enough) solve the same problem. Its just programming and scripting.
Retrieving data ...