Performance Metrics for AMS 2100, 2300 and 2500

Discussion created by Legacy HDS Forums on Jan 18, 2012
Latest reply on Jan 25, 2012 by Legacy HDS Forums

Like • Show 0 Likes0
Comment • 13

Originally posted by: Newbie

Hello,

I might sound a novice while asking this question. Could someone please explain me how can I calculate the max number of IOPS which could be supported by single port for an AMS 2100, 2300 and 2500. I have read on this forum that for a USP it practically gives arouond 2500 IOPS per port.
I am looking for similar numbers for a AMS2000 series arrays.

Regards,
Yo Ko

Outcomes

13 Replies

Legacy HDS Forums Jun 7, 2013 8:34 PM
Originally posted by: cris

Looking at frontend ports alone is a poor indicator for performance. Different blocks sizes greatly effect the number of IOPS that can be sustained by a frontend port. If you think frontend ports are a bottleneck I suggest you at the average tag count. If this is anywhere near 512 constantly then it might be a problem.
Like • Show 0 Likes0
Actions
Legacy HDS Forums Jun 7, 2013 8:34 PM
Originally posted by: stephen2615

As far as I am concerned, the worst application ever invented would be Microsoft Exchange. You could have a billion IOPs available and Exchange would find some way of screwing up the performance of the FEP. I once had a Solaris system with Oracle using VxVM where if I got 1500 IOPs per USP port I was over the moon as the system was going ballistic. During the nightly backup phase, I often saw 4000 IOPs on the ports and it was not a bother to me. During the day would have been very different. Same goes for data warehouse application loading where the IOPs were maxed out. You have to take the application and time of day into account before you get concerned about what's available.
Like • Show 0 Likes0
Actions
Legacy HDS Forums Jun 7, 2013 8:34 PM
Originally posted by: Newbie

Thanks Cris and Stephen.
I am looking for this information to decide how many servers can I map to a single port, something like a fan out ratio. The information you provided me is definitely going to help.
Just to add, I was working on an AMS 2300 right now and I did see that the IOPS did go as high as 2400 on a single port, with an average of 1600 IOPS.

Regards,
Yo Ko
Like • Show 0 Likes0
Actions
Legacy HDS Forums Jun 7, 2013 8:34 PM
Originally posted by: cris

In mixed workloads I've seen anywhere between 1000 to 2000 IOPS being the breaking point. But, It really depends on the block size. Exchange/SQL uses 8K blocks, data warehouse can use massive blocks (I've seen up to 512K).
You are better off segregating your ports based on workload profiles, if your environment has these sorts of characteristics. Because the workload profiles are so different, large blocks tend to "fill the pipe" and put less stress on the controller, smaller blocks tend to leave the pipe empty and increase processor utilisation, splitting them will give you better understanding the effect they are having. This is important because the AMS does not use dedicated ASICs on handle FEP operations, which means the controller does everything.

And again my advise would be that you use tag count if you looking at obtaining a fan out ratio. If you have large block workloads then you also need to look at overall throughput (MB/s).

As far as I am concerned, the worst application ever invented would be Microsoft Exchange. You could have a billion IOPs available and Exchange would find some way of screwing up the performance of the FEP. I once had a Solaris system with Oracle using VxVM where if I got 1500 IOPs per USP port I was over the moon as the system was going ballistic. During the nightly backup phase, I often saw 4000 IOPs on the ports and it was not a bother to me. During the day would have been very different. Same goes for data warehouse application loading where the IOPs were maxed out. You have to take the application and time of day into account before you get concerned about what's available.

Thanks Cris and Stephen.
I am looking for this information to decide how many servers can I map to a single port, something like a fan out ratio. The information you provided me is definitely going to help.
Just to add, I was working on an AMS 2300 right now and I did see that the IOPS did go as high as 2400 on a single port, with an average of 1600 IOPS.

Regards,
Yo Ko
Like • Show 0 Likes0
Actions
Legacy HDS Forums Jun 7, 2013 8:34 PM
Originally posted by: Newbie

Thanks Cris.
Could you please help me understand if there is a way I can find out the tag count for an AMS2100/2300 from Tuning Manager?
Like • Show 0 Likes0
Actions
Legacy HDS Forums Jun 7, 2013 8:34 PM
Originally posted by: cris

I don't use tuning manager for AMS performance tuning. I output the performance files and use some homegrown software I developed. The metric in peformance monitor is tag count. The metric is also known as Queue depth, so you can try to search for a standard report that displays this information. Tuning Manager is VERY limited on a AMS in comparison to the USP/VSP because the AMS doesn't have seperate CHP and DKA processors.

Thanks Cris.
Could you please help me understand if there is a way I can find out the tag count for an AMS2100/2300 from Tuning Manager?
Like • Show 0 Likes0
Actions
Legacy HDS Forums Jun 7, 2013 8:34 PM
Originally posted by: Newbie

Thank you Cris.
So if I understand correctly tag count is queue depth, which in case of an AMS2x00 is 512.
Like • Show 0 Likes0
Actions
Legacy HDS Forums Jun 7, 2013 8:34 PM
Originally posted by: stephen2615

Thank you Cris.
So if I understand correctly tag count is queue depth, which in case of an AMS2x00 is 512.

I find this queue depth discussion generally rather confusing but not on the storage side. I have seen Windows 2008 R2 servers with queue depths miles deep when they are attached to a brand new USP V that is not doing anything at all. I read somewhere that the maximum queue depth that a Windows server should have at any one time is eight (8). My all time favourite windows application (Exchange) often sits on a queue depth of 20 plus and the Exchange guys used to get on my case about the "performance" of Exchange. I used to tell them (not what you think) to balance the data in Exchange as they would invariably have all the data on one lun and the other luns would be near empty. Windows perfmon could have response times of 20 ms (at the very top of the acceptable response time for Exchange) and above but the storage would have sub 2 ms response time.

The amount of time spent trying to get a good storage solution for Exchange was ridiculous. No matter what I did, Exchange would complain. I am talking about RAID 4D+4D for the data and logs because Microsoft said RAID 5 was bad for Exchange. As if Microsoft even knows what storage is. Multiple ports between the server and the storage for data and logs and it would still complain albeit slightly less. At the same time we had a MS SQL database 10 times bigger than Exchange that would be moving 20 times the data and not have a problem at all. It would run hot during an ETL load but that was to be expected.

So the moral of this story is what? Having 512 or 1024 as a number for the amount of available queue depths on the storage array means little for something like Exchange. If you have a junk application or a very poorly designed bit of software, you can throw those numbers in the trash can.
Like • Show 0 Likes0
Actions
Legacy HDS Forums Jun 7, 2013 8:34 PM
Originally posted by: cris

[font="Calibri"][size="3"]AH queue depth... [/size][size="3"]mytwo cents.[/size][/font]

[font="Calibri"][size="3"]Stephen, what you are seeing in something like prefmon is the host's perspective of queue depth. [/size][size="3"]There are actually a number of queues throughout the I/O path.[/size][/font]

[font="Calibri"][size="3"]1. Application ((maybe), depending on how it interacts withthe I/O subsystem of the operating system)
[/size][/font][font="Calibri"][size="3"]2. Queue depth in the software (SCSI driver/OS/LUN host side) - this is from a hosts perspective
[/size][/font][font="Calibri"][size="3"]3. Queue depth in the HBA
[/size][/font][font="Calibri"][size="3"]4. Queue depth in Target port (512 max on an AMS)
[/size][size="3"]5. Queue depth on the disks (2 for SATA and 8 for SAS/FC (from memory))[/size][/font]

[size="3"][font="Calibri"]NOTE: No all point refer to it as queue depth, some refer tothem as pending or outstanding I/Os, tags, etc. [/font][/size]

[size="3"][font="Calibri"]There is also a LUN queue depth on the array side, but this is calculated by: (fan-out Initiators/maximum queue depth for the port/numberof LUNs attached to the port).[/font][/size]

[font="Calibri"][size="3"]How and when each queue transmits I/O to the next device inthe I/O path depends on the underlying software/firmware. [/size][size="3"]It is possible for one queue to be filling while other around in remain empty (which is why Stephens sees Exchange with a queue depth of 20, when the array is doing nothing) (BTW do you know M$ best practice is a queue depth of 1!!!!! haha).[/size] [/font]

[font="Calibri"][size="3"]So, queues 2-4 above are generally the ones we deal with inthe real world, the disk based queues (queue 5) are somewhat abstracted and thetrue impact on how heavily they are utilised is determined by the profile ofthe data and how well its being managed in cache. [/size][size="3"]Things like write coalescing (gathered writes in HDS terms),[/size] [size="3"]opportunities, pool/raidgroup configurations, etc all effect how busy the disks queues are.[/size][size="3"] Queue 4 is extremely important because if the queue reaches its limit a QFULL/BUSY (SCSI) is returned and its cascades down to queue 3 (and possibly 2) to reduce the inflow rate.[/size] [size="3"]This is why it is important that target ports (FEDs), aren't too heavily utilised.[/size] [size="3"]As for the other queues, queue 1 for example, is controlled by local caching in the application and its design.[/size] [size="3"]A software package may choose to cache I/O requests in memory while build a set before it writes it out (such as in transaction processing systems build on ACID) we generally have no control here.[/size][size="3"] Queues 2 and 3 some storage vendors have some best practices which try to prevent queue 4 from reaching a QFULL/BUSY condition, but its always a fluffy calculation; and this is the problem, [/size] [size="3"]its not a science, but rather a guestimation.[/size] [size="3"]I don't know how[/size] [size="3"]many times I've had calls out to disk vendors (HDS, EMC, Netapp, etc) and they've[/size] [size="3"]complained about incorrect HBA or SCSI driver settings and yet there have been no performance problems or the performance problem was corrected by changing something else.[/size] [size="3"]The main cause of this is because queues 2 to 4 are generally so heavily oversubscribed its just not funny. Combine this with a mixed workload and the resulting patterns are almost chaotic in nature, making them difficult to predict, ergo such unpredictability also means[/size][size="3"] it is difficult to design for (enter fluffy best practices). [/size] [size="3"]But the best practices themselvescan have a [/size] [size="3"]detrimental [/size] [size="3"]effect,[/size] [size="3"]adjusting queues host side (queues 2-3) to match max target size queues (queues 4) can hinder an applications ability or operating systems ability to flush burst I/Os. [/size][/font]

[font="Calibri"][size="3"]That's not to say queues aren't important, I have seen an environment that did benefit greatly from queue depth optimisation, but it wasan extreme case (high fan-out ratio, big ESX hosts). [/size][size="3"]In most environment (small to medium), If you adhere to some good common sense in regards to fan-out ratios and try tobalance you workload evenly across all resources, its unlikely to be a majorcause of concern, other theoretical limits such as disk utilisation are morelikely to occur long before queues before to become a real issue.[/size] [/font]

[size="3"][font="Calibri"]I guess what I'm trying to say is don't get too caught up onqueues, they are EXTREMELY difficult to tune correctly and are really unlikelyto be the cause of a problem in most environments. Evenly distribute workloads, have realistic expectations on fan-out ratios and if you must, investigate startingfrom queue 4 downwards. If the target ports are running well below the maximum (512 on the AMS), there is generally no need to look further (well not with queues anyway).[/font][/size]

EDIT: Grrr at Copy and Paste removing spaces

Thank you Cris.
So if I understand correctly tag count is queue depth, which in case of an AMS2x00 is 512.

I find this queue depth discussion generally rather confusing but not on the storage side. I have seen Windows 2008 R2 servers with queue depths miles deep when they are attached to a brand new USP V that is not doing anything at all. I read somewhere that the maximum queue depth that a Windows server should have at any one time is eight (8). My all time favourite windows application (Exchange) often sits on a queue depth of 20 plus and the Exchange guys used to get on my case about the "performance" of Exchange. I used to tell them (not what you think) to balance the data in Exchange as they would invariably have all the data on one lun and the other luns would be near empty. Windows perfmon could have response times of 20 ms (at the very top of the acceptable response time for Exchange) and above but the storage would have sub 2 ms response time.

The amount of time spent trying to get a good storage solution for Exchange was ridiculous. No matter what I did, Exchange would complain. I am talking about RAID 4D+4D for the data and logs because Microsoft said RAID 5 was bad for Exchange. As if Microsoft even knows what storage is. Multiple ports between the server and the storage for data and logs and it would still complain albeit slightly less. At the same time we had a MS SQL database 10 times bigger than Exchange that would be moving 20 times the data and not have a problem at all. It would run hot during an ETL load but that was to be expected.

So the moral of this story is what? Having 512 or 1024 as a number for the amount of available queue depths on the storage array means little for something like Exchange. If you have a junk application or a very poorly designed bit of software, you can throw those numbers in the trash can.
Like • Show 0 Likes0
Actions
Legacy HDS Forums Jun 7, 2013 8:34 PM
Originally posted by: stephen2615

Well said.

If only Exchange was never invented..

You never got onto the topic of the better HBA when it comes to adjusting the settings. Emulex v Qlogic. My last place was Qlogic mainly with blades using Mezzanine cards. Not much can be done with them. It didn't help that the server team had no idea whatsoever about using the interconnect modules. I nearly got into fisticuffs trying to get them to agree to a SOE for what to use in the interconnect slots.
Like • Show 0 Likes0
Actions
Legacy HDS Forums Jun 7, 2013 8:34 PM
Originally posted by: cris

Believe it on not exchange is getting better!!! Most of its problems in storage land stem from its DBMS implmenetation. Since exchange 5.5, the backend has run off something called Blue Jet which is a modified version of Red Jet (which is the engine that Microsofts Access uses). This is still the case with Exchange 2010, but considerable work has gone into improving how it handles I/O (some numbers say up to 30% better then 2003). M$ did test Exchange 2010 with a SQL backend, but it never made it to the release. It is rumored in some circles that this was because of some of the BLOB limitations which M$ has been making inroads to recently and the fact that it wasn't proven in large real world environments. Chances are now that Sharepoint is heavily invested into SQL and storing all documents in BLOBs and it has proven that it can scale in some very large environments, the next version of Exchange will be backed by SQL.

Well said.

If only Exchange was never invented..

You never got onto the topic of the better HBA when it comes to adjusting the settings. Emulex v Qlogic. My last place was Qlogic mainly with blades using Mezzanine cards. Not much can be done with them. It didn't help that the server team had no idea whatsoever about using the interconnect modules. I nearly got into fisticuffs trying to get them to agree to a SOE for what to use in the interconnect slots.
Like • Show 0 Likes0
Actions
Legacy HDS Forums Jun 7, 2013 8:34 PM
Originally posted by: Newbie

Thank you Cris and Stephen for your inputs and this discussion I am getting to quite a bit of helpful information :)
Like • Show 0 Likes0
Actions
Legacy HDS Forums Jun 7, 2013 8:34 PM
Originally posted by: stephen2615

Thank you Cris and Stephen for your inputs and this discussion I am getting to quite a bit of helpful information :)

Happy to be of service. Interestingly enough, I have never used an AMS 1000 or AMS 2000 series for mixed traffic out the ports. I have used them for database systems where one or two ports was for data, one for logs and one for backup. More often than not, they were used to hold SATA disks for a virtualised backup system for USP and USP V.
Like • Show 0 Likes0
Actions

Jive

Hitachi Vantara Community

Performance Metrics for AMS 2100, 2300 and 2500

Outcomes

Related Content

Recommended Content