AnsweredAssumed Answered

Storage Tiering "in reality"

Question asked by Rahul Bhat on Aug 21, 2013
Latest reply on Aug 28, 2013 by Mark Perino

Like • Show 1 Like1
Comment • 6

Hi,

I would like to ask to share the experiences in the Community, if anyone has used the HDT feature in their environment . I am going to test the HDT but before I do I would like to know what people so far have experienced. The feedback points I am looking at :

Response time degradation during the page relocation ?
Monitoring the HDT and it's challenges in a shared environment in terms of costing it ?
Impact on the running application if any , while relocating the pages ?
With 42 MB chunk movement, what savings and Performance benefits were found ? I would say even a smaller chunk relocation would have benefited and accommodated more application with less flash usage.
Policy configuration, individual application or IO profile grouped policies and their cycle of relocation used ?
Any tuning capabilities used to change the HDT mid-way ?
SATA pool write-verify causing performance dip ( every write involves a read to confirm the write happened )

I can Google it and there are like 1000 pages of pdf telling me all about it, but I wanted to get the feedback from the real production environment or people who have actually used it and worked on it for a longer period of time , so the information is more practical then theoretical.

Even if I get some feedback only few points is fine, but would appreciate a response .

Cheers

Janet Hutchison

Aug 27, 2013 11:08 AM

Correct Answer

Hello, Rahul.

I have worked fairly extensively with HDT in our training environment and have written technical courses on HDT. Are your applications going to be that sensitive to I/O response times? And with HDS storage, the I/O response truly comes out of the storage system's cache! There is no or negligible impact to I/O performance with HDT. It is easy to configure and manage and is very flexible!! You can change pool configuration while the pool is in full operation - now here is where you need to be a bit careful - adding volumes to a pool and especially shrinking a pool, now those operations can have an impact on performance. So while the pool continues to operate, it is very wise to carefully schedule physical pool changes.

Pool Monitoring is easy and robust and supported both through GUI and CLI. You can certainly configure pro-active threshold warnings.

Let's see, Hitachi's choice on 42mb page size for migration is actually much smaller than some of the competition - where the relocation chunk size is 1GB or more. There is really no flexibility or choice on the 42mb page size. Hitachi seems to make "balanced" choices on internal architecture design and ,in general, these seem to work quite well for the majority of applications. (Of course, there are always the exceptions....)

There is a lot of granular configuration and migration policy management at the individual volume level. However, Hitachi's intent is to let the HDT wisdom manage the environment! For many skilled and long-time administrators who are used to a lot of fine-detail control, turning this over to the system's logic is sometimes the most difficult challenge!

You have control over the SATA write verify configuration but this is an installation attribute of the physical RAID group in the storage system. It is a good idea to be sure this is set correctly according to your needs when the HDDs are installed in the system as it is rather difficult (and disruptive to the RAID group) to change it.

I will look around to see if I can locate some more specific performance data that could be shared with you.

In the meantime, HDT, try it! I think you'll like it!

Janet Hutchison

HDS Academy

janet.hutchison@hds.com

See the reply in context

No one else had this question

Outcomes

Janet Hutchison

Aug 27, 2013 11:08 AM

Hello, Rahul.

Pool Monitoring is easy and robust and supported both through GUI and CLI. You can certainly configure pro-active threshold warnings.

I will look around to see if I can locate some more specific performance data that could be shared with you.

In the meantime, HDT, try it! I think you'll like it!

Janet Hutchison

HDS Academy

janet.hutchison@hds.com

Actions

Rahul Bhat @ Janet Hutchison on Aug 27, 2013 11:56 AM

Dear Janet Hutchison

Thanks very much for taking time for the write up.

Actually the application responsiveness is something which we need to check and though some might not be too sensitive others might show jitters , as they say "it depends".

Another interesting thing is , if we can use external storage ( HUS Virtualization behind VSP for Tier 2 (cache mode enabled of course) ) Tier 0 (SSD), Tier 1 ( SAS within VSP) ? What do you think might be the challenges here ?

One interesting point you made regarding "increasing pool " which in our environment might happen quite often due to the fact that we are talking about service model based on multi-tenant environment , do you know what should we expect in perf dip ?

With technical complexities we might still be able to manage , but what is most difficult is to charge the customer for a shared environment for per use basis. Do you have any such case with HDT implemented for multi tenant environment, would be really interesting information how the charging is done.

We can implement for a dedicated customer but thats hardly going to fetch any revenues , although performance might improve. This is the biggest headache for us. I am in talks with HDS Engineering on this to find a way out.

Well I agree page size is lower but there are vendors which move the plex at around 700k as well, so I still think 42mb is pretty big chunk for relocation but I am told , this will not change , so this discussion is irrelevant for now.

I am gonna start with some tests pretty soon and hope I can share some results. But meanwhile if you would be so kind to put some insight on my further queries, that would really be appreciated.

Again, thank you for taking time out for explaining your experiences , it was really helpful.

Regards

Actions

Janet Hutchison

@ Rahul Bhat on Aug 27, 2013 12:16 PM

SSD is good, but cache is better. Cache is truly "Tier 0." How much cache is on your VSP?

You can create separate HDT pools. You can create separate Cache Logical Partitions (CLPRs) in the VSP. You could charge different rates for the higher performance of a pool that had access to more cache.

The biggest challenge is that nobody really knows what page is going to be called for next so we have to depend on the algorithms' prediction of what to keep in cache and what to keep in the different pool tiers. When the user can tell you which data (LUNs, volumes) need higher service then you can "manually" place them accordingly. But the whole point of HDT is to rely on the carefully designed, very smart Hitachi algorithms. You will also be able to monitor, collect and analyze access profiles by LUN/volume in the HDT environment! Just have to collect the data over time. You can then use this information to either adjust the tier policy for the volume or to make a decision about fixing it in the correct (probably highest) tier. When you have data that needs to be on SSD to meet the performance, then don't put that in a pool, or put it in an SSD-only pool, and charge for that level of service! There is also a feature on the VSP called Cache Residency - the data is kept in cache!. Are you familiar with that? Be careful, though, it's not a good option for volatile (high update) data.

Regarding the charges per usage - you will be able to collect and analyze metrics about what portion or percent of each LUN/volume is in each tier. You may have to collect mulitple data points and average them over the charging period or select a "reference point in time" figuring that it will either be representative or will average out over time.

Are you or will you also be using Tuning Manager? How do you intend to measure the performance?

It has been a while - I would have to check/double-check the increase pool size issue. The most recent I know is that there is a configuration setting that controls whether or not HDP (yes, the basic Dynamic Provisioning logic) actually moves data to the newly added pool volumes at the time they are added. There was seen to be some wisdom in trying to keep the data evenly balanced across all the available pool volumes. So if the system has to do a bit of "mass" data movement when the pool is expanded, this could have an overhead impact. You might want to add this to your discussion with the HDS folks you are working with.

Again, I think you will find that HDT works very well for most or all your and your customers' needs!

Yes, I will keep an eye out for more case studies I can share.

(Right now my focus is on our Compute Blades and Unified Compute Platforms.)

Your project sounds exciting and fun.

Wishes for all success!

Actions

Janet Hutchison

@ Janet Hutchison on Aug 27, 2013 12:27 PM

Oh, Rahul, by the way..... When you have the opportunity to take some HDS training, look for course CSI2143 Hitachi Dynamic Tiering Concepts and Operations (3 days)

(It's a very good course - I wrote it....)

Actions

Rahul Bhat @ Janet Hutchison on Aug 27, 2013 2:37 PM

Thanks Janet Hutchison for your feedback, it was surely very useful and interesting. I would know more once I jump in . Regarding the course, will check it.

Thanks again for taking time , your feedback is much appreciated.

PS : I will keep the thread open for some more time.

Actions

Mark Perino

Aug 28, 2013 6:26 AM

As Janet Correctly mentioned the performance of the HDT pool, relocation, and other things depends on a lot of different factors.

1. Similar to the way HDP works. How much data is moved, when it is moved, and how it is evaluated are tunable using the settings.

2. It helps to think of it in this way. Tier 1 is "bonus" performance, Tier 2 is acceptable SLA meeting performance, and Tier 3 is savings where cold data can live. So The key is that Tier 2 needs to be Fat enough to hold any data that has an SLA. The usage of Tier 3 for cold data is where you can measure savings. In a shared environment you potentially have the benefit of scaling larger to achieve savings due to averages in larger pools, but reduce the isolation of silos. From a costing model one could measure the usage of each tier for each customer in a shared pool. In that way you would reward applications which have more cold data living on cheaper storage, and cost more accurately applications which have higher demand. If an application grows and demands isolation or special treatment you have options.. Either PIN it in a tier, or migrate it to an isolated set of storage (POOL/CLPR/Etc..) As long as you have unallocated capacity/performance you can easily move things into and out of any HDT pool.

3. Re-locations are at a low priority, but do consume resources (MPB, cache, disk busy). Depending on cycle time, sufficient Tier-2 space, etc. You can reduce or increase impact based on your needs.

4. It could, but there are many reasons for using the 42 MB size. Optimality and balance of performance is achieved with 42MB chunk size.

5. Best determined, through an analysis of the workload.

6. Yes absolutely. Cycle time, pinning in tiers, adding capacity in tiers, and many other settings.

7. SATA would be tier-3 and should largely be used for either cold data, or data that is written once and mostly read after that. There should be sufficient capacity in the tier above SATA so that all new writes are to a non-sata tier (then migrated down to SATA if the data is cold).

1 person found this helpful

Actions

6 Replies

Janet Hutchison Aug 27, 2013 11:08 AM
Unmark Correct
Correct Answer
Hello, Rahul.
I have worked fairly extensively with HDT in our training environment and have written technical courses on HDT. Are your applications going to be that sensitive to I/O response times? And with HDS storage, the I/O response truly comes out of the storage system's cache! There is no or negligible impact to I/O performance with HDT. It is easy to configure and manage and is very flexible!! You can change pool configuration while the pool is in full operation - now here is where you need to be a bit careful - adding volumes to a pool and especially shrinking a pool, now those operations can have an impact on performance. So while the pool continues to operate, it is very wise to carefully schedule physical pool changes.
Pool Monitoring is easy and robust and supported both through GUI and CLI. You can certainly configure pro-active threshold warnings.
Let's see, Hitachi's choice on 42mb page size for migration is actually much smaller than some of the competition - where the relocation chunk size is 1GB or more. There is really no flexibility or choice on the 42mb page size. Hitachi seems to make "balanced" choices on internal architecture design and ,in general, these seem to work quite well for the majority of applications. (Of course, there are always the exceptions....)
There is a lot of granular configuration and migration policy management at the individual volume level. However, Hitachi's intent is to let the HDT wisdom manage the environment! For many skilled and long-time administrators who are used to a lot of fine-detail control, turning this over to the system's logic is sometimes the most difficult challenge!

You have control over the SATA write verify configuration but this is an installation attribute of the physical RAID group in the storage system. It is a good idea to be sure this is set correctly according to your needs when the HDDs are installed in the system as it is rather difficult (and disruptive to the RAID group) to change it.

I will look around to see if I can locate some more specific performance data that could be shared with you.
In the meantime, HDT, try it! I think you'll like it!
Janet Hutchison
HDS Academy
janet.hutchison@hds.com
Like • Show 4 Likes4
Actions
- Rahul Bhat @ Janet Hutchison on Aug 27, 2013 11:56 AM
  Mark Correct
  Correct Answer
  Dear Janet Hutchison
  Thanks very much for taking time for the write up.
  Actually the application responsiveness is something which we need to check and though some might not be too sensitive others might show jitters , as they say "it depends".
  
  Another interesting thing is , if we can use external storage ( HUS Virtualization behind VSP for Tier 2 (cache mode enabled of course) ) Tier 0 (SSD), Tier 1 ( SAS within VSP) ? What do you think might be the challenges here ?
  One interesting point you made regarding "increasing pool " which in our environment might happen quite often due to the fact that we are talking about service model based on multi-tenant environment , do you know what should we expect in perf dip ?
  
  With technical complexities we might still be able to manage , but what is most difficult is to charge the customer for a shared environment for per use basis. Do you have any such case with HDT implemented for multi tenant environment, would be really interesting information how the charging is done.
  We can implement for a dedicated customer but thats hardly going to fetch any revenues , although performance might improve. This is the biggest headache for us. I am in talks with HDS Engineering on this to find a way out.
  
  Well I agree page size is lower but there are vendors which move the plex at around 700k as well, so I still think 42mb is pretty big chunk for relocation but I am told , this will not change , so this discussion is irrelevant for now.
  
  I am gonna start with some tests pretty soon and hope I can share some results. But meanwhile if you would be so kind to put some insight on my further queries, that would really be appreciated.
  
  Again, thank you for taking time out for explaining your experiences , it was really helpful.
  
  Regards
  RB
  Like • Show 0 Likes0
  Actions
  - Janet Hutchison @ Rahul Bhat on Aug 27, 2013 12:16 PM
    Mark Correct
    Correct Answer
    SSD is good, but cache is better. Cache is truly "Tier 0." How much cache is on your VSP?
    You can create separate HDT pools. You can create separate Cache Logical Partitions (CLPRs) in the VSP. You could charge different rates for the higher performance of a pool that had access to more cache.
    The biggest challenge is that nobody really knows what page is going to be called for next so we have to depend on the algorithms' prediction of what to keep in cache and what to keep in the different pool tiers. When the user can tell you which data (LUNs, volumes) need higher service then you can "manually" place them accordingly. But the whole point of HDT is to rely on the carefully designed, very smart Hitachi algorithms. You will also be able to monitor, collect and analyze access profiles by LUN/volume in the HDT environment! Just have to collect the data over time. You can then use this information to either adjust the tier policy for the volume or to make a decision about fixing it in the correct (probably highest) tier. When you have data that needs to be on SSD to meet the performance, then don't put that in a pool, or put it in an SSD-only pool, and charge for that level of service! There is also a feature on the VSP called Cache Residency - the data is kept in cache!. Are you familiar with that? Be careful, though, it's not a good option for volatile (high update) data.
    
    Regarding the charges per usage - you will be able to collect and analyze metrics about what portion or percent of each LUN/volume is in each tier. You may have to collect mulitple data points and average them over the charging period or select a "reference point in time" figuring that it will either be representative or will average out over time.
    Are you or will you also be using Tuning Manager? How do you intend to measure the performance?
    
    It has been a while - I would have to check/double-check the increase pool size issue. The most recent I know is that there is a configuration setting that controls whether or not HDP (yes, the basic Dynamic Provisioning logic) actually moves data to the newly added pool volumes at the time they are added. There was seen to be some wisdom in trying to keep the data evenly balanced across all the available pool volumes. So if the system has to do a bit of "mass" data movement when the pool is expanded, this could have an overhead impact. You might want to add this to your discussion with the HDS folks you are working with.
    
    Again, I think you will find that HDT works very well for most or all your and your customers' needs!
    Yes, I will keep an eye out for more case studies I can share.
    (Right now my focus is on our Compute Blades and Unified Compute Platforms.)
    Your project sounds exciting and fun.
    Wishes for all success!
    Like • Show 2 Likes2
    Actions
    - Janet Hutchison @ Janet Hutchison on Aug 27, 2013 12:27 PM
      Mark Correct
      Correct Answer
      Oh, Rahul, by the way..... When you have the opportunity to take some HDS training, look for course CSI2143 Hitachi Dynamic Tiering Concepts and Operations (3 days)
      (It's a very good course - I wrote it....)
      Like • Show 1 Like1
      Actions
    - Rahul Bhat @ Janet Hutchison on Aug 27, 2013 2:37 PM
      Mark Correct
      Correct Answer
      Thanks Janet Hutchison for your feedback, it was surely very useful and interesting. I would know more once I jump in . Regarding the course, will check it.
      
      Thanks again for taking time , your feedback is much appreciated.
      
      PS : I will keep the thread open for some more time.
      Like • Show 0 Likes0
      Actions
Mark Perino Aug 28, 2013 6:26 AM
Mark Correct
Correct Answer
As Janet Correctly mentioned the performance of the HDT pool, relocation, and other things depends on a lot of different factors.

1. Similar to the way HDP works. How much data is moved, when it is moved, and how it is evaluated are tunable using the settings.
2. It helps to think of it in this way. Tier 1 is "bonus" performance, Tier 2 is acceptable SLA meeting performance, and Tier 3 is savings where cold data can live. So The key is that Tier 2 needs to be Fat enough to hold any data that has an SLA. The usage of Tier 3 for cold data is where you can measure savings. In a shared environment you potentially have the benefit of scaling larger to achieve savings due to averages in larger pools, but reduce the isolation of silos. From a costing model one could measure the usage of each tier for each customer in a shared pool. In that way you would reward applications which have more cold data living on cheaper storage, and cost more accurately applications which have higher demand. If an application grows and demands isolation or special treatment you have options.. Either PIN it in a tier, or migrate it to an isolated set of storage (POOL/CLPR/Etc..) As long as you have unallocated capacity/performance you can easily move things into and out of any HDT pool.
3. Re-locations are at a low priority, but do consume resources (MPB, cache, disk busy). Depending on cycle time, sufficient Tier-2 space, etc. You can reduce or increase impact based on your needs.
4. It could, but there are many reasons for using the 42 MB size. Optimality and balance of performance is achieved with 42MB chunk size.
5. Best determined, through an analysis of the workload.
6. Yes absolutely. Cycle time, pinning in tiers, adding capacity in tiers, and many other settings.
7. SATA would be tier-3 and should largely be used for either cold data, or data that is written once and mostly read after that. There should be sufficient capacity in the tier above SATA so that all new writes are to a non-sata tier (then migrated down to SATA if the data is cold).
1 person found this helpful
Like • Show 0 Likes0
Actions

Jive

Hitachi Vantara Community

Storage Tiering "in reality"

Outcomes

Related Content

Recommended Content

Incoming Links