Hitachi Ops Center​

 View Only

 CPU Load Balancer Automation

  • Hitachi Ops Center
  • Hitachi Ops Center
William Jansen Van Nieuwenhuizen's profile image
William Jansen Van Nieuwenhuizen posted 09-04-2020 09:11

Hello

 

We recently had a request where a client asked us about MP Load balancing. Using the Analyzer Connection setup in Automator as well as the CMREST Connection we created a quick MP load balancer that seems to work great on our G900 and G5000's.

 

Give it a try...

 

It identifies the busiest resources based on your performance metric and then finds the most impactful resources that it believes will bring balance based on that metric.

Service Template

 

 

The effect from the script can be see here in this example:

balancing effect

 

 

Requirements:

Need to setup your Analyzer and CMRest connection in Automator.

 

Special Shout out to Johann and Johnny too.

@Johnny Koen​ @Johannes Du Toit​ 


#HitachiOpsCenter
Arnoud Dekker's profile image
Arnoud Dekker

Nice one!

A small note, which could be my complete inexperience with Automator so far.

I seem to have to change the Python interpreter path as in service.xml it is set to /usr/local/bin/python3, overriding the default set for python in Administration > Shared Properties Settings in Automator.

William Jansen Van Nieuwenhuizen's profile image
William Jansen Van Nieuwenhuizen

Yes. This automation uses python 3.6+.

I tried and tested this on Opscenter 10.3 Ova.

Just wanted to check, its working fine for you now right?

Thanks for the feedback :-)

Arnoud Dekker's profile image
Arnoud Dekker

Thanks William, I found the plugin-doc pointing to Python 3. Sadly, on this RHEL7.8 install the default python version for python is 2.7 (in $PATH). I do have /bin/python3 as 3.6.8 so I expected this to work but so far no luck yet. Might be some missing dependencies somewhere. I'll have to dive into this when I find more time.

William Jansen Van Nieuwenhuizen's profile image
William Jansen Van Nieuwenhuizen

Yes that should work. I believe the only additional packages I use is the requests package.

So, you might need to:

  • 'Copy and Edit' this version. Head to the python plugins and change the interpretor on the plugin to use your local python3. Creating a symbolic link to /usr/local/bin/python3 might also do the trick.
  • Install requests in python. If you have pip3 available you should just be able to do a pip3 install requests

If that doesnt work we can always schedule a teams/webex/zoom session. This way we can solve it for the next guy :-)

 

William Jansen Van Nieuwenhuizen's profile image
William Jansen Van Nieuwenhuizen

Moved here with the latest version:
Community Automator Library - CPU Load Balancer

Miran Kreuzer's profile image
Miran Kreuzer

Hi, first of all thanks for the Service Template! So far I have managed to setup the Automator and import the Template. But when i click on "submit and view task" it runs into some Errors. Maybe somebody can give me a hint how to solve the Problem? Below the Errors from the Log. I have also tried to solve the problem with the installation of python 3.6.11 and changed the python interpreter path to python3. That changed the below Error Messages. But it still didn't work. Is a specific Python Version needed for the Template?

Thanks in advance!

Miran

0022 2023/10/13 06:49:55.739 Automation 23ECF465 6343A67C KNAE08461-I The specified Python script is starting. 0023 2023/10/13 06:49:56.096 Automation 23ECF465 6343A67C KNAE08465-E ER An error occurred while running the Python script. Revise the Python code specified in the body of the script, and then run the service again. 0024 2023/10/13 06:49:56.113 Automation 23ECF465 6343A67C KNAE08002-I Plug-in execution completed (task name: Storage CPU Load Balancer_20231013064946, task ID: 923211, step ID: /CPUBalancer, execution ID: , plug-in return code: 2). 0025 2023/10/13 06:49:56.114 Automation 23ECF465 6343A67C KNAE08004-I schema_version=2.0 0026 2023/10/13 06:49:56.115 Automation 23ECF465 6343A67C KNAE08004-I vendor=com.hitachi.software.dna 0027 2023/10/13 06:49:56.115 Automation 23ECF465 6343A67C KNAE08004-I name=PythonPlugin 0028 2023/10/13 06:49:56.115 Automation 23ECF465 6343A67C KNAE08004-I version=01.00.01 0029 2023/10/13 06:49:56.115 Automation 23ECF465 6343A67C KNAE08004-I plugin_type=javaClass 0030 2023/10/13 06:49:56.115 Automation 23ECF465 6343A67C KNAE08004-I vendor_display_name=Hitachi, Ltd. 0031 2023/10/13 06:49:56.115 Automation 23ECF465 6343A67C KNAE08004-I display_name=Python Plug-in 0032 2023/10/13 06:49:56.115 Automation 23ECF465 6343A67C KNAE08004-I short_description=This plug-in runs script written in Python. 0033 2023/10/13 06:49:56.115 Automation 23ECF465 6343A67C KNAE08004-I tags=Execute Script 0034 2023/10/13 06:49:56.115 Automation 23ECF465 6343A67C KNAE08004-I enable_SSH_charset_detection=(none) 0035 2023/10/13 06:49:56.116 Automation 23ECF465 6343A67C KNAE08004-I elevate_privileges=(none) 0036 2023/10/13 06:49:56.116 Automation 23ECF465 6343A67C KNAE08004-I run_as_system=(none)

0047 2023/10/13 06:49:56.117 Automation 23ECF465 6343A67C KNAE08004-I property=/CPUBalancer/standardErrorOutput, value=Traceback (most recent call last): File "/var/opt/hitachi/Automation/data/task/923211/PythonPlugin_step2/main.py", line 26, in <module> import urllib3 ModuleNotFoundError: No module named 'urllib3'

William Jansen Van Nieuwenhuizen's profile image
William Jansen Van Nieuwenhuizen

Hello Miran

What Ops Center version are you running? 

Looks like the python path you're using doesn't have requests installed. If your ops center hosts has internet access could you do a "pip3 install requests"? Should solve the dependency. 

Miran Kreuzer's profile image
Miran Kreuzer

Hello William, thanks for your super quick reply! We have the latest OPS Center Version installed >> 10.9.3. Requests should be installed:

[root@LNZOPSC01 ~]# python3 -m pip show requests
Name: requests
Version: 2.13.0
Summary: Python HTTP for Humans.
Home-page: http://python-requests.org
Author: Kenneth Reitz
Author-email: me@kennethreitz.com
License: Apache 2.0
Location: /usr/local/lib/python3.7/site-packages/requests-2.13.0-py3.7.egg
Requires:
Required-by:

Maybe wrong version?

Regards

William Jansen Van Nieuwenhuizen's profile image
William Jansen Van Nieuwenhuizen

Weird, might be that with my previous version on Ops Center it was included.

You could try a pip3 install urllib3.

Below all my installed packages. However a lot of these are also used by other automations. But just for comparison:

bcrypt==4.0.1
cffi==1.14.5
chardet==4.0.0
configshell-fb==1.1.28
cryptography==36.0.1
dbus-python==1.2.18
distro==1.5.0
file-magic==0.4.0
future==0.18.3
gpg==1.15.1
idna==2.10
iotop==0.6
kmod==0.1
libcomps==0.1.18
netifaces==0.10.6
netmiko==3.4.0
nftables==0.1
ntc_templates==4.0.0
nvmetcli==0.7
paramiko==3.3.1
pciutils==2.3.6
pexpect==4.8.0
ply==3.11
ptyprocess==0.6.0
pycparser==2.20
PyGObject==3.40.1
PyNaCl==1.5.0
pyOpenSSL==19.0.0
pyparsing==2.4.7
pyserial==3.5
PySocks==1.7.1
python-dateutil==2.8.1
python-dmidecode==3.12.2
PyYAML==5.4.1
requests==2.13.0
rhnlib==2.8.6
rpm==4.16.1.3
scp==0.14.5
selinux==3.5
sepolicy==3.5
setools==4.4.1
six==1.15.0
sos==4.5.1
systemd-python==234
tenacity==8.2.3
textfsm==1.1.3
urllib3==1.26.5
urwid==2.1.2

Miran Kreuzer's profile image
Miran Kreuzer

Hi,

finally i managed to get it running without any error. I have installed urllib3 into my python3 Version. 

Here some output from the log:
0126 2023/10/20 10:50:56.367 Automation 23ECF465 100F800B KNAE08468-I NOTHING TO DO, we have a good array @ 445247 0127 2023/10/20 10:50:56.367 Automation 23ECF465 100F800B KNAE08468-I NOTHING TO DO, we have a good array @ 448655 0128 2023/10/20 10:50:56.367 Automation 23ECF465 100F800B KNAE08468-I ========================================================= 0129 2023/10/20 10:50:56.367 Automation 23ECF465 100F800B KNAE08468-I HHHEEEEEEEEEEYYYYYYYYY 0130 2023/10/20 10:50:56.367 Automation 23ECF465 100F800B KNAE08468-I Looks like you need some balance there @ 445282! 0131 2023/10/20 10:50:56.367 Automation 23ECF465 100F800B KNAE08468-I Adjustment needed by about 6% 0132 2023/10/20 10:50:56.367 Automation 23ECF465 100F800B KNAE08468-I This Array is currently doing: 727 so lets move 43.62 IOPS 0133 2023/10/20 10:50:56.367 Automation 23ECF465 100F800B KNAE08468-I ========================================================= 0134 2023/10/20 10:50:56.367 Automation 23ECF465 100F800B KNAE08468-I ------------------------------- 0135 2023/10/20 10:50:56.368 Automation 23ECF465 100F800B KNAE08468-I ARRAY: 445282 0136 2023/10/20 10:50:56.368 Automation 23ECF465 100F800B KNAE08468-I ------------------------------- 0137 2023/10/20 10:50:56.368 Automation 23ECF465 100F800B KNAE08468-I MP BALANCE: {1: 0, 0: 0} 0138 2023/10/20 10:50:56.368 Automation 23ECF465 100F800B KNAE08468-I What I need to do is: 0139 2023/10/20 10:50:56.368 Automation 23ECF465 100F800B KNAE08468-I Average: 0.0 0140 2023/10/20 10:50:56.368 Automation 23ECF465 100F800B KNAE08468-I This means 0141 2023/10/20 10:50:56.368 Automation 23ECF465 100F800B KNAE08468-I 1 - TOTAL: 0 so please add: 0.0 0142 2023/10/20 10:50:56.368 Automation 23ECF465 100F800B KNAE08468-I 0 - TOTAL: 0 so please add: 0.0 0143 2023/10/20 10:50:56.368 Automation 23ECF465 100F800B KNAE08468-I FINAL MP BALANCE COUNT: {1: {'total': 0, 'desired': 0.0, 'current_value': 0}, 0: {'total': 0, 'desired': 0.0, 'current_value': 0}} 0144 2023/10/20 10:50:56.368 Automation 23ECF465 100F800B KNAE08468-I What I've done is the following... 0145 2023/10/20 10:50:56.368 Automation 23ECF465 100F800B KNAE08468-I [] 0146 2023/10/20 10:50:56.368 Automation 23ECF465 100F800B KNAE08468-I ========================================================= 0147 2023/10/20 10:50:56.368 Automation 23ECF465 100F800B KNAE08468-I HHHEEEEEEEEEEYYYYYYYYY 0148 2023/10/20 10:50:56.368 Automation 23ECF465 100F800B KNAE08468-I Looks like you need some balance there @ 454250! 0149 2023/10/20 10:50:56.368 Automation 23ECF465 100F800B KNAE08468-I Adjustment needed by about 6% 0150 2023/10/20 10:50:56.368 Automation 23ECF465 100F800B KNAE08468-I This Array is currently doing: 25 so lets move 1.5 IOPS 0151 2023/10/20 10:50:56.368 Automation 23ECF465 100F800B KNAE08468-I =========================================================



May i ask some questions about how the Template works or maybe for a link for Template Documentation? 

Questions:
- Which timeframe does it use to analyze the MP Load differences? (last 10 Minutes, or longer?)
- According to the above log, did the template anything? On one hand it shows that it needs adjustment by about 6 % on the other hand it notes "Average: 0.0... 0 so please add: 0.0..." What i've done is the following... []

Thanks for your time!

Regards

Miran

William Jansen Van Nieuwenhuizen's profile image
William Jansen Van Nieuwenhuizen
Hello Miran
 
I've written this some time ago so had to check what I did. It looks like the period I use is a variable set at 0.1 of a day. I think, from 2 hours back till current time . 
 
Let me try and break down what happened form the log:
 
Automation 23ECF465 100F800B KNAE08468-I NOTHING TO DO, we have a good array @ 445247 
# Here it found that the balance was ok for array 445247, so it skipped this array
 
Automation 23ECF465 100F800B KNAE08468-I NOTHING TO DO, we have a good array @ 448655
# Here it found that the balance was ok for array 448655, so it skipped this array
 
Automation 23ECF465 100F800B KNAE08468-I ========================================================= 
Automation 23ECF465 100F800B KNAE08468-I HHHEEEEEEEEEEYYYYYYYYY 
Automation 23ECF465 100F800B KNAE08468-I Looks like you need some balance there @ 445282! 
Automation 23ECF465 100F800B KNAE08468-I Adjustment needed by about 6%
Automation 23ECF465 100F800B KNAE08468-I This Array is currently doing: 727 so lets move 43.62 IOPS
Automation 23ECF465 100F800B KNAE08468-I =========================================================
Automation 23ECF465 100F800B KNAE08468-I -------------------------------
Automation 23ECF465 100F800B KNAE08468-I ARRAY: 445282 
Automation 23ECF465 100F800B KNAE08468-I ------------------------------- 
Automation 23ECF465 100F800B KNAE08468-I MP BALANCE: {1: 0, 0: 0}
Automation 23ECF465 100F800B KNAE08468-I What I need to do is: 
Automation 23ECF465 100F800B KNAE08468-I Average: 0.0 
Automation 23ECF465 100F800B KNAE08468-I This means
Automation 23ECF465 100F800B KNAE08468-I 1 - TOTAL: 0 so please add: 0.0 
Automation 23ECF465 100F800B KNAE08468-I 0 - TOTAL: 0 so please add: 0.0
Automation 23ECF465 100F800B KNAE08468-I FINAL MP BALANCE COUNT: {1: {'total': 0, 'desired': 0.0, 'current_value': 0}, 0: {'total': 0, 'desired': 0.0, 'current_value': 0}}
Automation 23ECF465 100F800B KNAE08468-I What I've done is the following... 
Automation 23ECF465 100F800B KNAE08468-I [] # This means it couldn't find 
# Here it found that the balance was not ok for array 445282, so it looked at the metric you selected. I'm guessing IOPS[Maybe try a different metric]? But for that time I found that there was no IOPS difference between the luns on the CPU's to balance it. It couldn't balance 0 IOPS, so it did nothing []. 
 
 
 
Automation 23ECF465 100F800B KNAE08468-I ========================================================= 
Automation 23ECF465 100F800B KNAE08468-I HHHEEEEEEEEEEYYYYYYYYY
Automation 23ECF465 100F800B KNAE08468-I Looks like you need some balance there @ 454250! 
Automation 23ECF465 100F800B KNAE08468-I Adjustment needed by about 6% 
Automation 23ECF465 100F800B KNAE08468-I This Array is currently doing: 25 so lets move 1.5 IOPS 0151
Automation 23ECF465 100F800B KNAE08468-I =========================================================
# Here it found that the balance was not ok for array 454250, so it looked at the metric you selected. I'm guessing IOPS? But for that time I found that there was a 6& difference between the luns on the CPU's to balance it. It calculated that the array is doing 25 ?IOPS? to bring this in balance(25IOPS*6%) it needed to move 1.50 IOPS. So it would look for a luns that's doing 1 or less IOPS and move it try and balance it. I can imaginge with such a small difference it couldn't find a ldev doing it so it might have also not done anyting.