AnsweredAssumed Answered

HCI 4 node cluster performance is very slow

on Sep 26, 2018
Latest reply on Oct 15, 2018 by Jon Chinitz

Like • Show 0 Likes0
Comment • 8

HI Team,

We are running a HCI system with 4 nodes, 64GB RAM and 500GB hard disk on each node. The file system is a share from HNAS consisting of around 170 million files. The workflow is running very slow and HCI 4 node cluster is not able to scan even 100K files per hour. Whereas servers utilisation is not going beyond 20-30% and lot of resources are free on servers. Initially the workflow was talking bunch of 10000+ documents, and scanning 1M files / Hour. Now its reduced to less than 1000 files as batch and taking minimum 12 hours to scan 1M files. We faced the issue before and logged a case earlier but the issue is not resolved.

taking the recommendations from the support team ,we tried doing some changes to the pipelines. Like we have removed most of the regex fields in date conversion stage, added many content_type fields in mime type detection stage etc... but none of these seem to give us consistent performance and the rate is very slow.

Going by this rate, it would take months for the customer's 170 million file FS to get scanned by HCI... Can anyone please suggest some workarounds or their recommendations if you have faced a similar situation before.

No one else has this question

Outcomes

Jon Chinitz

Sep 27, 2018 6:23 AM

If you have opened a case for this, and it is not resolved, please have it escalated to Engineering.

Also please download the logs from all 4 servers and upload them to the case.

Some things to check:

1. Are these physical or virtual servers?

2. How much CPU (cores) does each server have?

3. Is the HNAS share mounted in the same path on all 4 nodes?

4. Are the nodes doing anything else? if they are virtual nodes can you verify what else is going on on that machine that might be causing this issue?

5. Are you seeing any document errors or task failures OR is the document processing rate simply decreasing?

Actions

Shaik Rizwan

@ Jon Chinitz on Sep 27, 2018 6:58 AM

Hello Jon,

Yes we are following up with engineering and in the mean time, i was looking for some workaround if our colleagues can help with. we have uploaded the logs too from all 4 servers. please find my answers inline and suggest us if there are any recommendations.

1. Are these physical or virtual servers?----------------- Virtual servers running on ESXi 6.5U1

2. How much CPU (cores) does each server have?------------ 8 CPU each

3. Is the HNAS share mounted in the same path on all 4 nodes?-------- yes on the same path

4. Are the nodes doing anything else? if they are virtual nodes can you verify what else is going on on that machine that might be causing this issue?--------------- we will definitely check this but these nodes are doing nothing else and we did check with the team handling VM environment and they found no issues related to performance on the ESXi servers and at the same time no specific hardware failures.

5. Are you seeing any document errors or task failures OR is the document processing rate simply decreasing?---- we haven't seen any document error or task failure till now and it is only the document procseeing rate which is simply going down drastically.

Actions

Jon Chinitz

@ Shaik Rizwan on Sep 27, 2018 7:11 AM

How many CPUs does the host have? If each HCI VM has 8 cores then 4 machines have 32. Does your host have 32 CPUs? More? Less? Even though they are "virtual" the host still needs the aggregate horsepower.

Actions

Troy Myers

Sep 27, 2018 7:56 AM

What are your workflow settings? Most importantly the heap settings in the workflow and the memory in the index. Please take a look at the document below. To make sure you are utilizing all the RAM in your configuration.

Workflow and index settings troy notes

Actions

Shaik Rizwan

@ Troy Myers on Oct 15, 2018 7:36 AM

Hello Troy,

Sorry I was busy with other commitments. the issue is still pending as it was earlier. here are our workflow settings.

the memory of the index service is as below.

Actions

Jon Chinitz

Oct 15, 2018 7:58 AM

Have you opened a ticket with GSC?

Actions

Shaik Rizwan

@ Jon Chinitz on Oct 15, 2018 8:35 PM

Hello Jon,

we did log a case with GSC . Case Number: 00688667 - logged on 27th of september. apart from asking us to upload logs, we are getting no response from the support team even after repeated follow ups.

Actions

Jon Chinitz

@ Shaik Rizwan on Oct 15, 2018 9:38 PM

I will look into this.

Actions

8 Replies

Jon Chinitz Sep 27, 2018 6:23 AM
Mark Correct
Correct Answer
If you have opened a case for this, and it is not resolved, please have it escalated to Engineering.
Also please download the logs from all 4 servers and upload them to the case.
Some things to check:
1. Are these physical or virtual servers?
2. How much CPU (cores) does each server have?
3. Is the HNAS share mounted in the same path on all 4 nodes?
4. Are the nodes doing anything else? if they are virtual nodes can you verify what else is going on on that machine that might be causing this issue?
5. Are you seeing any document errors or task failures OR is the document processing rate simply decreasing?
Like • Show 1 Like1
Actions
- Shaik Rizwan @ Jon Chinitz on Sep 27, 2018 6:58 AM
  Mark Correct
  Correct Answer
  Hello Jon,
  
  Yes we are following up with engineering and in the mean time, i was looking for some workaround if our colleagues can help with. we have uploaded the logs too from all 4 servers. please find my answers inline and suggest us if there are any recommendations.
  1. Are these physical or virtual servers?----------------- Virtual servers running on ESXi 6.5U1
  2. How much CPU (cores) does each server have?------------ 8 CPU each
  3. Is the HNAS share mounted in the same path on all 4 nodes?-------- yes on the same path
  4. Are the nodes doing anything else? if they are virtual nodes can you verify what else is going on on that machine that might be causing this issue?--------------- we will definitely check this but these nodes are doing nothing else and we did check with the team handling VM environment and they found no issues related to performance on the ESXi servers and at the same time no specific hardware failures.
  5. Are you seeing any document errors or task failures OR is the document processing rate simply decreasing?---- we haven't seen any document error or task failure till now and it is only the document procseeing rate which is simply going down drastically.
  Like • Show 0 Likes0
  Actions
  - Jon Chinitz @ Shaik Rizwan on Sep 27, 2018 7:11 AM
    Mark Correct
    Correct Answer
    How many CPUs does the host have? If each HCI VM has 8 cores then 4 machines have 32. Does your host have 32 CPUs? More? Less? Even though they are "virtual" the host still needs the aggregate horsepower.
    Like • Show 0 Likes0
    Actions
Troy Myers Sep 27, 2018 7:56 AM
Mark Correct
Correct Answer
What are your workflow settings? Most importantly the heap settings in the workflow and the memory in the index. Please take a look at the document below. To make sure you are utilizing all the RAM in your configuration.

Workflow and index settings troy notes
Like • Show 2 Likes2
Actions
- Shaik Rizwan @ Troy Myers on Oct 15, 2018 7:36 AM
  Mark Correct
  Correct Answer
  Hello Troy,
  
  Sorry I was busy with other commitments. the issue is still pending as it was earlier. here are our workflow settings.
  
  the memory of the index service is as below.
  Like • Show 0 Likes0
  Actions
Jon Chinitz Oct 15, 2018 7:58 AM
Mark Correct
Correct Answer
Have you opened a ticket with GSC?
Like • Show 0 Likes0
Actions
- Shaik Rizwan @ Jon Chinitz on Oct 15, 2018 8:35 PM
  Mark Correct
  Correct Answer
  Hello Jon,
  
  we did log a case with GSC . Case Number: 00688667 - logged on 27th of september. apart from asking us to upload logs, we are getting no response from the support team even after repeated follow ups.
  Like • Show 0 Likes0
  Actions
  - Jon Chinitz @ Shaik Rizwan on Oct 15, 2018 9:38 PM
    Mark Correct
    Correct Answer
    I will look into this.
    Like • Show 0 Likes0
    Actions

Jive

Hitachi Vantara Community

HCI 4 node cluster performance is very slow

Outcomes

Related Content

Recommended Content