Gustavo Castets

Configure Brocade DCX 8510 with enough bandwidth for HUR Fast Copy Pace

Blog Post created by Gustavo Castets Employee on Aug 3, 2018

In an HUR two data center (2DC) configuration if you want to use HUR for initial copy and you want to use high copy pace to speed the initial copy process, then the switches connecting both data centers must have enough available bandwidth else the switch ports will fail.

In an experimental test case two Hitachi F900 storage systems were arranged in an HUR 2DC setup.  The connection between both sites consisted of a pair of FX8-24 DCX-8501 Brocade blades: in total there were four 10 GbE XGE ports (VE_Ports) connecting both sites. Each of the four connections included an IXIA Network Emulator II to emulate distance between the sites.

With the aforementioned configuration the theoretical available bandwidth was 5,120 MB/s. Nevertheless when running with an emulated distance of 1,500 Km the switch connections would fail: the VE_Ports would go offline and the tunnels we go into a permanent “InProg” condition.

After some analysis we realized that the minimum (-b) and maximum (-B) committed rates in the circuits of the tunnels were set to 5000000/5000000:

Distance0:admin> portshow fcipcircuit all

Tunnel Circuit  OpStatus Flags    Uptime  TxMBps RxMBps ConnCnt CommRt Met/G

--------------------------------------------------------------------------------

2/12 0 2/xge0  Up      ---4-xs 3d18h1m    0.00    0.00 25  5000/5000  0/-

2/12 1 2/xge1  Up      ---4--s  3d18h1m 0.00    0.00   28 5000/5000  0/-

2/22 0 2/xge0  Up      ---4--s 3d18h1m    0.00    0.00 25  5000/5000  0/-

2/22 1 2/xge1  Up      ---4-xs 3d18h1m    0.00    0.00 28  5000/5000  0/-

7/12 0 7/xge1  Up      ---4--s 3d18h6m    0.00    0.00 26  5000/5000  0/-

7/12 1 7/xge0  Up      ---4-xs 3d18h6m    0.00    0.00 28  5000/5000  0/-

7/22 0 7/xge0  Up      ---4--s 3d18h6m    0.00    0.00 28  5000/5000  0/-

7/22 1 7/xge1  Up      ---4-xs 3d18h6m    0.00    0.00 26  5000/5000  0/-

--------------------------------------------------------------------------------

Flags (circuit): s=sack v=VLAN Tagged x=crossport 4=IPv4 6=IPv6

                 L=Listener I=Initiator

Distance0:admin>

And we also recognized the crossports that were configured in the FX8-24 blades:

Distance0:admin> portshow ipif

Port IP Address                     / Pfx  MTU   VLAN Flags

--------------------------------------------------------------------------------

2/xge0 192.168.0.94                   / 24   1500  n/a U R M

2/xge0 192.168.0.93                   / 24   1500 n/a   U R M X

2/xge1 192.168.1.93                   / 24   1500  n/a U R M

2/xge1 192.168.1.94                   / 24 1500  n/a   U R M X

7/xge0 192.168.0.98                   / 24   1500  n/a U R M

7/xge0 192.168.0.97                   / 24   1500 n/a   U R M X

7/xge1 192.168.1.97                   / 24   1500  n/a U R M

7/xge1 192.168.1.98                   / 24   1500 n/a   U R M X

--------------------------------------------------------------------------------

Flags: U=Up B=Broadcast D=Debug L=Loopback P=Point2Point R=Running

       N=NoArp PR=Promisc M=Multicast S=StaticArp LU=LinkUp X=Crossport

Distance1:admin>

When trying to increase the committed rates to 5000000/10000000 we were getting a “Tunnel/Circuit Bandwidth Exceeded” error message.  From our reading of the Brocade Fabric OS FCIP Administrator’s Guide we realized that the crossports in the FX8-24 blades were in fact consuming bandwidth.  So the solution would consist in removing the crossports and setting the commit rates of the circuits to 5000000/10000000 so as to get a higher maximum commit rate.

Note: because we are not in a production environment we can afford not having crossports in order to maximize the available bandwidth which is needed for some of our tests.

The simplest procedure was to delete the existing tunnels which had the crossports and recreate them but this time without crossports and also with commit rate values of -b=5000000 and -B=10000000.  This is how we did it for blade 2 of one of the switches, then the same operation was done for blade 7 and for also for blades 2 and 7 on the other switch:

Distance0:admin> portcfg fciptunnel 2/12 delete

Operation Succeeded

Distance0:admin> portcfg fciptunnel 2/22 delete

Operation Succeeded

Distance0:admin> portcfg fciptunnel 2/12 create --remote-ip 192.168.1.93 --local-ip 192.168.1.90 -b 5000000 -B 10000000

Operation Succeeded

Distance0:admin> portcfg fciptunnel 2/22 create --remote-ip 192.168.0.94 --local-ip 192.168.0.91 -b 5000000 -B 10000000

Operation Succeeded

Distance0:admin>

These were the tunnels once they were redefined, this time doubling the maximum commit rate:

Distance0:admin> portshow fcipcircuit all

Tunnel Circuit  OpStatus Flags    Uptime  TxMBps RxMBps ConnCnt CommRt Met/G

--------------------------------------------------------------------------------

2/12   0 2/xge1 Up      ---4--s   19m46s 0.00    0.00    1 5000/10000 0/-

2/22   0 2/xge0 Up      ---4--s   19m13s 0.00    0.00    1 5000/10000 0/-

7/12   0 7/xge1 Up      ---4--s    1m17s 0.00    0.00    1 5000/10000 0/-

7/22   0 7/xge0 Up      ---4--s      52s 0.00    0.00    1 5000/10000 0/-

--------------------------------------------------------------------------------

Flags (circuit): s=sack v=VLAN Tagged x=crossport 4=IPv4 6=IPv6

L=Listener I=Initiator

Distance0:admin>

After completing the reconfiguration of the tunnels we run an HUR initial copy process with copy pace high and distance 1,500 Km.  In this opportunity a 4,857 MB/s throughput between sites was observed and the DCX-8510 ports did not fail.

Outcomes