Best Practice for Connecting IBM Storage to Power Systems-2016.pdf

Best Practices for Performance, Design and Troubleshooting IBM Storage Connected to IBM Power System Chuck Laing Senior Technical Staff Member (STSM) IBM GTS SO/IS Delivery Organization

© Copyright IBM Corporation 2016. Technical University/Symposia materials may not be reproduced in whole or in part without the prior written permission of IBM.

Knowledge is POWER!

As System’s Administrators – we don’t always KNOW what we don’t know about storage  Ask for storage leveraging what you know  Avoid bottlenecks  Use tools available  Speed problem isolation

 Make more informed architectural decisions

What we are NOT going to do today:  Try to turn you into storage administrators  Boil the ocean IBM Systems Technical Events | ibm.com/training/events


2

2

Throughput and Performance Key Optimization Factors • Throughput – Spreading and balancing IO across hardware resources

• IO Performance Tuning – Using utilities, functions and features to tweak (backend, frontend)

• Controllers • Ports & zoning connections • PCI’s Cards

• Qdepths • HBA transfer rates

• CPUs, RAM • Disks spindles

• LVM striping vs spreading • Data Placement

– Compression – Thin Provisioning

– Easy Tier - SSD

– FC adapters

– Random versus sequential

– Spreading versus Isolation – Application characteristics

• Etc….

Configuring throughput optimally increases potential performance scalability IBM Systems Technical Events | ibm.com/training/events


3

Foundation What causes performance degradation and Outages?

• The 3 most common root causes are: • Configuration changes • Hardware component failure • IO load shift or increase

• You should consider designing the environment to withstand load shift in the event of half the environment failing such as: • Controller outages • Fabric outages • Server Outages

• You should design configurations to known Best Practices • …Just because you can do something …should you? IBM Systems Technical Events | ibm.com/training/events


4

Agenda -Troubleshooting – as growth and change occur Top 5 most common things that go wrong by configuration design: • From Field Experience - Deviation from Best Practices/Trending •

Bottlenecks in the Physical Design 1. 2. 3. 4. 5.

•

Bottlenecks in the SAN Configurations Design 1. 2. 3. 4. 5.

•

Suboptimal zoning practices Port Placement for Best IO spread and redundancy Slow Draining Devices -Bad or dirty SFP connections Fillwords Port Speeds –fixed vs. auto negotiate

Bottlenecks in the Storage Configurations Design 1. 2. 3. 4. 5.

•

Rack and Stack Design Manageability – designing for physical growth Fibre cabling practices Fabric Topology Cooling

Data Layout - Storage Pools and Tiered pools - IO Spread vs IO isolation IO Spread and Load Balancing across CPU Complexes IO Spread and Load Balancing across DA Pairs IO Spread and Load Balancing across Disk Enclosures IO spread and Load Balancing across ports

Bottlenecks in Host Configurations Design 1. 2. 3. 4. 5.

Using the right Host Multipath Drivers to match Storage Microcode Striping , Spreading, Data Layout, HBA transfer rates Adjusting Qdepth Workload IO increases due to Data Growth

IBM Systems Technical Events | ibm.com/training/events


5

Bottlenecks in the physical design • A deeper dive into the Physical Topology



6

Bottlenecks in Physical setup – Rack and Stack Best Practices Rear View

Front View

A “right way” example Clean and Neat



7

Bottlenecks in Physical setup – Rack and Stack • # 1 Incorrect ways to rack and stack • #2 Manageability for growth • #3 Fibre cabling practices

• #4 Fabric topology

IBM Systems Technical Events | ibm.com/training/events 8


8

Bottlenecks in Physical setup – Rack and Stack • Determine if placement follows best practices taking advantage of hot/cold aisle and perforated tiles. The hot exhaust of one device blowing into the cold air intake of another could lead to overheating.



9

Bottlenecks in physical setup - Cooling #5 • What’s wrong here?  It was off in a corner with no surrounding air flow.  Impact: Shortened component life span from over heating.  Everytime the cabinet door was opened for service, it blew:

– A power supply – Fans always failing – Application outages



10

Quiz #1 .

Unserviceable fabric rack 1. Bend Radius exceeded 2. Insufficient strain relief Cable weight pull on other cables

3. Cables loose on floor Susceptible to pinching, getting caught in door, being stepped on…etc. IBM Systems Technical Events | ibm.com/training/events

11


11

Quiz #2 • What is the impact of this?  Can’t service  Blocked air exhaust



12

Quiz #3 • What’s Wong!!



13

Bottlenecks in the SAN Fabric - Best Practice

Suboptimal zoning practices



14

#1 Correct Zoning to Multiple Storage devices

• Create separate zones for each Storage device V7000 2 4

1 1

2

1

2

2

3

4

1

2

3

CAUTION

4 1 3

DS8K-2

2 1

2 Disconnect all supply power for complete isolation

1 3

2 4

1

CAUTION

1 3

2 1

2

3

4

1

2

3

2 4

4

CAUTION

XIV Zones

Fabric1 Core1

DS8K-1

Disconnect all supply power for complete isolation

V7K Zones

XIV

2 4


2

1

1

2 4

1 3

2 4

1 3

CAUTION 1 3


DS8K-1 Zones

DS8K-2 Zones


V7K Zones

XIV Zones

Fabric 2 Core2

DS8K-1 Zones

DS8K-2 Zones


15

#1 - Incorrect Zoning to Multiple Storage devices • Zone each device to only one core to avoid extra “ISL IO traversing” cutting down ISL hops • Spread Multiple Storage devices as evenly possible across Cores • Create separate zones for each Storage device

V7000 1 3

2 4

1 1

2

1

2

2

3

4

1

2

3

CAUTION

4

2 4

1 3

2 4


2

1

1

XIV

DS8K-1

DS8K-2

2 1

2

Disconnect all supply power for complete isolation 1 3

2 4

CAUTION 1 3



1 3

2 4

1

1 3

2 1

2

3

4

1

2

3

2 4

4

CAUTION

CAUTION

Storage Zone-1

Fabric1 Core1


Storage Zone-2

Fabric 2 Core2


16

One Zone per SVC Node per Fabric for each back-end storage device Making 1 zone per Node per Fabric with the same ports from a single backend storage unit, will reduce the overheads associated with large numbers of logins

Production SAN Fabric A

STG Zone-2

STG Zone-3

STG Zone-1

Physical port number 16G HBA 8G HBA

1 2 1 2

1 2 2 2

1 2 3 4 Slot 1

2 2 1 2

2 2 3 4

5 6 7 8 Slot 2

3 3 3 3 5 5 5 5 1 2 3 4 1 2 3 4 9 10 11 12 13 14 15 16

Slot 3

Node 1

Slot 5

1 2 1 2

1 2 2 2

1 2 34 Slot 1

2 2 1 2

2 2 3 4

5 6 7 8 Slot 2

Production SAN Fabric B

STG Zone-4

3 3 3 3 5 5 5 5 1 2 3 4 1 2 3 4 9 10 11 12 13 14 15 16

Slot 3

Node 2

Slot 5

Logical port with wwpn # embedded

I/O Group 0

DH8 1 Building Block – 16 FC ports per node IBM Systems Technical Events | ibm.com/training/events


17

#1 - Storage Zone Type – Good Core Edge Design – 1ISL Hops Back-end Storage HBA1

Host X

HBA2

P P 1 2

B1

A1

P P 1 2 Host Zone-1

STG Zone-2

STG Zone-1

Host Zone-2

IO request 1 ISL Hop1 Fabric1


1 2 1 2

1 2 2 2

1 2 3 4 Slot 1

Core1

2 2 1 2

Fabric 1

2 2 3 4

5 6 7 8 Slot 2

Edge 1

Fabric2

3 3 3 3 5 5 5 5 1 2 3 4 1 2 3 4 9 10 11 12 13 14 15 16

Slot 3

Node 1

Slot 5

1 2 1 2

1 2 2 2

1 2 34 Slot 1

2 2 1 2

2 2 3 4

5 6 7 8 Slot 2

Edge1

Fabric 2

Core 1

3 3 3 3 5 5 5 5 1 2 3 4 1 2 3 4 9 10 11 12 13 14 15 16

Slot 3

Node 2

Slot 5


I/O Group 0



18

The same back-end ports are used for each Node Zone XIV Storage Module1 HBA1

HBA2

P P 1 2

STG SAN Zone1

P P 3 4

Module2 HBA1 P P 1 2

STG Zone3

HBA2 P P 3 4

Module3 HBA1

HBA2

P P 1 2

Fabric 1

P P 3 4


STG SAN Zone2

HBA2 P P 3 4

STG Zone4


HBA2 P P 3 4


HBA2 P P 3 4

Fabric 2

Up to 16 zones for an 8 node Cluster 2 zones /fabric/ node Up to 16 Ports max from any back-end storage



19

Host Zones – 1 host initiator to 2 SVC node target ports • Should be (1:2) zoning

Host A1

B1

1 host initiator port

Fabric2 Edge1

Fabric1 Edge1 2 SVC Node Target ports


1 2 1 2

1 2 2 2

1 2 3 4 Slot 1

2 2 1 2

2 2 3 4

5 6 7 8 Slot 2

3 3 3 3 5 5 5 5 1 2 3 4 1 2 3 4 9 10 11 12 13 14 15 16

Slot 3

Node 1

Slot 5

1 2 1 2

1 2 2 2

1 2 34 Slot 1

2 2 1 2

2 2 3 4

5 6 7 8 Slot 2

3 3 3 3 5 5 5 5 1 2 3 4 1 2 3 4 9 10 11 12 13 14 15 16

Slot 3

Node 2

Slot 5


I/O Group 0



20

Host Zones – Too few • No (1:1) zoning

Host A1

B1

1 host initiator port

Fabric2 Edge1

Fabric1 Edge1 1 SVC Node Target ports


1 2 1 2

1 2 2 2

1 2 3 4 Slot 1

2 2 1 2

2 2 3 4

5 6 7 8 Slot 2

3 3 3 3 5 5 5 5 1 2 3 4 1 2 3 4 9 10 11 12 13 14 15 16

Slot 3

Node 1

Slot 5

1 2 1 2

1 2 2 2

1 2 34 Slot 1

2 2 1 2

2 2 3 4

5 6 7 8 Slot 2

3 3 3 3 5 5 5 5 1 2 3 4 1 2 3 4 9 10 11 12 13 14 15 16

Slot 3

Node 2

Slot 5


I/O Group 0



21

Host Zones – 4 paths per Vdisk • 2 preferred – 1 red & 1 blue • 2 non preferred – red & blue

Host A1

B1

Fabric2 Edge1

Fabric1 Edge1

Non Preferred Paths

Preferred Paths


1 2 1 2

1 2 2 2

1 2 3 4 Slot 1

2 2 1 2

2 2 3 4

5 6 7 8 Slot 2

3 3 3 3 5 5 5 5 1 2 3 4 1 2 3 4 9 10 11 12 13 14 15 16

Slot 3

Node 1

Slot 5

1 2 1 2

1 2 2 2

1 2 34 Slot 1

2 2 1 2

2 2 3 4

5 6 7 8 Slot 2

3 3 3 3 5 5 5 5 1 2 3 4 1 2 3 4 9 10 11 12 13 14 15 16

Slot 3

Node 2

Slot 5


I/O Group 0



22

Host Zones – too many paths and target ports Host A1

B1

Fabric2 Edge1

Fabric1 Edge1

Non Preferred Paths

Preferred Paths


1 2 1 2

1 2 2 2

1 2 3 4 Slot 1

2 2 1 2

2 2 3 4

5 6 7 8 Slot 2

3 3 3 3 5 5 5 5 1 2 3 4 1 2 3 4 9 10 11 12 13 14 15 16

Slot 3

Node 1

Slot 5

1 2 1 2

1 2 2 2

1 2 34 Slot 1

2 2 1 2

2 2 3 4

5 6 7 8 Slot 2

3 3 3 3 5 5 5 5 1 2 3 4 1 2 3 4 9 10 11 12 13 14 15 16

Slot 3

Node 2

Slot 5


I/O Group 0



23

Host Zones – Too few • Spectrum Virtualize - DH8

Host A1

B1

Fabric2 Edge1

Fabric1 Edge1

Non Preferred Paths

Preferred Paths


1 2 1 2

1 2 2 2

1 2 3 4 Slot 1

2 2 1 2

2 2 3 4

5 6 7 8 Slot 2

3 3 3 3 5 5 5 5 1 2 3 4 1 2 3 4 9 10 11 12 13 14 15 16

Slot 3

Node 1

Slot 5

1 2 1 2

1 2 2 2

1 2 34 Slot 1

2 2 1 2

2 2 3 4

5 6 7 8 Slot 2

3 3 3 3 5 5 5 5 1 2 3 4 1 2 3 4 9 10 11 12 13 14 15 16

Slot 3

Node 2

Slot 5


I/O Group 0



24

Quiz #1 - Right or Wrong?

Fabric_A Zone Definitions A1=10:00:00:00:c9:3f:75:37 A2=10:00:00:00:c9:3f:80:54 A3=10:00:00:00:c9:42:89:ac A4=10:00:00:00:c9:42:80:16

Fabric_B Zone Definitions B1=10:00:00:00:c9:3f:74:16 B2=10:00:00:00:c9:3f:f7:2f B3=10:00:00:00:c9:42:b4:4b B4=10:00:00:00:c9:42:b6:18

b03vio210

A1 A2 A3 A4 B1 B2 B3 B4

DIR2 SAN Fabric

DIR1 SAN Fabric

P1 P2 P3 P4 HBA 1

HBA 2

Node 1

P1 P2 P3 P4 P1 P2 P3 P4 HBA 1

HBA 2

Node 2

HBA 1

HBA 2

Node 3

SVC Host Definitions P1 P2 P3 P4 HBA 1

Node 4

iogrp 0

iogrp 1

2048 LUNs max

2048 LUNs max

wwpn 5005076801

HBA 2

4 Node SVC MAX Vdisk 4096


id:8 name:b03vio210_allvg 10000000C942B618 10000000C942B44B 10000000C9428016 10000000C94289AC 10000000C93FF72F 10000000C93F7416 10000000C93F8054 10000000C93F7537

In this example we show 1 SVC Host definitions and 2 Fabric zones, per host per Fabric. This does not allow proper failover from Primary to Alternate nodes within the SVC iogrp. See the next page for proper zoning


25

Right

b03vio210_1

A3

B2

A2

Fabric_B Zone Definitions B1=10:00:00:00:c9:3f:74:16 Fabric_A Zone Definitions A1=10:00:00:00:c9:3f:75:37

Fabric_B Zone Definitions B2=10:00:00:00:c9:3f:f7:2f

Fabric_A Zone Definitions A3=10:00:00:00:c9:42:89:ac

SVC Host Definitions

HBA 2

Node 1

P1 P2 P3 P4 P1 P2 P3 P4 HBA 1

HBA 2

Node 2

HBA 1

HBA 2

Node 3

P1 P2 P3 P4 HBA 1

iogrp 1

2048 LUNs max

2048 LUNs max

wwpn

HBA 2

Node 4

iogrp 0 5005076801

B4

Fabric_A Zone Definitions A4=10:00:00:00:c9:42:80:16

Fabric_B Zone Definitions B4=10:00:00:00:c9:42:b6:18

DIR2 SAN Fabric

DIR1 SAN Fabric

HBA 1

A4

B3

Fabric_B Zone Definitions B3=10:00:00:00:c9:42:b4:4b

Fabric_A Zone Definitions A2=10:00:00:00:c9:3f:80:54

P1 P2 P3 P4

b03vio210_4

b03vio210_3

b03vio210_2

B1

A1

id:1 name:b03vio210_1 10000000C93F7537 10000000C9428016 id:2 name:b03vio210_2 10000000C93F8054 10000000C93FF72F id:3 name:b03vio210_3 10000000C94289AC 10000000C942B44B

In this example we show 4 seperate SVC Host definitions and 2 seperate Fabric zones, per host per Fabric This does allow proper failover from Primary to Alternate nodes within the SVC iogrp.

4 Node SVC MAX Vdisk 4096


id:4 name:b03vio210_4 10000000C93F7416 10000000C942B618


26

Summary - 3 factors determine right/wrong paths

• Example of correct/incorrect number of datapaths with Host to Spectrum Virtualize • Correct datapathing has 3 factors • • •

Proper zoning Proper Spectrum Virtualize Host definitions (Spectrum Virtualize logical config of the host def) Proper redundancy for the Spectrum Virtualize preferred /non preferred pathing



27

Dual VIOS to Multiple LPARs Map LUNs within an LPAR in a round robin fashion to the active pseudo hosts. For example: • Vdisk1 to Pseudo1 – Vdisk2 to Pseudo2 – Vdisk3 to Pseudo1 – Vdisk4 to Pseudo2 VIO Server1

P1 FCA1 P2

P5

P3 FCA2 VFCA1.1 P4

P7

VIO Server2 FCA3

SAN

Pseudo1 VP7.1aVFCA

FCA4 VFCA5.1 P8 VP5.1a VP5.1i VFCA6.1

VP2.2i VP2.2a VFCA3.1

VP6.2a

VP3.1a

Spectrum Virtualize

VFCA4.1 VP4.2i

Avoiding this

Active Client Logical Partition (LPAR1)

VP5.1aVFCA Pseudo2

VP1.1i VP1.1a VFCA2.1

VP3.1i

VP1.1aVFCA

P6

VP4.2a

VFCA VP3.1a

VP6.2a

VFCA7.1 VP7.1a VP7.1i VFCA8.1 VP8.2a VP8.2i

VP2.2aVFCA

Pseudo1 VFCA VP8.2a VP6.2aVFCA I


Pseudo 2 VFCA VP4.2a

Fame1 Hypervisor



28

VIO Server1

FCA1 FCA2

P1 P2 P3

SAN

P4

P5 VIO Server2 FCA3 P6 P7 FCA4 P8

VP1.1i VP1.1a

VP5.1a

VP2.2i VP2.2a

VP6.2a VP6.2a

SVC

VP3.1i VP3.1a

VP5.1i

VP1.1aVFCA Pseudo1 VP7.1aVFCA VP5.1aVFCA Pseudo2 VP3.1aVFCA VP2.2aVFCA Pseudo1 VP8.2aVFCA

VFCA4.1VP7.1i VP7.1a VP8.2a VP8.2i

VP4.2i VP4.2a

I VFCA VP6.2a Pseudo 2 VP4.2aVFCA

inactive & active vWWPN pairs



LPM

Fame1 Hypervisor

.

P1 P2

P3 P4

SAN

•VFCA VFCA

P3 … P64

Fame2 Hypervisor

.

P1 P2

•VFCA VFCA

VP1.1i VFCA

I

P3 P4

SAN P3 … P64 I

LPM Could go to Frame2 or Frame3

Pseudo1 VP7.1i VFCA VP5.1i VFCA Pseudo2 VP3.1i

Active Client Logical Partition (Pseudo LPAR1b) During LPM

VFCA

During LPM the number of paths double from 4 to 8 Starting with 8 paths per vdisk will render an unsupported 16 paths during this time - could lead to IO interruption

Fame3 IBM Systems Technical Events | ibm.com/training/events Hypervisor


29

#2 - Port Placement for Best Practice IO spread and redundancy

Port_groups IBM Systems Technical Events | ibm.com/training/events

Spread across blades and port_groups © Copyright IBM Corporation 2016. Technical University/Symposia materials may not be reproduced in whole or in part without the prior written permission of IBM.

30

#2 – Incorrect Port Placement for IO spread and redundancy

All 24 ports on blade 2, are taken by the DS8K IBM Systems Technical Events | ibm.com/training/events


31

#3 – Slow Draining Devices • Slow Drain Explained • A Slow Drain Device is a device that does not accept frames at the rate generated by the source. In the presence of slow devices, Fibre Channel networks are likely to lack frame buffers, resulting in switch port credit starvation and potentially impacting Inter-Switch Links (ISLs) and/or port channels. • Slow Drain typically occurs when either a storage or end device is not receiving or sending frames fast enough to ensure that frames are flowing through the fabric • •

This creates a situation in which back pressure is felt behind the devices creating the slowness Symptoms are usually felt fabric wide

• **Slow drain can impact devices not directly involved in the problem



32

#3 - Slow Drain Causes • Edge devices •

Server performance problems •

•

Application or Operating System issues

Host bus adapter (HBA) problems •

•

driver or hardware issue

Speed mismatches •

•

one fast device and one slow device

Non-graceful virtual machine exit on a virtualized server •

•

resulting in frames held in HBA buffers

Storage subsystem performance problems •

•

including overload, runaway process, software bug

Physical Layer issues •

Failing SFP, Bad Optical Cable, Bad Panel

• Inter Switch Links (ISL) •

Lack of B2B credits for the distance the ISL is traversing •

• •

The existence of slow drain edge devices cause ISLs to stop moving frames Edge devices with faster speeds than ISLs even when port-channeled •

•

Example: 4 credits per KM @ 8Gbps

Example: 16Gbps Storage device traversing 8Gbps ISLs

Applies to port channels as well



33

#3 - Slow Drain Troubleshooting for Cisco • Use DCNM Host/Enclosure and pathtrace view to help understand end-to-end connectivity • Use Port-Mon alerts to target specific ports • Use DCNM Slow Drain analysis to find slow draining ports • Start tracing the ports through the fabric • •

If F-Port is the problem recover if not move to next upstream port Continue moving from each port to upstream port to find source

Manual steps to recover

• Identify the port involved and remedy • • • • •

Kill process Reboot server Fix optical cable or SFP Bounce or block port to clear buffers If speed mismatch move to upgrade slower links or makes faster links slower by forcing speed changes on the port

• Monitor that port with Port-mon to ensure reoccurrence does not happen IBM Systems Technical Events | ibm.com/training/events


34

#4 – Bad or Dirty SFP connection



35

#5 – Port Speeds – Best Practice of Fixed versus Auto Negotiate? • #1 Top Reason - Improper configuration changes or even worse “No” configuration changes Severity Why the error has occurred

Minor This error has occurred because, port speed on ISL / IFL are not set to fixed speed ie: A good example should look like this: 128 1 16 658000 id 4G Online E-Port 10:00:00:05:1e:36:05:b0 "" (downstream)(Trunk master) 129 1 17 658100 id 4G Online E-Port (Trunk port, master is Slot 1 Port 16 ) 130 1 18 658200 id 4G Online E-Port (Trunk port, master is Slot 1 Port 16 ) 131 1 19 658300 id 4G Online E-Port (Trunk port, master is Slot 1 Port 16 ) A bad example could be any of these: 48 4 0 653000 id N4 Online E-Port 10:00:00:05:1e:36:05:b0 "" (Trunk master) 49 4 1 653100 id N4 Online E-Port (Trunk port, master is Slot 4 Port 0 ) 50 4 2 653200 id N4 Online E-Port (Trunk port, master is Slot 4 Port 0 ) 51 4 3 653300 id N4 Online E-Port (Trunk port, master is Slot 4 Port 0 ) 60 4 12 653c00 id N4 Online E-Port (Trunk port, master is Slot 4 Port 13 ) Note : When ever you use the command "portcfgspeed" command, port will go offline and will come online, hence it is disruptive for that particular link. Implement the change in an appropriate time.

Potential Risks

Having ISL / IFL ports in "Auto Negotiate" mode switches will keep on check for the connectivity. Which will lead to both the switches to exchange the capabilities which may lead into principal switch polling

Actions to correct the Make sure that you set the port speeds of ISL / IFL on all switches to a fixed value. error IBM Systems Technical Events | ibm.com/training/events


36

#5 - Brocade Best practice to Trunk - on all Fibrechannel ISLs • #1 Top Reason - Improper configuration changes or even worse “No” configuration changes

Severity

Major

Why the error has occurred

One or more ISL does not have Trunking enabled.

Potential Risks

1. Running with a ISL without Trunking, ie. a single Fibre connection(s) between 2 SAN switch's there are some risks: • •

Single Point of Failure, causing Fabric Segmentation and loss of connectivity if the only (or last) connection between 2 switch's are lost Performance bottleneck, ISL Trunking is designed to significantly reduce traffic congestion in storage networks.

2. If there are 2 ISLs between the switches there are multiple scenarios why a trunk is not formed • • • • Actions to correct the error

Either there are no Trunking license The links are cabled in different ASICs in either end The difference in length of the cables are to long There are "noise" in one cable, could be a bad connector or patch panel, a too much bend cable etc.

Add more connections between this switch and the neighbor switch running with a single connection and/or purchase a Trunking License.



37

Bottlenecks in the Storage Design • The Top 5 common things that are or go wrong with Storage Best Practices



38

What causes THRASHING?

#1 -Data Layout/Placement

Placing Applications on the same LUNs/Pools result in IO contention

Most commonly when workloads peak at the same time or log files and data files share physical spindles made of strips on the outerstorage edge of the DDMs (1s) server also could have App A Raid-5 7+P monitoring tools to For existing LUN1 applications, use and performance understand current application workload characteristics such as:

•

1 1 Read/Write ratio 2 2

1 2

1 2

5 4 3

1 2

5 4 3

• • • • • •

5 5 4 4 Random/sequential 3 3

•

Remote mirroring link utilization and throughput

5 ratio3 4

1 2

5 4 3

1 2

5 4 3

1 2

5 4 3

Average transfer size (blocksize) Peak workload (I/Os per second for random access, and MB per second for sequential access) made of strips in the middle of the DDMs (3s) also could have App B Raid-5 7+P PeakLUN3 workload periods (time of day, time of month) Copy services requirements (Point-in-Time Copy, Remote Mirroring) Host connection utilization and throughput (HBA Host connections)

Extent pool or 8 Ranks

Strip1 Strip2 Strip3 Strip4 Strip5



39

39

Data Placement on Power Systems #1 - Data Layout Summary  Does data layout affect IO performance more than any tunable IO parameter?  Good data layout avoids dealing with disk hot spots – An ongoing management issue and cost  Data layout must be planned in advance – Changes are generally painful

 iostat might and filemon can show unbalanced IO  Best practice: evenly balance IOs across all physical disks unless TIERING

 Random IO best practice: – Spread IOs evenly across all physical disks unless dedicated resources are needed to isolate specific performance sensitive data • For disk subsystems

• Create RAID arrays of equal size and type • Create VGs with one LUN from every array • Spread all LVs across all PVs in the VG IBM Systems Technical Events | ibm.com/training/events


40

40

#1 - Storage Pools Easy Tier v3: Support for up to 3 Tiers • Support any combination of 1-3 tiers

Tier 0

Tier 1

Tier2

Flash/SSD

ENT

NL

Flash/SSD

ENT

NONE

Flash/SSD

NL

NONE

NONE

ENT

NL

Flash/SSD

NONE

NONE

NONE

ENT

NONE

NONE

NONE

NL



41

#1 - Storage Pools - “Example Only” Drive Selection in an Easy Tier Environment



42

Distributed Storage: Standard Building Block For MES to existing Storwize Virtual cluster or ESC designs

Intelligent ILM recommended w/auto-compression

Tunable based on historical or provided performance guidance Virtualization Layer

2145-DH8 SAN Volume Controller

TIER 0

FlashSystem 900 TIER 1 Using SVC EasyTier

XIV GEN3 15 module w/SSD Cache & FlashSystem 900

TIER 2 XIV GEN3 15 module w/SSD

TIER 3 Using SVC EasyTier

v5030 GEN2


1PB usable with 8 node cluster Can go higher with historical performance stats

Optional: Flash only pools are not recommended as you get bigger bang for buck with leveraging in mixed pools w/ET. If required by client need then you can use. Use subset of 10% in Flash only pool.

10% FlashSystem 900 (tier0) 50% XIV 2/3/4TB (tier1) (Drive size based on capacity needs) Optional: TIER2 pool with XIV only is optional. May be good fit for larger environments leveraging 4 or 6TB drives

40% v5030 Easy Tier Pools Up to six (6) 4TB NL-SAS RAID6 & one (1) 1.2TB 10k SAS arrays per pool (p 31) © Copyright IBM Corporation 2016. Technical University/Symposia materials may not be reproduced in whole or in part without the prior written permission of IBM.

43

#1 Storage Pools ‘Block’ Decision Tree Non-mission Critical data/appl/workload? Client acceptable to off premise Cloud use? No

Yes

Spectrum Accelerate on bare metal in Softlayer – See slides in backup (40-46)

Enterprise On Premise below

Application Highly Sensitive to IO Response Time?

Yes

Virtualization Needed?

No

See slide 2 for SAN strategy (DS8K, XIV)

Yes

No Virtualization Needed?

Yes

SVC 4 or 8 node Cluster Recommended

SVC 4 or 8 node Cluster Recommended

No Yes TIER2 needed?

See TIER2b for v5k Options or alternates

See slide 2/3 for SAN strategy (XIV, v5k)

No See Tier3 for v5k Options or alternates

See pages extra’s for Guidance on Tier Performance Levels If mission critical, SDS in Cloud is not approved



44

#2 - IO Spread and Load Balancing across CPU Complexes Even numbered extpools Primary IO Data flow ownership

Odd numbered extpools Primary IO Data flow ownership

Balance Just like an onion -virtualization has many layers

45

A


B

Arrays across Enclosures © Copyright IBM Corporation 2016. Technical University/Symposia materials may not be reproduced in whole or in part without the prior written permission of IBM.

45

45

#3 - IO Spread and Load Balancing across DA Pairs Quiz – Which has better throughput? Unbalanced I/O to DA Cards DA_0

DA_1

DA_2

DA_3

DA_4

DA_5

DA_6

DA_7

S1=Array_ 0 S3=Array_ 10 S5=Array_


44 S7=Array_ 52 S9=Array_ 18 S11=Array_


26 S13=Array_ 34

30 S14=Array_ 37

S15=Array _1 S17=Array_ 11 S19=Array_

S16=Array _6 S18=Array_ 15 S20=Array_



27 S27=Array_ 35 S29=Array_


40 S31=Array _2 S33=Array_














21 S59=Array_ 29

25 S60=Array_ 33

DA_0

DA_1

DA_2

DA_3

DA_4

DA_5

DA_6

DA_7


Balanced I/O to DA Cards DA_0

DA_1

DA_2

DA_3

DA_4

DA_5

DA_6

DA_7

S1=Array_ 0 S3=Array_ 2S5=Array_

S2=Array_ 1 S4=Array_ 3S6=Array_



10 S13=Array_ 12

11 S14=Array_ 13





















56 S59=Array_ 58

57 S60=Array_ 59

DA_0

DA_1

DA_2

DA_3

DA_4

DA_5

DA_6

DA_7


46

#4 - IO Spread and Load Balancing across Disk Enclosures

Array 1

Array 2 Array 3



47

#5 - IO spread and Load Balancing across ports Examples of correct Host to SVC Volume Balancing vdisk3

vdisk1

vdisk4

vdisk2

Preferred path for vdisk1 is SVC N1P2 & N1P3

Preferred path for vdisk2 is SVC N2P2 & N2P3

Non Preferred path for vdisk1 is SVC N2P2 &N2P3

Non Preferred path for vdisk2 is SVC N1P2 &N1P3

vdisk1

vdisk2

vdisk3

vdisk4



48

#5 - IO spread and Load Balancing across ports Quiz - Which has better throughput?

1

SAN

Fabric

SAN

Fabric

SAN

Fabric

SAN

Fabric

2



49

#5 - IO spread and Load Balancing across ports Quiz - XIV example – Right or Wrong?



50

Bottlenecks in the Host Design • The Top 5 common things that are or go wrong with Host Best Practices



51

#1 - Using the right Host Multipath Drivers/ Storage Microcode • #1 Top Reason - Improper configuration changes or even worse “No” configuration changes

Severity Why the error has occurred

Potential Risks

Actions to correct the error

Warning Microcode is below minimum targets • • • • • •

Lower levels of microcode susceptible to problems inherent to that level. If hardware support is engaged for any reason, the first recommended course of action will be to upgrade the microcode Lower code levels will prolong PD/PR activities Perform microcode update pursuant to Vendor specific support recommendation of level Test to verify server redundancy on each server attached to SAN prior to any concurrent microcode upgrades Coordinate microcode upgrades with storage Vendors, Account and Storage team SAs, CE's and Storage Admins should jointly monitor and coordinate microcode upgrades



52

#1 - Using the right Host Multipath Drivers/ Storage Microcode -Things that change - Create a script can show current status • For AIX VIO - driver used - sddpcm –

https://w3-connections.ibm.com/wikis/home?lang=enus#!/wiki/Global%20Server%20Management%20Distributed%20SL/page/Script%20to%20Capture%20SAN%20Path%20Info

– Multipath installed = Yes, Set to: 4 paths/ hdisk, fcsi_settings:2145: fast_fail, –

Multipath Policy =load_balance

• For Linux/ESX/VMWare –

https://w3-connections.ibm.com/files/app#/file/1ba027ec-5d20-4c60-a281-f18f16192f7a

– Device –mapper – multipath HBA elements=4,

– For Windows – driver used = MPIO=SDDDSM –

https://w3-connections.ibm.com/files/app#/file/3e52f54c-a445-4b17-aa5d-a5da43d4bedb

–

Multipath installed = Yes, HBA elements = 4, MPIO Policy = Optimized

• For Solaris – driver MPxIO •

Https://w3-connections.ibm.com/files/app#/file/66ea3228-4b26-48bd-a8fd-55751a02fc42

•

Multipath installed = MPxIO, Path Subscription= 4, MPIO Policy = round-robin

Content input from : Bill Marshall, Jason Moras, Brad Worthen, Ramesh Palakodeti 53



53

#1 - SDD Driver Testing for Proper HBA Failover • On an AIX vio server, check the AIX system to verify IO activity still continues on the alternate ports by testing with SDD and /or SDDPCM commands •

The Server Admin - Create a mount point for logical volume that can be manipulated to generate IO traffic for the purpose of this test The Server Admin - Verify and record selected (targets yet to be determined) datapaths for preferred and alternate status (active and inactive) by using the SDD "pcmpath query device" or "datapath query device" command on the AIX vio server Note the path selection counts on the multiple paths. There should only be two paths under the "Select" column, above zero (0). These are the two open paths on the preferred node. (if paths 0 and 2 show numbers under the "Select " column, other than zer0, then do the following:

•

•

1. 2.

3. 4. 5.

Take one path off-line by issuing the command (pcmpath set device 0 path 0 offline) or (datapath set device 0 path 0 offline) Path 0 should now be in a dead state . Go to the mount point of lv and edit a file to create traffic. After creating the traffic, reissue the pcmpath or datapath query command "pcmpath query device" or "datapath query device" and look at the path selection numbers. Notice only path selection count for Path 2 increased for the other preferred path Close Path 2 by issuing the command "pcmpath set device 0 path 2 offline" or "datapath set device 0 path 2 offline" Return to the mount point and add or edit files to create IO. Execute the "pcmpath query device" or "datapath query device“ command, to look at the path selection count. Disk access should now be via the other paths. (This is now load balancing to the nonpreferred SVC node for this Vdisk) Reestablish both preferred paths by executing the following commands: "pcmpath set device 0 path 0 online" and "pcmpath set device 0 path 2 online" or "datapath set device 0 path 0 online" and "datapath set device 0 path 2 online”



54

#1 - Non SDD Driver Testing for Proper HBA Failover •

Further "proper" testing should be done during a maintenance window •

Testing the redundancy between the Fabric and the host 1. 2.

Open a change record to reflect the change (Make sure all necessary approvers are notified) The Server Admin - Identify and verify which host HBA's are active for I/O activity by performing a test read and write to the SAN disk from the host 3. The Server Admin - Stop I/O between the host and the Disk Storage 4. The SAN Admin - On the San Fabric, disable the Switch port on the "even" fabric zoned between the host and the storage device. 5. The Server Admin - Perform another read/write test to the same LUN 6. The Server Admin - Identify and verify which host HBA is active for I/O activity 7. The SAN Admin - On the even SAN fabric enable the Switch port 8. The SAN Admin - On the San Fabric, disable the Switch port on the "odd" fabric zoned between the host and the storage device. 9. The Server Admin - Perform another read/write test to the same LUN 10. The Server Admin - Identify and verify which host HBA is active for I/O activity 11. The SAN Admin - the odd SAN fabric enable the Switch port

•

•

If the I/O activity toggles between the two HBA's then the test is successful

•

When a new Host server or Storage device is added to the environment testing is strongly recommended

Note: Ideally this type of test is best done during the initial implementation of new equipment, before it is turned over to the customer or placed in production



55

Data Placement on Power Systems #2 - Random IO Data layout 1

What does random LV creation order, help prevent? 1

2

2

3

4

5

3

datavg 4 5

# mklv lv1 –e x hdisk1 hdisk2 … hdisk5 # mklv lv2 –e x hdisk3 hdisk1 …. hdisk4 ….. Use a random order for the hdisks for each LV

RAID array LUN or logical disk PV Slide Provided by Dan Braden IBM Systems Technical Events | ibm.com/training/events


56

Data Placement on Power Systems #2 - Data Layout - OS Spreading versus Striping  Is there is a difference? What’s the diff?

– Do you know what your volumes made of!

File system spread



57

Data Placement on Power Systems #2 - Data Layout Summary  Does data layout affect IO performance more than any tunable IO parameter?  Good data layout avoids dealing with disk hot spots – An ongoing management issue and cost  Data layout must be planned in advance – Changes are generally painful

 iostat might and filemon can show unbalanced IO  Best practice: evenly balance IOs across all physical disks unless TIERING

 Random IO best practice: – Spread IOs evenly across all physical disks unless dedicated resources are needed to isolate specific performance sensitive data • For disk subsystems

• Create RAID arrays of equal size and type • Create VGs with one LUN from every array • Spread all LVs across all PVs in the VG IBM Systems Technical Events | ibm.com/training/events


58

58

#3 & 4- Tips – Most Common OS IO Tuning Parameters • Device Queue Depth – Queue Depth can help or hurt performance per LUN • aware Queueof Depth is central to the following fundamental performance • Be Queue Depth when planning system layout,formula: adjust only if necessary • •

• calculate IO Rate = Number of is Commands Response Time per Command To - best thing to do go to each *device “Information Center” URLs listed in link slide • For example: What are the default Queue Depths? ___

– IO Rate = 32 Commands per Second / .01 Seconds (10 milliseconds) per Command = 3200 IOPs Some real-world examples: • OS=Default Queue Depth= Expected IO Rate adapters

• HBA transfer rates – FC

• • • •

AIX Standalone = 16 per LUN = 1600 IOPs per LUN AIX VIOS = 20 per LUN = 2000 IOPs per LUN AIX VIOC = 3 per LUN = 300 IOPs per LUN Windows = 32 per Disk = 3200 IOPS per LUN

• LVM striping vs spreading • Data Placement – Random versus sequential • – Spreading versus Isolation

Content provided by Mark Chitti

What are the most common/Important OS I/O tuning parameters? IBM Systems Technical Events | ibm.com/training/events


59

#5 - Workload IO increases due to Data Growth • • • •

Should you ever stripe with previrtualized volumes? We recommend not striping or spreading in SVC, V7000 and XIV Storage Pools Avoid LVM spreading with any striped storage pool You can use file system striping with DS8000 storage pools • •

Across storage pools with a finer granularity stripe Within DS8000 storage pools but on separate spindles when volumes are created sequentially •

i.e (when the pools are created using the rotateext vs.rotatevol parameter

• Striped Pools

No Host Stripe

• Sequential Pools Host Stripe

S t r i p e

Host Stripe

Host Stripe - Raid-0 only



60

#5 - Workload IO increases due to Data Growth - careful tracking helps Data placement in Pools • On the host server using SDD- Mapping Virtual LUNS to Physical Disks DEV#: 81 DEVICE NAME: hdisk81 TYPE: 2145 SERIAL: 60050768019002F4A8000000000005C7

ALGORITHM:

Load Balance

====================================================================== Path# Adapter/Path Name State Mode Select Errors 0 fscsi0/path0 FAILED NORMAL 89154 2 1* fscsi0/path1 FAILED NORMAL 63 0 2 fscsi1/path2 OPEN NORMAL 34014 3 3* fscsi1/path3 OPEN NORMAL 77 0

LUN to Pool to Array

• Ask the StoAdmin to find disk/device UID or Raid-group in Storage Pool

StorAdmin cross-references Storage Pool UID with Controller’s Arrays in Pools



61

Summary • •

Knowing - what's inside will help you make informed decisions? You should make a list of the things you don’t know – Talk to the Storage Administrator or those who do know

•

A better Admin understands 1. 2. 3. 4.

• • – – – –

The backend physical makeup The backend virtual makeup What's in a Storage Pool for better data placement Avoids the Pitfalls associated with IO Tuning

5. Know where to go to get right multipathing device drivers 6. Knows why documentation matters 7. Keep Topology Diagrams 8. keep Disk Mapping documentation 9. Be able to use Storage Inquiry Tools to find answers 10. Understand how to troubleshoot storage performance bottlenecks



62

62

Questions-



63

Session Evaluations

YOUR OPINION MATTERS!

1

2

3

4

Submit four or more session evaluations by 5:30pm Wednesday to be eligible for drawings! *Winners will be notified Thursday morning. Prizes must be picked up at registration desk, during operating hours, by the conclusion of the event.



64

EXTRAS Use a troubleshoot method to divide the problem in half each time



65

Consistent Troubleshooting Methodology



66

#1 - Storage Pools Tiered Storage Classification

COST / PERFORMANCE / AVAILABILITY

TIER TIER0 – Flash Systems Preferred Solid State drives alternate

TIER1(a)

TIER1(b)

Description Ultra High Performance. Meet QoS for High End

Technical Examples – High level Guidance – Local variations on technology exist

IBM (block) SVC Recommended for Open

900 FlashSystem RAID5 -- Excellent ‘Turbo TIER1’ when coupled with XIV GEN3 (SSD cache) – -- V9000 not approved for use --

High Performance. Drive up utilization of high-ended storage subsystems and still maintain performance QoS objectives.

For low capacity requirements smaller less powerful devices may meet tier definition

Fibre Channel Only for block

DS8886 w/SAS 600GB 15K disk drive RAID5/RAID6 arrays *** -300GB 15K to be used only if DISK Magic shows need DS8884 can be used for smaller mainframe solutions – 50TB approx. demark -- RAID6 should be seriously considered for improved resiliency --- Mainframe only. See slide 5 & 6 for more detailed guidance

XIV* (GEN3) model 214 with 2TB SAS drives (11 Module or greater) XIV* (GEN3) model 214 with 3TB/4TB SAS drives (11 Module or greater) -- SSDs (solid state drives) required --- XIV GEN2 removed from strategy --- XIV is a lower cost option. DS8 recommended for mainframe. Exception based only for distributed envs. --


Performance range capability

DS8886 -> Greater than 250,000 IOPs/5500+ MBs 900 FlashSystem  Greater than 500,000 IOPs mixed work load (70/30). Greater than 14000 IOPs/TB (RAID 5) DS8886 -> 200,000+ IOPs / 5000 MBs • Greater than 600 IOPs/TB in a full 1536 drives configuration RAID 5 • Greater than 1200 IOPs/TB in a full 1536 drives configuration RAID 1 FAS8040 -> 1024 IOPS/TB

V9000 -> TBD XIV 2TB XIV (GEN3 15 mod) less than 130,000 IOPs/3,400 MBs Around 1000 IOPs/TB XIV (GEN3 11 mod) less than 95,000 IOPs/3,200 MBs Less than 1000 IOPs/TB XIV 3TB XIV (GEN3 15 mod) less than 120,000 IOPs/3,400 MBs Less than 700 IOPs/TB XIV (GEN3 11 mod) less than 85,000 IOPs/3,200 MBs Less than 500 IOPs/TB


67

RAID5 not recommended on drives greater than 900GB

Tier rating is based on performance AND reliability

#1 - Storage Pools Tiered Storage Classification


TIER

Description

Medium Performance. Meet QoS for applications/data that resides here.

TIER2

For low capacity requirements smaller less powerful devices may meet tier definition Fibre Channel Only for block

Technical Examples – High level Guidance – Local variations on technology exist IBM (block) SVC Recommended for Open/DS8K Recommended Direct Attached (NAS) strategy Performance range capability For Mainframe DS8886 w/SAS 600GB 15k disk drive RAID5/RAID10 arrays -Mainframe only. See slide 5 & 6 for more detailed guidance v5030 w/SAS 600GB 15K using RAID5 and SSD Easy Tier See slide 29/30/31 -- V5020 acceptable for 3rd location for Quorum disk for ESC w/600GB SAS or for small remote locations -- V5010 not approved -- V7000 remove from strategy

FAS8040 3TB SATA 1TB FlashCache/Node Increments - Min Controllers - 2 - Min Disks/Shelves – 2xDS4246 1 p/node - Max shelves per node before new node – 13 FAS2552 – remote office use 1TB SATA Min Controllers – 2 Min Disk/Shelves – 2 Max Shelves - full

DS8886 -> less than 80,000 IOPs or less than 3,000 MBs • Less than 600 IOPs/TB in a full 1536 drives configuration RAID 5 • Less than 1200 IOPs/TB in a full 1536 drives configuration RAID 1

v5030 Gen2 (block) -> less than 75,000 IOPs or less than 2,500 MBs •Less than 250 IOPs/TB in a full 1056 drives configuration RAID 5 •Less than 450 IOPs/TB in a full 1056 drives configuration RAID 1 FAS8040 -> 256 IOPS/TB FAS2552 -> 256 IOPS/TB

SoNAS & V7000U not supported

Medium performance/Clou d and Commodity Storage TIER2b Fibre Channel Only for block minus SDS. SDS will be iSCSI

SDS Block Cloud Spectrum Accelerate. See best practices (and slide later in deck). 50TB min. If smaller needed use Endurance or Performance offering from SL. XIV* (GEN3) model 214 with 6TB SAS drives (11 Module or greater unit). 800 GB SSDs required for CACHE


File Interface SDS File Offering Quantastor – 800GB SSD drives running RAID5. See Quantastor best practices.

Object Interface SwiftStack SoftLayer Object– Workload and storage both in SoftLayer File + Object Interface Swiftstack w/TwinStrata SoftLayer NAS and Object – Workload and storage in SoftLayer

XIV -> less than 100,000 IOPs or less than 2,500 MBs Less than 400 IOPs/TB (XIV Gen3 4TB 11 mod) QS SDS Block -> less than 50,000 IOPs or less than 1,000 MBs • Less than 2000 IOPs/TB in a full 34 drives configuration RAID 5 • Less than 3000 IOPs/TB in a full 34 drives configuration RAID 1 SA SDS Block -




68

#1- Storage Pools Tiered Storage Classification TIER

Description


Low Performance. Meet QoS for applications/data that resides here.

TIER3

Fibre Channel Only for block

Low Performance. Meet QoS for applications/data that resides here. TIER3b Fibre Channel Only for block minus SDS. SDS will be iSCSI

Technical Examples – High level Guidance – Local variations on technology exist IBM (block) SVC Direct Attached (NAS) strategy Performance range capability Recommended for Open FAS8040 DS8886 with NL-SAS tech DS8886 -> less than 25,000 IOPs or 3TB SATA using RAID6 when customer less than 1,000 MBs NO FlashCache already has DS8K with room • Less than 120 IOPs/TB in a full 1536 for this storage -- Mainframe Increments drives configuration RAID 5 only. See slide 5 & 6 for • Less than 220 IOPs/TB in a full 1536 -Min Controllers – 2 more detailed guidance drives configuration RAID 1 -Min Disks/Shelves – 2xDS4246 1 p/node v5030 model with 4 TB, or 6 v5030 (block) -> less than 30,000 -Max shelves per node before new TB NL-SAS using RAID-6, IOPs or less than 300 MBs node – 50 gen2 only. Drive size •Less than 70 IOPs/TB in a full 1056 drives configuration RAID 5 selection based on I/O FAS2520 – remote office use •Less than 140 IOPs/TB in a full 1056 density and capacity drives configuration RAID 1 3TB SATA requirements. Include 1.2TB drives and use Min Controllers – 2 Min Disk/Shelves – 2 SVC EasyTier for this TIER3 FAS8040 -> 64 IOPS/TB Max Shelves - full FAS2520 -> 64 IOPS/TB pool See slide 29/30/31 SoNAS & V7000U not supported

SDS Block Cloud Spectrum Accelerate. See best practices (and slide later in deck). 50TB min. If smaller needed use Endurance or Performance offering from SL.

SDS File Offering Quantastor – 4TB or 6TB drives running RAID10. See Quantastor best practices. Low end use only.

SDS & File Block: up to 35K and 280 MB/sec and when configured as specified

SDS Object Offering SwiftStack

SDS Object: Performance is high configuration dependent and measured in GETs and PUTs.

SoftLayer Shared Object– Workload and storage both in SoftLayer Object + Gateway Swiftstack w/TwinStrata SoftLayer Shared NAS and Object – Workload and storage in SoftLayer

TIER4

Archival, long term retention, backup

Virtual Engines, Tape ATLs, ProtecTIER


Less than 25,000 IOPs or less than 300MB/sec • Less than 200 IOPs/TB in a full 34 drives configuration RAID 5 Quantastor 4 TB • Less than 300 IOPs/TB in a full 34 drives configuration RAID 1 Quantastor 4 TB



N/A Tier based on features. © Copyright IBM Corporation 2016. Technical University/Symposia materials may not be reproduced in whole or in part without the prior written permission of IBM.

69

Spectrum Virtualize (SVC) ESC – Best Practice Use Case example

Host Server

Host Server

Clustered Servers where applicable

Node 1

Mission Critical vols

Node 2 Example 1 Stretched I/O Group

SVC Cluster 3 SVC Cluster 0 Site-1

Storage Pool 1

Site-2

SVC Global Mirror

Node 1

Node 2

ASYNC • Quorum

DR Site

Storage Pool 2 Storage Pool 3 B COPY C COPY

Mission Critical Block for Applications HA Storage: can sustain .3-1 ms latency hit) IBM Systems Technical Events | ibm.com/training/events

DR, Dev, Test, QA .. etc. © Copyright IBM Corporation 2016. Technical University/Symposia materials may not be reproduced in whole or in part without the prior written permission of IBM.

70

Spectrum Virtualize (SVC) ESC – Site-1 & Site-2 current building block example Example “Use Case” Scenario description: Public VSAN

Public VSAN

3X / node

• Nodes are stretched across sites to make an SVC Enhanced Stretch Cluster (ESC) for High Availability (HA) application requirements

3X / node

Private VSAN

Private VSAN SVC DH8/4nodes / 3HBAs @ 8Gb

Cisco MDS 9710 Core

2X

2X

• A Core Private Cisco VSAN is configured for Node to Node Communication port connections

2X

2X

Node to node

• A Core Public enterprise class Cisco VSAN is configured for Host/Storage port connections

Cisco MDS 9710 Core

2X

2X

• 12 Port Nodes are used with 4 ports each at 8Gb port speeds

Node to node 2X

2X

• Edge Switches per port count requirements are used for a true BP Core-Edge design to allow future growth scalability

GM 4X/Flash, 6X/XIV, 4X/V7K

4X/Flash, 6X/XIV, 4X/V7K

GM 4x ISL / Edge

4x ISL / Edge

• Storage building blocks of Flash, XIV and V7000 are used for a 3 tier or mixed tier approach 9396 Edge x 3

9396 Edge x 3

Host Servers

1X / host

• Separate Storage zones are created for each device providing isolated zones from each device type • Host Servers are connected to the Edge Switches

1X / host

• Edge Switches are ISLed to the Core Public VSANs Servers 1X / SVC Node

1X / SVC Node

FCIP Router 9250i FCIP Router 9250i IBM Systems Technical Events | ibm.com/training/events

• FCIP Routers are used for GM to the DR site. SVC GM ports are plugged directly into the FCIP Routers © Copyright IBM Corporation 2016. Technical University/Symposia materials may not be reproduced in whole or in part without the prior written permission of IBM.

71

Spectrum Virtualize (SVC) ESC – DR site current building block example 16X (2x/ SVC Node)

FCIP Router 9250i

GM

GM

Example “Use Case” Scenario description:

FCIP Router 9250i

Cisco MDS 9710 Core

Cisco MDS 9710 Core

Public VSAN Private VSAN

3x / nod e

3x / nod e

2 X

Public VSAN Private VSAN

• A Core Private Cisco VSAN is configured for Node to Node Communication port connections

SVC DH8 2 X

2 X

• 12 Port Nodes are used with 4 ports each at 8Gb port speeds

2 X 2 X

2 X 2 X

2 X

2 X 2 X 2 X 2 X

2 X

• Edge Switches per port count requirements are used for a true BP Core-Edge design to allow future growth scalability

2 X 2 X 2 X

4x/Flash, 6x/XIV, 2x/V7K

• Storage building blocks of Flash, XIV and V7000 are used for a 3 tier or mixed tier approach • Separate Storage zones are created for each device providing isolated zones from each device type

4x/Flash, 6x/XIV, 2x/V7K

8x ISL / Edge

8x ISL / Edge

Quorum 9396 Edge x 3

9396 Edge x 3

• A Core Public enterprise class Cisco VSAN is configured for Host/Storage port connections

• Host Servers are connected to the Edge Switches • Edge Switches are ISLed to the Core Public VSANs

??? Servers 1X / host

1X / host

IBM Systems Technical Events | ibm.com/training/events Servers

• FCIP Routers are used for GM to the DR site. SVC GM ports are plugged directly into the FCIP Routers © Copyright IBM Corporation 2016. Technical University/Symposia materials may not be reproduced in whole or in part without the prior written permission of IBM.

72

Spectrum Virtualize (SVC) ESC site-1 & site-2 synchronous with DR asynchronous Putting it all together in the example template 2X @ 16Gb/s LW CWDM SFP 2X @ 16Gb/s LW CWDM SFP

Site1

Site2 FCIP GM Priv Dark Fibre

Pub Dark Fibre

4X 4X

4X

4X 2X @ 16Gb/s CWDM LW SFP 2X @ 16Gb/s CWDM LW SFP

DR Site



73

Continue growing your IBM skills

ibm.com/training provides a comprehensive portfolio of skills and career accelerators that are designed to meet all your training needs.

If you can’t find the training that is right for you with our Global Training Providers, we can help. Contact IBM Training at [email protected]

Global Skills Initiative



74

Best Practice for Connecting IBM Storage to Power Systems-2016.pdf

Recommend Documents