Configuring SAN Storage on VIO server and Assigning to Client LPARS as VSCI LUNS Scope of this Document:
This document covers the steps involved and best practices for allocation of SAN LUNs from VIO server to client LPARs using VSCSI. This document does no t cover assigning LUNs using NPIV and the general best practices for VIO server configuration and maintenance . 1.0 Configuration on the VIO Server 1.1 Set the Fiber channel SCSI Attributes for Each Fiber Channel Adapter
Enable 'dynamic tracking' and 'fast fail' for high availability and quick failover for each fiber channel adapter. Example: As 'padmin' # chdev -dev fscsi0 -attr fc_err_recov=fast_fail dyntrk=yes -perm
fscsi0 changed Note: This should be run for all the adapters in 'lsdev -Cc adapter |grep fcs' output. # lsdev -dev fscsi0 -attr
attribute value description user_settable attach switch How this adapter is CONNECTED False dyntrk yes dyntrk yes Dynamic Tracking of FC Devices True
f {}c_err_recovfast_fail FC Fabric Event Error RECOVERY Policy True scsi_id 0x6f0058 Adapter SCSI ID False sw_fc_class 3 FC Class for Fabric True Note: Reboot of VIO server is required for the 'dyntrk' and 'fc_err_recov' parameters to take into effect. 1.2 Prepare the VIO server for Configuring the SAN Storage
Prepare the VIO server to configure the SAN storage at OS level. The steps involved would be same as AIX stand alone host . High level steps are: 1.2.1 Install ODM file sets
Install the necessary the AIX ODM file sets, host attachment scripts, this software would be supplied by the storage vendor, i.e., IBM, EMC, Hitachi etc. 1.2.2 Install Multipath Software
Install the Multipath software i.e., SDD PCM, EMC Power path, Hitachi HDLM etc as required. 1.2.3 Validate that multi pathing is operating normally
Verify that multipaths are active using the path query commands Example: For SDDPCM # pcmpath query adapter
For EMC Power Path # powermt display
For HDLM# dlnkmgr view -path 1.3 Logical disk configuration
Set the reserve policy on the logical disk that would be used to create virtual devices, i.e., hdisk, vpath, hdiskpower etc. The logical disk that would be used to create a virtual device would depend upon the multipath software. For example if IBM SDD is being used as multipath software it would be 'vpath', IBM SDD PCM is being used as multipath software it would be 'hdisk', whereas using EMC Powerpath it would be 'hdiskpower'. 1.3.1 Assign pvid and set the 'reserve policy' on the AIX logical disk Example: For SDD PCM, or Hitachi HDLM PCM versions:
As 'padmin' # chdev -dev hdiskx -attr pv=yes
# chdev -dev hdisk -attr reserve_policy=no For EMC Power path devices:
# chdev -dev hdiskpower0 -attr pv=yes # chdev -dev hdiskpower0 -attr reserve_lock=no Note: Setting reserve policy and enabling pvid should be performed on all the VIO servers in the frame,. 1.3.2 Queue depth setting for logical devices on VIO server
Check the queue depth settings on the logical devices that would be used to create the virtual devices i.e., vpath, hdisk, hdispower etc. IBM and EMC ODM file sets the queue depth value to a reasonable value and does not require initial tuning. For example, IBM storage ODM file sets the queue depths to 20, EMC ODM file sets it to 32, However Hitachi ODM configuration sets the queue depth to 2. Queue-depth of 2 is very low and and it is recommended to work with SAN team/storage support vendor support to change the initial setting of queue depth to a reasonable value. In case of Hitachi devices start with a value in the range of 8 to 16. Example:
In the following example please make a note of 'queue_depth' and 'rw_timeout' lsdev -dev hdiskpower1 -attr
attribute value description user_settable clr_q yes Clear Queue (RS/6000) True location Location True lun_id 0x1000000000000 LUN ID False lun_reset_spt yes FC Forced Open LUN True max_coalesce 0x100000 Maximum coalesce size True max_transfer 0x100000 Maximum transfer size True pvid 00f6095a2c486cec0000000000000000 Physical volume identifier False pvid_takeover yes Takeover PVIDs from hdisks True q_err no Use QERR bit True
q_type simple Queue TYPE False queue_depth 32 Queue DEPTH True
reassign_to 120 REASSIGN time out value True reserve_lock no Reserve device on open True rw_timeout 30 READ/WRITE time out True
scsi_id 0x710008 SCSI ID False start_timeout 60 START unit time out True ww_name 0x50001441001bcff1 World Wide Name False lsdev -dev hdisk32 -attr
attribute value description user_settable PCM PCM/friend/hitachifcp N/A True PR_key_value 0x104f967d6 Reserve Key True algorithm round_robin N/A False clr_q no Device CLEARS its Queue on error True dvc_support N/A False location Location Label True lun_id 0x1c000000000000 Logical Unit Number ID False max_transfer 0x40000 Maximum TRANSFER Size True node_name 0x50060e8005722303 FC Node Name False pvid 0004f967fea332590000000000000000 Physical Volume ID False q_err yes Use QERR bit False q_type simple Queue TYPE True queue_depth 8 Queue DEPTH True
reassign_to 120 REASSIGN time out True reserve_policy no_reserve Reserve Policy True r {}w_timeout 60 READ/WRITE time out True scsi_id 0x770000 SCSI ID False start_timeout 60 START UNIT time out True ww_name 0x50060e8005722303 FC World Wide Name False 1.4 Create Virtual devices (assign LUNs to VIO client LPARS)
Map the vhosts that belong to each client lpar, these are the vhosts that would be used to allocate LUNs to client lpars. Example: lsdev -slots and lsmap -all |grep vhost would h elp determine the vhosts available and their mapping to clients # lsmap -all |grep vhost vhost0 U8233.E8B.10095AP-V2-C2 0x00000004 vhost1 U8233.E8B.10095AP-V2-C3 0x00000005 vhost2 U8233.E8B.10095AP-V2-C4 0x00000004 vhost3 U8233.E8B.10095AP-V2-C5 0x00000005 vhost4 U8233.E8B.10095AP-V2-C6 0x00000004 vhost5 U8233.E8B.10095AP-V2-C7 0x00000005 vhost6 U8233.E8B.10095AP-V2-C8 0x00000004 vhost7 U8233.E8B.10095AP-V2-C9 0x00000005 vhost8 U8233.E8B.10095AP-V2-C10 0x00000004 vhost9 U8233.E8B.10095AP-V2-C11 0x00000005 If needed use 'lshwres' command on HMC To create a mapping between the client and server LPAR virtual adapters.
Example: # lshwres -r virtualio --rsubtype scsi --m Server-9117-570-SN6539DEC --filter lpar_ids=04
Where 04 is the lpar id of the client lpar Once the vhost is identified based on the mapping, create the virtual device as shown below: In the following example the LUN is being assigned to vhost2 (lpar id 004) and the logical disk is hdiskpower0. The naming convention for the device (-dev) is client lpar name plus the AIX logical disk name. The naming convention could be site specific and the same convention should be used consistently on all the VIOS mkvdev -vdev hdiskpower0 -vadapter vhost2 -dev dbp02_hdpowr0 lsmap -all |grep hdiskpower0 to verify that virtual device is created
Run the mkvdev command on the second VIO server also , which would provide the second path to the disk from the client Note: To identify the logical disk (hdisk, vpath, hdiskpower etc) that matches with a LUN serial number from storage use the respective multipath software commands or lscfg commands. Example: pcmapth query device |grep 020B powermt display dev=all |grep 02C
Where 02C and 02B are LUN serial number's last three alphanumeric characters Alternatively generate an output file using the ab ove commands so that multiple serial numbers can be mapped without having to run the pcmpath, powermt , dlnkmgr commands for every disk. Note: The number of LUNs that can be mapped to a vhost
The total number of LUNs (disks) that can be mapped to a vhost depends on the queue depth of the SAN LUN. The best practice is to set the queue depth on the virtual disk (client) same as the physical disk (VIOS). The maximum number of LUNs per vhost (virtual adapter) = (512-2)/ (3+queue depth)
From the examples above, For EMC storage with queue depth of 32 the maximum number of disks per virtual adapter would be (512-2)/(3+32) = 14 For IBM storage where the queue depth is typically set at 20, the maximum number of disks per virtual adapter would be (512-2)/(3+20) = 22 For Hitachi storage with a typical queue depth value of 16, the maximum number of disks per virtual adapter would be (512-2)/(3+16) = 26 Note: If needed create additional server and client adapters for allocating the SAN LUNs. 2.0 Configuration on Client LPAR 2.1 Set the Attributes for each Virtual SCSI Adapter
Set the 'vscsi_path_to' and 'vscsi_err_recov' to the values shown below for each vscsi adapter on the client lpar. # chdev -l vscsi1 -a vscsi_path_to=30 -P # chdev -l vscsi1 -a vscsi_err_recov=fast_fail -P
Note: vscsi_err_recov cannot be modified on older AIX 5.3 TLs and VIOS 2.x is needed for it to work. # lsattr -El vscsi1
vscsi_err_recov fast_fail N/A True vscsi_path_to 30 Virtual SCSI Path Timeout True Reboot of Client LPAR is required to for the above parameters to take into effect. 2.2 Configure VSCSI disks and Set Device Characteristics on the Client LPARs 2.2.1 Run 'cfgmgr'
run 'cfgmgr ' and verify that the disks assingned to vhosts on VIOS appear on the client lpars, use ' lspv |grep pvid ' to match the disks from VIOS to client lpars, also c heck UDID using the following command on both client and server. Not all multipath software and versions would use UDID method to identify the devices. # odmget -q attribute=unique_id CuAt (first and last numbers will be different from VIOS and client)
on the newer versions of VIO the following command can also be used to query the unique id # chkdev -dev hdiskx -verbose 2.2.2 Verify multipath to disks at Client LPAR
Verify that two paths are enabled to each vscsi disk (one from each vio server) lspath should show two paths to each vscsi disk (hdisk) on the client LPAR. # lspath |grep hdisk16
Enabled hdisk16 vscsi4 Enabled hdisk16 vscsi5 2.2.3 Set 'health_check' interval and 'queue depth' for disks on Client LPAR
Once the hdisks get configured on the client lpars, set the following device characteristics hcheck_interval queue_depth Set the hcheck_interval to at least equal to or smaller than the' rw_timeout' value of the SAN LUN on VIO server. For IBM and Hitachi Storage the rw-timeout value is typically 60 so hcheck_interval can be set to 60 or less. Check the 'rw_timeout' value from VIOS using 'lsattr' as shown above in 1.3.2 Example: # chdev -l hdisk2 -a hcheck_interval=50 -P
For EMC devices rw-timeout value is 30, so set the hecheck_interval to a value equal to or less than 30 # chdev -l hdisk2 -a hecheck_interval=20 -P
Set the queue depth value to the same value on the physical device on the VIOS Example: if the queue depth value on the VIOS is 20 # chdev -l hdisk16 -a queue_depth=20 -P
If the queue depth value on the VIOS is 32
# chdev -l hdisk16 -a queue_depth=32 -P
The values set using -P option would require a reboot of client lpar. 2.2.4 Path Priority settings
Path priority settings would facilitate load balancing across the VIO servers. There are two approaches to load balance from client lpars to VIO server lpars, in the first approach set the path priority for all the disks from one client lpar to one VIO sever, from the second client lpar to the second VIO server and so on. In the second approach set the path priority for some disks to first VIO lpar and for other disks to second VIO lpar, i.e., use all VIO servers from all client lpars. The first approach i.e., using one VIO server from one client LPAR would be easier to administer and provides load balancing. Example for path priority setting: # chpath -l hdisk16 -p vscsi4 -a priority=2 ( to set the path priority) # lspath -AHE -l hdisk16 -p vscsi4 (to verify the path priority)
attribute value description user_settable priority 2 Priority True 3.0 MPIO/VIO Testing
After the configuration steps on VIO server and client lpars have been completed, perform MPIO failover testing by shutting down one VIO server at a time. The path on client LPAR should failover to the second path, 'lspath' and 'errpt' could be used to validate the path failover. The volume groups should remain active while the VIOS is down. Start the VIO server, once the VIO server is up, the failed path on the client LPAR should go to 'Enabled' state.
