PROCEDURE TO LOW LEVEL FORMAT THE PASSIVE NODE CONDITIONS: WARNING: Do not perform this procedure unless instructed to by the next level of support. It is a dangerous procedure. WARNING: This procedure should be used in very rare cases, do not make the use of this procedure a common place thing. WARNING: When data disks are experiencing problems, any attempt to repair problems could escalate to a complete outage. Be prepared to handle an outage situation. WARNING: When removing and plugging in the SCSI cables, there is a risk that some of the disks on working node can fail or develop some media errors. WARNING: Do not power off the node during the formatting, that will most likely result in a dead drive. This procedure is dangerous to the data disks and the node. If a drive is dead, the node will have to be exchanged. WARNING: Do not reboot the node during the formatting, that could cause problems with the drives. If a failed drive is reported due to an unexpected reboot during this procedure, a node power cycle might he lp clear the situation. DO NOT POWER OFF A NODE DURING A FORMAT OPERATION. ATTENTION: ATTENTION: Make sure some spare spare nodes are available available in case of troubles. troubles. ATTENTION: ATTENTION: Make sure valid APG backups are available available on tape or stored stored on external system. system. ATTENTION: ATTENTION: Data disk configuration configuration info MAY MAY be lost by performing performing this procedure. procedure. Make sure a DT is available available to recreate the APG40 application configuration. ATTENTION: ATTENTION: Torx #8 screwdriver screwdriver will be needed to disconnect disconnect the APG40s cables. cables. Please have have the appropriate tools tools on hand before starting the procedure.
1. APG40 data disks are experiencing a lot of problems (e.g. medium errors, failed disk, failed RAID). 2. A low-level data disk format will be performed to attempt to solve the data disk problems. 3. This low-level format will be performed on one node at a time. 4. One node will always be running the cluster. 5. This procedure can be used to format all three disks in one node at the same time. 6. Low-level data disk format is done in parallel, so all the disks are formatting at the same time. This keeps the procedure length to a minimum. 7. This procedure is specific for C/4 hardware. It is not appropriate for C/2 hardware. Command: hwver
PROCEDURE: 1. Please consult with the next level of support before performing this procedure. Decide with the next level of support which nodes should be low-level formatted and in which order the nodes will be formatted.
2. Connect to the first node that will have the three data disks low-level formatted. Connect locally with a Monitor/Keyboard/Mouse. Login with an administrative account and open a CMD window. 3. Stop the Cluster Service and associated services. Commands: sc config acs_fch_server start= demand sc config acs_prc_clustercontrol start= demand sc config clussvc start= demand net stop acs_fch_server net stop acs_prc_clustercontrol net stop clussvc
4. Connect to the active node. 5. Perform a quick health check to make sure all is still as expected. 6. From the active node, power off the node that will be low-level formatted. Respond "y" if prompted. Command: fcc_save_to_remove other -f
7. When the MIA light is lit, remove the power from the node. 8. Disconnect the SCSI cables in the following order: ATTENTION: Always disconnect the SCSI connectors of the active node before removing the SCSI connectors of the powered-down node. -
Disconnect Disconnect Disconnect Disconnect
the the the the
upper lower upper lower
SCSI SCSI SCSI SCSI
connector connector connector connector
on on on on
the the the the
active node active node powered off node powered off node.
9. Power up the node by reconnecting the power cable. 10. Enter the LSI MegaRAID SCSI BIOS by pressing
when “Press to run MegaRAID Configuration Utility” message appears on the screen. 11. From Management Menu select Objects -> Adapter -> Clear Configuration. Select "Yes" to clear the Configuration. 12. Go back to Management Menu by pressing . Press to exit and confirm the Exit by selecting "Yes". Reboot the node by pressing . 13. After the reboot, enter again in the LSI MegaRAID SCSI BIOS by pressing when “Press to run MegaRAID Configuration Utility” message appears on the screen. 13.1. From Management Menu select Objects -> Adapter -> Fast Initialization. Set Fast initialization Mode to OFF. Answer YES to change the parameter. 13. 2. Go back to Management Menu by pressing . Press to exit and confirm the Exit by selecting "Yes". Reboot the node by pr essing . 13.3. After the reboot, enter again in the LSI MegaRAID SCSI BIOS by pressing when “Press to run MegaRAID Configuration Utility” message appears on the screen. 14. From Management Menu select Format
15. Select the disks that need to be formatted. In case you have earlier identified the disk which need to be formatted, select it by pressing . Move using /. Disk is selected when it starts to blink. In case all the disks need to be formatted, select all three disks. 16. When the disks that need to be formatted are selected, press to start formatting. Answer YES when asked for confirmation. 17. Formatting will start. Status of the disk will be changed to FRMTx where x is the disk number. Wait until formatting is completed. The formatting of the three disks in will take approximately 1 hour 45 minutes. 18. In case formatting was successful, the text "Formatting physical drive" will change to "Physical Drives are Formatted Successfully. Press Any key To Continue" press any key. (The blinking will continue until you have pre ssed any key.) In case formatting fails or takes more than 4 hours, replace the node by following the "Node Change OPI". 19. Status of the disks should be READY. 20. Leave LSI MegaRAID SCSI BIOS by pressing twice. Confirm Exit by selecting "Yes". Reboot node by pressing . 21. After the reboot, enter again in the LSI MegaRAID SCSI BIOS by pressing when “Press to run MegaRAID Configuration Utility” message appears on the screen. 22. From Management Menu select Objects->Adapter and press . 23. Make sure that "Enable BIOS" option appears in the menu ( it means that BIOS is currently DISABLED ). Only if "Disable BIOS" option appears, select it ( this will Disable BIOS). Move using / and press . Confirm the change by pressing “Yes”. 24. Set Fast Initialization to “On”. Move using / and press . Set the value to “On” and press again . Confirm the change by pressing “Yes”. 25. Select "Other Adapter Options" and press . Make sure that "Cluster Mode" is set to ENABLED and "Initiator ID" is set to 7 for Node A. Note: For Node B, "Cluster Mode" must also be set to ENABLED and "Initiator ID" must be set to 6. 25.1. In case you are prompted to reboot the node afte changing the SCSI Initiator ID, do so by pressing and enter again in the LSI MegaRAID SCSI BIOS by pressing when “Press to run MegaRAID Configuration Utility” message appears on the screen. Recheck that settings are correct. 26. Go back to Management Menu by pressing Leave LSI MegaRAID SCSI BIOS by pressing again. Confirm Exit by selecting "Yes", Then reboot the node with .
27. Wait for the node to start up.
Note: 1) If node does not reboot then r emove power cable and put it back to reboot the node. 2) If you receive a message saying there is an unresolved mismatch between disk(s) and NVRAM on the adapter and prompted to "Press A Key to Run Configuration Utility",enter the Configuration Utility and exit with ESC, then press Ctrl+Alt+Del to reboot the node again.
28. Start the Cluster Service and associated services.
ATTENTION: Some of the services may already be started, that is OK.
Commands: sc config acs_fch_server start= auto sc config acs_prc_clustercontrol start= auto sc config clussvc start= auto net start clussvc net start acs_fch_server net start acs_prc_clustercontrol 29. Perform a health check to make sure the node is passive. Login to the active node. 1.
Verify that no major alarms are present. AP Command: alist
2.
Verify that all resource are online. AP Command: cluster res cluster res|findstr -ive online
3.
Use command "megarc" as shown below. All Physical Target entries must be reported as ONLINE. AP Command: megarc -ldinfo -a0 –lall
4.
Verify that charging data is being received from the CP into the APG40 Message Store, using AP command "msdls" as shown below. The lastMessNo should be increasing to indicate charging data is being received from the CP. The lastAckMess should be increasing to indicate charging data is being fetched by the billing subsystem (RDT,FOS or RTR). The lastMessNo should be greater than or equal to the lastAckMess. The msdls command should be repeated several times to confirm the lastMessNo is increasing. GSM, WCDMA and Wireline AP Command: msdls -m CHS -s cp0ex
CDMA AP Command: msdls -m CHSMOB -s cp0ex
5.
Verify that charging data is being processed by the billing subsystem. RTR AP Command: rtrls -s
FOS AP Command: fosview -s
RDT AP Command: rdtview -s
6.
Verify that charging is being transferred to the billing centre. RTR AP Command: afpls -ls
FOS AP Command: fosview -b
RDT AP Command: rdtview -b
7.
Steps 7-9 are only applicable on an APG with APIO. Verify that mml and CPT connections are working. AP Command: mml PTCOI; PTWSP; PTCOE; EXIT;
8.
Verify that IO devices are being routed OK. No Faulty devices should be found. AP Command: cpdlist -l
9.
Verify that FMS is working. AP Command: cpfls
Make sure that the CP file system can communicate with FMS in the APG. Commands: mml SYBFP:FILE; exit;
Step 10 needs to be performed from a CP connected terminal. Depending on the site configuration this can be performed via the mml prompt of an APG, an IOG terminal or a TW. 10. Verify status of CP-AP communication and the state of IPNs. Only ACTIVE/PASSIVE and WO should be seen. Note: On an APZ 212 40 system the OCIPN command will return FAULT CODE 108 IPN NOT SUPPORTED IN THIS APZ. CP Command: DPWSP; APAMP; OCINP:IPN=ALL;
Step 11 is only applicable to an APG with STS (AP1). 11. Check the STS configuration. AP Command: stmotls stmrp -Ll stmmp –Ll
12. Verify there have been no recent process restarts.
AP Command: ispprint
13. Verify that no major alarms are present. AP Command: alist
