APG40, Node, Change, APG40C/4 Page 1 of 16

    OPERATIONAL INSTRUCTION     9/154 31-CRZ 222 05 Uen A     

© Ericsson AB 2006 - all rights reserved

APG40, Node, Change, APG40C/4

Contents
1 Introduction
1.1 Scope

2 Procedure
2.1 Prerequisites
2.2 Actions

3 Additional Information
3.1 Update AM FW
3.2 Delete customer data on the faulty node
3.3 Update RAID FW

4 Glossary

5 References
5.1 Operational Instructions
5.2 Printout Descriptions

http://localhost:9032/alexserv?AC=LINK&ID=31118&FN=9_15431-CRZ22205Uen.A.ht... 5/22/2023
APG40, Node, Change, APG40C/4 Page 2 of 16

1 Introduction

1.1 Scope
This Operational Instruction describes the procedure to change a node in the Adjunct Processor
(AP) cluster. This procedure must be performed on site.

http://localhost:9032/alexserv?AC=LINK&ID=31118&FN=9_15431-CRZ22205Uen.A.ht... 5/22/2023
APG40, Node, Change, APG40C/4 Page 3 of 16

2 Procedure

2.1 Prerequisites

2.1.1   Conditions

The following conditions must apply before this procedure can be completed:

• The alarm AP FAULT is received or there is a work order received.

• A technician is available on site.
• The node to replace is faulty. The other node is the active working node.
• The user account names and passwords that were set in the system at the time the backup
was taken are also available.
• The user name and password valid for a user with administrative rights on the replaced
node.
• The new node (spare part) must contain a correct LBB software installation, to be able to
restore the node.
• A valid backup of the node to be replaced available on the APG or on DAT-Tape or on FTP
site.

2.1.2   Data

The following data should be known before starting this procedure:

• The user name and password valid for a user with administrative rights on the active node.
• The hostname of the active node.
• The public IP address of the active node.
• The hostname of the faulty node.
• The hostname of the replacement node.
• The public IP address of the replacement node.
• The subnet mask of the replacement node.
• The gateway address of the replacement node.

2.1.3   Special Aids

• A VGA monitor.
• A keyboard with PS/2 or USB connector.
• A mouse with PS/2 or USB connector.
• An adapter to connect the keyboard and monitor to the same cable.
• A grounding strap. To be used when exchanging the equipment.

http://localhost:9032/alexserv?AC=LINK&ID=31118&FN=9_15431-CRZ22205Uen.A.ht... 5/22/2023
APG40, Node, Change, APG40C/4 Page 4 of 16

2.2 Actions

Find out which Node to Replace

1. Find out which node to replace by examining the alarm printout from AP FAULT or the work
order.

Logon to Faulty Node

2. Connect to the faulty node by inserting the adapter to the mouse/keyboard input on the
AP. Connect the keyboard and the mouse to the adapter and connect the monitor to the
monitor input. Logon as a user with administrative rights.

Note:  
If it is not possible to login to the faulty node then proceed with Step 6. In that case
stopping the cluster server is not possible as in steps from Step 3 to Step 5.

Stop Cluster Service

3. Stop ACS_PRC_EventAnalyser on the faulty node. Enter the following command in a

Command Prompt window:

cluster res ACS_PRC_EventAnalyser_<0/1> /off /wait

Note:  
Select either '0' or '1' depending on the last letter of the faulty AP node's hostname.
If the last letter is 'a' select '0' and if the last letter is 'b' select '1'.

4. Stop acs_prc_clustercontrol on the faulty node. Enter the following command in a

Command Prompt window:

net stop acs_prc_clustercontrol

5. Stop Cluster Service on the faulty node. Enter the following command in a Command
Prompt window:

net stop clussvc

Note:  
Confirm the question, cluster service will also stop "ACS_FCH_Server" service with
"yes".

Logon to Active Node

6. Connect to the active node by inserting the adapter to the mouse/keyboard input on the
AP. Connect the keyboard and the mouse to the adapter and connect the monitor to the
monitor input. Logon as a user with administrative rights.

Check Drive Status

7. Ensure that all the drives residing in the active node have the status optimal and also
check their capacity. Enter the following command in a Command Prompt window:

megarc -dispcfg -a0

Note:  

http://localhost:9032/alexserv?AC=LINK&ID=31118&FN=9_15431-CRZ22205Uen.A.ht... 5/22/2023
APG40, Node, Change, APG40C/4 Page 5 of 16

All three of them should be in the state optimal. Under the column Capacity you can
see the capacity. Take a note of this value. If the drives on the active node have a
status other than optimal then this Operational Instruction cannot correct the
problem.

8. Are all the drives in the active node optimal?

Yes
Go to Step 9.

No
Go to Step 69.

9. Check which firmware version that is installed. Enter the following command in a
Command Prompt window:

megarc -ctlrinfo -a0

Note:  
Under the column firmware version you can see the version. Note this down.

Note down Heartbeat IP addresses

10. Enter the following command in a Command Prompt window:

ipconfig

11. Note down the IP addresses of Heartbeat 1 and Heartbeat 2.

Bring Cluster into Safe State

12. Set cluster's hardware and software in a state that allows replacement of the faulty node.
Enter the following command in the active node:

fcc_save_to_remove other

Note:  
Confirm the question with "yes".

Note:  
If you were unable to perform the steps from Step 3 to Step 5 the command may
give an error message since the command will try to stop the cluster service on the
faulty node.

Note:  
If the fcc_save_to_remove command hang (for more than 10 minutes) or does not
shut down the node properly then log on to the faulty (passive) node and shut it
down with START->Shut Down and select Shut down.

Disconnect Faulty Node

13.

Warning!

http://localhost:9032/alexserv?AC=LINK&ID=31118&FN=9_15431-CRZ22205Uen.A.ht... 5/22/2023
APG40, Node, Change, APG40C/4 Page 6 of 16

In case of a suspected hardware fault, the node shall be exchanged. It's not
allowed to exchange boards since that will violate any warranties.

Warning!
There is a special procedure for removing the SCSI cables. This procedure must
be followed to ensure that the SCSI bus will be terminated correctly.

Wait until the Manual Intervention Allowed (MIA) LED is lit and then disconnect the faulty
node. Disconnect the cables from the faulty node in the following order:
Table 1
1 Power cable
2 Public ethernet cable
3 IPN ethernet cable
4 External alarms cable
5 Alarm panel cable
6 Both Heartbeat ethernet cables
7 Both Serial cables (to CPU board and AM board)
8 Monitor cable
9 Keyboard/Mouse adapter cable

Note:  
The SCSI cables should be removed in the correct order. See next step.

Disconnect SCSI Cables

14. Always disconnect the SCSI connectors of the active node before removing the SCSI
connectors of the powered-down (faulty) node. Disconnect the SCSI cables in the following
order:
Table 2
1 Disconnect the upper SCSI connector on the active node
2 Disconnect the lower SCSI connector on the active node
3 Disconnect the upper SCSI connector on the faulty node
4 Disconnect the lower SCSI connector on the faulty node

Replace Node

15. Replace the faulty node with the replacement node.

16. Connect the cables in the following order:

Table 3
1 Both Heartbeat cables
2 Public ethernet cable
3 Monitor cable
4 Keyboard/mouse adapter cable
5 Power cable

Logon to Replaced Node

http://localhost:9032/alexserv?AC=LINK&ID=31118&FN=9_15431-CRZ22205Uen.A.ht... 5/22/2023
APG40, Node, Change, APG40C/4 Page 7 of 16

17. Logon as a user with administrative rights.

Stop Cluster Service

18. Stop the Cluster Service on the replaced node. Enter the following commands in a
Command Prompt window:

net stop clussvc

Check Heartbeat IP addresses

19. Check the IP addresses of Heartbeat 1 and Heartbeat 2. Enter the following command in a
Command Prompt window:

ipconfig

20. Were the Heartbeat IP addresses of the replaced node conflict with the Heartbeat IP
addresses of the active node?

Yes
Go to Step 21.

No
Go to Step 22.

21. Change the IP addresses of the Heartbeat 1 to "192.168.202.3" and Heartbeat 2 to

"192.168.201.3".

Note:  
Go to Start->all programs->accessories->communications->network connections.
Select the corresponding Heartbeat connection and double click on it. Click the
properties button in the general tab. Select Internet Protocol (TCP/IP) in the general
tab and double click on it.

Check Disk Capacity

22. Check which capacity the drives have on the replaced node. Enter the following command
in a Command Prompt window:

megarc -LogPhysInfo -a0

Note:  
Under the column Capacity you can see the capacity. Take a note of this value.

23. Was the capacity of the disks equal to or bigger than the disks in the active node?

Yes
Go to Step 25.

No
Go to Step 24.

Note:  
Refer to Step 7 for the Disk Capacity in the active node.

24. This spare part cannot be used. The disks in the spare part must have equal or bigger
capacity than the disks in the active node. Go to Step 69.

http://localhost:9032/alexserv?AC=LINK&ID=31118&FN=9_15431-CRZ22205Uen.A.ht... 5/22/2023
APG40, Node, Change, APG40C/4 Page 8 of 16

Unpack Backup to System Disk

25. On which media do you have a valid backup?

DAT Tape
Go to Step 26.

FTP site
Go to Step 34.

The shared disk M:\Images\node(a/b)

Go to Step 35.

26. In the active node, insert a tape containing the backup.

27. Check that the correct backup is inserted in the tape drive. Enter the following command in
a Command Prompt window:

emfls

28. Is the backup valid?

YES
Go to Step 29.

NO
Go to Step 25.

29. Copy the backup file to be restored from the tape containing the backup file to M:\Images.
Enter the following command in a Command Prompt window:

emfcopy -m FROMTAPE <BACKUPFILENAME>.zip M:\Images\node(a/b)

Note:  
The backup should be valid for the node to be restored.

Note:  
Select either a or b in (a/b) depending on the last letter of the replaced AP node's
hostname.

30. Enter the following command in a Command Prompt window:

emfinfo -n <NODENAME>

Note:  
<NODENAME> is the name of the node on which the copy operation was initiated.

31. Is the copy operation finished?

YES
Go to Step 32.

NO
Go to Step 30.

32. When copying is finished, eject the tape.

33. Go to Step 35.

http://localhost:9032/alexserv?AC=LINK&ID=31118&FN=9_15431-CRZ22205Uen.A.ht... 5/22/2023
APG40, Node, Change, APG40C/4 Page 9 of 16

34. Use FTP to transfer the compressed backup. The backup should be transferred to the
directory M:\Images\node(a/b). The backup should be valid for the node to be restored.
Select either a or b in (a/b) depending on the last letter of the restored AP node's
hostname. See AP, File, Transfer and then return to Step 35 in this Operational Instruction.

35. Access the Replaced Node.

36. Enter the following command :

net use Z: \\<heartbeat net>\images/<user> <password>

Note:  
<heartbeat net> is the IP address of either heartbeat 1 or Heartbeat 2 on the
working AP node where the shared directory 'images' is located. When prompted,
logon with a user with administrative rights.

Note:  
Refer to Step 10 for the IP addresses of Heartbeat 1 and Heartbeat 2 on the active
node.

37. Enter the following command in a Command Prompt window:

cd /d c:\Program Files\pkware\pkzipc

38. Enter the following command:

pkzipc -extract Z:\node(a/b)\<NAMEOFBACKUP>.zip "program

files\ap\acs\bin\burrestore.exe"

Note:  
The backup should be valid for the replaced node.

Note:  
Select either a or b in (a/b) depending on the last letter of the replaced AP node's
hostname.

39. Enter the following command:

burrestore -a Z:\node(a/b)\<NAMEOFBACKUP>.zip -d D:\

Note:  
The backup should be valid for the node to be restored.

Note:  
Select either a or b in (a/b) depending on the last letter of the replaced AP node's
hostname.

Note:  
Confirm the question with "yes". The prompt is not returned until the backup is
restored.

Check RAID FW

40. In a Command Prompt window, enter the following command in replaced node:

megarc -ctlrInfo -a0

41. Was the RAID firmware (FW) version in the replaced node equal to the one in the active
node?

http://localhost:9032/alexserv?AC=LINK&ID=31118&FN=9_15431-CRZ22205Uen.A.ht... 5/22/2023
APG40, Node, Change, APG40C/4 Page 10 of 16

YES
Go to Step 43.

NO
Go to Section 3.3 and then return to Step 42.

Note:  
Refer to Step 9 for the RAID FW version in the active node.

42. In a Command Prompt window, enter the following command:

shutdown /r /t 1

Check AM FW

43. Check which AM FW version is used on the active node. Enter the following command in
the active node:

fcc_amversion own

44. Check which AM FW version is used on the replaced node. Enter the following command in
the replaced node:

fcc_amversion own

45. Are the AM FW versions identical on the active and on the replaced node?

YES
Go to Step 50.

NO
Go to Step 46.

Note:  
Compare the printouts received in Step 43 and Step 44.

46. Enter the following command:

dir /d c:\Program Files\Force\am_services

47. Is the correct version available on the replaced node?

YES
Go to Section 3.1 and return to Step 50.

NO
Go to Step 48.

Note:  
The file used when upgrading the AM FW should be named
<lbb_am_appl_x_y_r.bin> where 'x' and 'y' is the AM FW version and should
correspond to the printout received in Step 43.

48. Copy the correct 'lbb_am_appl_x_y_r.bin' file from the active node. Enter the following
command:

copy \\<active node hostname>\C$\Program

Files\Force\am_services\<lbb_am_appl_x_y_r.bin> d:\Program
Files\Force\am_services

http://localhost:9032/alexserv?AC=LINK&ID=31118&FN=9_15431-CRZ22205Uen.A.ht... 5/22/2023
APG40, Node, Change, APG40C/4 Page 11 of 16

49. Go to Section 3.1 and return to Step 50.

Restoring From Backup

50. Enter the following command in the replaced node:

burrestore -r <NAME> -s

Note:  
The NAME is to recognize the present LBB OS.

51. Confirm the question with "yes".

Shutdown Replaced Node

52. In a Command Prompt window, enter the following command:

shutdown /s /t 1

Note:  
Wait until "It is now safe to turn off your computer" message appears.

53. Disconnect the power cable from the replaced node.

Reconnect SCSI Cables

54. Always connect the SCSI connectors of the replaced (powered-down) node before
connecting the SCSI connectors of the active node. Connect the SCSI cables in the
following order:
Table 4
1 Connect the upper SCSI connector on the replaced node
2 Connect the lower SCSI connector on the replaced node
3 Connect the upper SCSI connector on the active node
4 Connect the lower SCSI connector on the active node

Note:  
The SCSI cables must be crossconnected (i.e. SCSI 1 on one node to SCSI 0 on the
other. The same for SCSI 0 to SCSI 1).

Reconnect Other Cables

55. Reconnect the remaining cables in the following order:

Table 5
1 IPN ethernet cable
2 External alarms cable
3 Alarm panel cable
4 Both Serial cables (to CPU board and AM board)
5 Power cable

Logon to Replaced Node

56. Logon as a user with administrative rights.

http://localhost:9032/alexserv?AC=LINK&ID=31118&FN=9_15431-CRZ22205Uen.A.ht... 5/22/2023
APG40, Node, Change, APG40C/4 Page 12 of 16

Setup Services

57. Enter the following command in a Command Prompt window:

setupservices

58. Enter the following command in a Command Prompt window:

prcboot

Note:  
The node will reboot and start up again.

59. Logon as a user with administrative rights.

Logon to Active Node

60. Connect to the active node by inserting the adapter to the mouse/keyboard input on the
AP. Connect the keyboard and the mouse to the adapter and connect the monitor to the
monitor input. Logon as a user with administrative rights.

Integrate Shared Mirrored Disk Drive

61. Integrate the shared mirrored disk drive. Enter the following command in a Command
Prompt window:

fcc_integrate other

62. Confirm action by entering 'y' followed by <ENTER>.

Note:  
The replaced node should now join the cluster and the mirroring of the data disk will
start. The mirroring process will take around 1-2 hours. The lower figure is for idle
disks with no traffic to them and the higher figure is for maximum specified disk
traffic.

Check RAID Mirroring

63. Ensure that the mirroring of the RAID is still ongoing. Enter the following command:

megarc -dispcfg -a0

Note:  
One of the RAID-1 should be in the status "rebuild" and the other two should be in
the status "rebuild/ pending".

Log Off from Active Node

64. Log off from the active node using START-> Log Off.

Logon to Replaced Node

65. Connect to the replaced node by inserting the adapter to the mouse/keyboard input on the
AP. Connect the keyboard and the mouse to the adapter and connect the monitor to the
monitor input. Logon as a user with administrative rights.

http://localhost:9032/alexserv?AC=LINK&ID=31118&FN=9_15431-CRZ22205Uen.A.ht... 5/22/2023
APG40, Node, Change, APG40C/4 Page 13 of 16

Wait for RAID Mirroring to Complete

66. Before proceeding with the next step, make sure that the mirroring of the RAID is
completed i.e. the RAID is optimal.

megarc -ctlrinfo -a0

Note:  
The mirroring process will take around 1-2 hours. The lower figure is for idle disks
with no traffic to them and the higher figure is for maximum specified disk traffic.

Backup System

67. Go to Operational Instruction AP, System Backup and Verify, Initiate.

Perform the steps in that Operational Instruction, and return to Step 68 in this Operational
Instruction.

68. Go to Step 70.

Consult Next Level of Maintenance Support

69. Consult the next level of maintenance support. Further action is outside the scope of this
Operational Instruction.

Prepare Faulty Node

70. Perform the actions in Section 3.2 and then return to Step 71.

Job Completed

71. Make a report.

Note:  
See Operational Instruction Report of Finished Work.

72. The job is completed.

http://localhost:9032/alexserv?AC=LINK&ID=31118&FN=9_15431-CRZ22205Uen.A.ht... 5/22/2023
APG40, Node, Change, APG40C/4 Page 14 of 16

3 Additional Information

3.1 Update AM FW
1. Start the AM configuration tool by using START->Programs->AM Configuration->AM
Configuration.
2. Select the tab Installation.
3. Press UpgradeAMgo.. button.
4. In the AMgo Upgrade window press the Browse button.
5. Select the file lbb_am_appl_x_y_r.bin (where 'x' and 'y' is the AM FW version).
6. Press the Open button.
7. Press the Start button.
8. Answer OK on the pop up menu and wait until the firmware is upgraded.
9. When the upgrade status has changed to Success the upgrade is finished.
10. Press OK to exit the AM configuration tool.
11. The AM firmware is now updated.

3.2 Delete customer data on the faulty node

Before the faulty node is sent for repair, the customer data must be deleted from the data disks.
If possible, perform these actions off site when this OPI has been completed successfully.

1. Connect only the keyboard and monitor to the faulty node, and then connect the power
cable.
2. Wait until Ctrl+M appears. Press Ctrl+M to activate the Raid Bios Configuration menu.
3. Use down arrow key to reach Initialize and press <Enter> to open submenu to display all
logical drives.
4. Use down arrow key to browse and select each drive by pressing< Space> key.
5. Press <F10> to initialize.
6. Select the "Yes" button with the tab key and press <ENTER> to confirm selection. Process
will take few minutes to complete.
7. Confirm the action with OK, the system reboot.

3.3 Update RAID FW

1. Enter the command in the Command Prompt window: cd /d c:\program
files\force\raid\firmware
2. Enter the command in the Command Prompt window: NTflash /r /f mflashw.exe

http://localhost:9032/alexserv?AC=LINK&ID=31118&FN=9_15431-CRZ22205Uen.A.ht... 5/22/2023
APG40, Node, Change, APG40C/4 Page 15 of 16

4 Glossary
-

http://localhost:9032/alexserv?AC=LINK&ID=31118&FN=9_15431-CRZ22205Uen.A.ht... 5/22/2023
APG40, Node, Change, APG40C/4 Page 16 of 16

5 References

5.1 Operational Instructions

AP, System Backup and Verify, Initiate

Report of Finished Work

5.2 Manual Pages

burrestore (1m)
prcboot(1m)

5.3 Printout Descriptions

AP FAULT

http://localhost:9032/alexserv?AC=LINK&ID=31118&FN=9_15431-CRZ22205Uen.A.ht... 5/22/2023