PCA X9-2 w 3.0.

1 EIS Checklist
Installation Checklist for the ORACLE® PRIVATE CLOUD APPLIANCE
(PCA) X9-2 / 3.0.1

Version Access Date Notes

1.0 Oracle Internal and Approved 24 Feb. Initial release

Partners ONLY 2022

1.1 02 March Updated the CN Field Installation Checklist as well as the Rack Rules in Appendix A
2022

1.2 15 March Added work around for NTP server problem, mysql server credential update, spelling errors
2022

1.3 19 April Changed the pre-day0 checks to point to KM

2022

1.4 16 June Updated Site Requirements and Network Requirements to reflect feedback from field. Reordered section on
2022 CN provisioning to better support patching.

1.5 15 July Added link to MOS article for those who don't have Confluence access. Added Appedix G: Firewall Ports.
2022

Customer:

Task Number:

Technician:

Date:

Overview
This document is provided as guidance to Oracle Field Personnel who will be installing the Private Cloud Appliance version 3.0.1. The EIS checklist
provides a framework for referencing the Private Cloud Appliance 3.0.1 Installation Guide but also includes important internal only content, taken from
various engineering specifications. Please be aware, while we strongly recommend installation services, the PCA X9-2 3.0.1 is customer installable. The
customer should reference the PCA X9-2 3.0.1 Installation Guide. Please NOT hand this checklist over to the customer.

Be sure to check the EIS web page for the latest versions prior to commencing the installation.

https://eis.us.oracle.com/checklists/

The purpose of this checklist is to help the installer achieve a "good" installation.
Installers must have attended the appropriate training classes. EIS checklists are not a replacement for proper training.
Use of a laptop is required.
Feedback on issues with EIS content or product quality is welcome. Oracle staff should enter comments section of the following confluence page:

PCA X9-2 w 3.0.1 EIS Checklist

Partners should contact the PartnerHelp Portal for assistance and feedback.

Table of Contents

Installation Checklist for the ORACLE® PRIVATE CLOUD APPLIANCE (PCA) X9-2 / 3.0.1
Overview
Table of Contents
Opening an SR and getting support
Glossary
Preparation Before Going on Site
Site Requirements
MOS Requirements
Patch Requirements
 Network Requirements
Connected Services Requirements
Plan for installing CN's in the field
Install Rack
Unpack and move into place
Install field installable CN's
Re-route PDU cables if necessary
Move Rack into place
Connect to Customer Networking Infrastructure
Power On for the First Time
Verify ZS Appliance is available and healthy
Verify Management Node ILOM Configuration
Boot and Verify the Management Node Cluster
Day 0 Configuration Prechecks
Day 0 Configuration
First Time Access to the Service Enclave
Connecting to ASR
Verify Health
Provision CN's
Software Patch/Upgrade
Verify local yum repository
Prepare PCA for Patching/Upgrade
Assess the Patches to be installed
Initiate patch/upgrade process
Verify patch/upgrade completed successfully
Install and Power on Field Installable Compute Nodes
Connect to Platinum Services
Change Your default passwords on all components
Install Complete
Appendix A: Rack Constraints
Rack Rules
Rack Elevations
Appendix B: Data Switch Cabling Reference
Cable Type and Part #'s
Data Switch Connection Reference
Appendix C: Management Switch Cabling Reference
Cable Type and Part #'s
Management Switch Connection Reference
Appendix D: ZS Appliance Cluster Cabling Reference
Appendix E: Power Scheme Reference
15KVA (Single and Three Phase)
Storage Enclosure Power Cabling
Compute Node and Switch Power Cabling
22KVA Single Phase
Storage Enclosure Power Cabling
Compute Node and Switch Power Cabling
24KVA Three Phase
Storage Enclosure Power Cabling
Compute Node and Switch Power Cabling
Appendix F: Default Logins and Passwords
Appendix G: Firewall Ports

Opening an SR and getting support

For hardware failures when installing a PCC/PCA system on-site, the process is the same as for other products:

1. Contact the HUB referencing the Installation Service Request and ask to have a new Technical Service Request created
a). GCH Handling Callbacks on Existing Technical SRs : GCSGCH (Doc ID 1803749.1)
b). If the FE is running into problems, they can ask for the Oncall Duty Manager for assistance

2. MOS GCH Handling Document

a). GCH Handling Callbacks on Existing Technical SRs : GCSGCH (Doc ID 1803749.1)
b). Regarding the process, the IC will need to create a technical SR and not a collab (to avoid chance of routing to the wrong group, ed.)
c). Recommend the ICs follow the same process the HUB engineer would follow on Doc ID 1803749.1 starting at task 55 and then relate the Install
SR with the Technical SR as a backup solution

Glossary
 Acronym Term Definition
(abbr.)

PCA Private Cloud Appliance

CN Compute Node

MN Management Node One three servers configured in a cluster (pcamn01, pcamn02, pcamn03)

SN Storage Node ZS9-2 Head

Aggregation Cisco 9336C Same Spine Switch (pcaswsp02, pcaswsp01)

Switch

Access Switch Cisco 9336C Same as Leaf Switch (pcaswlf02, pcaswlf01)

Management Cisco 9348GC-FXP Management Switch (pcaswmn01)

Switch

Flex Bay Flex Bay A grouping of 4 RU slots that are configurable by the customer, as Storage or Compute. A PCA supports
4 Flex Bay's:

Flex Bay 4 - RU41 - 38

Flex Bay 3 - RU37 - 34

Flex Bay 2 - RU23 - 20

Flex Bay 1 - RU19 - 16

Rule of Three Rule of Three The number of CN's must be a multiple of three.

ULN Unbreakable Linux

Network

ASR Auto Service Request

Day 0 Day 0 The wizard that walks the user through the initial setup

Preparation Before Going on Site

Reference Time Check
Site Requirements
Please refer to the Installation Guide Sections 1 and 2 for detailed customer facing checklist content. Or the FE PCA- PCA-PreSiteChecklist (Link 4 hours
PreSiteChecklist checklist in editable form available here. to download from MOS)

7.1 System
Components
Checklist
7.2 Data Center
Room Checklist
7.3 Data Center
Environmental
Checklist
7.4 Access Route
Checklist
7.5 Facility Power
Checklist
7.6 Safety Checklist
7.7 Logistics Checklist

Prior to on-site, it may be useful to download or print out any required reference material such as the [PCA 3.0.x] Day0
Installation Guide, KM documents, run books, etc. that are referenced throughout this document. Pre_Checks and
Post_Checks (Doc ID
2859427.1)

How is power delivered to the rack in the data center? Form a trough above the rack or below through the floor?

NOTE: Currently, by default, PDU cables are routed down through the bottom of the rack. If the PDU cables need to
be routed up through the top of the rack, due to the density of the rack, please plan on an hour of work to re-route
the cables. We advise this should be done with two people.

Installing a Bastion Server in the PCA rack is supported by exception only. Therefore, not covered in the EIS PCA engineering
documentation.
 References Time Check
MOS Requirements
Confirm customer MOS access and account settings. Doc ID 2 hours
1329200.1
Asset must be in CSI and the customer must have administrative access to the asset, This will be needed for both the local YUM
repository and ASR activation CSI
Administration

References Time Check

Patch Requirements
Ensure a local YUM repository is available and functional. A patch is likely to be required at install time, prior to putting the system ULN
into production. Registration

If the customer already has a local yum server configured and running they can simply add the CSI to their account at linux.oracle. Setting up a
com and add the PCA channels to that local yum server. If not, they will need to create an OL instance either on baremetal or a local ULN
VM external to the PCA. Follow the standard documentation on setting up a local yum server. OL7 is recommended as the uln- mirror
yum-mirror RPM for OL8 is not functional at the time this is written.

A local yum server is asynchronous to ULN (linux.oracle.com), which means: Patching

Guide
Sites not currently connected to the internet can still be used to update their client PCA(s).

Dark or Secure sites typically use a system that bridges between a secure internal network and the internet, so as above the
server can be disconnected from the internet (shutdown external interface, bring up internal interface) when client PCA(s) on the
secure network need to be updated. 1 day

Note: There are also methods to get patching done such as manually copying the contents of the patching directory to a system
on the network and running a simple HTTP server (e.g. "python -m SimpleHTTPServer 8000") from a directory with the correct
permissions and then pointing the client PCA to it.

There are likely to be five channels for the purpose of patching the PCA:

PCA 3.0.1 Container Images

PCA 3.0.1 Firmware

PCA 3.0.1 Hypervisor

PCA 3.0.1 MN

PCA 3.0.1 OCI Compute Images

References Time Check

Network Requirements
Fully read and understand the following. Pay close attention to the following details 1 day
Network Configuration Spreadsheet.
Physical topology that will be used for uplink ports (Port speeds, number of links and (Link to download from MOS)
physical layout) Concept guide section 2.3
If you will be using static or dynamic (BGP) routing Install guide
Type of DNS configuration that will be used (Zone delegation or Manual configuration) Section 3.1
If the admin traffic will be separated from the data traffic Section 7.8 Network
Customer must confirm that DNS and NTP are accessible from the IP subnet the PCA Specification Checklist
will be attached to. Section 7.9 Initial Installation
Customer must verify that all required ports called out in the Network Configuration Checklist
Spreadsheet are accessible to/from the PCA prior to installation.

During install, connectivity and access is REQUIRED to be done from an FE laptop. 10 min

In rack bastion is supported by exception only. Engineering will need to assess power usage. PCA Engineering Exception process

References Time Check

Connected Services Requirements
Will the customer connect to Automatic Service Requests (ASR)? Administration Guide, Section 5.5 Using Auto Service Requests

ASR Configuration and Activation Checklist.xlsx

(Link to download from MOS)

Will the customer connect to Platinum Services? Platinum Support TBD - Contact Product Management.

References Time Check

Plan for installing CN's in the field
The Configurator only allows for orders that adhere to the "rule of 3". Meaning, the number of CN's is required to Appendix A: Rack Constraints
be 3, 6, 9, 12, 15, 18. Exceptions are necessary to order a number of CN's that does not conform to the rule of 3.

The factory only installs CN's based on the rule of 3. Any additional CN's are to be installed in the field. CN Field Install Checklist.xlsx
(Link to download from MOS)
 1 hour
PDU Type dictates RU usage and the number of supported components. Appendix E: Power Scheme
Reference

Verify Network Connections Appendix B: Data Switch

Cabling Reference

Appendix C: Management
Switch Cabling Reference

Install Rack
Reference Time Check
Unpack and move into place
Note: Please ensure any local state, federal, country, rules and regulations are followed in an appropriate manner.

Please follow the instructions printed on either end of the cardboard carton for removing the shrink wrap, banding and 30 min
cardboard. Note and follow the seven steps for removing the rack from the pallet shown on labels on either ramp attached to the
pallet.

Reference Time Check

Install field installable CN's
A strategy for installing CN's in the field should have been determined in the CN Field Install Checklist. CN Field Install
Checklist.

If power and network cabling is required, install the cabling and server BEFORE moving into the data center.

Apply cable label's to all cables

30 min
Install and route Flex Bay cabling from pre-determined Flex Bay RU's to the power outlets and switch ports. Be sure to use per CN
routing and anchor points as close to factory as possible.

Install CN into pre-determined Flex Bay RU's.

NOTE: Do NOT connect power cables at this time. Power cables will be connected after Day 0 configuration is complete.

If power and network cabling is NOT required, the Compute Node can be added to the Flex Bay at any time.

Reference Time Check

Re-route PDU cables if necessary
Remove the right side panel (looking from the rear) 2 hour for 1 person

At the top of the PDU remove the two outside torx screws from the PDU end plate 1 hour for 2 people

At the bottom unscrew the lower torx screws that connect the bottom bracket to the rack rail so it is free from the rack

You'll be able to move the PDU out of the way

Remove the PDU cables from the shipping brackets mid rack

Feed the cables (roughly 6 feet of lenth) through the side rails so that the entire cable is outside the right side of the rack

Release the velcro scraps anchoring the PDU cables to the lower half of the PDU's

Velcro the PDU cables to the anchor points on the upper half of the PDU's.

Feed the cables through the top RU and top of rack

Reference Time Check

Move Rack into place
Details for installing the rack can be found in the The Installation Guide Section 4.1 through 4.3. Please refer to the PCA-PreSiteChecklist (
Installation Guide Checklists sections 7.1 through 7.7. Or the downloadable checklist found above in the Preparation Link to download from
Before Going on Site section. MOS)

Install Guide:

7.1 System 60 min

Components
Checklist
7.2 Data Center
Room Checklist
7.3 Data Center
Environmental
Checklist
7.4 Access Route
Checklist
7.5 Facility Power
Checklist
7.6 Safety
Checklist
7.7 Logistics
Checklist

Reference Time Check

Connect to Customer Networking Infrastructure
Connect PCA Spine Switch to Customer Networking Infrastructure Network Configuration Spreadsheet 30 min

(Link to download from MOS)

Connect a Laptop for Initial Access to the PCA X9
To gain initial access to the Oracle Private Cloud Appliance Dashboard, you must
connect directly to the Cisco Nexus 9348GC-FXP management switch. FE laptops
should connect to Port 1. Customer Bastion or workstations should connect to Port 2.
Install Guide does not address use of Port 1, Service
Port. Nor does it address setting up the correct IP
Address and netmask to be able to access the ILOM's.
15 min

Connect an FE laptop to the Ethernet cable in Port 1 with the following IP address and
netmask: 100.96.3.253/22.

NOTE: This differs from from the Install Guide in that this will allow access to the subnet
ranges 10.96.0.0 - 100.96.1.255 and range 100.96.2.0 - 100.96.3.255, which provides
access to the component ILOM ports.

To connect a customer provided Bastion or workstation to Port 2 use the following IP Install Guide Section 5.3 15 min
address and netmask: 100.96.3.254/23

Note: this only provides access to subnet ranges 100.96.2.0 - 100.96.3.25. To access
ILOM ports you will need to log into the Management Node, then ssh to the ILOM ports.

Reference Time Check

Power On for the First Time
Connect the PDU cables to Data Center power Install Guide,
Section 4.5.2
30 min
Power on the PDU breakers

Reference Time Check

Verify ZS Appliance is available and healthy
From the Service laptop / workstation connected to the Cisco Switch, ssh to the internal ZFSSA ip that floats with the capacity pool 10 Min

Log in = root/Welcome1

100.96.2.4 Capacity Pool

100.96.0.3 ilom-pcasn02
100.96.0.2 ilom-pcasn01
100.96.2.3 pcasn02
100.96.2.2 pcasn01

Caution

You will likely see the host name of the storage heads show up as 'sn01AKxxxxxxxx' and 'sn02AKxxxxxxxx'. While
this is inconsistent depending on which interface you're viewing, it is expected.

Run a basic status command to verify general health. The output is not important. We are only looking to see that the command is
responsive. This will show that the ZFSSA management software (akd) is alive and responsive.

ssh root@100.96.2.4
Password:
Warning: Permanently added '100.96.2.4' (ECDSA) to the list of known hosts.
Last login: Thu Dec 16 05:13:22 2021 from 100.96.2.34

sn0XXXXXXXXXXX:> status activity show

Activity:
CPU 0 %util Sunny
FTP 0 bytes/sec Sunny
Disk 159 ops/sec Sunny
iSCSI 1 ops/sec Sunny
NFSv3 0 ops/sec Sunny
NFSv4 180 ops/sec Sunny
Network 304K bytes/sec Sunny
SMB2 0 ops/sec Sunny
sn0XXXXXXXXXXX:>

Verify cluster status. If it is not in CLUSTERED CLUSTERD status issue a failback

sn0XXXXXXXXXXX:> configuration cluster

sn0XXXXXXXXXXX:configuration cluster> ls
Properties:
state = AKCS_CLUSTERED
description = Active
peer_asn = 2bd74634-945e-4a62-9471-e1f3089b66a5
peer_hostname = sn022126XLF011
peer_state = AKCS_CLUSTERED
peer_description = Active

Children:
resources => Configure resources

sn0XXXXXXXXXXX:configuration cluster>
If the cluster status is something other than the status above, issue a failback. The fail back should take about 30 to 60 seconds
and then return the prompt. The status should be checked to verify the CLUSTERED CLUSTERED status. If the verification is
correct, then ZFS Verification is Complete.

sn0XXXXXXXXXXX:configuration cluster> failback

sn0XXXXXXXXXXX:> exit

References Time Check

Verify Management Node ILOM Configuration
From the Service laptop / workstation connected to the Cisco Switch, Log into each Management node ILOM to verify the Product
Identity Record and System Identity Records.
10 min
Use the 'show /System' command:

Log in = root/Welcome1

100.96.0.33 ilom-pcamn01
100.96.0.34 ilom-pcamn02
100.96.0.35 ilom-pcamn03
Verify each Management Nodes Key Identity Properties using "show /System". The critical attributes are:

serial_number = AK00842951 <--------------------------------------------------- Originates from the Product Identity Record

component_model = ORACLE SERVER X9-2 <------------------------------------- Originates from the System Identity Record
component_part_number = 8209083 PCA X9-2 MN <---------------------------- Originates from the System Identity Record
component_serial_number = 2139XLD01B <-------------------------------------- Originates from the System Identity Record
chassis_model = ORACLE SERVER X9-2 <----------------------------------------- Originates from the Chassis Identity Record
chassis_part_number = 8209083 <------------------------------------------------ Originates from the Chassis Identity Record
chassis_serial_number = 2139XLD01B <------------------------------------------- Originates from the Chassis Identity Record
system_identifier = Oracle Private Cloud Appliance X9-2 AKxxxxxxxx <------------- Originates from ILOM

-> show /System

/System
Targets:
Open_Problems (0)
Processors
Memory
Power
Cooling
Storage
Networking
PCI_Devices
Firmware
BIOS
Log

Properties:
health = OK
health_details = -
open_problems_count = 0
type = Rack Mount
model = PCA X9-2 Base
qpart_id = Q13719
part_number = 7603900
serial_number = AK00842951
rfid_serial_number = 341A583DE58000000007E354
component_model = ORACLE SERVER X9-2
component_part_number = 8209083 PCA X9-2 MN
component_serial_number = 2139XLD01B
chassis_model = ORACLE SERVER X9-2
chassis_part_number = 8209083
chassis_serial_number = 2139XLD01B
system_identifier = Oracle Private Cloud Appliance X9-2 AKxxxxxxxx
system_fw_version = 5.0.2.20.a
primary_operating_system = Not Available
primary_operating_system_detail = Comprehensive System monitoring is not
available. Ensure the host is running
with the Hardware Management Pack. For
details go to
http://www.oracle.com/goto/ilom-redirect
/hmp
host_primary_mac_address = a8:69:8c:0a:2a:30
ilom_address = 100.96.0.33
ilom_mac_address = A8:69:8C:0A:2A:33
locator_indicator = Off
power_state = On
actual_power_consumption = 357 watts
action = (Cannot show property)

Commands:
cd
reset
set
show
start
stop
Verify and adjust each management nodes ILOM time.

Caution

At this point there is no NTP or automated time sync process. Please check the time in each Management Nodes
ILOM. The best practice is to get the times as close as possible. Once Day 0 is complete the time will resync based
on the NTP server in the customers network.

set /SP/clock datetime=MMDDhhmmYYYY timezone=timezone

Where MMDDhhmmYYYY is the month, date, hour, and minute as two digits, and the year as four digits.

timezone is the 3 or 4 alphanumeric string representing the time zone - Use UTC

-> show /SP/clock

Targets:

Properties:
datetime = Wed Feb 9 19:29:42 2022
timezone = GMT (GMT)
uptime = 38 days, 15:13:03
usentpserver = enabled

Commands:
cd
set
show

->

References Time Check

Boot and Verify the Management Node Cluster
The Management Node have autoboot disabled because they are dependent on resources made available from the ZS 10 min
appliance. Once the ZS appliance health and availability are verified the Management Nodes can be brought up.

Be aware, the three management nodes boot into a cluster. The best practice is to boot "pcamn01" first, wait a minute or two,
then power both "pcamn02" and "pcamn03". The Management Nodes can be brought up in two ways:

1. press the Standby Button (Power Button)

2. from the ILOM promt, execute "start /SYS

Please wait for System Login Prompt for all three nodes before continuing.

-> start /SYS

Are you sure you want to start /SYS (y/n)? y

Oracle Linux Server 7.9

Kernel 5.4.17-2102.203.6.el7uek.x86_64 on an x86_64

pcamn01 login:
From the Service laptop / workstation connected to the Cisco Switch, log into the MN VIP (100.96.2.32) and verify Management
Node / Cluster Health:

login/password = root/Welcome1

Management Node VIP 100.96.2.32

pcamn01 100.96.2.33
pcamn02 100.96.2.34
pcamn03 100.96.2.35

Verify there are 3 nodes configured, all are online and all 'Resource Groups' are 'Started'.

[root@pcamn01 ~]# pcs status

Cluster name: mncluster
Stack: corosync
Current DC: pcamn02 (version 1.1.23-1.0.1.el7-9acf116022) - partition with quorum
Last updated: Thu Jan 6 15:55:05 2022
Last change: Thu Jan 6 15:08:27 2022 by root via crm_resource on pcamn02

3 nodes configured
11 resource instances configured (1 DISABLED)

Online: [ pcamn01 pcamn02 pcamn03 ]

Full list of resources:

scsi_fencing (stonith:fence_scsi): Stopped (disabled)

Resource Group: mgmt-rg
vip-mgmt-int (ocf::heartbeat:IPaddr2): Started pcamn02
vip-mgmt-host (ocf::heartbeat:IPaddr2): Started pcamn02
vip-mgmt-ilom (ocf::heartbeat:IPaddr2): Started pcamn02
vip-mgmt-lb (ocf::heartbeat:IPaddr2): Started pcamn02
vip-mgmt-ext (ocf::heartbeat:IPaddr2): Started pcamn02
l1api (systemd:l1api): Started pcamn02
haproxy (ocf::heartbeat:haproxy): Started pcamn02
pca-node-state (systemd:pca_node_state): Started pcamn02
dhcp (ocf::heartbeat:dhcpd): Started pcamn02
hw-monitor (systemd:hw_monitor): Started pcamn02

Daemon Status:
corosync: active/enabled
pacemaker: active/enabled
pcsd: active/enabled

Verify each management node (by IP address) is a member of the cluster with a status of 'joined'.

[root@pcamn01 ~]# corosync-cmapctl | grep members

runtime.totem.pg.mrp.srp.members.1.config_version (u64) = 0
runtime.totem.pg.mrp.srp.members.1.ip (str) = r(0) ip(253.255.0.33)
runtime.totem.pg.mrp.srp.members.1.join_count (u32) = 1
runtime.totem.pg.mrp.srp.members.1.status (str) = joined
runtime.totem.pg.mrp.srp.members.2.config_version (u64) = 0
runtime.totem.pg.mrp.srp.members.2.ip (str) = r(0) ip(253.255.0.34)
runtime.totem.pg.mrp.srp.members.2.join_count (u32) = 1
runtime.totem.pg.mrp.srp.members.2.status (str) = joined
runtime.totem.pg.mrp.srp.members.3.config_version (u64) = 0
runtime.totem.pg.mrp.srp.members.3.ip (str) = r(0) ip(253.255.0.35)
runtime.totem.pg.mrp.srp.members.3.join_count (u32) = 1
runtime.totem.pg.mrp.srp.members.3.status (str) = joined
Configure the NTP servers to allow the management network access.

This step is needed to allow the components of the PCA to use the NTP servers on the management nodes. If this
step is skipped, the management nodes will not allow NTP requests despite the clients being properly configured.

From the active management node run the following command.

[root@pcamn02 ~]# for f in pcamn01 pcamn02 pcamn03 ;do echo $f; ssh $f "grep -qxF
'allow 100.96.0.0/22' /etc/chrony.conf || echo 'allow 100.96.0.0/22' >> /etc/chrony.
conf; systemctl restart chronyd.service";done

References Time Check

Day 0 Configuration Prechecks
It is important to ensure the system is health up top this point in the process. The following pre-checks will give us confidence all 10 Min
is well and the install can continue. Log into the Primary Management Node; ssh to the MN VIP (100.96.2.32)

Caution

The Primary MN of the cluster will have capabilities the other MN's do not. Be sure you are connected to the Primary
MN unless instructed otherwise. Connecting to the MN VIP

will automatically drop you into the Primary MN.

Follow KM 2859427.1 for all the required pre day 0 checks.

Caution

Following the checks in the above KM is a critical step to make sure the installation is successful.

Verify compute node ILOM and HOST should be seen.

ilom-pcacn001
ilom-pcacn002
ilom-pcacn003
pcacn001
pcacn002
pcacn003

The status of the compute nodes will transition as the discovery process progresses, until they get to 'ok', 'Ready_to_provision'.

Note

It can take a significant amount of time for the discover process to complete depending on the number of compute
nodes in the rack being installed. The general rules are:

60 minutes to discover a single CN.

6 CN's can be discovered in parallel. Once the first CN starts, each next CN will start within a minute or two.
The 7th CN will wait for the first CN to complete, then the 7th discovery process will being. And so forth for each
CN thereafter, until all CN's are discovered.
The CN discovery process is complete when it moves to a "Ready to Provision" state.

The discovery process and the Day 0 process can run in parallel. Therefore, move to the Day 0 process, do NOT wait
for the CN discovery process to complete.
[root@pcamn01 ~]# python3 /usr/lib/python3.6/site-packages/pca_foundation/server/api
/test_hardware.py 253.255.0.31 list
Querying: https://253.255.0.31:8000/hardware?action=list
list :
[1, 'A8:69:8C:0B:6D:D3', '100.96.0.2', 'ilom-pcasn01', '', 'zfs-ilom',
'5.0.2.23', 'root', 1, 'ilom-AK00842951', 'OK', 'ignore', '3', None, None, None,
None, '100.96.2.2']
[2, 'A8:69:8C:0A:4D:0B', '100.96.0.3', 'ilom-pcasn02', '', 'zfs-ilom',
'5.0.2.23', 'root', -1, 'ilom-AK00842951', 'OK', 'ignore', '4', None, None, None,
None, '100.96.2.3']
[3, '3c:fd:fe:87:72:ca', '100.96.2.2', 'pcasn01', '', 'zfs', 'ak/SUNW,
maguroZ9@2013.06.05.8.40,1-2.40.4958.2', 'root', 1, 'AK00842951', 'On', 'Ready',
'1', '1', None, None, None, '100.96.0.2']
[4, '3c:fd:fe:92:27:72', '100.96.2.3', 'pcasn02', '', 'zfs', 'ak/SUNW,
maguroZ9@2013.06.05.8.40,1-2.40.4958.2', 'root', 1, 'AK00842951', 'On', 'Ready',
'2', 'None', None, None, None, '100.96.0.3']
[5, 'A8:69:8C:0A:2A:33', '100.96.0.33', 'ilom-pcamn01', '', 'mgmt-ilom', '',
'root', 1, 'ilom-AK00842951', 'OK', 'ignore', '13', None, None, None, None,
'100.96.2.33']
[6, 'A8:69:8C:0A:B8:7B', '100.96.0.34', 'ilom-pcamn02', '', 'mgmt-ilom', '',
'root', 1, 'ilom-AK00842951', 'OK', 'ignore', '14', None, None, None, None,
'100.96.2.34']
[7, 'A8:69:8C:15:81:5F', '100.96.0.35', 'ilom-pcamn03', '', 'mgmt-ilom', '',
'root', 1, 'ilom-AK00842951', 'OK', 'ignore', '15', None, None, None, None,
'100.96.2.35']
[8, '54:9f:c6:0d:df:a7', '100.96.2.1', 'pcaswmn01', '', 'switch-mgmt', '9.3(2)',
'admin', 1, 'FDO24451GK3', 'On', 'Ready', None, '26', None, None, None, None]
[9, 'bc:d2:95:a6:cb:74', '100.96.2.20', 'pcaswsp01', '', 'switch-spine', '9.3
(2)', 'admin', 1, 'FLM251507N9', 'On', 'Ready', None, '31', None, None, None, None]
[10, '34:73:2d:03:32:08', '100.96.2.21', 'pcaswsp02', '', 'switch-spine', '9.3
(2)', 'admin', 1, 'FLM251507N1', 'On', 'Ready', None, '32', None, None, None, None]
[11, '4c:5d:3c:40:bf:20', '100.96.2.22', 'pcaswlf01', '', 'switch-leaf', '9.3
(2)', 'admin', 1, 'FLM251503A7', 'On', 'Ready', None, '24', None, None, None, None]
[12, '34:73:2d:03:35:b0', '100.96.2.23', 'pcaswlf02', '', 'switch-leaf', '9.3
(2)', 'admin', 1, 'FLM251507MN', 'On', 'Ready', None, '25', None, None, None, None]
[13, 'a8:69:8c:0a:2a:30', '100.96.2.33', 'pcamn01', '', 'mgmt', '3.0.1', 'root',
1, 'AK00842951', 'On', 'ignore', '5', '5', None, None, None, '100.96.0.33']
[14, 'a8:69:8c:0a:b8:78', '100.96.2.34', 'pcamn02', '', 'mgmt', '3.0.1', 'root',
1, 'AK00842951', 'On', 'ignore', '6', '6', None, None, None, '100.96.0.34']
[15, 'a8:69:8c:15:81:5c', '100.96.2.35', 'pcamn03', '', 'mgmt', '3.0.1', 'root',
1, 'AK00842951', 'On', 'ignore', '7', '7', None, None, None, '100.96.0.35']
[16, 'a8:69:8c:15:61:2f', '100.96.0.64', 'ilom-pcacn001', '', 'compute-ilom',
'5.0.2.20.a', 'root', 1, 'ilom-2139XLD01R', 'OK', 'Ready_to_provision', '21', None,
None, None, None, '100.96.2.64']
[17, 'a8:69:8c:15:82:2f', '100.96.0.65', 'ilom-pcacn002', '', 'compute-ilom',
'5.0.2.20.a', 'root', 1, 'ilom-2139XLD01K', 'OK', 'Ready_to_provision', '22', None,
None, None, None, '100.96.2.65']
[18, '00:0b:38:be:22:34', '100.96.1.243', '', '', '', '', '', '', '', '',
'ignore', None, None, None, None, None, None]
[19, '00:0b:38:be:22:35', '100.96.1.244', '', '', '', '', '', '', '', '',
'ignore', None, None, None, None, None, None]
[20, 'a8:69:8c:15:82:3b', '100.96.0.66', 'ilom-pcacn003', '', 'compute-ilom',
'5.0.2.20.a', 'root', 1, 'ilom-2139XLD01P', 'OK', 'Ready_to_provision', '23', None,
None, None, None, '100.96.2.66']
[21, 'a8:69:8c:15:61:2c', '100.96.2.64', 'pcacn001', '', 'compute', 'PCA
Hypervisor:3.0.1-b526', 'root', 1, '2139XLD01R', 'On', 'ignore', '16', '10', 'b8:ce:
f6:96:ea:0c', '64', '1024', '100.96.0.64']
[22, 'a8:69:8c:15:82:2c', '100.96.2.65', 'pcacn002', '', 'compute', 'PCA
Hypervisor:3.0.1-b526', 'root', 1, '2139XLD01K', 'On', 'ignore', '17', '9', 'b8:ce:
f6:96:ea:7c', '64', '1024', '100.96.0.65']
[23, 'a8:69:8c:15:82:38', '100.96.2.66', 'pcacn003', '', 'compute', 'PCA
Hypervisor:3.0.1-b526', 'root', 1, '2139XLD01P', 'On', 'ignore', '20', '8', 'b8:ce:
f6:3e:68:7e', '64', '1024', '100.96.0.66']

References Time Check

Day 0 Configuration
 Warning!

The following data points are immutable values that cannot be edited or corrected after the Day 0 process
is committed. Please ensure these values are exactly correct according to customer expectations:

availability_domain
domain_name
System Name
fault_domain
realm
region
routing type (dynamic or static)
Public IP's - You may add to the list but you cannot modify entried already committed

At this point you will need the completed PCA-X9-Network-Configuration Worksheet that was filled out by the Customer. Install Guide,
Section 5.1
From the Service laptop / workstation connected to the Cisco Switch, using a web browser, connect to the MN VIP to launch
the to the Day 0 wizard. The Wizard will guide you through various interactive screens. Section 5.2

Section 7.9 Initial

Install Checklist
https://100.96.2.32:30099

Note

This will bring you to the "Private Cloud Appliance First Boot" Screen where you will enter the customer admin
account credentials.

Enter system information: 15 min

Warning

System name and Domain are immutable parameters. Once they are committed they cannot be changed.

Availability Domain
System Name
Domain
Rack Name
Description
Enter Routing information 15
minutes

Warning

The "routing type" field is an immutable parameter. One it is committed it cannot be changed.

For a Static routing strategy the following data will be required:

Static*
Uplink gateway IP Address*
Spine virtual IP* (comma-separated values if using the 4 port dynamic mesh topology)
Uplink VLAN
Uplink HSRP Group

For a Dynamic Routing Strategy the following data will be required:

Dynamic*
Peer1 IP*
Peer1 ASN*
Peer2 IP
Peer2 ASN
Uplink Gateway
Oracle ASN
BGP Topology
BGP KeepAlive Timer
BGP HoldDown Timer
Enable MD5 Authentication

Configure the Management Node customer facing network parameters:

Management Node Virtual IP Address*

Management Node Virtual IP Hostname*
Management Node 1 IP Address
Management Node 1 Hostname
Management Node 2 IP Address
Management Node 2 Hostname
Management Node 3 IP Address
Management Node 3 Hostname

Configure the customer facing data path networking parameters:

Spine Switch 1 IP Address*

Spine Switch 2 IP Address*
Uplink Port Speed*
Uplink Port Count*
Uplink VLAN MTU*
Uplink Netmask*
Uplink Port FEC

Configure the local NTP server

Caution

The NTP input screen will only accept one IP address.

You can optionally segregate administrative appliance access from the data traffic. To configure this network, enter the
following information for your datacenter, then click Next.

Enable Admin Networking

Admin Port Speed
Admin Port Count
Admin HSRP Group
Admin VLAN
Admin VLAN MTU
Admin Port FEC
Admin Gateway IP
Admin Netmask
Admin CIDR
Admin Spine1 IP
Admin Spine2 IP
Admin Spine Virtual IP

Configure your DNS servers

Configure the customers public IP address ranges ("public" meaning the customers enterprise access IP to the PCA system.
Not the internet.)

References Time Check

First Time Access to the Service Enclave
To log into Service Enclave UI using the administrative account created in the Day 0 wizard, in you browser, enter the following
URL, where pcasys1 is the name of your Oracle Private Cloud Appliance and example.com is your
domain.

https://adminconsole.pcasys1.example.com

Note

During the first login to the Service Enclave UI, you will be presented with the ASR Configuration screen. At this time,
it is recommended to configure ASR. If you choose to opt out of configuring ASR at this time you will be able to
configure ASR any time in the future.

To log into the Service Enclave CLI using the administrative account created in the Day 0 wizard, ssh to the PCA-ADMIN> shell.

Caution

Please be aware there are two pca-admin shells. The Service Enclave Administrator Account "PCA-ADMIN>" and
the root "(pca-admin)" shell. Each provides unique functionality.

Be sure you are logged into the correct shell.

From Management Node VIP root login

[root@pcamn02 ~]# ssh admin@localhost -p 30006

Password authentication
Password:
PCA-ADMIN>
From a remote bastion, laptop or workstation

[root@XXXXXXXXXX-your-laptop ~]# ssh admin@100.96.2.32 -p 30006

Password authentication
Password:
PCA-ADMIN>

Note

The same data can be used with putty or similar tool

References Time Check

Connecting to ASR
Using the Service Enclave UI, Select "ASR Phone Home" in the main pulldown menu. ASR Configuration and 10
Activation Checklist minutes
Select "Register" in the upper right and enter the following

Username*: Enter your Oracle Single Sign On (SSO) credentials, which can be obtained from My Oracle
Support.
Password*: Enter the password for your SSO account.
Proxy Username: To use a proxy host, enter a username to access that host.
Proxy Password: To use a proxy host, enter the password to access that host.
Proxy Host: To use a proxy host, enter the name of that host.
Proxy Port: To use a proxy host, enter the port used to access the host.
Endpoint: Destination endpoint ASR telemetry will be sent. Will be a either a "Direct Connection" or
through an "ASR Relay" such as a Platinum OASG or stand alone ASRManager.

Direct connection: https://transport.oracle.com

OASG Relay connection: http://<ip_address of OASG>:8234/asr
ASR relay connection: http://<ip_address of ASRM>:16161/asr

Alternatively you may use the Service Enclave CLI to configure ASR

PCA-ADMIN> showallcustomcmds
Operation Name: <Related Object(s)>
-----------------------------------
abort: Job
asrClientDisable: ASRPhonehome
asrClientEnable: ASRPhonehome
asrClientRegister: ASRPhonehome
asrClientSendTestMsg: ASRPhonehome
asrClientUnregister: ASRPhonehome

[...]

Reference Time Check

Verify Health
Log into the Service Enclave UI using the administrator account

Select the pulldown Menu in the upper left

Select "Rack Units"

Verify:

State = "ON"

Provision State = "Ready to Provision"

All component exist, according to your customers order

Note

pcasn02 will show as 'not available'. But is health if the state is "on".

Log into Grafana by selecting "Monitoring" in the upper left of the Service Enclave UI

admin/Welcome1

In the Welcome to Grafana screen, select: Dashboards manage -> PCA 3.0 Service Advisor -> Platform Health Check

To see the logs for "not health" services: Explore Loki Log Labels Jobs <select service presented as not healthy in Platform
Health Check>

Alternative access to Grafana in a new browser window:

https://grafana.pcasys1.us.example.com/

Alternative method to verify health from the CLI's, log into the management node VIP (root/Welcome1)

[root@pcamn01 ~]# kubectl get node -o wide

NAME STATUS ROLES AGE VERSION INTERNAL-
IP EXTERNAL-IP OS-IMAGE KERNEL-
VERSION CONTAINER-RUNTIME
pcamn01 Ready control-plane,master 4h59m v1.20.6+1.
el7 253.255.0.33 <none> Oracle Linux Server 7.9 5.4.17-2102.203.6.
el7uek.x86_64 cri-o://1.20.2
pcamn02 Ready control-plane,master 4h58m v1.20.6+1.
el7 253.255.0.34 <none> Oracle Linux Server 7.9 5.4.17-2102.203.6.
el7uek.x86_64 cri-o://1.20.2
pcamn03 Ready control-plane,master 4h56m v1.20.6+1.
el7 253.255.0.35 <none> Oracle Linux Server 7.9 5.4.17-2102.203.6.
el7uek.x86_64 cri-o://1.20.2
[root@pcamn01 ~]#
Log into to the Service Enclave CLI (PCA-ADMIN) using the account created in the Day 0 wizard.

Note

Multiple commands are needed to capture similar data shown in the SEUI Rack Units screen. Please keep in mind the
data is pulled from the identical underlying

structures.

PCA-ADMIN> list RackUnit

Command: list RackUnit
Status: Success
Time: 2022-02-22 21:51:34,381 UTC
Data:
id objtype name
-- ------- ----
1b8a2e07-6adc-438d-807c-2b5c696f6cd7 ComputeNode pcacn001
810c7890-d156-4be0-afe4-fc4179e0412b ComputeNode pcacn002
98eb2209-f85d-4167-81db-10b74b3e2071 ComputeNode pcacn003
b53d6b04-781c-4681-9eb0-be5af7d51364 LeafSwitch pcaswlf01
a93a0460-1278-4e7c-a6c0-ff9fac018cfc LeafSwitch pcaswlf02
7f6aaefa-9c0c-4cc4-9d7a-529832741e45 ManagementNode pcamn02
4f227c4e-a95a-45dc-9d18-f6c96f36255f ManagementNode pcamn01
b288202f-95fe-4c93-832e-dbc17f88b478 ManagementNode pcamn03
5fa20818-e33f-4565-8111-e8d9552c696a ManagementSwitch pcaswmn01
6a980120-3d70-47dd-9e20-2b49c942a0d7 SpineSwitch pcaswsp02
f3c354a3-14d1-4761-a804-701dd1d3f3ac SpineSwitch pcaswsp01
67f4e572-3ae4-4535-9466-1a2d753457d2 ZFSAppliance pcasn02
cfe6e0a7-49cf-48ce-a2c1-9f8a9977bb96 ZFSAppliance pcasn01

PCA-ADMIN> list computeNode

Command: list computeNode
Status: Success
Time: 2022-02-22 21:52:30,605 UTC
Data:
id name provisioningState
provisioningType
-- ---- -----------------
----------------
810c7890-d156-4be0-afe4-fc4179e0412b pcacn002 Provisioned KVM
98eb2209-f85d-4167-81db-10b74b3e2071 pcacn003 Ready to Provision Unspecified
1b8a2e07-6adc-438d-807c-2b5c696f6cd7 pcacn001 Provisioned KVM
 PCA-ADMIN> list ?
AuthorizationGroup
ComputeNode
Event
Fault
FaultHistoryLog
IdentityProvider
IdpGroupMapping
Job
LeafSwitch
ManagedSession
ManagementNode
ManagementSwitch
Rack
RackUnit
SpineSwitch
Tenant
User
ZFSAppliance

References Time Check

Provision CN's
Log into Service Enclave using the user account created in the Day 0 wizard

For example, https://adminconsole.pcasys1.example.com where pcasys1 is the name of your Oracle Private Cloud
Appliance and example.com is your domain.
3 min
Select "Rack Units" per CN
Select "Actions" button for CN you want to provision
Select "Provision"

NOTE: Allow all CN's to complete the provisioning process before moving on.

References Time Check

Software Patch/Upgrade
Verify local yum repository
Verify that the correct repositories appear on your local mirror.

# sudo yum repolist

Loaded plugins: langpacks, rhnplugin, ulninfo

This system is receiving updates from ULN.

repo id repo name status

pca301_x86_64_containers 3.0.1 Container Images 39

pca301_x86_64_fw PCA 3.0.1 Firmware 2

pca301_x86_64_hypervisor PCA 3.0.1 Hypervisor 9

pca301_x86_64_mn PCA 3.0.1 MN 193

pca301_x86_64_oci PCA 3.0.1 OCI Compute Images 33

repolist: 1,674

Patching Guide,
Prepare PCA for Patching/Upgrade Chapter 2 Step 4

Configure the management nodes to receive yum updates from the local YUM repository Chapter 3 All steps
Verify you have permissions to perform patching operations, RPM's are available and the PCA is ready to Patch
/Upgrade
 Patch Guide
Assess the Patches to be installed
In the event multiple patches are released it is required to step through applying the patches in a specific order as
follows:

1. Host
2. MySQL Cluster
3. Vault and ETCD (these can be done in either order but must be done consecutively)
4. Kubernetes Cluster
5. Platform
6. Compute
7. Any Firmware

Patch Guide 5 hours

Initiate patch/upgrade process
Chapter 4
Log into Service Enclave using the administrative account created in the Day 0 wizard

For example, https://adminconsole.pcasys1.example.com where pcasys1 is the name of your Oracle Private
Cloud Appliance and example.com is your domain.

Pull down left menu and select "Patch and Upgrade"

Verify patch/upgrade completed successfully

Example of checking logs (coming soon)

Referrences Time Check

Install and Power on Field Installable Compute
Nodes
If field installable compute nodes were not installed before moving into the data center, install compute nodes now. CN Field install 30 Min per
checklist CN

Power on all Compute Nodes

Verify Compute Node status 10 min per

CN
PCA-ADMIN>

PCA-ADMIN> list computenode

References Time Check

Connect to Platinum Services
Platinum Certification in progress. Target completion end of March 2022. TBD TBD

References Time Check

Change Your default passwords on all components
In the pca-admin shell (lower case), change the root password for all rack components (ILOM, Host OS, switch, ZFS). In the pca-
admin shell there is a unique command for each component type. When executed, all components of that type will have their
passwords updated. For example, "change password compute" will change the password for all compute nodes in the rack.

Caution

Do not change passwords on individual components. For example, do not change an ILOM password by logging into
the ILOM of a component. Always use the pca-admin shell.

To access the root pca-admin shell, type "pca-admin" at the root prompt. No password is required:

[root@pcamn01 ~]# pca-admin

(pca-admin)
(pca-admin) help

Documented commands (use 'help -v' for verbose/'help <topic>' for details):
===========================================================================
alias exit help macro quit

Application commands (type help <topic>):

=========================================
add key change password zfs remove key
change password compute change password zfs-ilom rollback
change password compute-ilom complete rollback list
change password mgmt help rollback service
change password mgmt-ilom hw-monitor rollback show
change password switch-leaf list keys version
change password switch-mgmt node
change password switch-spine node list

(pca-admin)

Change the password for the mysql database by running the following from the active managment node.

[root@pcamn02 ~]# /var/lib/pca-foundation/scripts/pca_change_mysql_root_password.py

Updating MySQL root password
Retrieving credentials for /v1/kv-v2/data/hw/server/mgmt/mysql/root
Successfully retrieved credentials
Password:
Verify:

References Check
Install Complete
Ensure customer has access to all necessary resources https://www.oracle.com/assets/services-ovca-ds-1990356.pdf

Service Enclave
Customer Enclave
CLI access

Turn Rack Over to Customer

Appendix A: Rack Constraints

Rack Rules
Base Rack

Minimum orderable configuration is 3 CN's in RU's 8, 9, 10

RU's 11 - 15 are "Reserved" for CN's and will be cabled from the factory
Reserved RU's MUST be used first, contiguous, bottom up (RU11 - 15)
Removing one CN afford power for 3 storage trays

Flex Bays:

Any Flex Bay can accommodate four compute nodes, two DE3-24P, or one DE3-24C
Unused Flex Bays are not cabled.
Components are installed in the following order, working from bottom up: CN's, DE3-23P then DE3-24C.
If one CN or one 24P is installed in a Flex Bay, the Flex Bay is committed and cabled for that component type (cannot mix component types)
The number of CN's installed in MFG is restricted to 3, 6, 9, 12, 15, 18, and 21. The "rule of 3". IF the number of CN's ordered is different, the odd
number

will be shipped separately and installed in the field.

There are three use cases where Flex Bay cabling will need to installed in the field, to support the field installed CN's

1) When the number of CN's ordered is 13, RU20 will need to be wired in the field

2) When the number of CN's ordered is 14, RU20 and RU21 will need to be wired in the field

3) When the number of CN's ordered is 17, RU34 will need to be wired in the field

Strings

Max string depth is 2 DE's

Will not support mixing 24P's and 24C's in a string

Rack Elevations
Appendix B: Data Switch Cabling Reference
Cable Type and Part #'s
Data Switch Connection Reference
Appendix C: Management Switch Cabling Reference
Cable Type and Part #'s

Management Switch Connection Reference

Appendix D: ZS Appliance Cluster Cabling Reference
Appendix E: Power Scheme Reference

15KVA (Single and Three Phase)

Storage Enclosure Power Cabling

Compute Node and Switch Power Cabling

Max of 9 Compute Nodes
22KVA Single Phase

Storage Enclosure Power Cabling

Compute Node and Switch Power Cabling
Max of 18 Compute Nodes
24KVA Three Phase

Storage Enclosure Power Cabling

Compute Node and Switch Power Cabling
Max of 20 Compute Nodes
Appendix F: Default Logins and Passwords
 Account User Name Password

Service Enclave UI and CLI User Defined during Day 0 setup

Grafana admin Welcome1

Management Node VIP and Host OS root Welcome1

pca-admin (lower case) None: Accessed through

root on the MN VIP

ILOM (MN, CN, ZS) root Welcome1

Cisco admin Welcome1

PDU admin Welcome1

Compute Node Host OS root Welcome1

Appendix G: Firewall Ports