Professional Documents
Culture Documents
Oracleflexasmflexcluster 131219101112 Phpapp01
Oracleflexasmflexcluster 131219101112 Phpapp01
1
Dell - Internal Use - Confidential
Agenda
2
Dell - Internal Use - Confidential
The new Flex Clusters and Flex ASM features are designed to provide scalable and
high-availability cluster infrastructure for database cloud and application cloud.
Oracle 12c Flex Clusters introducing a new two-layered hub-andspoke topology to
the cluster architecture. The number of Hub Nodes in an Oracle Flex Cluster can be
as many as 64. The number of Leaf Nodes can be many more. Hub Nodes are similar
to Oracle Grid Infrastructure nodes in an Oracle Clusterware standard Cluster
configuration: they are tightly connected, and have direct access to shared storage.
Leaf Nodes are different from standard Oracle Grid Infrastructure nodes, in that they
do not require direct access to shared storage, but instead request data through Hub
Nodes. Hub Nodes can run in an Oracle Flex Cluster configuration without having any
Leaf Nodes as cluster member nodes, but Leaf Nodes must be members of a cluster
that includes at least one Hub Node.
In Oracle Flex ASM only a small number of cluster nodes run Oracle ASM instances.
These ASM instances are connected by all the database instances on the cluster to
provide storage access to these database instance, when a ASM fails database
instance that connects the failed asm instance will switch to other ASM instance.
Oracle Flex ASM is an option in Oracle 12c ASM which you can enable or disable. If
there is a standard cluster installed we can enable or disable flex ASM any time. With
Oracle Database 12c, the database instances and ASM instances can now be on
different servers, so ASM instances require a password file that is used to
authenticate a database instance connecting to an ASM instance
3 within the cluster.
For
configurations
Dell most
- Internal
Use - Confidential a default ASM password file is created during installation and
5
Dell - Internal Use - Confidential
In above pics, 4 node hub cluster is tightly connected with private interconnected which is same as 11gr2 RAC. Outside the hub
center, 12 leaf nodes form 4 groups. None of the leaf nodes are connected to other leaf nodes. Oracle rac db can run on leaf
nodes, leaf node dont have direct access to storage server, so the rac db on leaf nodes will need to get the database through
hub nodes. The technology to implement this remote storage access is called as Oracle Flex ASM. For this reason, when you
configure a Flex Cluster, Oracle Flex ASM is automatically enabled. Benefit of running the loosely coupled architecture of the
Flex Cluster is to provide the high availability of the Oracle Clusterware of the applications tier. We can run application tiers on
the Flex Cluster, which provides high availability, such as failover capability, against server hardware failover and planned
maintenance.
With this hub-andspoke topology, the Flex Cluster in Oracle 12cR1 is designed to scale up 64 Hub nodes and many more Leaf
nodes. OCR and voting disk accessible only to Hub nodes and not Leaf nodes. if we query the voting disks or OCR access from
a Leaf node:
$ crsctl query css votedisk
CRS-1668: operation is not allowed on a Leaf node
$ ocrcheck
PROT-605: The 'ocrcheck' command is not supported from a Leaf node.
If a Leaf node fails, this node will be evicted from the cluster. The services running on the failed Leaf node are failed over to
other Leaf nodes that are connected the same Hub node. In this way, the other part of the cluster nodes will be not impacted by
this Leaf nodes failure. The network heartbeat is used to maintain network connectivity between a Leaf node and the Hub node
heartbeat is tolerable is defined by the leafmisscount setting which by default is 30 seconds. If the heartbeat failure passes this
leafmisscount setting, then the Leaf node either will be reconnected to the other Hub node or will be evicted from the cluster.
$ crsctl get css leafmisscount
CRS-4678: Successful get leafmisscount 30 for Cluster Synchronization Services
6
Dell - Internal Use - Confidential
7
Dell - Internal Use - Confidential
Oracle Flex ASM fundamentally changes the ASM cluster architecture. Before the introduction of Oracle Flex ASM in Oracle
Database 12c, an ASM instance ran on every server in a cluster. These ASM instances communicated with other ASM
instances on other servers in the cluster and collectively they presented shared Disk Groups to the database clients running in
the cluster. This collection of ASM servers formed an ASM cluster. If an ASM instance were to fail, then all the database
instances running on the same server as the failing ASM instance failed as well.
In Oracle Database release 12c, only a smaller number of ASM instances need run on a subset of servers in a cluster. The
number of ASM instances running is called the ASM cardinality. If a server fails that is running an ASM instance, Oracle
Clusterware starts a replacement ASM instance on a different server to maintain the ASM cardinality. If an ASM instance fails
for whatever reason, then active Oracle 12c database instances that were relying on that ASM instance will reconnect to
another surviving ASM instance on a different server. Furthermore, database instances are connection load balanced across
the set of available ASM instances. The default ASM cardinality is 3, but that can be changed with a Clusterware command.
These features are collectively called Oracle Flex ASM.
8
Dell - Internal Use - Confidential
Oracle ACFS and ADVM through Oracle ASM proxy instance: In Flex ASM configuration, Oracle ASM proxy is introduced to provide support
for Oracle ACFS and Oracle ASM Dynamic Volume Manager (Oracle ADVM). In order for Flex ASM clients to connect to remote ASM
instances, Flex ASM introduces a new type of network called the ASM network between Flex ASM clients and ASM instances. With this new
type of network, inside of Oracle Clusterware 12c, there are four types of networks:
1. Public network, which usually connects the cluster nodes to the Corporate network;
2. Private network for the interconnect communication between the cluster nodes;
3. ASM network for ASM clients to connect to ASM instances;(needed only for flex ASM)
4. Storage network for the cluster nodes to access the shared storage.
In Flex Clusters, Leaf nodes are not required to have direct storage access. Leaf nodes to access ASM disks on the shared storage through
the Hub nodes, Oracle Flex ASM is required in Oracle Flex Clusters.
If you convert a standard cluster to a Flex Cluster after the OUI installation, you need to enable the Flex ASM on the cluster before changing
the cluster mode to the Flex Cluster. In a Flex Cluster configuration, ASM instances have to run the Hub nodes because Hub nodes are
required to have direct access the shared storage.
ASM Network
With Flex ASM Oracle 12c, a new type of network is called the ASM network. it is used for communication between ASM and its clients and
is accessible on all the nodes. All ASM clients in the cluster have access to one or ore ASM network. Also it is possible to configure single
network can perform both function as a private and an ASM network (oifcfg getif, you should see cluster_interconnect,asm)
ASM Listeners
To support FLex ASM, a set of ASM listeners are configured for every ASM network. Up to three ASM listener addresses are registred as
remote listeners in each client database instance. All clients connections are load balanced across the entire set of ASM instances
ora.ASMNET1LSNR_ASM.lsnr
ONLINE ONLINE
auvmdbacoeora07
ONLINE ONLINE
auvmdbacoeora08
STABLE
STABLE
9
Dell - Internal Use - Confidential
The default Grid listener listens for ASM,APX, Grid Infrastructure Management and Database instances
LSNRCTL for Linux: Version 12.1.0.1.0 - Production on 11-NOV-2013 11:19:56
Copyright (c) 1991, 2013, Oracle. All rights reserved.
Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))
STATUS of the LISTENER
-----------------------Alias
LISTENER
Version
TNSLSNR for Linux: Version 12.1.0.1.0 - Production
Start Date
11-NOV-2013 03:14:50
Uptime
0 days 8 hr. 5 min. 7 sec
Trace Level
off
Security
ON: Local OS Authentication
SNMP
OFF
Listener Parameter File /u01/app/12.1.0/grid/network/admin/listener.ora
Listener Log File
/u01/app/oracle/diag/tnslsnr/auvmdbacoeora07/listener/alert/log.xml
Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.49.41.72)(PORT=1521)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.49.41.101)(PORT=1521)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcps)(HOST=auvmdbacoeora07.local)(PORT=5500))
(Security=(my_wallet_directory=/u01/app/oracle/product/12.1.0/dbhome_1/admin/ora12c/xdb_wallet))(Presentation=HTTP)(Session=RAW))
Services Summary...
Service "+APX" has 1 instance(s).
Instance "+APX1", status READY, has 1 handler(s) for this service...
Service "+ASM" has 1 instance(s).
Instance "+ASM1", status READY, has 2 handler(s) for this service...
Service "ora12c.local" has 1 instance(s).
Instance "ora12c1", status READY, has 1 handler(s) for this service...
Service "ora12cXDB.local" has 1 instance(s).
Instance "ora12c1", status READY, has 1 handler(s) for this service...
Service "pdbora12c.local" has 1 instance(s).
Instance "ora12c1", status READY, has 1 handler(s) for this service...
The command completed successfully
10
Dell - Internal Use - Confidential
ADVM Proxy
ASM Dynamic Volume Manager (ADVM) Proxy is a special Oracle instance. It enables ADVM to connect to Flex ASM and is required to
run on the same node as ADVM and ACFS. It can be shutdown when ACFS is not running.
The ADVM proxy instance has its ORACLE_SID set to +APX<node number>
ora.proxy_advm
ONLINE ONLINE
auvmdbacoeora07
STABLE
ONLINE ONLINE
auvmdbacoeora08
STABLE
[oracle@auvmdbacoeora07 bin]$ ps -elf | grep pmon | grep APX
0 S oracle 23994 1 0 78 0 - 350500 - 03:15 ?
00:00:00 apx_pmon_+APX1
[oracle@auvmdbacoeora07 bin]$ ./srvctl status asm -proxy
ADVM proxy is running on node auvmdbacoeora07,auvmdbacoeora08
[oracle@auvmdbacoeora07 bin]$ ./srvctl stop asm -proxy -node auvmdbacoeora07
[oracle@auvmdbacoeora07 bin]$ ./srvctl status asm -proxy
ADVM proxy is running on node auvmdbacoeora08
[oracle@auvmdbacoeora07 bin]$ ps -elf | grep pmon | grep APX
[oracle@auvmdbacoeora07 bin]$
[oracle@auvmdbacoeora07 bin]$ ./srvctl start asm -proxy -node auvmdbacoeora07
[oracle@auvmdbacoeora07 bin]$ ps -elf | grep pmon | grep APX
0 S oracle 22234 1 0 78 0 - 350500 - 11:33 ?
00:00:00 apx_pmon_+APX1
[oracle@auvmdbacoeora07 bin]$ ./srvctl status asm -proxy
ADVM proxy is running on node auvmdbacoeora07,auvmdbacoeora08
11
Dell - Internal Use - Confidential
12
Dell - Internal Use - Confidential
As a part of the configuration, you need to specify the Hub nodes and Leaf nodes. below figure shows a seven-node Flex Cluster with
three Hub nodes and four Leaf nodes.
13
Dell - Internal Use - Confidential
As a part of Flex Cluster configuration, Oracle Flex ASM is implicitly enabled. After the successful installation of the GI, you
should be able to verify the cluster is in Flex Cluster.
$ crsctl get cluster mode status
Cluster is running in "flex" mode
After that we need to restart oracle high availbility services on this node.
# crsctl stop crs
# crsctl start crs wait
$crsctl get node role config
Node 'knewracn2' configured role is 'hub
If you need to change the leaf node to hub node, you need to check vip exist on that node or not, if not we can add vip on
this node.
$ srvctl config vip -n knewracn5
PRKO-2310 : VIP does not exist on node knewracn5
# srvctl add vip -n knewracn5 -A 172.16.150.201/255.255.255.0/eth0 -k 1
$crsctl get node role status -node knewracn5
Node 'knewracn5' active role is 'leaf
To check the css misscount and leafmisscount setting, run the following commands:
$ crsctl get css misscount
CRS-4678: Successful get misscount 30 for Cluster Synchronization Services.
$ crsctl get css leafmisscount
CRS-4678: Successful get leafmisscount 30 for Cluster Synchronization Services
14
Dell - Internal Use - Confidential
The Flex ASM configuration is completed as a part of the GI installation process. By default, three-node Hub nodes will be
chosen to run the ASM instances and the rest of the cluster nodes will be Flex ASM clients. The Flex ASM listener will also be
created. You also can enable the Flex ASM on a standard ASM cluster by converting the standard ASM to the Flex ASM.
However, as the prerequisite, prior to the conversion, you need to have at least one network interface for the ASM network.
Hub-and-spoke topology is the key architecture feature that segments the cluster into groups of nodes. Two
fundamental impacts: 1) limiting the size of the hub to reduce contention to OCR and voting disks and 2) less heartbeats
network traffic exchange between the nodes
Clients on Leaf Nodes use GNS (Grid Naming Service) to locate Hub Node services. This requires access to GNS through
a fixed VIP running on one of the nodes so that Leaf Node clients have reliable naming service within the cluster.
You can disable or enable Flex cluster functionalities. By the default, Flex cluster functionality is disabled.
15
Dell - Internal Use - Confidential
Managing Flex ASM Once you initially configure a Flex ASM instance through OUI or through the conversion method, there is
not much specific work you need to do for this Flex ASM feature. You can do all the administrative work on a Flex ASM instance
in the same way that you do for a standard ASM instance. There is not any specific instance parameter designed for the Flex
ASM instance, and all the instance parameters for a Flex ASM instance are same as for a standard ASM instance. You can
check whether or not Oracle ASM is enabled in your cluster environment using the asmcmd command:
$asmcmd showclustermode
ASM cluster : Flex mode enabled
The SRVCTL status command shows the cluster nodes where ASM instances run:
$ srvctl status asm -detail
ASM is running on knewracn2,knewracn1,knewracn4
ASM is enabled.
And the SRVCTL config command shows more details about the Flex ASM configurations:
$ srvctl config asm
ASM home: /u01/app/12.1.0/grid
Password file: +DATA1/orapwASM
ASM listener: LISTENER
ASM instance count: 3
Cluster ASM listener: ASMNET1LSNR_ASM
16
Dell - Internal Use - Confidential
After configuration you can see one more instance are created +APX1 and +APX2 on both nodes. APX is nothing but a asm
proxy to support acfs and advm. If you dont want to use you can disable it.
[root@auvmdbacoeora07 bin]# ps -ef|grep pmon
oracle 2475 1 0 02:03 ?
00:00:00 ora_pmon_ora12c1
oracle 4172 1 0 02:04 ?
00:00:00 apx_pmon_+APX1
root
9613 27483 0 02:12 pts/1 00:00:00 grep pmon
oracle 31860 1 0 02:03 ?
00:00:00 asm_pmon_+ASM1
[oracle@auvmdbacoeora07 bin]$ ./crsctl get cluster mode status
Cluster is running in "flex" mode
[root@auvmdbacoeora08 bin]# ./oifcfg getif
eth0 10.49.41.64 global public
eth1 10.49.16.0 global cluster_interconnect
eth2 192.168.41.0 global asm
18
Dell - Internal Use - Confidential
19
Dell - Internal Use - Confidential
20
Dell - Internal Use - Confidential
21
Dell - Internal Use - Confidential