10

Diagnosing Oracle Clusterware

and RAC Components

Copyright 2007, Oracle. All rights reserved.

Objectives

After completing this lesson, you should be able to:

Collect Oracle Clusterware diagnostic files
Use Cluster Verify

10 - 2

Copyright 2007, Oracle. All rights reserved.

One Golden Rule in RAC Debugging

Always make sure that your nodes have exactly the

same system time to:
Facilitate log information analysis
Ensure accurate results when reading GV$ views
Avoid untimely instance evictions

The best recommendation is to synchronize nodes

using Network Time Protocol.

10 - 3

Copyright 2007, Oracle. All rights reserved.

Oracle Clusterware Main Log Files

ORA_CRS_HOME

/etc

ORACLE_HOME

oracle
log

log
<nodename>.oprocd.log

<hostname>

crsd

cssd

<hostname>

evmd

racg

client

racgeut

racgevtf

racgmain

client

racg

alert<nodename>.log
racgeut

10 - 4

racgimon racgmain

Copyright 2007, Oracle. All rights reserved.

racgmdb

Diagnostics Collection Script

Script to collect all important log files:

Must be executed as root
Is located in $ORA_CRS_HOME/bin/
Is called diagcollection.pl

Generates the following files in the local directory:

#
#
#
#
#

10 - 5

basData_<hostname>.tar.gz
crsData _<hostname>. tar.gz

ocrData _<hostname>. tar.gz

oraData _<hostname>. tar.gz

export ORACLE_HOME=/u01/app/oracle/product/10.2.0/db_1
export ORA_CRS_HOME=/u01/crs1020
export ORACLE_BASE= =/u01/app/oracle
cd $ORA_CRS_HOME/bin
./diagcollection.pl -collect

Copyright 2007, Oracle. All rights reserved.

Managing Diagnostic Data in RAC

Support Workbench
DIAGNOSTIC_DEST

ADR
Base

Shared
ADR base

diag

asm

rdbms

+asm

DB
Name

+ASM1

+ASMn

SID1

SIDn

ADR
Home

ADRCI

10 - 6

GV$DIAG_INFO

Copyright 2007, Oracle. All rights reserved.

Cluster Verify: Overview

To verify that you have a well-formed cluster for Oracle

Clusterware and RAC:
Installation
Configuration
Operation

Full stack verification

Nonintrusive verification
Diagnostic mode seeks to establish a reason for the
failure of any verification task.
Easy-to-use interface:
Stage commands
Component commands
10 - 7

Copyright 2007, Oracle. All rights reserved.

Cluster Verify Stages

-pre dbcfg

$ cluvfy stage -list

Configures
RAC DB

-pre dbinst
Installs
RAC

-pre crsinst

Installs
CRS

-pre cfs

-post crsinst
Sets up OCFS
(OPT)

-post cfs
User sets up the
hardware,
network, & storage

10 - 8

-post hwos

Copyright 2007, Oracle. All rights reserved.

Cluster Verify Components

An individual subsystem or a module of the RAC cluster

is known as a component in CVU.
The availability and integrity of a cluster component can
be verified.
Components can be simple like a specific storage
device, or complex like the Oracle Clusterware stack:
Space availability
Shared storage
accessibility
Node connectivity
Cluster File System integrity
Oracle Clusterware integrity

10 - 9

Cluster integrity
Administrative
privileges
Peer compatibility
System requirements
$ cluvfy comp -list

Copyright 2007, Oracle. All rights reserved.

Cluster Verify Locations

Download it from OTN:

Create a local directory.
Copy and extract cvu_<OS>.zip.

Oracle software DVD:

Disk1 directory
runcluvfy.sh

Oracle Clusterware home:

$ORA_CRS_HOME/bin/cluvfy

Oracle Home:
$ORACLE_HOME/bin/cluvfy

10 - 10

Copyright 2007, Oracle. All rights reserved.

Cluster Verify Configuration File

$ cat cvu_config
# Configuration file for CVU
# Version: 011405
#
#CV_ORACLE_RELEASE=11gR1
#CV_NODE_ALL=
CV_RAW_CHECK_ENABLED=TRUE
CV_ASSUME_DISTID=Taroon
#CV_XCHK_FOR_SSH_ENABLED=TRUE
#ORACLE_SRVM_REMOTESHELL=/usr/bin/ssh
#ORACLE_SRVM_REMOTECOPY=/usr/bin/scp
10 - 11

Copyright 2007, Oracle. All rights reserved.

Cluster Verify: Examples

$ cluvfy comp sys -n node1,node2 -p crs -verbose

$ cluvfy comp ssa -n all -s /dev/sda1

$ cluvfy comp space -n all -l /home/product -z 5G

$ cluvfy comp nodereach -n node2 -srcnode node1

$ cluvfy comp nodecon -n node1,node2 i eth0 -verbose 5

$ cluvfy comp admprv -n all -o user_equiv -verbose

$ cluvfy comp nodeapp -n all -verbose

$ cluvfy comp peer -n all verbose | more

10 - 13

Copyright 2007, Oracle. All rights reserved.

Cluster Verify Output: Example

$ cluvfy comp crs -n all -verbose
Verifying CRS integrity
Checking CRS integrity...
Checking daemon liveness...

Liveness of all the daemons

Node Name
CRS daemon
CSS daemon
EVM daemon
------------ ------------- --------------- ---------atlhp9
yes
yes
yes
atlhp8
yes
yes
yes
Checking CRS health...
Check: Health of CRS
Node Name
CRS OK?
---------------------------------- -----------------------atlhp9
yes
atlhp8
yes
Result: CRS health check passed.
CRS integrity check passed.
Verification of CRS integrity was successful.
10 - 15

Copyright 2007, Oracle. All rights reserved.

Summary

In this lesson, you should have learned how to:

Collect Oracle Clusterware diagnostic files
Use Cluster Verify

10 - 16

Copyright 2007, Oracle. All rights reserved.

Practice 10: Overview

This practice covers the following topics:

Identifying Oracle Clusterware log files
Fixing voting disk corruptions

10 - 17

Copyright 2007, Oracle. All rights reserved.