Oracle RAC

PSU Update as ROOT:

 /u01/app/11.2.0/grid_1/OPatch/opatch auto /export/home/oracle/install/psu/2019/q1/28813878/ -ocmrf /export/home/oracle/ocm.rsp

Main command: Clusterware Resource Status Check :

crsctl status resource -t 

or shorter:

crsctl stat res -t

Shutdown and Start sequence of Oracle RAC components

Stop Oracle RAC (11g, 12c)

1. emctl stop dbconsole (11c only. In 12c DB Express replaces dbconsole and doesn’t have to be stopped )
2. srvctl stop listener [-listener listener_name] [-node node_name] [-force] (stops all listener services)
3. srvctl stop database -d db_unique_name [-stopoption stop_options] [-eval(12c only)] [-force] [-verbose]
4. srvctl stop asm [-proxy] [-node node_name] [-stopoption stop_options] [-force]
5. srvctl stop nodeapps [-node node_name] [-gsdonly] [-adminhelper] [-force] [-relocate] [-verbose]
6. crsctl stop crs

Start Oracle RAC (11g, 12c)

1. crsctl start crs
2. crsctl start res ora.crsd -init
3. srvctl start nodeapps [-node node_name] [-gsdonly] [-adminhelper] [-verbose]
4. srvctl start asm [-proxy] [-node node_name [-startoption start_options]]
5. srvctl start database -d db_unique_name [-eval(12c only)]] [-startoption start_options] [-node node_name]
6. srvctl start listener [-node node_name] [-listener listener_name] (start all listener services)
7. emctl start dbconsole (11c only)

To start resources of your HA environment if that are still down(e.g. ora.ons, Listener):

crsctl start resource -all

DEBUG

Starting with Oracle 12c, the log and trace files of the clusterware files are stored in the Automatic Diagnostic Repository (ADR) under the ADR_HOME location $ADR_BASE/diag/crs/`hostname`/crs.

$ adrci

adrci> show homes

Manage low level cluster resources: CRS, HAS, cluster

How to display the status of resources in RAC?

Clusterware Resource Status Check : crsctl status resource -t (or shorter: crsctl stat res -t)

Find offline resources: crs_stat -t | grep -i offline

How to check the current status of a cluster?

crsctl check cluster
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online

To know the cluster name: olsnodes -c

How to check the current status of CRS?

crsctl check crs
CRS-4638: Oracle High Availability Services is online (has)
CRS-4537: Cluster Ready Services is online (crs)
CRS-4529: Cluster Synchronization Services is online (css)
CRS-4533: Event Manager is online

How to Stop/Start the local node?

crsctl stop has
This command will also abort the database and CRS. Local Listeners will stop and VIP listeners will migrate elsewhere.

crsctl start has
This command will start all the CRS components, listeners and the database.

How to Stop/Start the whole cluster?

crsctl stop cluster -all
crsctl start cluster -all

How to To start and stop oracle clusterware (CRS)?

crsctl stop crs
crsctl start crs

Manage Network components

How to display global public and global cluster_interconnect?

C:\Windows\system32>oifcfg ge34f
Heartbeat 194.56.67.0 global cluster_interconnect,asm
Production 10.356.3.0 global public

How to check if nodeapps running on a node?

srvctl status nodeapps [-n my-node]

For each VIP address: network enabled/disabled, running on node host1 or not running.

Nodeapps are standard set of oracle application services which are started automatically for RAC. Node apps Include:

1) VIP
2) Oracle Net listeners
3) Global Service Daemon
4) Oracle Notification Service (ONS).

Nodeapp Services run on each node of the cluster. They switch over to other nodes through VIP during a failover.

How to check the SCAN Configuration?

The SCAN makes it possible to add or remove nodes from the cluster without needing to reconfigure clients.

Using CLUVFY to Confirm DNS is Correctly Associating the SCAN addresses.

cluvfy comp scan
Verifying Single Client Access Name (SCAN) …PASSED
Verification of SCAN was successful.
CVU operation performed: SCAN
Date: Oct 19, 2017 1:17:59 PM
CVU home: C:\…\grid_home\bin\..\
User: .\VFENOLL

How to display the current configuration of the SCAN VIPs?

srvctl config scan
SCAN name: MY-CLUSTER-SCAN, Network: 1
Subnet IPv4: 10.104.2.0/255.255.255.0/Production, static
Subnet IPv6:
SCAN 1 IPv4 VIP: 10.404.2.677
SCAN VIP is enabled.
SCAN VIP is individually enabled on nodes:
SCAN VIP is individually disabled on nodes:
SCAN 2 IPv4 VIP: 10.404.2.618
SCAN VIP is enabled.
SCAN VIP is individually enabled on nodes:
SCAN VIP is individually disabled on nodes:
SCAN 3 IPv4 VIP: 10.404.2.619
SCAN VIP is enabled.
SCAN VIP is individually enabled on nodes:
SCAN VIP is individually disabled on nodes:

How to display the status of SCAN VIPs and SCAN listeners?

srvctl status scan
SCAN VIP scan1 is enabled
SCAN VIP scan1 is running on node my-node1
SCAN VIP scan2 is enabled
SCAN VIP scan2 is running on node my-node2
SCAN VIP scan3 is enabled
SCAN VIP scan3 is running on node my-node1

If you want to add or modify a scan VIP: srvctl add | modify scan -n my-scan

To delete it: srvctl remove scan

How to display the status of SCAN listeners?

srvctl status scan_listener
SCAN Listener LISTENER_SCAN1 is enabled
SCAN listener LISTENER_SCAN1 is running on node my-node1
SCAN Listener LISTENER_SCAN2 is enabled
SCAN listener LISTENER_SCAN2 is running on node my-node2
SCAN Listener LISTENER_SCAN3 is enabled
SCAN listener LISTENER_SCAN3 is running on node my-node1

If you want to add or remove a scan_listener:

srvctl add | remove scan_listener

o change the port:

srvctl modify scan_listener -p

Manage the Oracle Cluster Registry (OCR)

How to verify the integrity of OCR?

cluvfy comp ocr -n all -verbose
Verifying OCR Integrity …PASSED
Verification of OCR integrity was successful.
CVU operation performed: OCR integrity
Date: Oct 11, 2017 4:56:01 PM
CVU home: C:\…grid_home\bin\..\
User: \VFENOLL

How to backup the OCR?

Oracle takes physical backup of OCR automatically every 3 hours. Default location is CRS_home/cdata/my_cluster_name/OCRBackup. The ocrconfig tool is used to make daily copies of the automatically generated backup files.

Show backups:

ocrconfig -showbackup

Change default location of physical OCR copies:

ocrconfig -backuploc

After that, you have to copy these files on tape or in another backup location (cp -p -R CRS_home/cdata/my_cluster_name /u03/backups )

To do a manual backup:

ocrconfig -export /u03/backups/exports/OCR_exportBackup.dmp

How to recover OCR from physical or export backup?

Pre-requisite: All RAC components shutdow

Recover OCR from automatic physical backups:

crconfig -restore CRS_home/cdata/my_cluster_name/OCRBackup/backup00.ocr

Recover OCR from export backup:

ocrconfig -import /u03/backups/exports/OCR_exportBackup.dmp

How to backup the Voting disks?

In older versions of Oracle Clusterware you have to backup voting disks with the dd command. Starting with Oracle Clusterware 11g Release 2 you no longer need to backup them. Voting disks are automatically backed up as a part of the OCR.

Manage database components

How to find the name of the database?

This name is useful as it is used in RAC commands with -d parameter. With SQL*Plus:

connect / as sysdba
show parameter db_unique_name

With crsctl:

crsctl status resource -t | grep db

How to inspect the database configuration?

srvctl config database -d my-db-name
Database unique name: my-db-name
Database name: my-db-name
Oracle home: D:\oracle\db\product\12.2.0\dbhome_1
Oracle user: nt authority\system
Spfile: +DATA/my-db-name/PARAMETERFILE/spfile.272.9460543263
Password file: +DATA/my-db-name/PASSWORD/pwdmy-db-name.256.998734039
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools:
Disk Groups: DATA,FRA
Mount point paths:
Services: my-db-name1,my-db-name2,srv1,srv2, srv3
Type: RAC
Start concurrency:
Stop concurrency:
Database instances: my-db-name1,my-db-name2
Configured nodes: node1,node2
CSS critical: no
CPU count: 0
Memory target: 0
Maximum memory: 0
Default network number for database services:
Database is administrator managed

How to display the name and the status of the instances in the RAC?

srvctl status database -d my-db-name
Instance my-db-name1 is running on node node1
Instance my-db-name2 is not running on node node2

To list just active nodes: olsnodes -s –t

How to start|stop the database?

srvctl stop database -d my-db-name -o immediate
srvctl start database -d my-db-name

How to start|stop one instance of the RAC?

srvctl start instance -d my-db-name -i my-db-name1
srvctl stop instance -d my-db-name -i my-db-name1
Use -force if the instance to stop is not on the local server

How to start and stop a PDB in Oracle RAC?

Stop a PDB

On the current node [or on all the nodes]:

ALTER PLUGGABLE DATABASE my-PDB-name CLOSE IMMEDIATE [Instances=all];
This will stop the associated service too.
Manually stopping the associated service will not close the PDB. You have to use this SQL command.

Start a PDB

On the current node [or on all the nodes]:

ALTER PLUGGABLE DATABASE my-PDB-name OPEN [Instances=all;]
You can also start the PDB with the associated service
This will NOT start the service(s) associated with this PDB.

How to stop and start a Listener?

srvctl stop listener -l LISTENER_NAME
srvctl start listener -l LISTENER_NAME

To relocate When you find all 3 scan listeners are running on single node, then you may relocate any one of the listener

srvctl relocate scan_LISTENER -i 1 -n node2

Author: Vincent Fenoll Oracle DBA Montreal