How to STOP and START processes in Oracle RAC and Log Directory Structure

You need to follow the below steps to Stop and Start processes in Oracle RAC

Shutdown RAC Database
You need to Shutdown Database instances on each node. You can either use Oracle Enterprise Manager or SVRCTL to shutdown the instances. If you are using EM Grid control then set a blackout in Grid control for processes that you intend to shutdown. So that records for these processes indicate that the shutdown was planned.
Use below command to stop Enterprise Manager/Grid Control
$ORACLE_HOME/bin/emctl stop dbconsole

Use below command to shutdown all oracle RAC instances on all nodes.
$ ORACLE_HOME/bin/srvctl stop database -d db_name

If you want to stop specific database instances use below command
$ ORACLE_HOME/bin/srvctl stop database -d db_name –i instance_name

Shutdown Oracle ASM Instance
Once the database is stopped, proceed with ASM Instance shutdown.
Use below command to shutdown ASM instances on all nodes
$ORACLE_HOME/bin/bin/srvctl stop asm -n node

Shutdown Node applications
Use below command to shutdown node apps on all RAC nodes
$ORACLE_HOME/bin/bin/srvctl stop nodeapps -n node

Shutdown Oracle Clusterware
You need to Shutdown oracle clusterware or CRS as root and run below command on each node in the cluster.
#crsctl stop crs

Please note that using above command will stop Oracle High availability services (OHAS) and Clustware stack in a single command

From 11g R2, you can do this in two stops

1. Stop Clustwerware stack on local node
#crsctl stop cluster

You can stop the clusterware stack on all nodes in the cluster
# Crsctl stop cluster –all

Where
-all Start clusterware on all nodes
-n Start clusterware on particular nodes

2. Stop Oracle High availability service demon on each node in the cluster.
# crsctl stop has

Check the Status of Cluster
Once all process stopped run the below command to check the status of CRSD,CSSD,EVMD process.
# crsctl check crs

If you see any process failed to stop then you can also use Force option to terminate the processes unconditionally.
$ crsctl stop crs –all –f

Start processes in Oracle RAC
Follow the reverse sequence to start all processes in oracle RAC
# crsctl start crs
$ORACLE_HOME/bin/bin/srvctl start nodeapps -n node
$ORACLE_HOME/bin/bin/srvctl start asm -n node
$ORACLE_HOME/bin/srvctl start database -d db_name

If you come across any issues during startup orshutdown, check the Oracle Clusterware Component Log Files.

Oracle Clusterware Log Directory Structure

CRS_HOME/log/hostname/crsd/ - The log files for the CRS daemon CRS_HOME/log/hostname/cssd/ - The log files for the CSS daemon
CRS_HOME/log/hostname/evmd/ - The log files for the EVM daemon
CRS_HOME/log/hostname/client/ - The log files for the Oracle Cluster Registry (OCR)
CRS_HOME/log/hostname/racg/ - The log files for the Oracle RAC high availability component
CRS_HOME/log/hostname/racg/ - The log files for the Oracle RAC high availability component
CRS_HOME/log/hostanme/alert.log – The alert.log for Clusterware issues.

Please note that the CRS_HOME is the directory in which the Oracle Clusterware software was installed and hostname is the name of the node

Regards,
Satishbabu Gunukula
http://www.oracleracexpert.com

Webinar: Oracle Real Application Cluster

Oracle Real application cluster allows multiple database instances on different servers in the cluster against a shared database. Oracle RAC provides Reliability, Recoverability, Error Detection and Continuous operations.

Date and time: Monday, November 14th 2011 8:00am - 9:00am PST

You will get an opportunity to learn

• What is Real Application Cluster?
• Oracle Cluster Benefits & Components
• Oracle Cluster Ready Services
• Interconnect & Cache fusion
• Oracle RAC Database & ASM
• Transparent Application Failover (TAF)
• Backup & Recovery
• New Features in Oracle 11g RAC

To register for Webinar, please send an email to SatishbabuGunukula@gmail.com and reserve your spot. You will receive an email confirmation with Webinar link.

Registration is limited.

Click here for presentation link

Regards,
Satishbabu Gunukula
http://www.oracleracexpert.com

How to find Master Node in Oracle RAC

I have seen many users asking how to find “Master node” in Oracle RAC, let me clear some of their doubts.

There are two types of Masters in Oracle RAC, one is Mater node at Oracle Clusterware level and other is Master node for specific resource or block or object.

The node which gets the active state during startup is authorized to be a master node by Cluster Synchronization Service.

Run the below command to find which node is master at Clusterware level
$cat $ORA_CRS_HOME/log/`hostname`/cssd/ocssd* |grep master
or
$ for x in `ls -tr $ORA_CRS_HOME/log/`hostname`/cssd/ocssd* `; do grep -i "master node" $x ; done | tail -1

The OCR Automatic backups are taken only by master node. If the Master fails, the OCR backups will be created on the new Master. The Master node which has OCR backups goes down due to failure then we cannot be recover the OCR that’s why Oracle recommends taking backups using “ocrconfig” and also integrating OCR backups with backup strategy.

Run the below command to find which node is OCR Master and taking automatic backups.
$ocrconfig –showbackup
testrac02 2010/08/30 16:29:52 /oracle/crs/cdata/crs
testrac02 2010/08/30 16:29:52 /oracle/crs/cdata/crs
testrac02 2010/08/30 12:29:49 /oracle/crs/cdata/crs
testrac02 2010/08/30 08:29:46 /oracle/crs/cdata/crs
testrac02 2010/08/29 00:29:23 /oracle/crs/cdata/crs

The block level masters are used by Cache fusion while transferring the block. Any node can become the master node of a particular block and you can also see which node acting as master in V$GES_RESOURCE table (MASTER_NODE column)

You can manually remaster an object with oradebug command:
SQL> oradebug lkdebug -m pkey "object_id"

Regards
Satishbabu Gunukula
http://www.oracleracexpert.com

How to move or replace and repair OCR (Oracle Cluster Registry)

Moving or Replacing Oracle Cluster Registry:
If you are replacing or moving an OCR/OCR MIRROR make sure that other OCR file is ONLINE and also clusterware is running on the node that you are performing the replace operation.

Please note that the OCR that you are replacing or moving can be either online or offline.

Run the below command to move or replace OCR
#ocrconfig -replace ocr
For ex:- ocrconfig -replace ocr /dev/raw/raw5

Run the below command to move or replace OCR MIRROR.

#ocrconfig -replace ocrmirror
For ex: - ocrconfig -replace ocr /dev/raw/raw6

Repairing Oracle Cluster Registry (OCR) Configuration:
You may need to repair an OCR configuration, if your OCR configuration changes ran (adding/removing/replacing OCR/OCR mirror) while the node is shutdown.

Make sure you run the command on the node which you have shutdown and you cannot perform this operation while oracle clusterware is running.

Run the below command to repair
# ocrconfig –repair ocrmirror device_name
for ex:- ocrconfig –repair ocrmirror /dev/raw/raw1

Click here to learn How to Add and Remove Oracle Cluster Registry

Regards,
Satishbabu Gunukula
http://www.oracleracexpert.com/

How to Add and Remove OCR (Oracle Cluster Registry)

Adding an OCR (Oracle Cluster Registry):-

You can also add an OCR after completing the Oracle RAC installation, if you didn’t mirror the OCR during installation. Please note that you must run all the commands as “root”.

Run the below command to add OCR MIRROR.
#ocrconfig -replace ocrmirror

Removing an Oracle Cluster Registry (OCR)
To remove OCR/OCR MIRROR the other OCR must be online.

Run the below command on any node to remove OCR.
# ocrconfig -replace ocr

Run the below command on any node to remove OCR Mirror.
# ocrconfig -replace ocrmirror

Please note that these commands update the OCR configuration on all of the nodes where Oracle Clusterware is running.

Click here to learn How to backup and recover Oracle Cluster Registry.
Regards,
Satishbabu Gunukula
http://www.oracleracexpert.com/

Voting disk Backup and Recovery

Voting disk manages node membership information and it is used by Cluster synchronization services demon (CSSD).

Backing up Voting Disks:-
Run the below command to back up the voting disk.
$ dd if=voting_disk_name of=backup_file_name
or
$ dd if=voting_disk_name of=backup_file_name bs=4k

Recovering Voting Disks:-
Run the below command to recover a voting disk
$ dd if=backup_file_name of=voting_disk_name

You can change the Voting Disk Configuration dynamically after the installation.
Please note that you need to run the command as “root”.

Run the below command to add a voting disk:
# crsctl add css votedisk_path

You can have upto 32 Voting disks.

Run the following command to remove a voting disk:
# crsctl delete css votedisk_path

Click here to learn Backup and Recovery of OCR

Regards,
Satishbabu Gunukua
http://www.oracleracexpert.com/

Restore and Recover OCR from backup (Oracle Cluster Registry)

Step 1: Locate physical the OCR backups using –showbackup command.
#ocrconfig – showbackup

Step 2: Review the contents
#ocrdump –backupfile backup_file_name

Step 3: Stop the Oracle clusterware on all the nodes.
#crsctl stop crs

Step 4: Restore the OCR backup
# ocrconfig –restore $CRS_HOME/cdata/crs/day.ocr
OR
Restore the OCR from export/logical backup.
# ocrconfig –import export_file_name
For ex: - # ocrconfig –import /backup/oracle/exp_ocrbackup.dmp

Step 5: Restart the Clusterware on all nodes.
#crsctl start crs

Step 6: Check the OCR integrity
# cluvfy comp ocr –n all

Click here to learn How to backup and recover OCR.

Regards,
Satishbabu Gunukula
http://www.oracleracexpert.com/

Backup and Recovery of OCR - Oracle Cluster Registry

Oracle Cluster Registry (OCR) file is a key component of the Cluster. It maintains the information about cluster node list, instance to node mapping and application resources profiles such as VIP address, services...Etc.

There are two methods for OCR Backup (Oracle Cluster Registry)

1. Automatically generated OCR files under $CRS_HOME/cdata/crs
2. OCR export/logical backup

The Oracle Clusterware automatically creates OCR backups
-Every four hours: last three copies
-At the End of the Day: last two copies
-At the end of the week: last two copies.

To backup OCR file, copy the generated file from $CRS_HOME/cdata/crs to your backup directory (/backup/oracle).

You must run the backup as “root”.

Run the below command to take OCR export backup.
# ocrconfig -export export_file_name

Click here to learn Restore and Recovery OCR from Backup.

Click here to learn Voting disk Backup and Recovery.

Regards,
Satishbabu Gunukula
http://www.oracleracexpert.com/