Oracle RAC and Grid new features in 19c

Oracle Database 19c has many exciting new features and in order to take advantage of these features you need to upgrade the databases from older versions to Oracle 19c

Join the Webinar to learn New Features in Oracle 19c RAC and Grid

Date and time: Nov 25th 2020 8:00am-9:00am
Pacific Daylight Time (San Francisco, GMT-07:00)


To register for this Webinar, please send an email to SatishbabuGunukula@gmail.com. Note that registrations are limited, first come and first serve basis only. You will receive an email confirmation with meeting session link.

For Presentation, link "Click here"

Thanks & Regards
http://www.oracleracexpert.com

Webinar: Data Guard Physical Standby Setup in Oracle?

This Webinar helps you to understand the benefits of Oracle Data Guard, available methods and how to setup and common issues.

Date and time: Feb 28th 2019 8:00am-9:00am
Pacific Daylight Time (San Francisco, GMT-07:00)

This Webinar covers following Topics.
· Oracle Data Guard and its Benefits
· Overview of different Data Guard Options
· Difference between RAC vs DG
· Prerequisites
· DataGuard Demonstration
· Common Issues
· Post verification Steps
· References
· Contact Info
· Q&A

To register for this Webinar, please send an email to SatishbabuGunukula@gmail.com.

Note that registrations are limited and first come and first serve basis.
You will receive an email confirmation with meeting session link.

For presentation link Click here

Regards,
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com

Physical standby redo log must be renamed

When creating standby database using database duplication it completed but received below standby redo error. This error will not stop the replication but user don’t want to see these ORA- errors.

Finished recover at 2019-02-12:15:02:32
ORACLE error from auxiliary database: ORA-19527: physical standby redo log must be renamed
ORA-00312: online log 1 thread 1: '/oracle/redo1/ORCL/ ORCL _1A.rdo'

RMAN-05535: warning: All redo log files were not defined properly.
ORACLE error from auxiliary database: ORA-19527: physical standby redo log must be renamed
ORA-00312: online log 2 thread 1: '/oracle/redo1/ORCL/ORCL _2A.rdo'

Though this didn't stop the data guard replication, the customer wanted to get-rid the message. Starting with v10g, this was an expected behavior to improve the switchover and failover. With v10g, when MRP is started, it will attempt to clear the online log files.

If PRIMARY and STANDBY has the same directory structure then you will see these errors. This is an expected behavior when you don’t set log_file_name_convert parameter. You can get rid-off the error message just setup dummy value

SQL> ALTER SYSTEM SET log_file_name_convert='dummy','dummy';

You should not see ORA messages any more in the alert.log file.

Thanks
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com

How to recreate oraInventory using Oracle Universal Installer (OUI)

I come across the missing oraInventory issue and I have recreated the inventory using Oracle Universal installer. In case inventory got corrupted, you can use the same steps.

1. Make sure you set the ORACLE_HOME variable
$ export ORACLE_HOME=/oracle/product/11203

2. Update the $ORACLE_HOME/oraInst.loc file to point to the inventory or wherever you want to keep the inventory.

Forex:- cat $ORACLE_HOME/oraInst.loc
inventory_loc=/oracle/oraInventory
inst_group=dba

3. Run below OUI command to register ORACLE_HOME

$cd $ORACLE_HOME/oui/bin
$ ./runInstaller -silent -invPtrLoc $ORACLE_HOME/oraInst.loc -attachHome ORACLE_HOME="/oracle/product/11203" ORACLE_HOME_NAME="OraDb11g_home1"

In case of RAC environment, pls use below syntax

$./runInstaller -silent -attachHome oracle_home="<oracle_home_location>"
"cluster_nodes={<node1, node2>}" local_node="<node_name>"

4. You can verify the inventory created or not using below command
$ORACLE_HOME/OPatch/opatch lsinventory -invPtrLoc $ORACLE_HOME/oraInst.loc

Deleting Oracle Home from Central Inventory

$./runInstaller -silent -detachHome -invPtrLoc $ORACLE_HOME/oraInst.loc 
ORACLE_HOME="<Oracle_Home_Location>"

Regards

Satishbabu Gunukula, Oracle ACE

http://www.oracleracexpert.com

Webinar: How to Convert Single Instance to RAC

I am conducting the webinar once again due to overwhelming response from Oracle community. After this Webinar I hope you should be able to understand Oracle RAC benefits and "How to Convert your single instance to RAC”.

Date and time: Thursday, Sept 15th, 2016 8:00am-9:00am
Pacific Daylight Time (San Francisco, GMT-07:00)

This Webinar covers following Topics.

• Oracle RAC and its Benefits
• Overview of different RAC conversion methods
• Overview of different Storage options
• Convert Single instance to Oracle RAC Using RMAN
• Overview
• Prerequisites
• Demonstration
• Check the logs
• Post Conversion Steps
• References
• Contact Info
• Q&A

To register for this Webinar, please send an email to SatishbabuGunukula@gmail.com and reserve your spot. Note that registrations are limited and first come and first serve basis.

You will receive an email confirmation with meeting session link.
For presentation link Click here .

Regards,
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com

Are you attending Oracle OpenWorld?

Oracle OpenWorld presents the world's largest, most in-depth selection of sessions by Oracle experts, customers, and partners over a course of five days.

Oracle OpenWorld 2015- October 25- 29, 2015

In Oracle Open World, you are sure to discover new ways to optimize your systems, new solutions that will help you take your business to the next level and new practices that will make you more successful in your job and in your industry.

At Oracle Open World you are going to get -
• 2500+ Technology sessions
• Labs & demos
• Exhibitors and partners
• Opportunities to meet with industry experts
• Fun at the Welcome Reception and Appreciation event at the end.

And lot more…!

Click here for Oracle Open World Agenda

Exhibit, Sell, Win – At Oracle Open World. This is the great opportunity to launch and demo your new products and solutions, meet people face to face with prospects. Follow the Three easy steps to Exhibit.

Click here to lean “How to Exhibit: Three Easy Steps”

Learn at Oracle OpenWorld
Oracle Open World provides more educational and networking opportunities than any other Oracle conference. Last year Oracle OpenWorld attracted 60,000+ visitors and this year you can expect more.

Whether you are an experienced developer or DBA, a novice technologist, or business manager or executive, you'll find how-tos, overviews, best-practice sessions, and customer panels targeted to your job role and level of expertise at Oracle OpenWorld.

Oracle Open World Streams and Tracks
• Applications
• Database
• Engineered Systems
• Enterprise Manager
• Industries
• Middleware
• MySQL
• Operating Systems
• Servers and Storage
• Support and Services
• Virtualization

Let Your Manager Know Why You Should Attend
Oracle OpenWorld offers thousands of sessions geared toward helping Oracle users get the most out of their IT infrastructure and investments—and helping them make the most cost-effective and strategic IT decisions for the future.

Oracle OpenWorld is also a great showcase for the latest developments in cloud, big data, security, governance, data center optimization, and other issues that we’re dealing with. Through presentations and hands-on sessions the conference offers hundreds of opportunities to learn money- and time-saving techniques.

Nowhere else you could get the detailed best practices learned by other companies facing challenges similar to yours. You can get advice directly from the people who know the technology best, and can participate in question-and-answer sessions and other roundtable meetings—getting access to high-value information from experts, peers, partners, and industry leaders.

If you still need some help convincing your manager to let you attend Oracle OpenWorld, Download a Customizable Justification Letter.

There are many packages in Oracle OpenWorld 2015 and see which one best suite your needs. Note that Early Bird and Group Registrations receive discounts.

Here is the available packages info.
https://www.oracle.com/openworld/register/index.html

Please see below link to compare benefits
https://www.oracle.com/openworld/register/compare.html#compare

Some of the sessions will be covering in Oracle Open world are:
• Technology- Oracle’s complete technology portfolio
• Applications - Enablement offerings, initiatives and service development options
• Engineered Systems - Ways to transform IT into more cost effective operation
• Industry - Discover ways to successfully position and deliver solutions to clients
• Cloud - Learn how to go to market and implement innovative cloud solutions

Click here to View all the sessions.

Become Part of the Experience, Meet up with your peers, find out when and where our leaders and most exciting events will occur, and participate in the discussion.

Click here to Register for Oracle OpenWorld 2015

Regards,
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com

Oracle RAC New features in 11gR2 (11.2.0.4)

Oracle 11g R2 (11.2.0.4) has new administration and deployment features for Oracle Real Application Clusters (Oracle RAC).

1. Oracle RAC Configuration Audit Tool 

The Oracle RAC Configuration Audit Tool (RACcheck) is designed to audit various important configuration settings within an Oracle RAC system. This tool evaluates single instance and Oracle RAC database installations for best practices, configuration issues, regular health checks, and pre- and post-upgrade best practices.

You can use RACcheck to audit configuration settings for

• Database parameters and configuration settings important to Oracle RAC
• Operating system kernel parameters
• Oracle Grid Infrastructure
• Operating system packages and configuration
• Upgrade readiness assessment to Oracle Database 11g release 2 (11.2) 
• Oracle Database and Oracle ASM (Oracle Automatic Storage Management

You can use RACcheck on the following platforms, please check documentation for detailed version info.

• Intel Linux (Enterprise Linux, RedHat and SuSE 9,10, 11)
• Oracle Solaris SPARC (Solaris 10 and 11), x86-64 (Solaris 10 and 11)
• IBM AIX
• HP-UX

2. Oracle Trace File Analyzer Collector

The Oracle Trace File Analyzer (TFA) Collector is a diagnostic utility, will be useful to simplify diagnostic data collection for Oracle Grid Infrastructure, Clusterware, and Oracle RAC systems.

TFA is installed into the Oracle Grid Infrastructure home when you install, or upgrade to 11.2.0.4. It discovers relevant trace file directories and analyzes the files. The TAF information is stored in a Berkeley database in the Grid home

You can use TFA either on demand and automatically.

• Use tfactl set to enable automatic collection of diagnostics
• Use tfactl diagcollect command to collect trimmed trace files for any component and time range

TFA starts automatically whenever a node starts. You can use following commands to start manually.

Starts the TFA daemon
        /etc/init.d/init.tfa start
Stops the TFA daemon
       /etc/init.d/init.tfa stop
Stops and then starts the TFA daemon
        /etc/init.d/init.tfa restart
Stops the TFA daemon and removes entries from the appropriate O/S configuration
        /etc/init.d/init.tfa shutdown


Regards,
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com

Oracle 12c - One Database and One Solution

Hi Everyone,

My articles published in Oracle Experts website media "Allthingsoracle.com" by RedGate

Please view the article using below link.
Oracle 12c – One Database and One Solution

In this article i have covered

• What is Multitenant Architecture?
• The benefits of managing many database as ONE database
•  Why Oracle 12c?
• Upgrade path to Oracle 12c
• Oracle Enterprise Manage Cloud Control

This article helps all Oracle community who don’t understand what Oracle 12c offers and how they can reduce IT Costs.

I hope you will like the article and it will be helpful to you.

Please leave your valuable comments.

Regards,
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com

Convert Single Instance to RAC – Part2: Manually Convert to RAC

Hi Everyone,

My articles published in Oracle Experts website media "Allthingsoracle.com" by RedGate

Please view the article using below link.
Convert Single Instance to RAC – Part2: Manually Convert to RAC

I hope you will like the article and it will be helpful to you.

Please leave your valuable comments.

Regards,
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com

Time for Change: Migrate your Non-RAC Database to RAC

Hi Everyone,

One of my articles published in "IOUG Select Journal" and distributed at Oracle Open World 2012.

Time for Change: Migrate your Non-RAC Database to RAC - Page 25
By Satishbabu Gunukula

Please view the IOUG 2012 Q3 article using below link.
http://www.ioug.org/p/do/sd/sid=2371&fid=2133&req=direct


I hope you will like the article and it will be helpful to you.

Please leave your valuable comments.

Regards,
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com

Convert Single Instance to RAC – Part1: Duplicate DB using RMAN

Hi Everyone,

My articles published in Oracle Experts website media "Allthingsoracle.com" by RedGate

Please view the article using below link.
Convert Single Instance to RAC - Part1: Duplicate DB using RMAN

I hope you will like the article and it will be helpful to you.

Please leave your valuable comments.

Regards,
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com

How to STOP and START processes in Oracle RAC and Log Directory Structure

You need to follow the below steps to Stop and Start processes in Oracle RAC

Shutdown RAC Database
You need to Shutdown Database instances on each node. You can either use Oracle Enterprise Manager or SVRCTL to shutdown the instances. If you are using EM Grid control then set a blackout in Grid control for processes that you intend to shutdown. So that records for these processes indicate that the shutdown was planned.
Use below command to stop Enterprise Manager/Grid Control
$ORACLE_HOME/bin/emctl stop dbconsole

Use below command to shutdown all oracle RAC instances on all nodes.
$ ORACLE_HOME/bin/srvctl stop database -d db_name

If you want to stop specific database instances use below command
$ ORACLE_HOME/bin/srvctl stop database -d db_name –i instance_name

Shutdown Oracle ASM Instance
Once the database is stopped, proceed with ASM Instance shutdown.
Use below command to shutdown ASM instances on all nodes
$ORACLE_HOME/bin/bin/srvctl stop asm -n node

Shutdown Node applications
Use below command to shutdown node apps on all RAC nodes
$ORACLE_HOME/bin/bin/srvctl stop nodeapps -n node

Shutdown Oracle Clusterware
You need to Shutdown oracle clusterware or CRS as root and run below command on each node in the cluster.
#crsctl stop crs

Please note that using above command will stop Oracle High availability services (OHAS) and Clustware stack in a single command

From 11g R2, you can do this in two stops

1. Stop Clustwerware stack on local node
#crsctl stop cluster

You can stop the clusterware stack on all nodes in the cluster
# Crsctl stop cluster –all

Where
-all Start clusterware on all nodes
-n Start clusterware on particular nodes

2. Stop Oracle High availability service demon on each node in the cluster.
# crsctl stop has

Check the Status of Cluster
Once all process stopped run the below command to check the status of CRSD,CSSD,EVMD process.
# crsctl check crs

If you see any process failed to stop then you can also use Force option to terminate the processes unconditionally.
$ crsctl stop crs –all –f

Start processes in Oracle RAC
Follow the reverse sequence to start all processes in oracle RAC
# crsctl start crs
$ORACLE_HOME/bin/bin/srvctl start nodeapps -n node
$ORACLE_HOME/bin/bin/srvctl start asm -n node
$ORACLE_HOME/bin/srvctl start database -d db_name

If you come across any issues during startup orshutdown, check the Oracle Clusterware Component Log Files.

Oracle Clusterware Log Directory Structure

CRS_HOME/log/hostname/crsd/ - The log files for the CRS daemon CRS_HOME/log/hostname/cssd/ - The log files for the CSS daemon
CRS_HOME/log/hostname/evmd/ - The log files for the EVM daemon
CRS_HOME/log/hostname/client/ - The log files for the Oracle Cluster Registry (OCR)
CRS_HOME/log/hostname/racg/ - The log files for the Oracle RAC high availability component
CRS_HOME/log/hostname/racg/ - The log files for the Oracle RAC high availability component
CRS_HOME/log/hostanme/alert.log – The alert.log for Clusterware issues.

Please note that the CRS_HOME is the directory in which the Oracle Clusterware software was installed and hostname is the name of the node

Regards,
Satishbabu Gunukula
http://www.oracleracexpert.com

Oracle Cluster wait events

What is wait event?
An event can be anything that Oracle has to perform on behalf of a set of instructions sent by the user interface. The term “wait” is used because every time a user connects to your application, a resource is allocated to perform tasks on its behalf. The waiting comes when a session is waiting for an action, sometimes from a user and at other times from the database

Wait events for Oracle can be divided as three categories
1.Time-based event
2.System-wide event
3.Session wait

Cluster Wait events
Cluster wait events are caused by multiple nodes in the cluster needing access to the same set of data. These must be tracked down on an individual basis. Usually there are multiple statements and tables that have contention problems.

Ideally, one node would only access one subset of data in order to reduce these conflicts. If possible, different types of operations should be done on each node. For example OLAP access could be targeted to node 1 while OLTP access could be targeted to node 2.

Usually there are multiple queries attempting update the same sets of data from different nodes. In order to determine what queries is producing the contention across the nodes, do the following:

Determine your beginning and ending snapshots from DBA_HIST_SNAPSHOT. Please note that we have used 4000 as beginning snapshot and 5000 as ending snapshot.

Use below SQL query to find event id
SQL> select event_id, event, count(1) cnt from dba_hist_active_sess_history where snap_id between 4000 and 5000 and wait_class_id=3871361733 and event in (’gc cr block busy’,’gc cr multi block request’) group by event_id, event order by 3;
SQL> Select sql_id, count(1) from dba_hist_active_sess_history where snap_id between 4000 and 5000 and event_id in (results from (2) above);
SQL> Select sql_text from dba_hist_sqltext where sqlid in .;

The steps above will tell you the SQL statements that cause the data contention.
The most important wait events for RAC include various categories, such as:
Block-oriented
•gc current block 2-way
•gc current block 3-way
•gc cr block 2-way
•gc cr block 3-way
Message-oriented
•gc current grant 2-way
•gc cr grant 2-way
Contention-oriented
•gc current block busy
•gc cr block busy
•gc current buffer busy
Load-oriented
•gc current block congested
•gc cr block congested

The major wait events in Oracle RAC are:
1. gc cr request
2. gc buffer busy

In a RAC environment the buffer cache is global across all instances in the cluster and hence the processing differs when compared single-database instance. When a process needs to read data, Oracle will first check to see if it exists in the local buffer cache. If not, it will review global buffer cache to see if another instance already has the data in their buffer cache. If global buffer cache has data then it send a request the data and sends the data via interconnect to avoid disk read. We need to monitor the amount of requests going back and forth via interconnect.

gc cr request( global cache cr request) – This event specifies the time it takes to receive data from remote cache. The main reasons for this event are

RAC traffic using slow connection – Normally we will use high speed inter connect to transfer data between instances, however, sometimes oracle may not able to select the high-speed interconnect and instead route traffic over the slower public network. This significantly increases the wait time for gc cr request wait event.

The oradebug command can be used to verify which network is being used for RAC traffic:
SQL> oradebug setmypid
SQL> oradebug ipc

The oradebug command can be used to manually remaster an object
SQL> oradebug lkdebug -m pkey "object_id"

Inefficient queries - This is mainly due to queries which are not written/tuned properly. The inefficient query increases the amount of data blocks requested by an oracle session. Due to this more often a block will need to be read from a remote instance via interconnect, which causes the wait event.

gc buffer busy(global cache buffer busy) – This wait event is similar to the buffer busy wait event in a single-instance database and this event specifies the time the remote instance locally spends accessing the required data block. You will encounter this wait event due to hot blocks or inefficient queries.

Hot Blocks – You will encounter this issue when multiple sessions requesting a block that either not in buffer cache or it is in incompatible mode. Also heavy delete and inserts operations on hot rows alleviate the problem. Adjusting pctfree and/or pctused parameters will help.

Inefficient Queries – Due to inefficient queries the more blocks requested from the buffer cache the more likelihood of a session having to wait for other sessions. Tuning queries will result to access fewer blocks and will often result in less contention for the same block.

Regards,
Satishbabu Gunukula
http://www.oracleracexpert.com

Flashback Data Archive (Oracle Total Recall)

Flashback Data archive is the new feature introduced in Oracle 11g. It provides the ability to track and store all transitional changes to a table over its lifetime. Flashback Data Archive (which can logically span one or more table spaces) specifies a space quota and retention period for the archive, and then creates the required tables in the archive. It allows long-term retention (for ex years) of changed data to a table or set of tables for a user-defined period of time.

The database can have multiple flashback data archives, but only a single default archive. The individual flashback archive consists of one or more tablespaces or parts of tablespace. Each flashback archive has a name, retention period and a quota on each associated tablespace.

When a DML transaction commits an operation on a flashback archive enabled table, the FBDA (Flashback Data Archiver) process stores the pre-image of the rows into a flashback archive, along with metadata of the current rows. The FBDA process is also responsible for managing the data within the flashback archives, such as purging data beyond the retention period.

To use this feature user should have below privileges
1.FLASHBACK ARCHIVE object privilege on the Flashback Data Archive that you want to use for that table
2.FLASHBACK ARCHIVE ADMINISTER system privilege to enable or disable Flashback data archive.

Creating a Flashback Data Archive:-
• CREATE FLASHBACK ARCHIVE flashback_archive TABLESPACE flashback_archive_tbs
QUOTA 15G RETENTION 5 YEARS;

Altering a Flashback Data Archive:-
You can add, modify and remove tablespaces, and change the retention, purge some or all of its data.

Add up to 2G of tablespace “flashback_archive_tbs1” to Flashback Data Archive flashback_archive1
• ALTER FLASHBACK ARCHIVE flashback_archive1 ADD TABLESPACE flashback_archive_tbs1 QUOTA 2G;

Change the maximum space that Flashback Data Archive “flashback_archive1” can use in tablespace flashback_archive_tbs1 to 5G
• ALTER FLASHBACK ARCHIVE flashback_archive1 MODIFY TABLESPACE flashback_archive_tbs1QUOTA 5G;

Remove tablespace “flashback_archive_tbs1” from Flashback Data Archive
• ALTER FLASHBACK ARCHIVE flashback_archive1 REMOVE TABLESPACE flashback_archive_tbs1;

Dropping a Flashback Data Archive:-
Remove Flashback Data Archive flashback_archive1 and all its historical data
• DROP FLASHBACK ARCHIVE flashback_archive1;

Enabling and Disabling Flashback Data Archive:-
By default, flashback archiving is disabled for any table and you need FLASHBACK ARCHIVE privilege to enable for any table. After flashback archiving is enabled for a table, you can disable it only if you either have SYSDBA or FLASHBACK ARCHIVE ADMINISTER privilege.

Create table EMP and store the historical data in the DEFAULT Flashback Data Archive
• CREATE TABLE EMP (EMPNO NUMBER(5) NOT NULL, ENAME VARCHAR2(20))
FLASHBACK ARCHIVE;

Create table EMP and store the historical data in the Flashback_Archive 1
• CREATE TABLE EMP (EMPNO NUMBER(5) NOT NULL, ENAME VARCHAR2(20))
FLASHBACK ARCHIVE Flashback_Archive1;

Enable flashback archiving for the table EMP and store data in the default Flashback Data Archive
• ALTER TABLE EMP FLASHBACK ARCHIVE;

Enable flashback archiving for the table EMP and store data in Flashback Data Archive Flashback_Archive1
• ALTER TABLE EMP FLASHBACK ARCHIVE Flashback_Archive1;

Disable flashback archiving for the table employee:
• ALTER TABLE EMP NO FLASHBACK ARCHIVE;

Regards,
Satishbabu Gunukula
http://www.oracleracexpert.com

How to find Master Node in Oracle RAC

I have seen many users asking how to find “Master node” in Oracle RAC, let me clear some of their doubts.

There are two types of Masters in Oracle RAC, one is Mater node at Oracle Clusterware level and other is Master node for specific resource or block or object.

The node which gets the active state during startup is authorized to be a master node by Cluster Synchronization Service.

Run the below command to find which node is master at Clusterware level
$cat $ORA_CRS_HOME/log/`hostname`/cssd/ocssd* |grep master
or
$ for x in `ls -tr $ORA_CRS_HOME/log/`hostname`/cssd/ocssd* `; do grep -i "master node" $x ; done | tail -1

The OCR Automatic backups are taken only by master node. If the Master fails, the OCR backups will be created on the new Master. The Master node which has OCR backups goes down due to failure then we cannot be recover the OCR that’s why Oracle recommends taking backups using “ocrconfig” and also integrating OCR backups with backup strategy.

Run the below command to find which node is OCR Master and taking automatic backups.
$ocrconfig –showbackup
testrac02 2010/08/30 16:29:52 /oracle/crs/cdata/crs
testrac02 2010/08/30 16:29:52 /oracle/crs/cdata/crs
testrac02 2010/08/30 12:29:49 /oracle/crs/cdata/crs
testrac02 2010/08/30 08:29:46 /oracle/crs/cdata/crs
testrac02 2010/08/29 00:29:23 /oracle/crs/cdata/crs

The block level masters are used by Cache fusion while transferring the block. Any node can become the master node of a particular block and you can also see which node acting as master in V$GES_RESOURCE table (MASTER_NODE column)

You can manually remaster an object with oradebug command:
SQL> oradebug lkdebug -m pkey "object_id"

Regards
Satishbabu Gunukula
http://www.oracleracexpert.com

Modifying the VIP Address or VIP Hostname in Oracle RAC Node

Oracle 10g or 11g uses Virtual IP address (VIP) in clustered environment for clients to connect to the database. During the installation of Oracle Clusterware users are prompted to enter VIP and VIP hostname for each node in the cluster. The VIP is a static IP with a hostname defined and also resolved through DNS.

The VIP information is stored in OCR (Oracle Cluster Registry) and also in different HA framework. Changing the VIP Address or VIP Hostname involves modification of the nodeapps, which includes the VIP,GSD, Listener, and ONS(Oracle Notification Services).The VIP can be changed while the nodeapps are running, but the changes will take effect only when nodeapps are restated.

Note that stopping nodeapps may cause other resources to be stopped for ex: - ASM, instance or database, so the change should be made during scheduled outage.

Follow the steps to change the VIP address or VIP hostname.

Step 1:- Check the original configuration before change
$ srvctl config nodeapps -n -a
Using '-a' will give you the current VIP hostname, VIP address and interface

Example:
# srvctl config nodeapps -n testrac01 -a
VIP exists.: /testrac01-vip/10.11.12.01/255.255.255.0/eth0

The VIP Hostname is 'testrac01-vip’
The VIP IP address is '10.11.12.01'
The VIP subnet mask is '255.255.255.0'
The Interface Name used by the VIP is called 'eth0'

Step 2:- Stop Instance,ASM,Nodeapps resources
$srvctl stop instance -d testdb -i tetdb1
$srvctl stop asm -n testrac01
$srvctl stop nodeapps -n testrac01

Step 3:- Verify the VIP Address is no longer running by using below command
$ifconfig -a

You can also check the resources status using crs_stat command.

Step 4:- Update /etc/hosts file with new VIP Address or VIP hostname on node1 and also update DNS to associate the new IP address with VIP hostname as per /etc/hosts file.

Step 5:- Modify VIP Address or VIP hostname on nodeapps by using srvctl command (Run as root)
#srvctl modify nodeapps -n [-o ] [-A ]
Where
-n < node_name> - Node name.
-o - Oracle Home for the cluster software (CRS-Home).
-A The node level VIP address (/netmask[/if1[|if2|...]]).

Example:- Modify the VIP Address to 10.11.12.11
#srvctl modify nodeapps -n testrac01 -A 10.11.12.11/255.255.255.0/eth0

Use below command to change to change VIP address using VIP hostname.The srvctl command will resolve the IP to hostname or the hostname to IP address. You can use the same command to change the VIP hostname from ”testrac01-vip” to “testrac01-v”

#srvctl modify nodeapps -n testrac01 -A testrac01-v/255.255.255.0/eth0

Step 6:- Verify the change by running below command
$srvctl config nodeapps -n testrac01 –a

Step7:- Start all resources
$srvct start nodeapps -n testrac01
$srvctl start asm -n testrac01
$srvctl start instance -d testdb –i testdb1

Step8:- Repeat the same steps on all remaining nodes in the cluster.

Regards,
Satishbabu Gunukula
http://www.oracleracexpert.com

Overview of Transparent application Failover (TAF) in Oracle

Transparent Application Failover (TAF) is a client-side feature that allows for clients to reconnect to surviving nodes in the event of a failure of an instance. The reconnect happens automatically from within the OCI (Oracle Call Interface) library. Any uncommitted transactions are rolled back and server side program variables and session properties will be lost. In some case the select statements automatically re-executed on the new connection with the cursor positioned on the row on which it was positioned prior to the failover.

For high availability and scalability, Oracle provides the Transparent Application Failover feature part of Oracle Real Application Clusters (RAC).

The failover is configured in tnsnames.ora file, the TAF settings are placed in CONNECT_DATA section of the tnsnames.ora using FAILOVER_MODES parameters.

FAILOVER_MODE contains the subparameters
----------------------------------------------------------
BACKUP: Specify a different net service name for backup instance connections. A backup should be specified when using PRECONNECT to pre-establish connections.

A sample configuration would look like

TESTDB1 =
(DESCRIPTION =
(LOAD_BALANCE = ON)
(FAILOVER = ON)
(ADDRESS = (PROTOCOL = TCP)(HOST = TESTRAC1-VIP)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = TESTRAC2-VIP)(PORT = 1521))
(CONNECT_DATA =
(SERVICE_NAME = testdb.oracleracexpert.com)
(FAILOVER_MODE = (TYPE = SELECT)(METHOD = PRECONNECT)(BACKUP=TESTDB2))
)
)

TYPE: TAF supports three types of failover types

1.SESSION failover - If a user's connection is lost, SESSION failover establishes a new session automatically created for the user on the backup node. This type of failover does not attempt to recover selects. This failover is ideal for OLTP (online transaction processing) systems, where transactions are small.

2.SELECT failover – If the connection is lost, Oracle Net establishes a connection to another node and re-executes the SELECT statements with cursor positioned on the row on which it was positioned prior to the failover. This mode involves overhead on the client side and Oracle NET keeps track of SELECT statements. This approach is best for data warehouse systems, where the transactions are big and complex

3.NONE: This setting is the default and failover functionality is provided. Use this setting to prevent failover.

A sample configuration would look like

TESTDB1 =
(DESCRIPTION =
(LOAD_BALANCE = ON)
(FAILOVER = ON)
(ADDRESS = (PROTOCOL = TCP)(HOST = TESTRAC1-VIP)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = TESTRAC2-VIP)(PORT = 1521))
(CONNECT_DATA =
(SERVICE_NAME = testdb.oracleracexpert.com)
(FAILOVER_MODE = (TYPE = SELECT)(METHOD = BASIC)(RETRIES = 10)(DELAY = 5))
)
)

METHOD: This parameters determines how failover occurs from the primary node to the backup node
BASIC: Use this mode to establish connections at failover time, no work on the backup server until failover time.
PRECONNECT: Use this mode to pre-established connections. This PRECONNECT mode provides faster failover but requires that the backup instance be capable of supporting all connections from every supported instance.

RETRIES: Use this parameter to specify number of times to attempt to connect after a failover. If DELAY is specified but RETRIES is not specified, RETRIES default to five retry attempts.

DELAY: Use this parameter to Specify the amount of time in seconds to wait between connect attempts. If RETRIES is specified but DELAY is not specified, DELAY default to one second.

A sample configuration would look like

TESTDB1 =
(DESCRIPTION =
(LOAD_BALANCE = ON)
(FAILOVER = ON)
(ADDRESS = (PROTOCOL = TCP)(HOST = TESTRAC1-VIP)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = TESTRAC2-VIP)(PORT = 1521))
(CONNECT_DATA =
(SERVICE_NAME = testdb.oracleracexpert.com)
(FAILOVER_MODE = (TYPE = SELECT)(METHOD = BASIC)(RETRIES = 10)(DELAY = 5))
)
)

Please note that you can pre-establish a connection to reduce the failover time using METHOD=PRECONNECT option.

To verify that TAF is correctly configured, you query FAILOVER_TYPE, FAILOVER_METHOD, and FAILED_OVER columns in the V$SESSION view.
SQL> SELECT MACHINE, FAILOVER_TYPE, FAILOVER_METHOD, FAILED_OVER, COUNT(*) FROM V$SESSION

Regards,
Satishbabu Gunukula
http://www.oracleracexpert.com

Oracle RAC load balancing and failover

LOAD BALANCING: The Oracle RAC system can distribute the load over many nodes this feature called as load balancing.

There are two methods of load balancing
1.Client load balancing
2.Server load balancing

Client Load Balancing distributes new connections among Oracle RAC nodes so that no one server is overloaded with connection requests and it is configured at net service name level by providing multiple descriptions in a description list or multiple addresses in an address list. For example, if connection fails over to another node in case of failure, the client load balancing ensures that the redirected connections are distributed among the other nodes in the RAC.

Configure Client-side connect-time load balancing by setting LOAD_BALANCE=ON in the corresponding client side TNS entry.

TESTRAC =
(DESCRIPTION =
(ADDRESS_LIST=
(LOAD_BALANCE = ON)
(ADDRESS = (PROTOCOL = TCP)(HOST = TESTRAC1-VIP)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = TESTRAC2-VIP)(PORT = 1521))
)
(CONNECT_DATA = (SERVICE_NAME = testdb.oracleracexpert.com))
)

Server Load Balancing distributes processing workload among Oracle RAC nodes. It divides the connection load evenly between all available listeners and distributes new user session connection requests to the least loaded listener(s) based on the total number of sessions which are already connected. Each listener communicates with the other listener(s) via each database instance’s PMON process.

Configure Server-side connect-time load balancing feature by setting REMOTE_LISTENERS initialization parameter of each instance to a TNS name that describes list of all available listeners.

TESTRAC_LISTENERS =
(DESCRIPTION =
(ADDRESS_LIST =(ADDRESS = (PROTOCOL = TCP)(HOST = TESTRAC1)(PORT = 1521)))
(ADDRESS_LIST =(ADDRESS = (PROTOCOL = TCP)(HOST = TESTRAC2)(PORT = 1521))))
)

Set *.remote_listener= TESTRAC_LISTENERS’ initialization parameter in the database’s shared SPFILE and add TESTRAC_LISTENERS’ entry to the TNSNAMES.ORA file in the Oracle Home of each node in the cluster.

Once you configure Server-side connect-time load balancing, each database’s PMON process will automatically register the database with the database’s local listener as well as cross-register the database with the listeners on all other nodes in the cluster. Now the nodes themselves decide which node is least busy, and then will connect the client to that node.

FAILOVER:

The Oracle RAC system can protect against failures caused by O/S or server crashes or hardware failures. When a node failure occurs in RAC system, the connection attempts can fail over to other surviving nodes in the cluster this feature called as Failover.

There are two methods of failover
1. Connection Failover
2. Transparent Application Failover (TAF)

Connection Failover - If a connection failure occurs at connect time, the application failover the connection to another active node in the cluster. This feature enables client to connect to another listener if the initial connection to the first listener fails.

Enable client-side connect-time Failover by setting FAILOVER=ON in the corresponding client side TNS entry.

TESTRAC =
(DESCRIPTION =
(ADDRESS_LIST=
(LOAD_BALANCE = ON)
(FAILOVER = ON)
(ADDRESS = (PROTOCOL = TCP)(HOST = TESTRAC1-VIP)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = TESTRAC2-VIP)(PORT = 1521))
)
(CONNECT_DATA = (SERVICE_NAME = testdb.oracleracexpert.com))
)

If LOAD_BALANCE is set to on then clients randomly attempt connections to any nodes. If client made connection attempt to a down node, the client needs to wait until it receives the information that the node is not accessible before trying alternate address in ADDRESS_LIST.

Transparent application Failover (TAF) – If connection failure occurs after a connection is established, the connection fails over to other surviving nodes. Any uncommitted transactions are rolled back and server side program variables and session properties will be lost. In some case the select statements automatically re-executed on the new connection with the cursor positioned on the row on which it was positioned prior to the failover.

TESTRAC =
(DESCRIPTION =
(LOAD_BALANCE = ON)
(FAILOVER = ON)
(ADDRESS = (PROTOCOL = TCP)(HOST = TESTRAC1-VIP)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = TESTRAC1-VIP)(PORT = 1521))
(CONNECT_DATA =
(SERVICE_NAME = testdb.oracleracexpert.com)
(FAILOVER_MODE = (TYPE = SELECT)(METHOD = BASIC)(RETRIES = 180)(DELAY = 5))
)
)

Regards,
Satishbabu Gunukula
http://www.oracleracexpert.com