Wednesday, June 30, 2010

Modifying the VIP Address or VIP Hostname in Oracle RAC Node

Oracle 10g or 11g uses Virtual IP address (VIP) in clustered environment for clients to connect to the database. During the installation of Oracle Clusterware users are prompted to enter VIP and VIP hostname for each node in the cluster. The VIP is a static IP with a hostname defined and also resolved through DNS.

The VIP information is stored in OCR (Oracle Cluster Registry) and also in different HA framework. Changing the VIP Address or VIP Hostname involves modification of the nodeapps, which includes the VIP,GSD, Listener, and ONS(Oracle Notification Services).The VIP can be changed while the nodeapps are running, but the changes will take effect only when nodeapps are restated.

Note that stopping nodeapps may cause other resources to be stopped for ex: - ASM, instance or database, so the change should be made during scheduled outage.

Follow the steps to change the VIP address or VIP hostname.

Step 1:- Check the original configuration before change
$ srvctl config nodeapps -n -a
Using '-a' will give you the current VIP hostname, VIP address and interface

# srvctl config nodeapps -n testrac01 -a
VIP exists.: /testrac01-vip/

The VIP Hostname is 'testrac01-vip’
The VIP IP address is ''
The VIP subnet mask is ''
The Interface Name used by the VIP is called 'eth0'

Step 2:- Stop Instance,ASM,Nodeapps resources
$srvctl stop instance -d testdb -i tetdb1
$srvctl stop asm -n testrac01
$srvctl stop nodeapps -n testrac01

Step 3:- Verify the VIP Address is no longer running by using below command
$ifconfig -a

You can also check the resources status using crs_stat command.

Step 4:- Update /etc/hosts file with new VIP Address or VIP hostname on node1 and also update DNS to associate the new IP address with VIP hostname as per /etc/hosts file.

Step 5:- Modify VIP Address or VIP hostname on nodeapps by using srvctl command (Run as root)
#srvctl modify nodeapps -n [-o ] [-A ]
-n < node_name> - Node name.
-o - Oracle Home for the cluster software (CRS-Home).
-A The node level VIP address (/netmask[/if1[|if2|...]]).

Example:- Modify the VIP Address to
#srvctl modify nodeapps -n testrac01 -A

Use below command to change to change VIP address using VIP hostname.The srvctl command will resolve the IP to hostname or the hostname to IP address. You can use the same command to change the VIP hostname from ”testrac01-vip” to “testrac01-v”

#srvctl modify nodeapps -n testrac01 -A testrac01-v/

Step 6:- Verify the change by running below command
$srvctl config nodeapps -n testrac01 –a

Step7:- Start all resources
$srvct start nodeapps -n testrac01
$srvctl start asm -n testrac01
$srvctl start instance -d testdb –i testdb1

Step8:- Repeat the same steps on all remaining nodes in the cluster.

Satishbabu Gunukula

Friday, June 4, 2010

Remove Grid control Agents or Targets from Repository

Many users facing issues while removing the Targets or Agents from Grid Control GUI. After removing the target, the Grid control GUI still shows the removed agent. The reason is the Repository still has reference of removed Targets or Agents.

Follow the manual steps to remove the Agent or Target completely from repository:
1. Make sure that agent/target stopped
For ex:- $ emctl stop agent
2. Connect to the Grid Control Repository as SYSMAN
3. Run below command to list all registered targets from OEM
SQL> select target_name from mgmt_targets where target_type=’oracle_emd’;
4. Remove the Agent/Target from the list using below command.
SQL> exec mgmt_admin.cleanup_agent(‘Target1:3673');

You may receive below error while running “mgmt_admin.cleanup_agent”

1. ERROR at line 1:
ORA-20206: Target does not exist: Target1:3673'
ORA-06512: at "SYSMAN.MGMT_ADMIN", line 846
ORA-06512: at line 1

Check for errors in emoms.trace file, which are referring to removed agent.
[AJPRequestHandler-ApplicationServerThread-19] ERROR eml.OMSHandshake java.? - OMSHandshake failed.(AGENT URL = https://agenthostname.domainname:port/emd/main/)(ERROR = KEY_MISMATCH)

Run the following EMDIag kit fix to resolve the issue
$ repvfy verify loaders -test 700 –fix

2. “ORA-20206: Target does not exist” Error message on mgmt_admin.cleanup_agent and there is no target in sysman.mgmt_targets, but there is a record in sysman.mgmt_targets_delete without delete_complite_time set.

If anyone comes across such situation then run below command for each target without delete_complete_time set.
SQL > Exec mgmt_admin.delete_target_internal(Target_name,Target_type)

3. Ora 20221: Insufficient privileges: /WITH_ADMIN
line 400 ORA-6512: at line 1

This issue can occur in due to a BUG 3574400. Download and apply patch 3574400 to resolve the issue.

Satishbabu Gunukula