Monday, April 22, 2013

Oracle RMAN backup fails with ORA-01031 insufficient privileges

RMAN automatically requests a connection to the target database as SYSDBA

In order to connect to the target using RMAN as SYSDBA, user must do one of the following:
  • The user account should be part of the operating system DBA group on the target database server. It gives the ability to connect with SYSDBA privilege without password.
  • Create a password file using orapwd command and enable initialization parameter REMOTE_LOGIN_PASSWORDFILE.
  • Make sure you are passing correct username and password.
If the target database does not have a password file, then the user you are logged in as must be validated with operating system authentication.

Recovery Manager: Release 11.2.0.3.0 - Production on Thu Apr 18 16:37:36 2013
Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved.
RMAN>
RMAN> connect target *
RMAN-00571: ============================================
RMAN-00569: ======== ERROR MESSAGE STACK FOLLOWS ============
RMAN-00571: ============================================
ORA-01031: insufficient privileges

RMAN> **end-of-file**
RMAN>
host command complete

RMAN> 2> 3> 4> 5> 6> 7>
RMAN-00571: ============================================
RMAN-00569: ========== ERROR MESSAGE STACK FOLLOWS ==========
RMAN-00571: ============================================
RMAN-03002: failure of allocate command at 04/18/2013 16:37:37
RMAN-06171: not connected to target database

RMAN>
RMAN-00571: =============================================
RMAN-00569: =========== ERROR MESSAGE STACK FOLLOWS =========
RMAN-00571: =============================================
RMAN-03002: failure of list command at 04/18/2013 16:37:37
RMAN-06171: not connected to target database

Cause:  In above scenario, RMAN denies logon to target database

Solution: - create a password file for the target database or add yourself to the administrator list in the operating system.

1. If you are using Password file then make sure that it should be in the format of orapw<SID> under $ORACLE_HOME/dbs directory and also make sure you are using REMOTE_LOGIN_PASSWORDFILE parameter in init.ora or spfile .

$ orapwd file=$ORACLE_HOME/dbs/orapwTESTDB password=xxxxx

SQL> alter system set REMOTE_LOGIN_PASSWORDFILE = EXCLUSIVE scope=spfile;
If using pfile then add or edit the parameter.

Query v$pwfile_users to see the list of users who have been granted SYSDBA and SYSOPER privilege as derived from password file.

SQL> select * from v$pwfile_users;
USERNAME SYSDB SYSOP SYSAS
----------------------------- ----- ----- -----
SYS TRUE TRUE FALSE

2. Operating system authentication, Check the sqlnet.ora in your$ ORACLE_HOME\network\admin directory and add to it the following line, if not exists.

SQLNET.AUTHENTICATION_SERVICES = (NTS)

If above entry exists and commented then uncomment the line. In case if sqlnet.ora file does not exist then create the file.

Below Authentication Methods Available with Oracle Net Services

NONE means not authentication mode. The user must use valid username and password can be used to access database
ALL means for all authentication modes
NTS means for Windows NT native authentication.

Find out that weather user is part of operating system DBA group, if not add the user to the group.

In unix, you can find the group using below command.
$ cat /etc/group | grep dba

Regards
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com

ORA-01111: name for data file is unknown in Standby

Many users encountered the issue UNNAMEDnnn file in Standby Database

MRP0: Background Media Recovery terminated with error 1111
ORA-01111: name for data file 536 is unknown - rename to correct file
ORA-01110: data file 10:'/home/oracle/11.2.0/dbs/UNNAMED00010'
ORA-01157: cannot identify/lock data file 10 - see DBWR trace file

Uses will encounter this error when standby_file_management is set to auto and file path which is added to the primary database does not exist on standby site. In this situation standby database creates a dummy data file such as UNNAMEDnnnn

If you encounter this error follow the steps
Find the files which are needs to be recovered

SQL> select * from v$recover_file where error=' FILE MISSING';
FILE# ONLINE ONLINE_ ERROR CHANGE# TIME
---------- ------- ------- -------------------- ---------- ------------------
10 ONLINE ONLINE FILE MISSING 0

Run below command to Identify the file name on Primary as well as on Standby
SQL>select file#,name from v$datafile where file#=10;

Set the standby_file_management to Manual
SQL> ALTER SYSTEM set standby_file_management=MANUAL;

Create the actual data file from dummy filename
SQL> ALTER DATABASE CREATE DATAFILE 'datafile_path/UNNAMEDnnnn' as '{standby_file_path}/datafile_name.dbf';

Users will also encounter the same error when they add a data file on Primary database and STANDBY_FILE_MANAGEMENT on standby set to MANUAL. When MRP (Recovery process) trying to apply the archive logs, it will create an UNNAMEDnnnn file under $ORACLE_HOME/dbs

Run below command to create the data file manually in Standby

SQL> ALTER DATABASE CREATE DATAFILE ‘/home/oracle/11.2.0/dbs/UNNAMED00010'as ‘/dbs01/ORAC/oradata/prd_data10.dbf’;

Regards
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com

Tuesday, March 26, 2013

OALL8 is in an inconsistent state

"OALL8 is in an inconsistent state" is a generic exception that occurs in JDBC 10.1, 10.2, and 11.1. It indicates that an internal inconsistency has been detected in the JDBC connection but it does not provide information on what caused the inconsistency.

OALL8 is in an inconsistent state.
java.sql.SQLException: OALL8 is in an inconsistent state.
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:138)
at oracle.jdbc.driver.T4C8Oall.init(T4C8Oall.java:325)
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:170)
at oracle.jdbc.driver.T4CPreparedStatement.execute_for_describe(T4CPreparedStatement.java:503)
at oracle.jdbc.driver.OracleStatement.execute_maybe_describe(OracleStatement.java:1029)
at oracle.jdbc.driver.T4CPreparedStatement.execute_maybe_describe(T4CPreparedStatement.java:535)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1126)
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3001)
at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3043)
at oracle_jdbc_driver_T4CPreparedStatement_Proxy.executeQuery()
at com.agile.pc.cmserver.util.ClusterLock.acquireClusterLock(ClusterLock.java:61)
at com.agile.pc.cmserver.notification.NotifyTask.run(NotifyTask.java:413)
at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)
<2013-02-27 22:59:37,734> <NotifyTask:ERROR> notification task exception


You may also see a "Protocol Violation error, the thin driver throws this exception when it reads something from the RDBMS that it did not expect. This means that the protocol engine in the Thin driver and the protocol engine in the RDBMS are out of synch. There is no way to recover from this error. The connection is dead and You should try to close it, but that will probably fail too.

If you are able to reproduce the error, please file a TAR with Oracle Global Support by specifying the exact version numbers of the JDBC driver and the RDBMS.

Protocol violation
java.sql.SQLException: Protocol violation
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:138)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:175)
at oracle.jdbc.driver.DatabaseError.check_error(DatabaseError.java:898)
at oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:656)
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:185)
at oracle.jdbc.driver.T4CPreparedStatement.execute_for_describe(T4CPreparedStatement.java:503)
at oracle.jdbc.driver.OracleStatement.execute_maybe_describe(OracleStatement.java:1029)
at oracle.jdbc.driver.T4CPreparedStatement.execute_maybe_describeT4CPreparedStatement.
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1126)
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3001)
at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3043)
at oracle_jdbc_driver_T4CPreparedStatement_Proxy.executeQuery()
at com.agile.pc.cmserver.util.ClusterLock.acquireClusterLock(ClusterLock.java:61)
at com.agile.pc.cmserver.notification.NotifyTask.run(NotifyTask.java:199)
at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)


Possible Solution: It is very hard to troubleshoot and the simplest solution is to upgrade your JDBC Driver.

This exception no longer occurs in JDBC 11.2.

Other scenarios & Reference
1. Oracle Transportation Management (OTM) - Version 5.5.04.02 to 5.5.06.03
If you are facing the issue with OTM then check below metalink note for Cause and Solution
java.sql.SQLException: OALL8 is in an inconsistent state [ID 754144.1]

2. Master Note: Understanding the "OALL8 is in an Inconsistent State" Exception [ID 944692.1]

Regards
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com

ORA-00283: recovery session canceled due to errors

We have restored a database from backup in different system and while doing the recovery we have received the following the error.

Users will see this error when a new Data file is created and it is not part of the backup. An UNNAMED datafile name is added to the control file and file is created under $ORACLE_HOME/dbs

Errors in file /home/oracle/saptrace/diag/rdbms/orac/ORAC/trace/ORAC_pr00_16269.trc:
ORA-00283: recovery session canceled due to errors
ORA-01111: name for data file 83 is unknown - rename to correct file
ORA-01110: data file 83: '/home/oracle/11.2.0/dbs/UNNAMED00083'
ORA-01157: cannot identify/lock data file 83 - see DBWR trace file
ORA-01111: name for data file 83 is unknown - rename to correct file
ORA-01110: data file 83: '/home/oracle/11.2.0/dbs/UNNAMED00083'
Recovery Slave PR00 previously exited with exception 283
ORA-283 signalled during: ALTER DATABASE RECOVER DATABASE UNTIL TIME '2012-10-02:23:15:00'USING BACKUP CONTROLFILE ...


Solution:
Find the UNNAMED data file using below command.
SQL> SELECT FILE#, NAME FROM V$DATAFILE
FILE# NAME
-------- -----------------------------------------
1 /dbs01/ORAC/system01.dbf
2 /dbs01/ORAC/sysaux01.dbf
4 /dbs01/ORAC/users01.dbf
5 /home/oracle/11.2.0/dbs/UNNAMED00083
....

Rename an UNNAMED Data file using below command.
SQL> alter database create datafile '/home/oracle/11.2.0/dbs/UNNAMED00083' as '/dbs01/ORAC/ /prd_data83.dbf';
Database altered.

Once you create data file proceed with recovery, if needed.

Regards
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com

RMAN Backup using BRTools failed with RMAN-03009, ORA-19504, ORA-27054 errors

Recently I have encountered below error when doing RMAN backup using BRTools. Make sure that you have mounted your backup mount point with below options.

Once you mount your backup mount point with below options you are able to run the backup successfully.

rw,bg,hard,rsize=32768,wsize=32768,vers=3,nointr,timeo=600,proto=tcp,suid

This solution is application for with or without BRTOOLS using RMAN.

BRTOOLS Error log file

BR0051I BRBACKUP 7.20 (0)
BR0055I Start of database backup: bekozflm.fnd 2013-02-19 15.46.46
BR0484I BRBACKUP log file: /oracle/ORAC/sapbackup/bekozflm.fnd
BR0477I Oracle pfile /oracle/ORAC/102_64/dbs/initORAC.ora created from spfile /oracle/ORAC/102_64/dbs/spfileORAC.ora

BR0280I BRBACKUP time stamp: 2013-02-19 15.46.47
BR0057I Backup of database: ORAC
BR0058I BRBACKUP action ID: bekozflm
BR0059I BRBACKUP function ID: fnd
BR0110I Backup mode: FULL
BR0077I Database file for backup: /oracle/ORAC/backup/cntrlORAC.dbf
BR0061I 6 files found for backup, total size 14242.117 MB
BR0143I Backup type: online
BR0112I Files will not be compressed
BR0130I Backup device type: disk
BR0106I Files will be saved on disk in directory: /oracle/backup/ORAC/backup/bekozflm

BR0280I BRBACKUP time stamp: 2013-02-19 15.46.47
BR0256I Enter 'c[ont]' to continue, 's[top]' to cancel BRBACKUP:c
BR0280I BRBACKUP time stamp: 2013-02-19 15.46.48
BR0257I Your reply: 'c'

BR0259I Program execution will be continued...

BR0370I Directory /oracle/backup/ORAC/backup/bekozflm created
BR0280I BRBACKUP time stamp: 2013-02-19 15.46.49
BR1390I RMAN prepared for backup
BR0202I Saving init_ora
BR0203I to /oracle/backup/ORAC/backup/ORAC ...
BR0202I Saving /oracle/ORAC/102_64/dbs/initORAC.sap
BR0203I to /oracle/backup/ORAC/backup/ORAC ...
BR0280I BRBACKUP time stamp: 2013-02-19 15.46.49
BR0198I Profiles saved successfully

BR0202I Saving /oracle/ORAC/sapdata2/sr3db_2/sr3db.data2
BR0203I to /oracle/backup/ORAC/backup/bekozflm/sr3db.data2.S ...
BR0278E Command output of 'SHELL=/bin/sh /oracle/ORAC/102_64/bin/rman nocatalog':

Recovery Manager: Release 10.2.0.4.0 - Production on Tue Feb 19 15:46:49 2013
Copyright (c) 1982, 2007, Oracle. All rights reserved.
RMAN>
RMAN> connect target *
connected to target database: ORAC (DBID=1058933594)
using target database control file instead of recovery catalog
RMAN> **end-of-file**

RMAN>
host command complete
RMAN> 2> 3> 4> 5> 6> 7>
allocated channel: dsk
channel dsk: sid=381 devtype=DISK
executing command: SET NOCFAU
Starting backup at 19-FEB-13
channel dsk: starting incremental level 0 datafile backupset
channel dsk: specifying datafile(s) in backupset
input datafile fno=00005 name=/oracle/ORAC/sapdata2/sr3db_2/sr3db.data2
channel dsk: starting piece 1 at 19-FEB-13
released channel: dsk
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on dsk channel at 02/19/2013 15:46:52
ORA-19504: failed to create file "/oracle/backup/ORAC/backup/bekozflm/sr3db.data2.S"
ORA-27054: NFS file system where the file is created or resides is not mounted with correct options
Additional information: 3
RMAN>

Recovery Manager complete.

BR0200I BR_TRACE: location BrRmanCall-44, commands for RMAN in: /oracle/ORAC/backup/.bekozflm.cmd
*** glibc detected *** /usr/sap/ORAC/SYS/exe/run/brbackup: free(): corrupted unsorted chunks: 0x0000000003dd4980 ***
======= Backtrace: =========
/lib64/libc.so.6[0x3d0ca70f0f]
/lib64/libc.so.6(cfree+0x4b)[0x3d0ca7136b]
/lib64/libc.so.6(fclose+0x14b)[0x3d0ca5f95b]
/usr/sap/ORAC/SYS/exe/run/brbackup[0x679172]
/usr/sap/ORAC/SYS/exe/run/brbackup[0x69cc48]
/usr/sap/ORAC/SYS/exe/run/brbackup[0x426a0f]
/usr/sap/ORAC/SYS/exe/run/brbackup[0x44d8a5]
/lib64/libc.so.6(__libc_start_main+0xf4)[0x3d0ca1d994]
/usr/sap/ORAC/SYS/exe/run/brbackup(xdr_string+0x7a)[0x405fda]

======= Memory map: ========
00400000-008c9000 r-xp 00000000 c7:2710 1675 /sapmnt/ORAC/exe/uc/linuxx86_64/brbackup
009c8000-009e0000 rw-p 004c8000 c7:2710 1675 /sapmnt/ORAC/exe/uc/linuxx86_64/brbackup
009e0000-00aa8000 rw-p 009e0000 00:00 0
03da5000-03ed0000 rw-p 03da5000 00:00 0 [heap]
3d0c600000-3d0c61c000 r-xp 00000000 fd:00 455243 /lib64/ld-2.5.so
3d0c81c000-3d0c81d000 r--p 0001c000 fd:00 455243 /lib64/ld-2.5.so
………….
3d0d616000-3d0d815000 ---p 00016000 fd:00 455376 /lib64/libpthread-2.5.so
3d0d815000-3d0d816000 r--p 00015000 fd:00 455376 /lib64/libpthread-2.5.so
………….
3d1cc00000-3d1cc0d000 r-xp 00000000 fd:00 455395 /lib64/libgcc_s-4.1.2-20080825.so.1
3d1cc0d000-3d1ce0d000 ---p 0000d000 fd:00 455395 /lib64/libgcc_s-4.1.2-20080825.so.1
……………
3d1f400000-3d1f4e6000 r-xp 00000000 fd:01 2192511 /usr/lib64/libstdc++.so.6.0.8
3d1f4e6000-3d1f6e5000 ---p 000e6000 fd:01 2192511 /usr/lib64/libstdc++.so.6.0.8
…………..
3d1f6ee000-3d1f700000 rw-p 3d1f6ee000 00:00 0
2b889d543000-2b889d545000 rw-p 2b889d543000 00:00 0
2b889d55b000-2b889d55c000 rw-p 2b889d55b000 00:00 0
2b889d55c000-2b889e78d000 r-xp 00000000 c7:2712 6022 /oracle/ORAC/102_64/lib/libclntsh.so.10.1
2b889e78d000-2b889e98c000 ---p 01231000 c7:2712 6022 /oracle/ORAC/102_64/lib/libclntsh.so.10.1
2b889e98c000-2b889ea30000 rw-p 01230000 c7:2712 6022 /oracle/ORAC/102_64/lib/libclntsh.so.10.1
2b889ea30000-2b889ea4d000 rw-p 2b889ea30000 00:00 0
2b889ea4d000-2b889ed3c000 r-xp 00000000 c7:2712 8126 /oracle/ORAC/102_64/lib/libnnz10.so
2b889ed3c000-2b889ee3b000 ---p 002ef000 c7:2712 8126 /oracle/ORAC/102_64/lib/libnnz10.so
2b889ee3b000-2b889eeec000 rw-p 002ee000 c7:2712 8126 /oracle/ORAC/102_64/lib/libnnz10.so
2b889eeec000-2b889eef2000 rw-p 2b889eeec000 00:00 0
2b889ef08000-2b889ef12000 r-xp 00000000 fd:00 455194 /lib64/libnss_files-2.5.so
2b889ef12000-2b889f111000 ---p 0000a000 fd:00 455194 /lib64/libnss_files-2.5.so
………….
2b889f136000-2b889f19a000 r--p 00000000 c7:2712 7254 /oracle/ORAC/102_64/oracore/zoneinfo/timezlrg.dat
2b889f19a000-2b889f19d000 rw-p 2b889f19a000 00:00 0
7fff6995b000-7fff6996f000 rwxp 7ffffffe9000 00:00 0 [stack]
7fff69970000-7fff69970000 rw-p 7fffffffe000 00:00 0
ffffffffff600000-ffffffffffe00000 ---p 00000000 00:00 0 [vdso]
*** glibc detected *** /usr/sap/ORAC/SYS/exe/run/brbackup: malloc(): memory corruption: 0x00002b889f19b010 ***

======= Backtrace: =========
/lib64/libc.so.6[0x3d0ca71bae]
/lib64/libc.so.6(__libc_malloc+0x6e)[0x3d0ca738de]
/lib64/libc.so.6[0x3d0ca6030a]
/usr/sap/ORAC/SYS/exe/run/brbackup(BrSigHandle+0x108c)[0x6bb65c]
/lib64/libc.so.6[0x3d0ca302f0]
/lib64/libc.so.6(gsignal+0x35)[0x3d0ca30285]
/lib64/libc.so.6(abort+0x110)[0x3d0ca31d30]
/lib64/libc.so.6[0x3d0ca6944b]
/lib64/libc.so.6[0x3d0ca70f0f]
/lib64/libc.so.6(cfree+0x4b)[0x3d0ca7136b]
/lib64/libc.so.6(fclose+0x14b)[0x3d0ca5f95b]
/usr/sap/ORAC/SYS/exe/run/brbackup[0x679172]
/usr/sap/ORAC/SYS/exe/run/brbackup[0x69cc48]
/usr/sap/ORAC/SYS/exe/run/brbackup[0x426a0f]
/usr/sap/ORAC/SYS/exe/run/brbackup[0x44d8a5]
/lib64/libc.so.6(__libc_start_main+0xf4)[0x3d0ca1d994]
/usr/sap/ORAC/SYS/exe/run/brbackup(xdr_string+0x7a)[0x405fda]
======= Memory map: ========
00400000-008c9000 r-xp 00000000 c7:2710 1675 /sapmnt/ORAC/exe/uc/linuxx86_64/brbackup
009c8000-009e0000 rw-p 004c8000 c7:2710 1675 /sapmnt/ORAC/exe/uc/linuxx86_64/brbackup
009e0000-00aa8000 rw-p 009e0000 00:00 0
03da5000-03ed0000 rw-p 03da5000 00:00 0 [heap]
2b889ef08000-2b889ef12000 r-xp 00000000 fd:00 455194 /lib64/libnss_files-2.5.so
2b889ef12000-2b889f111000 ---p 0000a000 fd:00 455194 /lib64/libnss_files-2.5.so
/oracle/ORAC/102_64/oracore/zoneinfo/timezlrg.dat
………….
2b889f19a000-2b889f19d000 rw-p 2b889f19a000 00:00 0
7fff6995b000-7fff6996f000 rwxp 7ffffffe9000 00:00 0 [stack]
7fff69970000-7fff69970000 rw-p 7fffffffe000 00:00 0
ffffffffff600000-ffffffffffe00000 ---p 00000000 00:00 0 [vdso]
BR0280I BRBACKUP time stamp: 2013-02-19 15.46.53
BR0261E BRBACKUP cancelled by signal 6
BR0054I BRBACKUP terminated with errors
#######################################################
BR0292I Execution of BRBACKUP finished with return code 0
BR0280I BRTOOLS time stamp: 2013-02-19 15.46.53
BR0668I Warnings or errors occurred - you can continue to ignore them or go back to repeat the last action
BR0670I Enter 'c[ont]' to continue, 'b[ack]' to go back, 's[top]' to abort:
BR0280I BRTOOLS time stamp: 2013-02-19 15.46.56
BR0261E BRTOOLS cancelled by signal 2

BR0280I BRTOOLS time stamp: 2013-02-19 15.46.56
BR0654I BRTOOLS terminated with errors