RMAN Backup using BRTools failed with RMAN-03009, ORA-19504, ORA-27054 errors

Recently I have encountered below error when doing RMAN backup using BRTools. Make sure that you have mounted your backup mount point with below options.

Once you mount your backup mount point with below options you are able to run the backup successfully.

rw,bg,hard,rsize=32768,wsize=32768,vers=3,nointr,timeo=600,proto=tcp,suid

This solution is application for with or without BRTOOLS using RMAN.

BRTOOLS Error log file

BR0051I BRBACKUP 7.20 (0)
BR0055I Start of database backup: bekozflm.fnd 2013-02-19 15.46.46
BR0484I BRBACKUP log file: /oracle/ORAC/sapbackup/bekozflm.fnd
BR0477I Oracle pfile /oracle/ORAC/102_64/dbs/initORAC.ora created from spfile /oracle/ORAC/102_64/dbs/spfileORAC.ora

BR0280I BRBACKUP time stamp: 2013-02-19 15.46.47
BR0057I Backup of database: ORAC
BR0058I BRBACKUP action ID: bekozflm
BR0059I BRBACKUP function ID: fnd
BR0110I Backup mode: FULL
BR0077I Database file for backup: /oracle/ORAC/backup/cntrlORAC.dbf
BR0061I 6 files found for backup, total size 14242.117 MB
BR0143I Backup type: online
BR0112I Files will not be compressed
BR0130I Backup device type: disk
BR0106I Files will be saved on disk in directory: /oracle/backup/ORAC/backup/bekozflm

BR0280I BRBACKUP time stamp: 2013-02-19 15.46.47
BR0256I Enter 'c[ont]' to continue, 's[top]' to cancel BRBACKUP:c
BR0280I BRBACKUP time stamp: 2013-02-19 15.46.48
BR0257I Your reply: 'c'

BR0259I Program execution will be continued...

BR0370I Directory /oracle/backup/ORAC/backup/bekozflm created
BR0280I BRBACKUP time stamp: 2013-02-19 15.46.49
BR1390I RMAN prepared for backup
BR0202I Saving init_ora
BR0203I to /oracle/backup/ORAC/backup/ORAC ...
BR0202I Saving /oracle/ORAC/102_64/dbs/initORAC.sap
BR0203I to /oracle/backup/ORAC/backup/ORAC ...
BR0280I BRBACKUP time stamp: 2013-02-19 15.46.49
BR0198I Profiles saved successfully

BR0202I Saving /oracle/ORAC/sapdata2/sr3db_2/sr3db.data2
BR0203I to /oracle/backup/ORAC/backup/bekozflm/sr3db.data2.S ...
BR0278E Command output of 'SHELL=/bin/sh /oracle/ORAC/102_64/bin/rman nocatalog':

Recovery Manager: Release 10.2.0.4.0 - Production on Tue Feb 19 15:46:49 2013
Copyright (c) 1982, 2007, Oracle. All rights reserved.
RMAN>
RMAN> connect target *
connected to target database: ORAC (DBID=1058933594)
using target database control file instead of recovery catalog
RMAN> **end-of-file**

RMAN>
host command complete
RMAN> 2> 3> 4> 5> 6> 7>
allocated channel: dsk
channel dsk: sid=381 devtype=DISK
executing command: SET NOCFAU
Starting backup at 19-FEB-13
channel dsk: starting incremental level 0 datafile backupset
channel dsk: specifying datafile(s) in backupset
input datafile fno=00005 name=/oracle/ORAC/sapdata2/sr3db_2/sr3db.data2
channel dsk: starting piece 1 at 19-FEB-13
released channel: dsk
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on dsk channel at 02/19/2013 15:46:52
ORA-19504: failed to create file "/oracle/backup/ORAC/backup/bekozflm/sr3db.data2.S"
ORA-27054: NFS file system where the file is created or resides is not mounted with correct options
Additional information: 3
RMAN>

Recovery Manager complete.

BR0200I BR_TRACE: location BrRmanCall-44, commands for RMAN in: /oracle/ORAC/backup/.bekozflm.cmd
*** glibc detected *** /usr/sap/ORAC/SYS/exe/run/brbackup: free(): corrupted unsorted chunks: 0x0000000003dd4980 ***
======= Backtrace: =========
/lib64/libc.so.6[0x3d0ca70f0f]
/lib64/libc.so.6(cfree+0x4b)[0x3d0ca7136b]
/lib64/libc.so.6(fclose+0x14b)[0x3d0ca5f95b]
/usr/sap/ORAC/SYS/exe/run/brbackup[0x679172]
/usr/sap/ORAC/SYS/exe/run/brbackup[0x69cc48]
/usr/sap/ORAC/SYS/exe/run/brbackup[0x426a0f]
/usr/sap/ORAC/SYS/exe/run/brbackup[0x44d8a5]
/lib64/libc.so.6(__libc_start_main+0xf4)[0x3d0ca1d994]
/usr/sap/ORAC/SYS/exe/run/brbackup(xdr_string+0x7a)[0x405fda]

======= Memory map: ========
00400000-008c9000 r-xp 00000000 c7:2710 1675 /sapmnt/ORAC/exe/uc/linuxx86_64/brbackup
009c8000-009e0000 rw-p 004c8000 c7:2710 1675 /sapmnt/ORAC/exe/uc/linuxx86_64/brbackup
009e0000-00aa8000 rw-p 009e0000 00:00 0
03da5000-03ed0000 rw-p 03da5000 00:00 0 [heap]
3d0c600000-3d0c61c000 r-xp 00000000 fd:00 455243 /lib64/ld-2.5.so
3d0c81c000-3d0c81d000 r--p 0001c000 fd:00 455243 /lib64/ld-2.5.so
………….
3d0d616000-3d0d815000 ---p 00016000 fd:00 455376 /lib64/libpthread-2.5.so
3d0d815000-3d0d816000 r--p 00015000 fd:00 455376 /lib64/libpthread-2.5.so
………….
3d1cc00000-3d1cc0d000 r-xp 00000000 fd:00 455395 /lib64/libgcc_s-4.1.2-20080825.so.1
3d1cc0d000-3d1ce0d000 ---p 0000d000 fd:00 455395 /lib64/libgcc_s-4.1.2-20080825.so.1
……………
3d1f400000-3d1f4e6000 r-xp 00000000 fd:01 2192511 /usr/lib64/libstdc++.so.6.0.8
3d1f4e6000-3d1f6e5000 ---p 000e6000 fd:01 2192511 /usr/lib64/libstdc++.so.6.0.8
…………..
3d1f6ee000-3d1f700000 rw-p 3d1f6ee000 00:00 0
2b889d543000-2b889d545000 rw-p 2b889d543000 00:00 0
2b889d55b000-2b889d55c000 rw-p 2b889d55b000 00:00 0
2b889d55c000-2b889e78d000 r-xp 00000000 c7:2712 6022 /oracle/ORAC/102_64/lib/libclntsh.so.10.1
2b889e78d000-2b889e98c000 ---p 01231000 c7:2712 6022 /oracle/ORAC/102_64/lib/libclntsh.so.10.1
2b889e98c000-2b889ea30000 rw-p 01230000 c7:2712 6022 /oracle/ORAC/102_64/lib/libclntsh.so.10.1
2b889ea30000-2b889ea4d000 rw-p 2b889ea30000 00:00 0
2b889ea4d000-2b889ed3c000 r-xp 00000000 c7:2712 8126 /oracle/ORAC/102_64/lib/libnnz10.so
2b889ed3c000-2b889ee3b000 ---p 002ef000 c7:2712 8126 /oracle/ORAC/102_64/lib/libnnz10.so
2b889ee3b000-2b889eeec000 rw-p 002ee000 c7:2712 8126 /oracle/ORAC/102_64/lib/libnnz10.so
2b889eeec000-2b889eef2000 rw-p 2b889eeec000 00:00 0
2b889ef08000-2b889ef12000 r-xp 00000000 fd:00 455194 /lib64/libnss_files-2.5.so
2b889ef12000-2b889f111000 ---p 0000a000 fd:00 455194 /lib64/libnss_files-2.5.so
………….
2b889f136000-2b889f19a000 r--p 00000000 c7:2712 7254 /oracle/ORAC/102_64/oracore/zoneinfo/timezlrg.dat
2b889f19a000-2b889f19d000 rw-p 2b889f19a000 00:00 0
7fff6995b000-7fff6996f000 rwxp 7ffffffe9000 00:00 0 [stack]
7fff69970000-7fff69970000 rw-p 7fffffffe000 00:00 0
ffffffffff600000-ffffffffffe00000 ---p 00000000 00:00 0 [vdso]
*** glibc detected *** /usr/sap/ORAC/SYS/exe/run/brbackup: malloc(): memory corruption: 0x00002b889f19b010 ***

======= Backtrace: =========
/lib64/libc.so.6[0x3d0ca71bae]
/lib64/libc.so.6(__libc_malloc+0x6e)[0x3d0ca738de]
/lib64/libc.so.6[0x3d0ca6030a]
/usr/sap/ORAC/SYS/exe/run/brbackup(BrSigHandle+0x108c)[0x6bb65c]
/lib64/libc.so.6[0x3d0ca302f0]
/lib64/libc.so.6(gsignal+0x35)[0x3d0ca30285]
/lib64/libc.so.6(abort+0x110)[0x3d0ca31d30]
/lib64/libc.so.6[0x3d0ca6944b]
/lib64/libc.so.6[0x3d0ca70f0f]
/lib64/libc.so.6(cfree+0x4b)[0x3d0ca7136b]
/lib64/libc.so.6(fclose+0x14b)[0x3d0ca5f95b]
/usr/sap/ORAC/SYS/exe/run/brbackup[0x679172]
/usr/sap/ORAC/SYS/exe/run/brbackup[0x69cc48]
/usr/sap/ORAC/SYS/exe/run/brbackup[0x426a0f]
/usr/sap/ORAC/SYS/exe/run/brbackup[0x44d8a5]
/lib64/libc.so.6(__libc_start_main+0xf4)[0x3d0ca1d994]
/usr/sap/ORAC/SYS/exe/run/brbackup(xdr_string+0x7a)[0x405fda]
======= Memory map: ========
00400000-008c9000 r-xp 00000000 c7:2710 1675 /sapmnt/ORAC/exe/uc/linuxx86_64/brbackup
009c8000-009e0000 rw-p 004c8000 c7:2710 1675 /sapmnt/ORAC/exe/uc/linuxx86_64/brbackup
009e0000-00aa8000 rw-p 009e0000 00:00 0
03da5000-03ed0000 rw-p 03da5000 00:00 0 [heap]
2b889ef08000-2b889ef12000 r-xp 00000000 fd:00 455194 /lib64/libnss_files-2.5.so
2b889ef12000-2b889f111000 ---p 0000a000 fd:00 455194 /lib64/libnss_files-2.5.so
/oracle/ORAC/102_64/oracore/zoneinfo/timezlrg.dat
………….
2b889f19a000-2b889f19d000 rw-p 2b889f19a000 00:00 0
7fff6995b000-7fff6996f000 rwxp 7ffffffe9000 00:00 0 [stack]
7fff69970000-7fff69970000 rw-p 7fffffffe000 00:00 0
ffffffffff600000-ffffffffffe00000 ---p 00000000 00:00 0 [vdso]
BR0280I BRBACKUP time stamp: 2013-02-19 15.46.53
BR0261E BRBACKUP cancelled by signal 6
BR0054I BRBACKUP terminated with errors
#######################################################
BR0292I Execution of BRBACKUP finished with return code 0
BR0280I BRTOOLS time stamp: 2013-02-19 15.46.53
BR0668I Warnings or errors occurred - you can continue to ignore them or go back to repeat the last action
BR0670I Enter 'c[ont]' to continue, 'b[ack]' to go back, 's[top]' to abort:
BR0280I BRTOOLS time stamp: 2013-02-19 15.46.56
BR0261E BRTOOLS cancelled by signal 2

BR0280I BRTOOLS time stamp: 2013-02-19 15.46.56
BR0654I BRTOOLS terminated with errors

Oracle Database Source Control made easy using Red-Gate new tool

The Red Gate launching new tool “Source Control for Oracle tool” on March 12, 2013.

How this tool can help you…

• Manual processes to script out database objects and check in them into SVN or TFS, can be time consuming and error prone. Source Control for Oracle automates scripting of objects and check-in to SVN or TFS.

•  Developers and DBAs can see a history of objects that have been checked in to source control, when and by who. They’ll be able to see the check-in message and a side-by-side diff view of the creation script before and after check in, object by object.

• Pending check-ins are indicated on the taskbar icon and within the tool to prompt developers to keep source control up-to-date.

• If developers are working on private schemas or a dedicated instance they can see other team members’ changes to objects and apply these from source control to their schemas before working on them.

• If developers are working on a shared database they can see all the objects that haven’t yet been checked in to source control. And a history of the objects that already have been, when, and buy who.

• The side-by-side diff view (before checking in changes, updated a local instance, or when looking at the audit trail under History) means errors are easy to spot.

You can DOWNLOAD the Beta version from below link
www.red-gate.com/source-control-for-oracle

Win a FREE 5 user license of Source Control for Oracle 

I’m giving away one five-user license of Source Control for Oracle by Red Gate Software. To enter the draw you need to do two things:

1. Like the page “oracleRACexpert.com” in Facebook before April 1st 2013
https://www.facebook.com/pages/OracleRACexpertcom/158925710927837

2. Using the comments section below, tell me your most painful experience which could have been easily avoided by using source control.

It doesn’t have to be a lack of database source control, it just has to be a story where, things could’ve been easier if you had the right source control tools.

On April 1st 2013 I’ll choose the best story and author will get a 5 user license of Source Control for Oracle.

Please make sure you comment under an account which I can use to get in touch with you if you make the top story. Particularly painful stories will earn extra points as will stories where the pain was generously shared with a larger team!

For Prize draw Terms and Conditions please visit below link
http://www.red-gate.com/products/oracle-development/source-control-for-oracle/source-control-oracle-prize-terms

Java SQL Exception “No more data to read from socket” in Oracle

Application using JDBC 10.1.0.5 fails with below error after upgrading your database from Oracle 9i to 10g.

JDBC Version 10.1.0.5 to 11.1.0.7 are affected with this issue.

java.sql.SQLException: OALL8 is in an inconsistent state.
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:138)
at oracle.jdbc.driver.T4C8Oall.init(T4C8Oall.java:325)
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:170)
at oracle.jdbc.driver.T4CPreparedStatement.execute_for_rows(T4CPreparedStatement.java:633)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1161)
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3001) Caused by: java.sql.SQLException: No more data to read from socket at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:138)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:175)
at oracle.jdbc.driver.DatabaseError.check_error(DatabaseError.java:898)
at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:994)
at oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:951)
at oracle.jdbc.driver.T4C7Ocommoncall.receive(T4C7Ocommoncall.java:100)
at oracle.jdbc.driver.T4CConnection.do_rollback(T4CConnection.java:478)
at oracle.jdbc.driver.PhysicalConnection.rollback(PhysicalConnection.java:1263)

Cause:- The error is due to the published Bug 5851267.

This issue is caused by constraints used by the optimizer getting attempted to be used at run time when they should not. This issue is documented in Note:463899.1

You should see the following error in the trace file during the execution of the transaction

ORA-07445: Exception aufgetreten: CORE Dump [delrefi()+34] [SIGSEGV] [Address not mapped to object] [0x8] [] []

Solution: This issue has been fixed in 11.1.0.7.

If you are running in 10g then you can apply the patch for Bug:5851267
Or

you can use the following workaround to resolve the issue.

Set database parameter "_optimizer_join_elimination_enabled" to false. By doing this you can disable the elimination transformation.

SQL> alter session set “_optimizer_join_elimination_enabled”=false;

For more information refer below metalink notes
BUG:5851267 - ORA-07445 DURING EXECUTION OF PACKAGE WHERE DELETE OCCURS
NOTE:463899.1 - ORA-7445 [Delrefi] Where Delete Occurs

Regards
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com

Oracle GoldenGate: A Real-time Data integration & Heterogeneous replication tool

Now a day, the majority of businesses are facing below challenges

Availability - Data must be accessible at or near 24/7/365 without performance degradation or interruption

Rduced latency -  Data must remain up-to-date without latency. 

Heterogeneity and IT flexibility - Integration and replication across different IT systems.

Transaction integrity - Data completeness

Oracle GoldenGate provides the solution for all these challenges. It is a high performance software application for real-time transactional change data capture, distribution, transformation and delivery across heterogeneous.

Oracle GoldenGate 11g Release 2 offers below new features

• Intelligent Conflict Detection and Resolution for Active-Active and Multi-Master implementations 
• Integrated Capture to support all Oracle Database and Exadata compression types, as well as XML Object  Relational, XML Binary data types 
• Expanded support for global deployments, including support for multi-byte and character set conversions 
• Improved security with added support for Federal Information Protection Standard (FIPS) 
• Enhanced manageability and monitoring utilizing Oracle Enterprise Manager 
• Increased extensibility by taking defined actions based on an event record in the transaction log or in the Trail file. 
• Expanded heterogeneity via: 
• Log-based capture from IBM DB2 on i-Series (AS/400) 
• Delivery to Postgres 
• New data types supported for Microsoft SQL Server, Sybase, and MySQL 
• Teradata multi-byte DDL support 
• IBM z/OS improved initial load performance
•  Certification for near zero downtime upgrade for Oracle Communications Billing and Revenue Management applications 

Oracle GoldenGate 11gR2 supports following scenarios

1. Zero-Downtime

2. Data Distribution

3. Disaster recovery and data protection

4. Query offloading.

Oracle GoldenGate also provides graphical monitoring tool for GoldenGate deployments and it consists of the following components:

Oracle GoldenGate Monitor Agent -  An Oracle GoldenGate Monitor Agent is installed on Oracle GoldenGate instance. It collects information of the instance and sends it to the Oracle GoldenGate Monitor Server.

Oracle GoldenGate Monitor Server - The Oracle GoldenGate Monitor Server coordinates the monitoring of multiple Oracle GoldenGate instances..

Oracle GoldenGate Monitor Repository - The Oracle GoldenGate Monitor Server uses a database as a central repository to store information about users and groups, process status, monitoring points, events, and other information.

Please visit the link: Oracle Management pack Key Features and Benefits for Oracle Golden Gate

Refer below links for Oracle GoldenGate Software, Installation Documentation, White papers & Best Practices

•  Download Oracle GoldenGate Software
•  Oracle Golden Gate Documentation
•  Oracle GoldenGate White papers, Podcasts and Webcasts

Regards
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com

Oracle Database upgrade fails with error “UnsatisfiedLinkError exception loading native library: njni10”

We have received following error when trying to upgrade database from 10g R2 10.2.0.2 to Oracle 10g R2 10.2.0.4 using “dbua” (Database upgrade assistant) tool

ORAC> dbua

UnsatisfiedLinkError exception loading native library: njni10

Exception in thread "main" java.lang.UnsatisfiedLinkError: get

at oracle.net.common.NetGetEnv.get(Native Method)

at oracle.net.config.Config.getNetDir(Unknown Source)

at oracle.net.config.Config.initConfig(Unknown Source)

at oracle.net.config.Config.<init>(Unknown Source)

at oracle.sysman.assistants.util.NetworkUtils.<init>(NetworkUtils.java:225)

at oracle.sysman.assistants.util.step.StepContext.<init>(StepContext.java:264)

at oracle.sysman.assistants.dbma.backend.CompManager.<init>(CompManager.java:610)

at oracle.sysman.assistants.dbma.ui.UICompManager.<init>(UICompManager.java:150)

at oracle.sysman.assistants.dbma.Dbma.getCompManager(Dbma.java:154)

at oracle.sysman.assistants.dbma.Dbma.execute(Dbma.java:98)

at oracle.sysman.assistants.dbma.Dbma.statusMain(Dbma.java:199)

at oracle.sysman.assistants.dbma.Dbma.main(Dbma.java:180)

The server Operating system is recently upgraded to newer version and suspected the issue might be related to library relink and ran “relink all”.

The “relink all” failed with below errors.

/oracle/ORAC/102_64/lib//libcore10.a(slftmp.o): In function `SlfTmpnam':

slftmp.c:(.text+0x10): warning: the use of `tmpnam_r' is dangerous, better use `mkstemp'

/usr/bin/ld: crti.o: No such file: No such file or directory

collect2: ld returned 1 exit status

make[1]: *** [/oracle/ORAC/102_64/sysman/lib/libnmemso.so] Error 1

make: *** [libnmemso] Error 2

/usr/bin/ld: crt1.o: No such file: No such file or directory

collect2: ld returned 1 exit status

make: *** [ctxhx] Error 1

gcc: unrecognized option '-h'

gcc: unrecognized option '-h'

gcc: unrecognized option '-h'

/usr/bin/ld: crt1.o: No such file: No such file or directory

collect2: ld returned 1 exit status

make[1]: *** [/oracle/ORAC/102_64/rdbms/lib/extproc32] Error 1

make: *** [extproc32] Error 2

Solution: Finally figured out that the issue is related to missing 32bit RPM "glibc-devel (32 bit)" and after installing the RPM the database upgraded successfully to Oracle 10g R2 10.2.04.

Regards,
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com