Showing posts with label Backup/Recovery. Show all posts
Showing posts with label Backup/Recovery. Show all posts

Monday, April 21, 2025

Data Recovery Advisor (DRA) in Oracle Database

I had the opportunity to work with the Data Recovery Advisor (DRA) feature in earlier versions of Oracle Database, and its great tool to automate the recovery process, reducing downtime and recovery time

In Oracle 11g, the Data recovery advisor is introduced, and this tool helps to reduce the recovery time by providing best automated repair option for the database. This tool automatically diagnoses data failures, determine best repair options, executes repairs at user request, it helps to reduce mean time to recover (MTTR).

The Data recovery advisor can help to limit damage caused by corruption as it can detect, analyze and repair failures before database process discovers it. In traditional method user manually determines the impact and repair options, in some cases users needs to determine right sequence of repair as well.

The Data recovery advisor commands are LIST FAILURE, ADVISE FAILURE, REPAIR FAILURE and CHANGE FAILURE.

  • LIST FAILURE: Lists the failures that have occurred in the database. 
        RMAN> LIST FAILURE;
  • ADVISE FAILURE: Provides the best repair options for the listed failures. 
        RMAN> ADVISE FAILURE;
  • REPAIR FAILURE: Executes the repair process based on the user's selection.
        RMAN> REPAIR FAILURE;
  • CHANGE FAILURE: Allows users to modify the failure status.
    RMAN> CHANGE FAILURE FAILURE #1 RESOLVED;
 
Deprecation of DRA in Oracle 19c

While the Data Recovery Advisor was a valuable tool in earlier Oracle releases, it was deprecated in Oracle 19c. This means that DRA will no longer be available for use in future versions beyond Oracle 19c. Additionally, the associated RMAN commands (LIST FAILURE, ADVISE FAILURE, REPAIR FAILURE, and CHANGE FAILURE) have also been deprecated. As a result, DBAs will no longer have access to these commands in Oracle 19c and beyond, marking the end of an era for this powerful recovery tool.

Thanks & Regards,
https://oracleracexpert.com

Monday, August 14, 2023

Article: Mastering Data History using Oracle's Flashback Data Archive Feature

The Historical data is key for business decisions and having reliable data is very important for organizations. There are frequent implications for both financial and legal data for organizations and having a secure and accurate history of change of data is crucial for business.

Oracle Flashback technology offers significant benefits for database management and recovery. 

Please see the below article how it will help to meet the organizational needs.



Thanks,
https://oracleracexpert.com

Wednesday, February 10, 2021

Patching Oracle Database Platform using Opatch or MOPatch

When applying the Oracle patches, make sure you follow the processes pre-checks, patch execution and post checks.

In SAP environment, MOPatch is very helpful and it has bundle patchs (SBP). Using this tool we can install multiple patches at single run, it will automate the oracle patching processes. However, note that MOPatch internally calls OPatch to install patches.

Pre-checks
  • Make sure you have enough space in $ORACLE_HOME file system
  • Take a Tar or backup of $ORACLE_HOME
tar cvf - $ORACLE_HOME/12201 | gzip > 12201.101020.tar.gz
  • Install latest Opatch and MOpatch (in case of SAP)
$ mv $ORACLE_HOME/OPatch $ORACLE_HOME/OPatch-pre-SBP_12201201020_101020
$ unzip -qd $ORACLE_HOME OPATCH12201P_2008-70001555.ZIP -x SIGNATURE.SMF
$ unzip -qd $ORACLE_HOME/sapbundle SAP12201P_2011-70001555.ZIP 'SBP_12201201020_101020/MOPatch/*'
$ test -d $ORACLE_HOME/MOPatch && mv $ORACLE_HOME/MOPatch $ORACLE_HOME/MOPatch-pre-SBP_12201201020_101020
$ mv $ORACLE_HOME/sapbundle/SBP_12201201020_101020/MOPatch $ORACLE_HOME/MOPatch
  • check the version of MOPatch & Opatch version
$ORACLE_HOME/MOPatch/mopatch.sh –h
$ORACLE_HOME/OPatch/opatch version

Execution
  • Stop all database applications
  • Backup the database and related files
  • Shut down the databases, listeners, and other processes running from the Oracle Home, use the fuser OS command to check for stale SQL*Plus sessions: $SBPFUSER $ORACLE_HOME/bin/oracle
  • Install the patch –
For SAP systems using MOPatch
env ORACLE_HOME=$ORACLE_HOME $ORACLE_HOME/MOPatch/mopatch.sh -v -s SAP12201P_2011-70001555.ZIP

You should see a message
MOPatch completed successfully
Overall Status : COMPLETE

For Other Oracle Systems using Opatch
$ cd /opt/patches/<patch number>
$ opatch apply

You should see a message
Opatch completed successfully
Overall Status : COMPLETE

Post-Checks
  • Start the database and verify the database processes and logs. The logs should not have any error messages
  • Run the post patch update steps for SAP system 
$ORACLE_HOME/sapbundle/catsbp
  • Apply _FIX_CONTROL, EVENT parameters if required.
Where _FIX_CONTROL, initialization parameter will give control over specific features and bug fixes. However, it should be set under the direction of Oracle Corporation

Ex:- ALTER SYSTEM SET "_FIX_CONTROL"='4545454:ON','4547778:OFF',……. COMMENT='SAP_12201201020_101020' SCOPE=SPFILE;

Where EVENT, initialization parameter will be used to generate information in form of a trace file in the context of the event. The trace file is located in a directory specified by the parameter USER_DUMP_DEST

ALTER SYSTEM SET EVENT= '10027','10028', '10142' COMMENT='SAP_12201171017_101020' SCOPE=SPFILE;
  • Restart the database and Take a backup
  • Start the application and check the application
SAP Oracle Patches Reference
  • Known issues related to the installation of Database Release Update 12.2.0.1.201020 and the other patches contained in this SBP are described in SAP Note 2507228 Database: Patches for 12.2.0.1.
  • Known issues related to the Oracle installation utilities, including MOPatch, OPatch, and the Oracle Universal Installer, are described in SAP Note 1915299 Troubleshooting Software Installation (12.1/12.2).
Oracle Patches reference

Please refer Oracle documentation
https://docs.oracle.com/cd/E24628_01/doc.121/e39376/opatch_overview.htm
Oracle Help Center
https://docs.oracle.com/cd/E25290_01/doc.60/e25224/patchesupdates.htm

Friday, December 18, 2020

ORA-39095: Dump file space has been exhausted: Unable to allocate 8192 bytes

Uses can receive below error during the export in case if you did not provide enough file names

$expdp xxx/xxxx directory=EXP_DIR DUMPFILE=exp_user.dmp LOGFILE=exp_user.log SCHEMAS=EMPUSER FILESIZE=20m
…..

ORA-39095: dump file space has been exhausted: unable to allocate 4096 bytes
Job "EMPUSER"."sys_export_table_01" stopped due to fatal error at 10:05:03

You will also receive this error if you specify one dump file or a number less than parallelism value. The slave processes locking the file does not release the lock even after finishing as it wait other processes to write to the file.

$expdp xxx/xxxx directory= EXP_DIR DUMPFILE=exp_user.dmp LOGFILE=exp_user.log SCHEMAS=EMPUSER PARALLEL=5

ORA-39095: dump file space has been exhausted: unable to allocate 8192 bytes

DONOT use PARALLEL clause to avoid this error.

In case if you have space issue on file system, then you can generate multiple dumps on multiple file systems or disks.


Thanks,
http://oracleracexpert.com
Oracle ACE


Tuesday, June 16, 2020

Duplicating On-premise Database to Oracle Cloud in Oracle 18c

From Oracle 18c, by using DUPLICATE command you can duplicate an on-prem database to Oracle Could. Oracle databases on Oracle Cloud are always encrypted even if no encryption clause is specified during duplication.

Using Oracle RMAN you to perform two main types of database duplications.
  • Backup-based duplication – In this method we will use pre-existing RMAN backups or copies of the source database. 
  • Active database duplication - The database will be duplicated by copying the live source database over the network to the auxiliary instance. 
Follow the steps to migrate on-prem database to Cloud:
1. Ensure the prerequisites for the DUPLICATION technique are met, see Prerequisites for Duplicating a Database".
2. Configure Recovery Manager to use Oracle Database Backup Cloud Service as the backup destination. Use CONFIGURE command. Pls refer Oracle Cloud Using Oracle Database Backup Service for more details.

Syntax:-
RMAN> CONFIGURE CHANNEL DEVICE TYPE sbt
PARMS='SBT_LIBRARY= SBT-library-location-for-backup-module,
SBT_PARMS=(OPC_PFILE=location-of-the-configuration file)';

Ex:-
RMAN> CONFIGURE CHANNEL DEVICE TYPE sbt
PARMS='SBT_LIBRARY=/oracle/18c/lib/libopc.so,
SBT_PARMS=(OPC_PFILE=/oracle/18c/dbs/emp.ora)';

3. Complete the planning tasks, as described in "Planning to Duplicate a Database

4. Prepare the auxiliary instance, as described in "Preparing the Auxiliary Instance"

• You must create auxiliary instance as CDB and start instance with enable_pluggable_database=TRUE in the initialization parameter file
• When instructed to create an initialization parameter file for the auxiliary instance, user must copy the file from the source database. This ensures that the auxiliary instance is also a CDB. After you copy the file you need to perform the following steps:
   – Modify the DB_NAME parameter
   – Modify the various destination/location parameters
• Start the auxiliary instance in NOMOUNT mode.

5. Start RMAN and connect to the root as a common user with the SYSBACKUP privilege or SYSDBA.

6. If the source CDB uses encryption, then open the Oracle keystore that contains the master key on the source CDB.

7. Configure RMAN channels, if necessary, as described in "Configuring RMAN Channels for Use in Duplication".

8. On the destination CDB, open the Oracle keystore from the source CDB. If the destination CDB uses a password-based software keystore, then you must specify the password used to open this keystore

SET DECRYPTION WALLET OPEN IDENTIFIED BY 'password';

9. Use the DUPLICATE command to duplicate the source CDB.

Use one of the following options of the DUPLICATE command:
  • DUPLICATE DATABASE or DUPLICATE...ACTIVE DATABASE - Use this command for duplicating non-CDBs and CDBs.
  • DUPLICATE DATABASE ... FOR STANDBY - Use this command create a standby database by duplicating the source.
  • Use the DUPLICATE DATABASE ... FOR FARSYNC – Use this command to create an Oracle Data Guard far sync instance using duplication.
  • DUPLICATE PLUGGABLE DATABASE – Use this command to duplicate one or more PDBs while connected to the root.
You can also use SET NEWNAME command to specify alternate names for duplicate database files,

Note that Using duplication to create a standby database to Oracle Cloud is not supported


Regards
Satishbabu Gunukula, Oracle ACE

Monday, May 11, 2020

Relocated PDBs in Oracle Database 18c

Oracle 18c allows RMAN backups created before the non-CDB or PDB was migrated into a different target CDB can be used for recovery operations. The COMPATIBLE parameter of the source and Oracle Cloud must be set to 18.0.0 or higher

The RMAN commands used to backup and recovery CDBs and PDBs are the same as those used for non-CDBs, with few variations in the syntax.

The backup and recovery operations performed on non-CDBs can also be performed on CDBs and PDBs. This includes the following:
– Full and incremental backups
– Complete and point-in-time recovery (PITR)
– Reporting operations (such as listing and cross-checking backups)
– Flashback Database

We need take metadata for the existing backups and available to the destination CDB. To export metadata user needs to run DBMS_PDB.EXPORTRMANBACKUP procedure on the source database.

EXECUTE DBMS_PDB.exportrmanbackup();
Or
EXECUTE DBMS_PDB.exportrmanbackup('EMP_PDB');

In case if you are unplug you no need to run this command as unplug already includes the metadata.

Convert NON-CDB to PDB : As we have metadata, now we can covert the NON-CDB instance to PDB.

Step1: Open the non-CDB instance in read-only mode and describe and shutdown

SQL> SHUTDOWN IMMEDIATE;
SQL> STARTUP OPEN READ ONLY;
SQL> BEGIN
DBMS_PDB.DESCRIBE( pdb_descr_file => '/oracle/empdb.xml');
END;
SQL> SHUTDOWN IMMEDIATE;

Step2: Create the new pluggable database using the non-CDB description file that we have taken in above step

SQL> CREATE PLUGGABLE DATABASE empdb_pdb USING '/oracle/empdb.xml' COPY;
SQL> ALTER SESSION SET CONTAINER= empdb_pdb;
SQL> @$ORACLE_HOME/rdbms/admin/noncdb_to_pdb.sql
SQL> ALTER PLUGGABLE DATABASE OPEN;
SQL> ALTER PLUGGABLE DATABASE SAVE STATE;

Step3:- Restore and recovery using pre-plugin backup

SQL> ALTER PLUGGABLE DATABASE empdb_pdb CLOSE IMMEDIATE;

RMAN> SET PREPLUGIN CONTAINER=db18cpdb;
RMAN> RESTORE PLUGGABLE DATABASE empdb_pdb FROM PREPLUGIN;
RMAN> RECOVER PLUGGABLE DATABASE empdb_pdb FROM PREPLUGIN;

Sometimes users may come across RMAN-06054 error in that case you need to CATALOG the missing archive log and start the recovery again.

RMAN-06054: media recovery requesting unknown archived log for thread 3 sequence 94983
RMAN>SET PREPLUGIN CONTAINER= empdb_pdb;
RMAN>CATALOG PREPLUGIN ARCHIVELOG 'oracle/archivelog/arc_empdb_pdb_3_94983.arc';
RMAN>RECOVER PLUGGABLE DATABASE empdb_pdb FROM PREPLUGIN;

Perform normal recovery and open the database

RMAN>RECOVER PLUGGABLE DATABASE empdb_pdb;
RMAN>ALTER PLUGGABLE DATABASE empdb_pdb OPEN;


Reference:
Check preplugin backups available to the CDB instance
RMAN> LIST PREPLUGIN BACKUP OF PLUGGABLE DATABASE empdb_pdb;

Preplugin backups are usable only on the destination CDB into which you plug in the source non-CDB or PDB

Regards
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com

Sunday, May 3, 2020

How to Register or Catalog Backups, Archive logs, data file copy, FRA...etc

When you backup using RMAN all backups  will be registered with catalog. If you have taken backup without catalog and you can also register manually.

Backup pieces copied manually to different location to catalog backup piece run below command.
RMAN> CATALOG BACKUPPIECE /oracle/backup/DB1_1978087_6446005710.bkp' ;

In case if you have taken backup using o/s command and to catalog a datafile copy use below command.
RMAN> CATALOG DATAFILECOPY '/oracle/backup/employee01.bak' LEVEL 0;

In case, archive log location full and you have moved backups to different location using o/s command then you can register the archive logs using below command

RMAN> CATALOG ARCHIVELOG ‘/oracle/backup2/db1_arch_99089_0876589.arc'
If you are using Flash recovery area and want to catalog all files in the FRA use below command.
RMAN> CATALOG recovery area;

This command will prompt to confirm that you want the found files to be cataloged. If you do not want to prompt use “noprompt” clause.
RMAN> CATALOG recovery area noprompt;

If you want to catalog only files Start with specific name or belongs to specific SID (db1) use below command
RMAN> CATALOG START WITH '/oracle/backup/db1'

If you run CROSSCHECK, it will go through the files that match criteria and will be marked as available, and files not found will be marked as EXPIRED.

RMAN> CROSSCHECK BACKUP ;
If you want to DELETE EXPIRED backups you can use below command, note that EXPIRED copy will no longer appear in RMAN. You have to be extra cautious to delete backups.

RMAN> DELETE EXPIRED BACKUP ;

If you find extra copies and want to delete those extra copies use DELETE BACKUPPIECE command:
RMAN> DELETE BACKUPPIECE 4555;
Hope this helps managing your backups taken outside of RMAN recovery catalog .

Thanks & Regards 
Satishbabu Gunukula, Oracle ACE
http://oracleracexpert.com



Wednesday, April 29, 2020

ORA-19815: WARNING: db_recovery_file_dest_size of 128849010 bytes is 99.03% used, and has 14136528 remaining bytes available.

You will receive ORA-19815 is reaching maximum size of flash backup recovery area (FRA) and note that this limitation will be managed by db_recovery_file_dest_size parameter

If you got this error as one time then you might be running adhoc load or some batch job. Also you will get below message in alert.log file. Please follow one of the choice to fix the issue.

ORA-19815: WARNING: db_recovery_file_dest_size of 128849010 bytes is 99.03% used, and has 14136528 remaining bytes available.
2020-02-21T11:31:19.892420-07:00

************************************************************************
You have following choices to free up space from recovery area:
1. Consider changing RMAN RETENTION POLICY. If you are using Data Guard,
then consider changing RMAN ARCHIVELOG DELETION POLICY.
2. Back up files to tertiary device such as tape using RMAN
BACKUP RECOVERY AREA command.
3. Add disk space and increase db_recovery_file_dest_size parameter to
reflect the new space.
4. Delete unnecessary files using RMAN DELETE command. If an operating
system command was used to delete files, then use RMAN CROSSCHECK and
DELETE EXPIRED commands.

In case if you archive logs are going to FRA then you should run RMAN backups and delete the unnecessary files to avoid Database hung or CRASH situation.

RMAN> backup archivelog all delete input;

If the Flash recovery area is 100% then you might receive below error when starting up the database

Database mounted.
ORA-03113: end-of-file on communication channel
Process ID: 9789
Session ID: 321 Serial number: 3


Pls make sure you follow below steps to add more space to flash recovery area (FRA)

SQL> startup mount;
SQL> alter system set db_recovery_file_dest_size=<size>G scope=both;
Replace <size> with a number larger than the current and ensure is sufficient space in the destination
SQL> alter database open;


To see what FRA has run below command

SQL> select * from v$flash_recovery_area_usage;
FILE_TYPE            PERCENT_SPACE_USED PERCENT_SPACE_RECLAIMABLE NUMBER_OF_FILES     CON_ID
-------------------- ------------------ ------------------------- --------------- ----------
CONTROL FILE                          0                         0               0          0
REDO LOG                              0                         0               0          0
ARCHIVED LOG                          0                         0               0          0
BACKUP PIECE                          0                         0               0          0
IMAGE COPY                            0                         0               0          0
FLASHBACK LOG                     99.76                         0             192          0
FOREIGN ARCHIVED LOG                  0                         0               0          0
AUXILIARY DATAFILE C                  0                         0               0          0

In my case I am using for FLASHBACK LOG, these logs are different from redo logs or undo. They are a separate, optional recording of database activity.

These logs are automatically deleted when FRA is our of space and logs is not needed for recvoery within the window specified in DB_FLASHBACK_RETENTION_TARGET parameter

To determine the disk quota and current disk usage in the flash recovery area, query the view
V$RECOVERY_FILE_DEST.

Thanks & Regards,
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com

Wednesday, April 8, 2020

ORA-00230: operation disallowed: snapshot control file enqueue unavailable

User might encounter below error, in case if RMAN backup is already running and user might have triggered another backup. The second backup must wait control file enqueuer to create a snapshot of the controlfile.

RMAN-03002: failure of backup command at 04/10/2020 16:40:33
ORA-00230: operation disallowed: snapshot control file enqueue unavailable

You can run below command to find out the backup that is running and causing the wait

SELECT S.SID, USERNAME , PROGRAM, MODULE,
ACTION, LOGON_TIME FROM V$SESSION s, V$ENQUEUE_LOCK l
WHERE l.SID = s.SID AND l.TYPE = 'CF' AND l.ID1 = 0 AND l.ID2 = 2;

To avoid this error make sure that backups are not overlapping each other.

Regards,
Satishbabu Gunukula, Oracle ACE
http://oracleracexpert.com

Tuesday, October 1, 2019

ORA-01033: ORACLE initialization or shutdown in progress

You will see this error very often during data guard setup, the main reason for this error is the password files are not copied to from primary to standby.

ORA-01033: ORACLE initialization or shutdown in progress.
Cause: An attempt was made to log on while Oracle is being started up or shutdown.
Action: Wait a few minutes. Then retry the operation 



If you don’t have any password file then create one using below command on primary and copy to standby node.

$orapwd file=orapwDG password=xxxxxx

In case of RAC, the password file might be on Primary cluster DB which was stored in ASM, but on standby you have copied the password file to $ORACLE_HOME/dbs

Run below commands to on both primary and standby

For ex:- On Primary
$ srvctl config database -d DG 
Database unique name: DG
Database name: DG
Oracle home: /home/oracle/product/12.2.0.1/dbhome_1
Oracle user: oracle
Spfile: +DISK1/DG/ parameterfile/spfileDG.ora
Password file: +DISK1/DG/ parameterfile/pwdDG

For ex:- On standby
$ srvctl config database -d DG_std
Database unique name: DG_std
Database name: DG
Oracle home: /home/oracle/product/12.2.0.1/dbhome_1
Oracle user: oracle
Spfile: +DISK1/DG/ parameterfile/spfileDG.ora
Password file: /home/oracle/product/12.2.0.1/dbhome_1/dbs/orapwDG

Also users see this error on normal database. There are many possible reasons for this error.
1. You may get this error if Database might not opened
2. Startup/shutdown is in the middle and hung up.
3. If you are trying to connect to wrong database
4. Database might be in down for recovery

Please make sure you check below

1. If the database is not opened then use command “alter database open” .
2. Check target DB is in the middle of startup or shutdown as system is very busy or has some issues. You need to verity alert.log to find out the real issue.
3. Verify all environment variable are set correct and In case if DB is down startup the database
4. In case if user doing recovery and not successful and below steps will help to recover the DB
SQL> startup mount
ORACLE Instance started
SQL> recover database
Media recovery complete
SQL> alter database open;
Database altered

Regards
Satishbabu Gunukula, Oracle ACE

Tuesday, May 14, 2019

Recovering Tables and Table Partitions using RMAN Backups

Oracle 12c offers new feature to recover tables and table partitions at a point-in-time using RMAN without affecting the other objects in the database. Use RECOVER TABLE command to recover tables or table partitions from an RMAN backup.

This feature is useful in the following scenarios:
  • Object has Logical corruption or dropped
  • There is no sufficient undo to perform Flashback table
  • DDL operation modified the structure and you want to recover the data (Flashback cannot rewind the structural changes)
  • Recover a small number of tables to a point-in-time
User should verify the pre-requisites before you prepare to recover the tables and table partitions. RMAN enables you to specify the point-in-time either using SCN, Time or sequence number.

The conditions must be met to perform table/ table Partition recovery:

Before you prepare to recover the tables and table partitions,
  • Database must be in ARCHIVELOG mode and read-write mode
  • At least one full backup is available along with archived logs
  • Enough disk space is available on the database server for auxiliary instance
  • If present, any dependent objects to include in recovery
  • At least 1 Gigabyte extra in memory for the auxiliary database
  • Tables and table partitions on standby databases cannot be recovered.
  • Tables with named NOT NULL constraints cannot be recovered with the REMAP option. · COMPATIBLE parameter must be set to 11.1.0 or higher to recover table partition

 
                    
RMAN enables recovery of selected tables without affecting remaining database objects. During the recovery process RMAN creates an auxiliary database, which is used to recover the tables or table partitions to a specified point-in-time. User need to specify the auxiliary database location using AUXILIARY DESTINATION clause in the RECOVERY command or SET NEWNAME command.

Please find the steps performed by RMAN during the recovery process:

1. Determine the backup which has the tables or table partitions that needs to recover to specified point-in-time
2. Create auxiliary database and recovery the tables or table partitions until specified point-in-time
3. Take a data dump export with recovered tables or table partitions
4. Import the dump into target database
5. Rename the recovered tables or table partitions in the target database

Please find an example to recovery TABLE1 table.

RECOVER TABLE TESTUSER.TABLE1
UNTIL SCN 384840289
AUXILIARY DESTINATION '/TEMP/ORCLDB/recover'
DATAPUMP DESTINATION '/TEMP/ORCLDB/dumpfiles'
DUMP FILE 'testdump.dat';

If source table exists then user can specify NOTABLEIMPORT or REMAP TABLE. Also user can use UNTIL TIME or UNTIL SEQUENCE clause to specify point-in-time recovery.

Please find an example to recovery TABLE1 table as TABLE1_REC to the state that it was 2 days before the current date.

RECOVER TABLE TESTUSER.TABLE1
UNTIL TIME ‘SYSDATE-2’
AUXILIARY DESTINATION '/TEMP/ORCLDB/recover'
REMAP TABLE 'TESTUSER'.'TABLE1':'TABLE1_REC';

There are some limitations recovering tables and table partitions:
  • We cannot recover table and table partitions belongs of SYS schema
  • We cannot recover table or table partitions from SYSAUX,SYSTEM tablespace Tables
  • We cannot recover tables with named NOT NULL constraint using REMAP option.
  • We cannot recover Table/Table partitions ON STANDBY database
  • We cannot recovery table partitions if version is prior Oracle Database 11g R1
Note that there are other methods available to recover tables to a point-in-time such as Oracle Flashback and Tablespace Point-in-Time Recovery.

Refer below link for

Flashback and Database Point-in-Time Recovery
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com


Sunday, May 5, 2019

Restoring and Recovering Files over Network in Oracle

Oracle 12c offers many new features and restoring, recovering files over the network is one of the new feature. Using RMAN you can restore and recover a database, datafile, controlfile, tablespace or spfile over the network from a physical standby database. To restore the database over the network, use the RESTORE… FROM SERVICE command and use the RECOVER…FROM SERVICE command to recover the database over the network. The FROM SERVICE clause specifies the service name of the physical standby.

You can also use multisection, compression and encryption to improve backup and restore performance.

  • Use SECTION SIZE with RESTORE command to perform multisection restore
  • Use SET ENCRYPTION clause before the RESTORE command to specify the encryption
  • Use USING COMPRESSED BACKUPSET clause to compress backup sets
This feature is useful to synchronize primary and standby database. Here are the few scenarios
Roll-forward a physical standby database to sync with the primary database
Restore the primary database using physical standby database.
Restore physical standby database using the primary database.

In the following example restoring data file over the network from physical standby to primary database

Connected to primary database implicitly
RMAN> CONNECT TARGET /

Restoring the datafile on the primary using datafile on physical database with service “standby_db”
RMAN> RESTORE DATAFILE '/oradata1/orcl/users.dbf' FROM SERVICE standby_db SECTION SIZE 1024M;

Prior to 12c , to restore and recover files over the network, you need to use RMAN BACKUP INCREMENTAL FROM SCN command to create a backup on the primary database that started at the current SCN of the standby, and was then used to roll the standby database forward in time. That manual, multi-step method is not necessary as of Oracle Database 12c.

Regards
Satishbabu Gunukula, Oracle ACE

ORA-39126: Worker unexpected fatal error in KUPW$WORKER.LOAD_METADATA

I have recently come across the below error when importing the data using DATA PUMP.

ORA-39126: Worker unexpected fatal error in KUPW$WORKER.LOAD_METADATA [SELECT..
......................................
ORA-39183: internal error -19 ocurred during decompression phase 2


ORA-06512: at "SYS.DBMS_SYS_ERROR", line 95
ORA-06512: at "SYS.KUPW$WORKER", line 9715

----- PL/SQL Call Stack -----
  object      line  object
  handle    number  name

0x64f77ff8     21979  package body SYS.KUPW$WORKER


Cause: Normally users will see ORA-39183 error when the dump got corrupted during the file transfer or user might not be using BINARY mode using FTP

I would recommend taking once more export and copy again, it should work.

Users might come across below known issue

ORA-39126: Worker unexpected fatal error in KUPW$WORKER.PUT_DDLS [PASSWORD_HISTORY:"<USER>"]
ORA-06502: PL/SQL: numeric or value error
LPX-00216: invalid character 0 (0x0)
ORA-06512: at "SYS.DBMS_SYS_ERROR", line 95
ORA-06512: at "SYS.KUPW$WORKER", line 8358

Cause: In this case hashes of the old passwords stored in user_history$ are corrupted.

Workaround: Please follow below steps, if you error doesn’t match please contact Oracle support

1. EXCLUDE=PASSWORD_HISTORY on impdp statment
or
2. Delete the user_history$ information pertaining to that user.
    SQL> delete from user_history$ where user# in (select user# from user$ where name='USER1');

Please refer below metatlink notes before making any change

RA-39126: "Worker unexpected fatal error in KUPW$WORKER.PUT_DDLS [PASSWORD_HISTORY: ...]" (Doc ID 2114832.1)

DataPump Import Errors ORA-39183 Internal Error -19 Ocurred During Decompression Phase 2 ORA-6512 [ID 1524577.1]

Regards
Satishbabu Gunukula, Oracle ACE
http://oracleracexpert.com

Tuesday, April 10, 2018

Oracle Flashback Technology and features

Oracle Flashback Technology is a group of Database features that that will help you to view past states of database objects or to return database objects to a previous state without using point-in-time media recovery.

With flashback features, you can do the following:
· Queries that return past data
· Recover rows or tables to a previous point in time
· Track and archive transactional data changes
· Roll back a transaction and its dependent transactions while the db online

Oracle flashback uses automatic undo management (AUM) for all flashback transactions.

Here are few Oracle flashback features.
· Flashback Database
· Flashback Table
· Flashback drop
· Flashback query
· Flashback Version/Transaction Query
· Flashback Data Archive (From Oracle 11g)
· Flashback Recovery Area:-

Flashback Database:- The FLASHBACK DATABASE is a fast alternative to performing an incomplete recovery. The database must be running in ARCHIVELOG mode, because archived logs are used in the Flashback Database operation. You must have a flash recovery area enabled, because flashback logs can only be stored in the flash recovery area.

To enable logging for Flashback Database, set the DB_FLASHBACK_RETENTION_TARGET initialization parameter and issue the ALTER DATABASE FLASHBACK ON statement.

DB_FLASHBACK_RETENTION_TARGET- length of the desired flashback window in minutes (1440 min)

SQL> ALTER SYSTEM SET DB_FLASHBACK_RETENTION_TARGET=4320; # 3 days

For ex-
FLASHBACK DATABASE TO TIMESTAMP my_date;
FLASHBACK DATABASE TO SCN my_scn;

Limitations:
· It cannot be used to repair media failures, or to recover from accidental deletion of data files.
· You cannot use Flashback Database to undo a shrink data file operation.
· If the database control file is restored from backup or re-created, all accumulated flashback log information is discarded
· Flashback database cannot be used against block corruptions.

Flashback Table The FLASHBACK TABLE used to restore an earlier state of a table in the event of human or application error. The time in the past to which the table can be flashed back is dependent on the amount of undo data in the system and cannot restore a table to an earlier state across any DDL operation.

SQL> FLASHBACK TABLE test TO TIMESTAMP (SYSTIMESTAMP - INTERVAL '1' minute);

Flashback Drop:

The Oracle 10g provides the ability to reinstating an accidentally dropped table from recyclebin.
SQL> FLASHBACK TABLE test TO BEFORE DROP;

Flashback Query The feature allows the DBA to see the value of a column as of a specific time, as long as the before-image copy of the block is available in the undo segment.

For ex:-
SQL> SELECT comments FROM employee AS OF TIMESTAMP TO_TIMESTAMP ('2004-03-29 13:34:12', 'YYYY-MM-DD HH24:MI:SS');

SQL> SELECT comments FROM employee AS OF SCN 722452;

Flashback Version/Transaction Query Flashback Query only provides a fixed snapshot of the data as of a time. Use Flashback Version Query feature to see the changed data between two time points.

Ex: - The following query shows the changes made to the table:

SQL> select versions_starttime, versions_endtime, versions_xid, versions_operation, rate from rates versions between timestamp minvalue and maxvalue order by VERSIONS_STARTTIME

Flashback transaction query can be used to get extra information about the transactions listed by flashback version queries using FLASHBACK_TRANSACTION_QUERY view. The UNDO_SQL column in the table shows the actual SQL Statement.

For Ex:-
SQL> SELECT xid, operation, start_scn, commit_scn, logon_user, undo_sql FROM flashback_transaction_query

Oracle 11g has more enhancements in Flashback Transaction and LogMiner. The LogMiner Viewer has been incorporated into Enterprise Manager and integrated with the new Flashback Transaction feature, making it simple to recover transactions. Flashback Transaction allows the changes made by a transaction to be undone.

Flashback Data Archive (From Oracle 11g) : Flashback data archive allows long-term retention (for ex years) of changed data to a table or set of tables for a user-defined period of time. Flashback Data Archive (which can logically span one or more table spaces) specifies a space quota and retention period for the archive, and then creates the required tables in the archive.

User needs the FLASHBACK ARCHIVE ADMINISTER, FLASHBACK ARCHIVE privileges

SQL> CREATE FLASHBACK ARCHIVE flash_back_archive

TABLESPACE flash_back_archive QUOTA 10G RETENTION 5 YEARS;

Flashback Recovery Area:-
Flash recovery area is a disk location in which the database can store and manage files related to Backup and Recovery. To setup a flash recovery area, you must choose a directory location or Automatic Storage Management disk group to hold the files.

Flash recovery area simplifies the administration of your database by automatically naming recovery-related files, retaining the files as long as they are needed for restore and recovery activities, and deleting the files when they are no longer needed to restore your database and space is needed for some other backup and recovery-related purpose.

To Setup Flash Recovery Area (FRA), you just need to specify below two parameters.
1. DB_RECOVERY_FILE_DEST_SIZE (Specifies max space to use for FRA)

2. DB_RECOVERY_FILE_DEST (Location of FRA)


Performance Guidelines for Oracle Flashback Technology
· For Oracle Flashback Version Query, use index structures.

· Not to scan entire tables and use indexes to query small set of past data. If you need to scan a full table then use parallel hint to the query

· Keep the statistics current as cost-based optimized relies on statistics and use the DBMS_STATS package to generate statistics for tables involved in a Oracle Flashback Query.

· In a Oracle Flashback Transaction Query, the xid column is of the type RAW(8). Use the HEXTORAW conversion function: HEXTORAW(xid)to take advantage of inbuilt function

· The I/O performance cost is mainly paging in the data and undo blocks I the buffer cache. The CPU performance cost is to apply undo.

· A Oracle Flashback Query against a materialized view does not take advantage of query rewrite optimization.

Regards
Satishbabu Gunukula, Oracle ACE

Thursday, March 8, 2018

ORA-31613: Master process DM00 failed during startup

While taking export backup using data pump user received following error.

ORA-31637: cannot create job SYS_EXPORT_SCHEMA_01 for user
ORA-06512: at "SYS.DBMS_SYS_ERROR", line 95
ORA-06512: at "SYS.KUPV$FT", line 1587
ORA-39062: error creating master process DM00
ORA-31613: Master process DM00 failed during startup.

Normally user receives this error when maximum numbers of processes are exceeded. You will see below error in trace files.

ORA-00020: maximum number of processes (150) exceeded

Solution: To fix the issue increase PROCESSES initialization parameter and restart the database.

If you don’t see the error ORA-00020 trace file then look for other error messages which will help to fix the issue. 

I had faced similar issue and  but don't see ORA-00020 error. After looking the other error messages I see that child processes are terminated and in my case restarting the database resolved the issue.

Regards,
Satishbabu Gunukula, Oracle ACE

Friday, May 26, 2017

How to manage Oracle RAC?

Oracle Real Application cluster is a group of independent servers that collaborate as singe system and provides single system image for management and configuration. All tools and utilities provided by oracle to manage system can be used from one system. The clusterware and Database can be installed, configured and managed from single location.

Rolling PatchOracle supports rolling patches, but this requires that each node has a separate Oracle Home. The patches are applied one node at a time while other nodes are operational. Oracle cluster patches are applied in rolling fashion, so no downtime required.

Refer https://docs.oracle.com/cd/B16240_01/doc/em.102/e15294/rac.htm

Rolling UpgradeOracle supports rolling upgrades and individual patches will be rolling upgradable. If any patch modifies the common structures shared between the instances or database will not be quailed for rolling upgrade. Oracle will certify the patches eligible for rolling upgrade and not all the patches are eligible, rolling upgrades are not approved for patch sets. From oracle 10g release1 supports database software upgrade in rolling fashion without any down time by using Data Guard SQL Apply.

Refer: https://docs.oracle.com/cd/B28359_01/install.111/b28263/procstop.htm#CWLIN341

Enterprise Grid ControlOracle Enterprise Grid Control is a GUI management tool provided by oracle to eliminate tedious work and manage your cluster environment more efficiently. This tool provides the centralized management of cluster databases, using this tool you can view system status, view alerts and set thresholds for alert generation, backup and recovery, monitor cluster wait events, monitor performance metrics across all database instances, and you can able to perform all maintenance activities.

Refer : https://docs.oracle.com/cd/E11857_01/em.111/e11982/overview.htm

Scalability
Oracle Real Application Cluster provides scalability for all your enterprise business applications. Oracle provides a wide array of tools and techniques for scaling, and you can use these tools to ensure seamless growth while minimizing the investment in hardware resources. You can allow RAC database to grow seamlessly from a small system to a big multinational enterprise applications.

Refer : https://docs.oracle.com/cd/B10501_01/rac.920/a96597/psscadtl.htm

Oracle RAC architecture automatically accommodates rapidly changing business requirements, adding a server to the cluster does not require an outage and as soon as the new Instance added the application can take advantage of extra resources. You need to make sure that all servers in the cluster must run the same OS and same Oracle version, but they do not have to be the exactly same capacity. Oracle automatically balances the user load among the multiple nodes in the cluster.

High Availability
Oracle Real application cluster provides the following important features for a High available data management.

Reliability – In Oracle real application cluster if an instance fails, the remaining instances in the server remain active and open for users. Oracle cluster monitors all oracle processes and immediately restarts any failed component.

Recoverability - If an instance fails in Oracle RAC database, it is recognized by other instances in the server pool and recovery will start automatically. Fast connection Failover (FCF) and Fast application notification (FAN) makes it easy for applications to mask component failures from the user.

Continuous Operations – Oracle Real application cluster provides continuous service for both unplanned and planned outages. If a server or instances fails, the database remains open and the application/users are able to access data from other surviving instances. Most of the database maintenance operations can be performed without downtime and many other maintenance tasks can be done in a rolling fashion so application downtime is minimized.

Error Detection - Oracle Cluster automatically monitors oracle databases and other oracle process (ASM instances, Listener, ONS…etc) and provides fast detection of problems in the RAC environment. It also automatically recovers from failures often before users noticed that a failure has occurred.
Scalability - Oracle Real Application Cluster provides scalability for all your enterprise business applications. Oracle provides a wide array of tools and techniques for scaling, and you can use these tools to ensure seamless growth while minimizing the investment in hardware resources. You can allow RAC database to grow seamlessly from a small system to a big multinational enterprise applications.

High Availability- Oracle Real application cluster provides Reliability, Recoverability, Continuous Operations, Error Detection features for a High available data management. If an instance fails in Oracle RAC database, the Cluster detects the problems immediately and recovery will start automatically. The remaining instances in the server remain active and open for uses.

Maintenance - In Oracle RAC, most of the database maintenance operations can be performed without downtime and many other maintenance tasks can be done in a rolling fashion (Rolling Patch, Rolling Upgrade) so application downtime is minimized

Regards
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com

Thursday, July 23, 2015

Why Virtual Private Catalog?

Hi Everyone,

My articles published in Oracle Experts website media "Allthingsoracle.com" by RedGate

Please see the below link.

Article: Why Virtual Private Catalog?

This article will help you to understand

  • What is Virtual Private Catalog
  • How to secure the boundaries between DBA's
  • How to segregate the duties in an Enterprise Environment.

I hope you will like the article and it will be helpful to you.

Please leave your valuable comments.

Regards,
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com

ORA-01163: SIZE clause indicates (blocks), but should match header

I have generated a control file using “backup control file to trace” but while recreating I have encountered below error

*
ERROR at line 1:
ORA-01503: CREATE CONTROLFILE failed
ORA-01163: SIZE clause indicates 32768 (blocks), but should match header 262144
ORA-01517: log member: '/oracle/TEST/redo1/TEST_redo1.rdo'


I see that the failure is due to log member size. But the current redo log file size on the disk and in the script is correct. I ran below query to find the redo log BLOCKSIZE and found that the size is 4096.

select * from v$log;

I tried couple of options to recreate the control file by changing the redo log size in KB/MB/Bytes..etc but nothing worked. After investigation I found that drop and recreating the redo log is only the option.

Other possible scenarios/errors
1. If you receive the same ORA-01163 error for DATA FILE then your size calculation might be incorrect.

    Use below formula to get the size.
    Expected size = Expected no of blocks * db_block_size / 1024

2. ORA-01378: The logical block size (4096) of file /oracle/TEST/redo1/TEST_redo1.rdo'is not
    compatible with the disk sector size (media sector size is 512 and host sector size is 512)

If you receive above error you ned to set below parameter
SQL> alter system set "_disk_sector_size_override"=TRUE scope=both;

Reference for more info.
ORA-00344: WHEN RESIZING YOUR REDO LOGS (Doc ID 1018307.102)

Regards
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com

Thursday, May 7, 2015

Terminating the instance due to error 16038 and unable to startup

I have encountered storage crash on one of the Database system. After storage is back online, not able to bring up the database. The database comes to mount stage and terminating the instance.

We are receiving below error.

ORA-03113: end-of-file on communication channel
Process ID: 5038
Session ID: 202 Serial number: 3


When I look into the alert.log I found below error messages

ARC2: Archival started
ARC1: Becoming the 'no FAL' ARCH
ARC2: Becoming the heartbeat ARCH
ARC3: Archival started
ARC0: STARTING ARCH PROCESSES COMPLETE
Tue Mar 10 10:24:10 2015
Incomplete read from log member '/oracle/ORAC/datafile/ORAC/onlinelog/ORAC_Redo1_.log'. Trying next member.
ARCH: All Archive destinations made inactive due to error 333
ARCH: Closing local archive destination LOG_ARCHIVE_DEST_1: '/oracle/ORAC/archiveORAC_ARCH_347548987.dbf' (error 333) (ORAC)
Committing creation of archivelog '/oracle/ORAC/archive/1_3541_854889367.dbf' (error 333)
Errors in file /oracle/diag/rdbms/orac/ORAC/trace/ORAC_ora_23498.trc:
ORA-16038: log 1 sequence# 3541 cannot be archived
ORA-00333: redo log read error block count
ORA-00312: online log 1 thread 1: '/oracle/ORAC/datafile/ORAC/onlinelog/ORAC_Redo1_.log'

USER (ospid: 23498): terminating the instance due to error 16038
Tue Mar 10 10:24:12 2015
System state dump requested by (instance=1, osid=23498), summary=[abnormal instance termination].
System State dumped to trace file /oracle/diag/rdbms/orac/ORAC/trace/ORAC_diag_23470.trc
Dumping diagnostic data in directory=[cdmp_20150310102412], requested by (instance=1, osid=23498), summary=[abnomal instance termination].
Instance terminated by USER, pid = 23498


By looking the error, I can see that one of the online redo is corrupted. Archive process cannot able to archive to disk. In order to bring up the database you need to clear the corrupted redo log.

From ALERT.LOG file, I am able to identify that group 1 redo log 1 corrupted. By using below command I have cleared the unarchived log file

SQL> alter database clear unarchived logfile ‘/oracle/ORAC/datafile/ORAC/onlinelog/ORAC_Redo1_.log'

If you find all multiplexed redo members are corrupted then you need to run below command to clear the GROUP.

SQL> alter database clear unarchived logfile group 1;

Now I am able to bring up the database. But note that you cannot able to bring the database back beyond corrupted point using old backups. Now you must take one FULL database backup.

Regards
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com
http://www.sqlserver-expert.com

ORA-16038: log 1 sequence# 3541 cannot be archived

I have encountered storage crash on one of the Database system. After storage is back online, not able to bring up the database. The database comes to mount stage, but going down.

We are receiving below error.

ORA-03113: end-of-file on communication channel
Process ID: 5038
Session ID: 202 Serial number: 3


When I look into the alert.log I found below error messages

ARC2: Archival started
ARC1: Becoming the 'no FAL' ARCH
ARC2: Becoming the heartbeat ARCH
ARC3: Archival started
ARC0: STARTING ARCH PROCESSES COMPLETE
Tue Mar 10 10:24:10 2015
Incomplete read from log member '/oracle/ORAC/datafile/ORAC/onlinelog/ORAC_Redo1_.log'. Trying next member.
ARCH: All Archive destinations made inactive due to error 333
ARCH: Closing local archive destination LOG_ARCHIVE_DEST_1: '/oracle/ORAC/archiveORAC_ARCH_347548987.dbf' (error 333) (ORAC)
Committing creation of archivelog '/oracle/ORAC/archive/1_3541_854889367.dbf' (error 333)
Errors in file /oracle/diag/rdbms/orac/ORAC/trace/ORAC_ora_23498.trc:
ORA-16038: log 1 sequence# 3541 cannot be archived
ORA-00333: redo log read error block count
ORA-00312: online log 1 thread 1: '/oracle/ORAC/datafile/ORAC/onlinelog/ORAC_Redo1_.log'

USER (ospid: 23498): terminating the instance due to error 16038
Tue Mar 10 10:24:12 2015
System state dump requested by (instance=1, osid=23498), summary=[abnormal instance termination].
System State dumped to trace file /oracle/diag/rdbms/orac/ORAC/trace/ORAC_diag_23470.trc
Dumping diagnostic data in directory=[cdmp_20150310102412], requested by (instance=1, osid=23498), summary=[abnomal instance termination].
Instance terminated by USER, pid = 23498


By looking the error, I can see that one of the online redo is corrupted. Archive process cannot able to archive to disk. In order to bring up the database you need to clear the corrupted redo log.

From ALERT.LOG file, I am able to identify that group 1 redo log 1 corrupted. By using below command I have cleared the unarchived log file

SQL> alter database clear unarchived logfile ‘/oracle/ORAC/datafile/ORAC/onlinelog/ORAC_Redo1_.log'

If you find all multiplexed redo members are corrupted then you need to run below command to clear the GROUP.

SQL> alter database clear unarchived logfile group 1;

Now I am able to bring up the database. But note that you cannot able to bring the database back beyond corrupted point using old backups. Now you must take one FULL database backup.

Regards
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com
http://www.sqlserver-expert.com