Oracle AutoUpgrade with source and Target Database on different servers

AutoUpgrade utility can be used on upgrading databases from 12c R2 release (12.2 + DBJAN2019RU and newer) and it has Analyze, Fixup, Deploy and Upgrade processing modes. 

Before you proceed make sure you meet following requirements.

• create config file
• Source Database should be running in original oracle home
• The Database server should be registered in DNS
• JAVA8 required for Autoupgrade to run. Oracle 12c R2 (12.2.0.1) or newer oracle homes have a java version by default.
• If you run AutoUpgrade in batch or script mode use “noconsole” parameter

From Oracle 19c (10.3) autoupgrade.jar file exists by default, for earlier releases you must download latest autoupgrade.jar file from Oracle support.

There are four AutoUpgrade modes and each mode performs different steps

•  Analyze Mode: Setup, Pre-checks.
• Fixups Mode: Setup, Pre-checks, and Pre-fixups.
• Deploy Mode: Setup, Guaranteed Restore Point, Pre-upgrade, Pre-checks, Pre-fixups, Drain,      Database Upgrade, Post-checks, Post-fixups, and Post-upgrade.
• Upgrade Mode: Setup, Database Upgrade, Post-checks, and Post-fixups.

Analyze Mode: This mode runs few checks to see weather database is ready for upgrade. In this mode it only reads data from database to analyze and does not perform any changes.

You can run this step on source database before you setup target.

Run below command to start AutoUpgrade Analyze

$ORACLE_HOME/jdk/bin/java -jar $ORACLE_HOME/rdbms/admin/autoupgrade.jar – config /home/oracle/autoupgrade/config.txt -mode analyze

Make sure you analyze output files for errors and run fixup on Source database server. You should see SID.html and SID_preupgrade.log files

Note that this mode also generates status files such as status.json, progress.json which can be located under cfgtoollogs/upgrade/auto/status.

Status.json – This file contains High level status of upgrade.
Progress.json – This file has contains status of progress of all upgrades

In case if target database is not on the same server then you must set the source home path in the configuration file, so that Autoupgrade analyze can run on source oracle home.

For example,
upgrade1.source_home=/home/oracle/product/12.2.0.1/dbhome_1 # Source ORACLE_HOME Path
upgrade1.target_home=/home/oracle/product/19.0.0.0/dbhome_1 # Target ORACLE_HOME Path

Fixups Mode : This mode performs all required automated fix up that are required before you start an upgrade. In this mode it creates guaranteed restore point.

Run below command on source Database server to run fixups.

$ORACLE_HOME/jdk/bin/java -jar $ORACLE_HOME/rdbms/admin/autoupgrade.jar – config /home/oracle/autoupgrade/config.txt -mode fixups

Deploy Mode: This mode performs all required actions for an upgrade, which include analyze, fixups, upgrade and post upgrade steps. You will use this mode when source and target Oracle Home’s are on the same server. This mode also creates guaranteed restore point.

Run below command on source Database server to run fixups.

$ORACLE_HOME/jdk/bin/java -jar $ORACLE_HOME/rdbms/admin/autoupgrade.jar – config /home/oracle/autoupgrade/config.txt -mode Deploy

Move the database from source to Target Database server and perform Upgrade mode.

Upgrade Mode: You will run this Autoupgrade mode when you are moving database to Target server or don’t have source Oracle home access. This mode is used only when you are moving to new server. But note that this mode doesn’t create guaranteed restore point and it doesn’t perform the post upgrade steps.

Before you perform upgrade make sure you copy the during_upgrade_pfile_dbname.ora from source to target Oracle Home with default name init<SID>.ora. You can also create the spfile using during_upgrade_pfile_dbname.ora which can be found under temp directory.

SQL> create spfile from pfile=' /home/oracle/autoupgrade/au21/TESTDB/temp/during_upgrade_pfile_testdb.ora';

On Target database server start the database in upgrade mode and run below command to upgrade

$ORACLE_HOME/jdk/bin/java -jar $ORACLE_HOME/rdbms/admin/autoupgrade.jar – config /home/oracle/autoupgrade/config.txt -mode upgrade

After upgrade make sure you perform post upgrade steps manually

· Copy network files sql.net ora, listener,ora, tnsnames.ora files..etc
· Restart of database in case of RAC
· Remove guaranteed restore point

Thanks & Regards,
https://oracleracexpert.com, Oracle ACE

DBMS_DST Package updating Database Time Zone File

Users normally see below message in alert.log file when upgrading the Database. You can use DBMS_DST package to upgrade the time zone file.

Database is using a timezone file older than version xx

The timezone has 2 file i.e large file and small file and these are located under oracore/zoneinfo under ORACLE HOME directory.

• The large versions are designated as timezlrg_version_number.dat., this file has all the time zones defined in the database

• The small versions are designated as timezone_version_number.dat, this file has all most commonly used time zones

You can run query V$TIMEZONE_FILE to identify time zone file version used by the database.

SQL> select filename, version, from v$timezone_file;
FILENAME VERSION
-------------------- ----------
timezlrg_26.dat 26

You can get the primary and secondary time zone versions details from DATABASE_PROPERTIES

SQL> column property_name format a40

SQL>column property_value format a30

SQL>select property_name, property_value from database_properties where property_name like 'DST_%' order by property_name;

PROPERTY_NAME PROPERTY_VALUE
---------------------------------------- ------------------------------
DST_PRIMARY_TT_VERSION 26
DST_SECONDARY_TT_VERSION 0
DST_UPGRADE_STATE NONE

After upgrade you can check upg_summary.log to identify Database time zone version and current release time zone version

Oracle Database Release 19 Post-Upgrade Status Tool 04-30-2021 11:07:0
Database Name: TDB1
Component Current Full Elapsed Time
Name Status Version HH:MM:SS
Oracle Server UPGRADED 19.10.0.0.0 00:09:54
JServer JAVA Virtual Machine VALID 19.10.0.0.0 00:01:12
Oracle XDK UPGRADED 19.10.0.0.0 00:00:38
Oracle Database Java Packages UPGRADED 19.10.0.0.0 00:00:05
Oracle Text UPGRADED 19.10.0.0.0 00:00:23
Oracle Workspace Manager UPGRADED 19.10.0.0.0 00:00:22
Oracle Real Application Clusters OPTION OFF 19.10.0.0.0 00:00:00
Oracle XML Database UPGRADED 19.10.0.0.0 00:00:51
Oracle Multimedia UPGRADED 19.10.0.0.0 00:01:47
Datapatch 00:01:41
Final Actions 00:01:44
Post Upgrade 00:00:16

Total Upgrade Time: 00:17:31

Database time zone version is 26. It is older than current release time
zone version 32. Time zone upgrade is needed using the DBMS_DST package.

Grand Total Upgrade Time: [0d:0h:20m:20s]

You can also get latest time zone version using below query

SQL> SELECT DBMS_DST.get_latest_timezone_version FROM dual;
GET_LATEST_TIMEZONE_VERSION
---------------------------
32

To upgrade time zone follow below steps

1. Shutdown the database and startup in upgrade mode

SQL> Shutdown immediate
SQL> Startup upgrade

2. Prepare the time zone version upgrade using BEGIN_PREPARE procedure

SQL> set serveroutput on

SQL> declare

l_tz_version pls_integer;

begin

l_tz_version := dbms_dst.get_latest_timezone_version;

dbms_output.put_line('l_tz_version=' || l_tz_version);

DBMS_DST.BEGIN_PREPARE(l_tz_version);

end;

/

PL/SQL procedure successfully completed.

The below query will show the upgrade version we are attempting to

SQL> column property_name format a40

SQL>column property_value format a30

SQL>select property_name, property_value from database_properties where property_name like 'DST_%' order by property_name;

PROPERTY_NAME PROPERTY_VALUE

---------------------------------------- ------------------------------

DST_PRIMARY_TT_VERSION 26

DST_SECONDARY_TT_VERSION 32

DST_UPGRADE_STATE PREPARE

3. You can find the affected tables by running below queries

SQL> exec DBMS_DST.find_affected_tables;
PL/SQL procedure successfully completed.

SQL> select count(*) from sys.dst$affected_tables;
COUNT(*)
----------
0
SQL> select * from sys.dst$error_table;
no rows selected

4. Use END_PREPARE procedure to end the prepare stage and begin upgrade using BEGIN_UPGRADE procedure

SQL> EXEC DBMS_DST.END_PREPARE;
PL/SQL procedure successfully completed.

SQL> set serveroutput on

SQL> declare

l_tz_version pls_integer;

begin

l_tz_version := dbms_dst.get_latest_timezone_version;

dbms_output.put_line('l_tz_version=' || l_tz_version);

DBMS_DST.BEGIN_UPGRADE(l_tz_version);

end;

/

l_tz_version=32
An upgrade window has been successfully started.

PL/SQL procedure successfully completed.

5. Open in normal mode and upgrade database time zone file using DBMS_DST.UPGRADE_DATABASE

SQL> shutdown immediate
SQL> startup

sql> set serveroutput on

sql> declare

l_failures pls_integer;

begin

DBMS_DST.UPGRADE_DATABASE(l_failures);

dbms_output.put_line('dbms_dst.upgrade_database : l_failures=' || l_failures);

DBMS_DST.END_UPGRADE(l_failures);

dbms_output.put_line('dbms_dst.end_upgrade : l_failures=' || l_failures);

end;

/

Table list: "GSMADMIN_INTERNAL"."AQ$_CHANGE_LOG_QUEUE_TABLE_S"
Number of failures: 0
Table list: "GSMADMIN_INTERNAL"."AQ$_CHANGE_LOG_QUEUE_TABLE_L"
Number of failures: 0
Table list: "MDSYS"."SDO_DIAG_MESSAGES_TABLE"
Number of failures: 0
Table list: "DVSYS"."SIMULATION_LOG$"
Number of failures: 0
Table list: "DVSYS"."AUDIT_TRAIL$"
Number of failures: 0
DBMS_DST.upgrade_database : l_failures=0
An upgrade window has been successfully ended.
DBMS_DST.end_upgrade : l_failures=0

PL/SQL procedure successfully completed.

Verify the time zone upgrade by running below query

SQL> column property_name format a40

SQL>column property_value format a30

SQL>select property_name, property_value from database_properties where property_name like 'DST_%' order by property_name;

PROPERTY_NAME PROPERTY_VALUE

---------------------------------------- ------------------------------

DST_PRIMARY_TT_VERSION 32

DST_SECONDARY_TT_VERSION 0

DST_UPGRADE_STATE NONE

Thanks & Regards,

https://oracleracexpert.com, Oracle ACE

Oracle RAC and Grid new features in 19c

Oracle Database 19c has many exciting new features and in order to take advantage of these features you need to upgrade the databases from older versions to Oracle 19c

Join the Webinar to learn New Features in Oracle 19c RAC and Grid

Date and time: Nov 25th 2020 8:00am-9:00am
Pacific Daylight Time (San Francisco, GMT-07:00)


To register for this Webinar, please send an email to SatishbabuGunukula@gmail.com. Note that registrations are limited, first come and first serve basis only. You will receive an email confirmation with meeting session link.

For Presentation, link "Click here"

Thanks & Regards
http://www.oracleracexpert.com

ORA-00060: deadlock resolved & ORA-12012: error on auto execute of job

We have recently upgraded a dataset to 12c and started receiving below errors in alert.log

ORA-00060: deadlock resolved; details in file /home/oracle/diag/rdbms/ORCL/ORCL/trace/ORCL_j001_200521.trc
ORA-12012: error on auto execute of job "SYS"."ORA$AT_SA_SPC_SY_5723"
ORA-00060: deadlock detected while waiting for resource < ORA-06512: at "SYS.DBMS_SPACE", line 2741 < ORA-06512: at "SYS.DBMS_HEAT_MAP_INTERNAL", line 716 < ORA-06512: at "SYS.DBMS_HEAT_MAP_INTERNAL", line 1164 < ORA-06512: at "SYS.DBMS_HEAT_MAP", line 228 < ORA-06512: at "SYS.DBMS_SPACE", line 2747

After investigation found that there is a bug in 12.2 Release and Oracle working to it. You can refer below oracle support notes on this.

Bug 24687075 - SPACE ADVISOR TASKS/JOBS HITTING DEADLOCKS WITH GATHER DB STATS JOBS
In 12.2 Auto Space Advisor Job Fails With ORA-60 (Doc ID 2321020.1)

If you have setup alerts for ORA- errors then either you add exclusions to your script or you can run below command to disable the job Until Bug 24687075 gets fixed

SQL> EXEC DBMS_AUTO_TASK_ADMIN.DISABLE('AUTO SPACE ADVISOR',NULL, NULL);

Regards

Satishbabu Gunukula, Oracle ACE

https://oracleracexpert.com

How to manage Oracle RAC?

Oracle Real Application cluster is a group of independent servers that collaborate as singe system and provides single system image for management and configuration. All tools and utilities provided by oracle to manage system can be used from one system. The clusterware and Database can be installed, configured and managed from single location.

Rolling PatchOracle supports rolling patches, but this requires that each node has a separate Oracle Home. The patches are applied one node at a time while other nodes are operational. Oracle cluster patches are applied in rolling fashion, so no downtime required.

Refer https://docs.oracle.com/cd/B16240_01/doc/em.102/e15294/rac.htm’

Rolling UpgradeOracle supports rolling upgrades and individual patches will be rolling upgradable. If any patch modifies the common structures shared between the instances or database will not be quailed for rolling upgrade. Oracle will certify the patches eligible for rolling upgrade and not all the patches are eligible, rolling upgrades are not approved for patch sets. From oracle 10g release1 supports database software upgrade in rolling fashion without any down time by using Data Guard SQL Apply.

Refer: https://docs.oracle.com/cd/B28359_01/install.111/b28263/procstop.htm#CWLIN341

Enterprise Grid ControlOracle Enterprise Grid Control is a GUI management tool provided by oracle to eliminate tedious work and manage your cluster environment more efficiently. This tool provides the centralized management of cluster databases, using this tool you can view system status, view alerts and set thresholds for alert generation, backup and recovery, monitor cluster wait events, monitor performance metrics across all database instances, and you can able to perform all maintenance activities.

Refer : https://docs.oracle.com/cd/E11857_01/em.111/e11982/overview.htm

Scalability

Oracle Real Application Cluster provides scalability for all your enterprise business applications. Oracle provides a wide array of tools and techniques for scaling, and you can use these tools to ensure seamless growth while minimizing the investment in hardware resources. You can allow RAC database to grow seamlessly from a small system to a big multinational enterprise applications.

Refer : https://docs.oracle.com/cd/B10501_01/rac.920/a96597/psscadtl.htm

Oracle RAC architecture automatically accommodates rapidly changing business requirements, adding a server to the cluster does not require an outage and as soon as the new Instance added the application can take advantage of extra resources. You need to make sure that all servers in the cluster must run the same OS and same Oracle version, but they do not have to be the exactly same capacity. Oracle automatically balances the user load among the multiple nodes in the cluster.

High Availability

Oracle Real application cluster provides the following important features for a High available data management.

Reliability – In Oracle real application cluster if an instance fails, the remaining instances in the server remain active and open for users. Oracle cluster monitors all oracle processes and immediately restarts any failed component.

Recoverability - If an instance fails in Oracle RAC database, it is recognized by other instances in the server pool and recovery will start automatically. Fast connection Failover (FCF) and Fast application notification (FAN) makes it easy for applications to mask component failures from the user.

Continuous Operations – Oracle Real application cluster provides continuous service for both unplanned and planned outages. If a server or instances fails, the database remains open and the application/users are able to access data from other surviving instances. Most of the database maintenance operations can be performed without downtime and many other maintenance tasks can be done in a rolling fashion so application downtime is minimized.

Error Detection - Oracle Cluster automatically monitors oracle databases and other oracle process (ASM instances, Listener, ONS…etc) and provides fast detection of problems in the RAC environment. It also automatically recovers from failures often before users noticed that a failure has occurred.

Refer: https://docs.oracle.com/cd/B28359_01/server.111/b28281/architectures.htm#g1012375

Benefits of using Oracle RAC:-

Scalability - Oracle Real Application Cluster provides scalability for all your enterprise business applications. Oracle provides a wide array of tools and techniques for scaling, and you can use these tools to ensure seamless growth while minimizing the investment in hardware resources. You can allow RAC database to grow seamlessly from a small system to a big multinational enterprise applications.

High Availability- Oracle Real application cluster provides Reliability, Recoverability, Continuous Operations, Error Detection features for a High available data management. If an instance fails in Oracle RAC database, the Cluster detects the problems immediately and recovery will start automatically. The remaining instances in the server remain active and open for uses.

Maintenance - In Oracle RAC, most of the database maintenance operations can be performed without downtime and many other maintenance tasks can be done in a rolling fashion (Rolling Patch, Rolling Upgrade) so application downtime is minimized

Regards
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com

error in invoking target 'agent nmhs' of makefile

I come across a situation that servers & Operating systems are (O/S) are getting upgraded and we cannot perform in place upgrade for easy roll backing capability.

Also new O/S does not support the Database version.

We need to restore the Oracle database into unsupported Server version. We have copied the Database binaries and during the Database upgrade we got below error.

error in invoking target 'agent nmhs' of makefile

Workaround:
During upgrade or clone  you will get this error when relink is running

Here is the workaround 
ls $ORACLE_HOME/sysman/lib/ins_emagent.mk

Search for the line 
$(MK_EMAGENT_NMECTL)
Change it to:
$(MK_EMAGENT_NMECTL) -lnnz11

https://community.oracle.com/thread/1093616

Goto Database upgrade window and click on "Retry" and upgrade should run successfully.

Regards
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com

Failed to get connections for connection [http://hostname:8000] using session locale [English]

Failed to get connections for connection [http://hostname:8000] using session locale [English].

I was trying to create a OLAP connection from CMC to HANA in BOXI4.1 SP6 for Analysis Office 2.x SSO setup and receive below error. All the server details verified and everything seems to be correct.

After doing some investigation I found that the issue is due to limitation in BI platform4.1 sp06 or lower.

It cannot able to connect to HANA in CMC because there is no data provider available.

Errors in MDAS log file
Creating a BICS application with alias [session-2:http://hostname:8000:dev:en_US:HTTP:CREDENTIALS:null]
JCoDestinationDataProvider adding destination session-2:http://hostname:8000:dev:en_US:HTTP:CREDENTIALS:null.
JCoDestinationDataProvider setting jco.destination.pool_capacity property to 0 because it was not specified. <exception> com.sap.ip.bi.base.service.connection.impl.ConnectionException: No connection factory found for type session-2:http://hostname:8000:dev:en_US:HTTP:CREDENTIALS:null/HTTP.
Possible Solution: Upgrade to BI 4.1 SP07 or higher then you should be able to connect using HTTP provider.

Refer below SAP Note for more info
2182585 - Cannot connect to the Hana via SAP Hana Http OLAP connection in CMC

Regards

Satishbabu Gunukula, Oracle ACE

http://www.oracleracexpert.com