Showing posts with label Trace File. Show all posts
Showing posts with label Trace File. Show all posts

Wednesday, February 7, 2024

SQL Firewall in Oracle 23c

SQL Firewall inspects all incoming statements and ensures only authorized SQL is run and it is embedded in the Oracle Database and unauthorized SQL Statements will be logged and blocked.

SQL firewall provides real-time protection from attacks and mitigate risks from SQL injection attacks, anomalous access, credential abuse or theft. SQL Firewall supports all commands except transaction control commands such as SAVEPOINT, COMMIT, ROLLBACK.

To administer SQL Firewall user must have SQL_FIREWALL_ADMIN role. To query DBA_SQL_FIREWALL* data dictionary the user must have SQL_FIREWALL_VIEWER role

You can Configure SQL Firewall using DBMS_SQL_FIREWALL package or Oracle Data Safe. SQL Firewall can be used in both root and Pluggable Database (PDB)


You can enable SQL Firewall using below command.
SQL> EXEC DBMS_SQL_FIREWALL.ENABLE;

Create and enable SQL Firewall capture for a user using below command

SQL> BEGIN
DBMS_SQL_FIREWALL.CAPTURE_CAPTURE (
Username => ‘SCOTT’
top_level_only => TRUE,
Start_capture => TRUE
);
END;


Enable SQL Firewall Allow List

SQL>BEGIN
DBMS_SQL_FIREWALL.ENABLE_ALLOW_LIST (
username => ‘SCOTT’,
enforce => DBMS_SQL_FIREWALL.ENFORCE_SQL,
block => TRUE );
END;

You can use below commands to START and STOP capture.

SQL> EXEC DBMS_SQL_FIREWALL.START_CAPTURE (‘SCOTT’);
SQL> EXEC DBMS_SQL_FIREWALL.STOP_CAPTURE (‘SCOTT’);

You can generate an allow list using below procedures.

DBMS_SQL_FIREWALL.ADD_ALLOWED_CONTEXT
DBMS_SQL_FIREWALL.DELETE_ALLOWED_CONTEXT
DBMS_SQL_FIREWALL.DELETE_ALLOWED_SQL

The SQL Firewall may generate large volume of capture logs and to minimize performance impact the database memory needs to be sized to handle the load. It is advised to add additional 2GB to LARGE_POOL_SIZE parameter and also it advised to have SGA_TARGET to 8GB or more.

To purge logs you can use below procedure

BEGIN
DBMS_SQL_FIREWALL.PURGE_LOG (
username => ‘SCOTT’,
purge_time => '2024-01-10 12:00:00.00 -08:00',
log_type => 'DBMS_SQL_FIREWALL.ALL_LOGS'
);
END;
/

You can also enable and disable SQL Firewall Trace using below commands. The trace level value should be LOW, HIGH, HIGHEST based upon how much detail tracing you want to have.

-Session level Tracing Enable and Disable
ALTER SESSION SET EVENTS 'TRACE SQL_FIREWALL DISK=trace_level
ALTER SESSION SET EVENTS 'TRACE SQL_FIREWALL OFF

-System Level Tracing Enable and Disable
ALTER SYSTEM SET EVENTS 'TRACE SQL_FIREWALL DISK=trace_level
ALTER SYSTEM SET EVENTS 'TRACE SQL_FIREWALL OFF


You can query below Data Dictionary Views for SQL Firewall protections
DBA_SQL_FIREWALL_ALLOWED_SQL - View shows allowed SQL and Accessed objects
DBA_SQL_FIREWALL_ALLOWED_IP_ADDR – View shows the Users allowed IP address
DBA_SQL_FIREWALL_CAPTURE_LOGS – View shows the Capture log entries
DBA_SQL_FIREWALL_VIOLATIONS – View shows the SQL Firewall Violations

Thanks & Regards
https://oracleracexpert.com
Oracle ACE Pro


Monday, October 16, 2023

WARNING: inbound connection timed out (ORA-3136)

The ORA-3136 will be written into alert.log when users fails to provide credentials and cannot authenticate within the set timeout value.

The default value set for below parameters in sqlnet.ora is 60 seconds.

SQLNET.INBOUND_CONNECT_TIMEOUT
INBOUND_CONNECT_TIMEOUT_listener_name

You will see below error in the alert.log file

Fatal NI connect error 12170.
VERSION INFORMATION:
TNS for Linux: Version 19.0.0.0.0 - Production
Oracle Bequeath NT Protocol Adapter for Linux: Version 19.0.0.0.0 - Production
TCP/IP NT Protocol Adapter for Linux: Version 19.0.0.0.0 - Production
Version 19.5.0.0.0
Time: 14-OCT-2023 18:06:02
Tracing not turned on.
Tns error struct:
ns main err code: 12535

TNS-12535: TNS:operation timed out
ns secondary err code: 12606
nt main err code: 0
nt secondary err code: 0
nt OS err code: 0
Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=xxx.xx.xxx.xx)(PORT=51290))
2023-10-12T16:06:02.222415-07:00
WARNING: inbound connection timed out (ORA-3136)
2023-10-12T16:15:18.866821-07:00


In the above log you can see the client IP and port, you can able to check corresponding entry in the listener log as well. Also, you can see the time stamp of user initiated the connection and time stamp when user got error, it will be more than 60 sec.

The main reason for the error is …

1. When user or application trying to connect using wrong credentials or no attend made under 1 min threshold default value of the instance.

You can able to reproduce the issue by entering wrong credentials
 
SQL> sqlplus HR/xxxx@orcl
SQL*Plus: Release 19.0.0.0.0 - Production on Fri Oct 13 19:31:30 2023
Version 19.18.0.0.0
Copyright (c) 1982, 2022, Oracle. All rights reserved.

ERROR:
ORA-01017: invalid username/password; logon denied

$ tail -f orcl_alert.log
Fri Oct 13 19:32:31 2023
WARNING: inbound connection timed out (ORA-3136)

2. If the database has some performance issues or any network delay can cause long time and cannot authenticate user within default time out i.e. 60 seconds

If the database has some load causing the slowness, then increasing SQLNET.INBOUND_CONNECT_TIMEOUT to higher value will help to reduce the errors.

No DB restart required when you make the change in sqlnet.ora but note that it will be applicable to next server process.

3. The server receives the request but cannot be able to authenticate with in default time out i.e. 60 seconds

Find out what causing the delay in authentication if required increase the time out value.

If you are receiving error frequently you can enable tracing using below command
SQL> alter system set events '3136 trace name errorstack level 3';
 
You can trun off tracing using
SQL> alter system set events '3136 trace name context off';

Note that tracing will be generated under USER_DUMP_DEST or BACKGROUND_DUMP_DEST

If you still seeing these warnings I would suggest to raise a ticket with oracle support

Thanks,
https://oracleracexpert.com

Wednesday, April 14, 2021

ORA-20011: Approximate NDV failed: ORA-08103: object no longer exists

The ORA-20011, can be found in alert.log and this error occurs when DBMS_STATS:GATHER_STATS_JOB gathering stats

ORA-20011: Approximate NDV failed: ORA-08103: object no longer exists

You identify the issue set the following trace events
SQL> alter session set events '10384 trace name context forever , level 16384';

After research found that object truncate and starts are gathering at same time. In case of partition set granularity to ‘PARTITION’

DB level
SQL> exec dbms_stats.set_global_prefs(pname=>'GRANULARITY',pvalue=>'PARTITION')

Table level
SQL> exec dbms_stats.set_table_prefs(ownname=>'Owner_Name',tabname=>'<Table_Name>',pname=>'GRANULARITY',pvalue=>'PARTITION')

Run the gather stats after DDL or DML have been completed.

Also note that “ORA-08103 object no longer exists” error occurs when another user has deleted object since the operation began. In case of incomplete recovery of the object, the work around would be to remove references to the object or delete the object.

When using Global temporary table with ON COMMIT DELETE ROWS options, may encounter ORA-08103: object no longer exists when commit statement that followed right after the delete statement.

In this case recreation of global temporary table with ON COMMIT PRESERVE ROWS clause will helps to safely fetch data

There are several bugs related to ORA-8103 error, check oracle support for more details.

Bug 13618170
Bug 5523799
Bug 5637976

Thanks & Regards,
http://oracleracexpert.com, Oracle ACE

Monday, August 3, 2020

ORA-00700: soft internal error, arguments

I have come across ORA-00700 error recently and found below error in the logs. This

ORA-00700: soft internal error, arguments: [dbgrfafr_1], [60], [60], [1], [0x7FB5DD11BC70], [], [], [], [], [], [], []
Incident details in: /oracle/diag/rdbms/orcl/ORCL/incident/incdir_75765/ORCL_ora_8274_i75765.trc


Incident details in the trace will have more detailed error. In my case below connections are coming from foglight monitoring tool . After research found that this error can occur due failing statement called from monitoring tools like TOAD,foglight..etc

Dump file /oracle/diag/rdbms/orcl/ORCL/incident/incdir_75765/ORCL_ora_8274_i75765.trc
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.5.0.0.0
Build label: RDBMS_19.3.0.0.0DBRU_LINUX.X64_190417
ORACLE_HOME: /oracle/product/19.0.0.0/dbhome_1
System name: Linux
Node name: server1
Release: 3.10.0-1062.12.1.el7.x86_64
Version: #1 SMP Thu Dec 12 06:44:49 EST 2019
Machine: x86_64
Instance name: ORCL
Redo thread mounted by this instance: 1
Oracle process number: 130
Unix process pid: 29807, image: oracle@server1

*** 2020-05-20T00:45:56.870884-07:00
*** SESSION ID:(204.25936) 2020-05-20T00:45:56.870895-07:00
*** CLIENT ID:() 2020-05-20T00:45:56.870900-07:00
*** SERVICE NAME:(ORCL) 2020-05-20T00:45:56.870904-07:00
*** MODULE NAME:(Foglight for Oracle <5.9.5.20>) 2020-05-20T00:45:56.870909-07:00
*** ACTION NAME:(isrgorcl-db-ORCL-DBO_Instance_General) 2020-05-20T00:45:56.870914-07:00
*** CLIENT DRIVER:(jdbcthin : 12.2.0.1.0) 2020-05-20T00:45:56.870919-07:00

[TOC00000]
Jump to table of contents
Dump continued from file: /oracle/diag/rdbms/orcl/ORCL/trace/ORCL_ora_29807.trc
[TOC00001]
ORA-00700: soft internal error, arguments: [dbgrfafr_1], [60], [60], [0], [0x7FFC2A3E6488], [], [], [], [], [], [], []

[TOC00001-END]
[TOC00002]
========= Dump for incident 201097 (ORA 700 [dbgrfafr_1]) ========

*** 2020-05-20T00:45:56.871792-07:00
dbkedDefDump(): Starting incident default dumps (flags=0x2, level=3, mask=0x0)
[TOC00003]

----- Current SQL Statement for this session (sql_id=gxpn94tgbwz4d) -----
select /*+ RULE */ version "db_version",
decode(log_mode,'NOARCHIVELOG','NO','YES') "is_archived",
decode(a.asm_files,0,'NO','YES') "is_asm",
case when dg.dataguard > 0 THEN 'YES'
when controlfile_type = 'STANDBY' THEN 'YES'
else 'NO'
end as "is_dataguard",
database_role,
decode(r.rman_conf,0,'NO','YES') "is_rman",
flashback_on "is_flashback",
instance_name,
instance_number,
open_mode,
decode(c.cell_count,0,'NO','YES') "is_exadata",
decode(al.al_count,0,'NO','YES') "is_alertlog",
'NO' "is_rds",
decode(cdb,'YES','YES','NO') is_pluggable
from sys.v$instance, sys.v$database,
(select count(*) asm_files from v$datafile where name like '+%' and rownum<2) a,
(select count(*) dataguard from sys.v$archive_dest where target = 'STANDBY' and rownum<2) dg,
(select count(*) rman_conf from sys.v$RMAN_STATUS where rownum<2) r,
(select count(*) cell_count from sys.v$cell where rownum<2) c,
(select count(*) al_count from sys.x$DBGALERTEXT where rownum<2) al
[TOC00003-END]

Also few other users encountered HIGH CPU consumption issue. This can happen when $DBGALERTEXT is populated from the XML alert log file situated in the ADR location. Note that this is an undocumented fixed table and when XML alert log is very large it takes time to access and also it can cause ORA-700 error.

You can query on X$DBGALERTEXT table to see high CPU taking a long time to complete.

SELECT count(*)
FROM X$DBGALERTEXT
WHERE to_date(to_char(originating_timestamp, 'dd-mon-yyyy hh24:mi'), 'dd-mon-yyyy hh24:mi') > to_date(to_char(systimestamp - .00694, 'dd-mon-yyyy hh24:mi'), 'dd-mon-yyyy hh24:mi') /* last 10 minutes */
AND (message_text = 'ORA-00600'
OR message_text LIKE 'útal%'
OR message_text LIKE '%error%'
OR message_text LIKE '%ORA-%'
OR message_text LIKE '%terminating the instance%');

It’s a good practice to purge the table regularly and it can be purged using ADRCI utility.

--First make sure you check the count
SQL> select count(*) from X$DBGALERTEXT;

--Connect to ADRCI utility

$adrci> show home
ADR Homes:
diag/rdbms/orcl/ORCL
diag/tnslsnr/server1/listener_orcl
$adrci> SET HOMEPATH /oracle/diag/rdbms/orcl/ORCL

--Run below command to purse logs older than 1 day.
adrci> purge -age 1440 -type alert
adrci> exit

If you still see the error from Monitoring tools like TOAD, FOGLIGHT contact the vendor. If the issue is not from monitoring tools then contact Oracle

Refer  Metalink note 2056666.1  for more info.

Regards
Satishbabu G, Oracle ACE

Tuesday, July 28, 2020

How to enable Oracle SQL*Net Client, Server and Listener Tracing

To enable tracing make sure TNS_ADMIN environment variable should be set and add below tracing parameters in sqlnet.ora. Pls note that any trace settings apply to all server processes so you need to careful when making changes on Prod, I would always suggest to test before applying changes on PRD.
Enable SQL*Net server and Client side tracing.
TRACE_LEVEL_CLIENT = SUPPORT
TRACE_UNIQUE_CLIENT = on
TRACE_LEVEL_SERVER = SUPPORT
TRACE_DIRECTORY_CLIENT = \tmp
TRACE_FILE_CLIENT = Client_Trace.log
TRACE_DIRECTORY_SERVER = \tmp
TRACE_FILE_SERVER = Server_Trace.log
TRACE_TIMESTAMP_ CLIENT = ON
DIAG_ADR_ENABLED= OFF

Starting from 11g and higher to enable client, server trace you need to add DIAG_ADR_ENABLED= OFF

SQL*Net Listener tracing
TRACE_FILE_LISTENER = Listener_Trace.log
TRACE_DIRECTORY_LISTENER = \tmp
TRACE_LEVEL_LISTENER = SUPPORT
DIAG_ADR_ENABLED_LISTENER_<LISTENER> = OFF

To start Listener Trace you need to set DIAG_ADR_ENABLED_<listener name> = OFF (must set to OFF)

To enable Tnsping Trace
TNSPING.TRACE_LEVEL = SUPPORT
TNSPING.TRACE_DIRECTORY = <Trace file path>

Enabling Dynamic Listener Tracing
LSNRCTL> set current_listener listener_ORCL
Current Listener is listener_ORCL
LSNRCTL> set trc_level 16
Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))
LISTENER parameter "trc_level" set to support
The command completed successfully
LSNRCTL> set trc_directory /oracle/diag/tnslsnr/host1/listener_orcl/trace
Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))
LISTENER parameter "trc_directory" set to /oracle/diag/tnslsnr/host1/listener_orcl/trace

Turn off dynamic tracing using:
LSNRCTL>set current_listener listener
LSNRCTL>set trc_level OFF


Thanks
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com

Friday, September 20, 2019

weblogic.jdbc.extensions.PoolDisabledSQLException

We have come across content pool suspended issue in WebLogic as pool got exhausted. All new connections failings and existing connections are running very slow. Which intern created issue in Agile PLM as it is using WebLogic.

weblogic.jdbc.extensions.PoolDisabledSQLException: weblogic.common.resourcepool.ResourceDisabledException: Pool AgileContentPool is Suspended, cannot allocate resources to applications..
        at weblogic.jdbc.common.internal.JDBCUtil.wrapAndThrowResourceException(JDBCUtil.java:265)
        at weblogic.jdbc.jts.Driver.newConnection(Driver.java:900)
        at weblogic.jdbc.jts.Driver.createLocalConnection(Driver.java:220)
        at weblogic.jdbc.jts.Driver.connect(Driver.java:170)
        at weblogic.jdbc.common.internal.RmiDataSource.getConnectionInternal(RmiDataSource.java:523)
        at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:516)
        at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:512)
<2019-09-10 10:38:56,366> <PCMHelperSessionBean_9xz6y2_Impl:ERROR> Session terminated...

To find the root causes examine the thread dumps while the error is occurring and pay attention to stuck thread.

If there are long running queries find out what it is running and users may see ORA-00020: Maximum number of processes reached.

Possible reasons :

1. Database running some custom or un-optimized long running queries
2. Application forgot to release the connection back to pool due to recent config changes, which can result connection leak.
3. Inactive connection timeout value may need to change to support system growth/load.

We have verified at DB level and everything working fine except more connections hanging around and not releasing. After investigation we found that most of the connections going though one server due to recent patch and didn’t update the Pcclient.jnlp, Jndiurl.properties..etc

After updating the config we see that user connections going trough all servers and connection load balance happening as expected and issue has been resolved.

Hope this post helps to resolve Agile Content pool suspended issue.

Thanks,
Satishbabu Gunukula, Oracle ACE

Saturday, May 4, 2019

ORA-04030: out of process memory when trying to allocate 4024 bytes (kxs-heap-b,kghsseg_1 : kokeibib)

Normally users see ORA-04030 error with shortage of RAM, small PGA and kernel parameter config.

 ORA-04030: out of process memory when trying to allocate 4024 bytes (kxs-heap-b,kghsseg_1 : kokeibib)
 Non critical error ORA-48913 caught while writing to trace file "/u01/home/oracle/diag/rdbms/ORCL/ORCL/incident/incdir_8937/ORCL_ora_11258_i8937.trc"
 Error message: ORA-48913: Writing into trace file failed, file size limit [5242880] reached
 ORA-04030: out of process memory when trying to allocate 169040 bytes (pga heap,kgh stack)
 ORA-04030: out of process memory when trying to allocate 4024 bytes (kxs-heap-b,kghsseg_1 : kokeibib)
 Non critical error ORA-48913 caught while writing to trace file "/u01/home/oracle/diag/rdbms/ORCL/ORCL/incident/incdir_8938/ORCL_ora_11258_i8938.trc"
 Error message: ORA-48913: Writing into trace file failed, file size limit [5242880] reached

Users can refer metalink Note 233869.1 titled "Diagnosing and Resolving ORA-4030 errors"

By looking the error it is clear that “out of process memory”, where oracle cannot get the required RAM. But note that you need to analyze the trace to find out the root cause

In my case I found that one of the un-optimized PL/SQL job consuming the resources and caused the issue.

Current SQL Statement for this session (sql_id=gjn38xnrxcfbj) -----
BEGIN procedure_1(:1 , :2 , :3 , :4 , :5 , :6 , :7 , :8 , :9 , :10 , :11 , :12 , :13 , :14 , :15 , :16 , :17 , :18 , :19 , :20 , :21 ) ; END;

57% pid 67: 4089 MB used of 4092 MB allocated <= CURRENT PROC
Dump of Real-Free Memory Allocator Heap [0x7f2c640b4000]
mag=0xfefe0001 flg=0x5000003 fds=0x6 blksz=65536
blkdstbl=0x7f2c640b4010, iniblk=66560 maxblk=524288 numsegs=115
In-use num=65424 siz=4290117632, Freeable num=0 siz=0, Free num=1 siz=20512768

In our case max_map_count value is not enough and needs to investigage why the pl/sql processes requires such a alarge amout of memory. In this case adjusting PGA_AGGREGATE_TARGET or MEMORY_TARGET will not limit the size processes can grow.

To find out current vale of max_map_count run below command

$cat /proc/sys/vm/max_map_count
65530

To change the value run below command
sysctl -w vm.max_map_count= <greater than 65530>

Increase the value by following metalink note
Modification of "vm.max_map_count" on Oracle Linux 6 (Doc ID 2195159.1)

Regards,
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com

Wednesday, February 27, 2019

How to Apply Oracle Latest patch set, OJVM and security patch...etc

When planning for Oracle patches make sure all pre-requisites are met and required backups are in place. Please refer below link to find out the required patches for 11g and 12c.

Oracle 11g and 12c Latest Database or OJVM Patch Set Update or Release Update

Pre-requisites

1. Before you apply patch make sure you are using latest Opatch version.

$./opatch version
OPatch Version: 11.2.0.3.19
OPatch succeeded.

Downloaded the latest opatch for 12.2 version using the link - https://updates.oracle.com/ARULink/PatchDetails/process_form?patch_num=6880880

To apply latest Opatch follow the instructions
OPatch - Where Can I Find the Latest Version of OPatch(6880880)? [Video] (Doc ID 224346.1)


2. Download the required PSU or JVN or intern patch and unzip

3. Make sure there are no conflicts and run below script to find out the conflicts. If you see any conflicts the make sure you fix the reported conflicts.

$ opatch prereq CheckConflictAgainstOHWithDetail -ph ./

In case of any conflicts you will see the message like below

There are no patches that can be applied now.
Following patches have conflicts. Please contact Oracle Support and get the merged patch of the patches :xxxxxxx, xxxxxx


In case if there are no conflicts then you will see the message like below

Invoking prereq "checkconflictagainstohwithdetail"
Prereq” CheckConflictAgainstOHWithDetail” passed.
Opatch Succeeded.


4. Using the -report option to simulate the application of the interim patch or PSU

Using “-report” option with "opatch apply" to simulate the apply without updating any files. Using this option you no need to shutdown the databases and it can be up and running.

$ opatch apply -report
...
Skip patching component oracle.rdbms, 11.2.0.4.0 and its actions.
The actions are reported here, but are not performed.

ApplySession skipping inventory update.

Verifying the update...
Inventory and System verification is performed here.
Patch xxxxxx successfully applied
Log file location: /oracle/product/11.2.0.4/cfgtoollogs/opatch/12834800_May_02_2019_10_27_17/apply2019-05-12_10-27-17AM_1.log

OPatch succeeded

5. Run utlrp.sql to recompile any invalid objects
$ORACLE_HOME/rdbms/admin/utlrp.sql

6. Check Oracle Components are VALID or not. Please make sure you fix INVALID component.
If NOT required then you might have to remove INVALID components.

SQL> select substr(comp_name,1,30) comp_name, substr(comp_id,1,10)
comp_id,substr(version,1,12) version,status from dba_registry; 
COMP_NAME                                                    COMP_ID              VERSION         STATUS
------------------------------------------------------------ -------------------- --------------- ----------
Oracle Multimedia                                            ORDIM                11.2.0.4.0      VALID
Oracle XML Database                                          XDB                  11.2.0.4.0      VALID
Oracle Expression Filter                                     EXF                  11.2.0.4.0      VALID
Oracle Rules Manager                                         RUL                  11.2.0.4.0      VALID
Oracle Workspace Manager                                     OWM                  11.2.0.4.0      VALID
Oracle Database Catalog Views                                CATALOG              11.2.0.4.0      VALID
Oracle Database Packages and T                               CATPROC              11.2.0.4.0      VALID
JServer JAVA Virtual Machine                                 JAVAVM               11.2.0.4.0      VALID
Oracle XDK                                                   XML                  11.2.0.4.0      VALID
Oracle Database Java Packages                                CATJAVA              11.2.0.4.0      VALID

7. Make sure you have taken TAR of the ORACLE_HOME, in case of any issues

tar -cvf oracle_home_12c.tar 12.2.0.1

Applying interim patch or PSU or JVM

1. Stop the databases and listener

Sqlplus / as sysdba
Shutdown immediate
$lsnrctl stop LISTENER

2. Make sure you are in the patch directory and run the below command to apply the patch

$cd <patch number>
$ORACLE_HOME/OPatch/opatch apply

3. Run post patch script against each database

Sqlplus /as sysdba
startup
@catbundle.sql psu apply
quit

Starting version 12.1 we need to use datapatch to complete post patch actions upon any 12c or later database restart. The datapatch contains the logic to identify if any post-patch SQL actions are pending.

Sqlplus / as sysdba
Startup upgrade

$ cd $ORACLE_HOME/Opatch
./datapatch -verbose

-verbose
Output additional information used for debugging

After datapatch restart the database.
Sqlplus / as sysdba
Shutdown immediate
startup

Post-verification:-
1. After patching check the inventory, to make sure patch has been updated or not
$ORACLE_HOME/OPatch/opatch lsinventory

Oracle Interim Patch Installer version 11.2.0.3.19
Copyright (c) 2019, Oracle Corporation. All rights reserved.

Oracle Home : /oracle/product/11.2.0/db_1
Central Inventory : /u01/app/oraInventory
from : /oracle/product/11.2.0/db_1/oraInst.loc
OPatch version : 11.2.0.3.19
OUI version : 11.2.0.4.0

Log file location : /oracle/product/11.2.0/db_1/cfgtoollogs/opatch/opatch2019-05-02_09-41-09AM_1.log
Lsinventory Output file location : /oracle/product/11.2.0/db_1/cfgtoollogs/opatch/lsinv/lsinventory2019-05-02_09-41-09AM.txt
……
OPatch succeeded.

2. Start the database and Listener, in case if they are still down

Sqlplus / as sysdba
startup

$lsnrctl start LISTENER

3. Run “utlrp.sql” to re-compile all invalid object

$ORACLE_HOME/rdbms/admin/utlrp.sql

4. Verify the patch history using from dba_registry_history using below query

SQL>select action_time,action from dba_registry_history;

ACTION_TIME ACTION

-------------------------------------------------- ------------------------------
02-MAY-19 06.35.23.191942 PM VIEW INVALIDATE
02-MAY-19 06.35.23.217686 PM UPGRADE
02-MAY-19 06.35.41.668820 PM APPLY

5. Please make sure all Oracle COMPONENTS are VALID

select substr(comp_name,1,30) comp_name, substr(comp_id,1,10)
comp_id,substr(version,1,12) version,status from dba_registry;

Thanks
Satishbabu Gunukula

Monday, November 12, 2018

ORA-04030: out of process memory when trying to allocate 4024 bytes (kxs-heap-b,kghsseg_1 : kokeibib

Normally users see ORA-04030 error with shortage of RAM or small PGA and kernel parameter config.

ORA-04030: out of process memory when trying to allocate 4024 bytes (kxs-heap-b,kghsseg_1 : kokeibib)
Non critical error ORA-48913 caught while writing to trace file "/u01/home/oracle/diag/rdbms/ORCL/ORCL/incident/incdir_8937/ORCL_ora_11258_i8937.trc"
Error message: ORA-48913: Writing into trace file failed, file size limit [5242880] reached
ORA-04030: out of process memory when trying to allocate 169040 bytes (pga heap,kgh stack)
ORA-04030: out of process memory when trying to allocate 4024 bytes (kxs-heap-b,kghsseg_1 : kokeibib)
Non critical error ORA-48913 caught while writing to trace file "/u01/home/oracle/diag/rdbms/ORCL/ORCL/incident/incdir_8938/ORCL_ora_11258_i8938.trc"
Error message: ORA-48913: Writing into trace file failed, file size limit [5242880] reached

Users can refer metalink Note 233869.1 titled "Diagnosing and Resolving ORA-4030 errors"

By looking the error it is clear that “out of process memory”, where oracle cannot get the required RAM. But note that you need to analyze the trace to find out the root cause

In my case I found that one of the un-optimized PL/SQL job consuming the resources and caused the issue.

Current SQL Statement for this session (sql_id=gjn38xnrxcfbj) -----
BEGIN orcl_server_bom_script.all_level(:1 , :2 , :3 , :4 , :5 , :6 , :7 , :8 , :9 , :10 , :11 , :12 , :13 , :14 , :15 , :16 , :17 , :18 , :19 , :20 , :21 ) ; END;
57% pid 67: 4089 MB used of 4092 MB allocated <= CURRENT PROC
Dump of Real-Free Memory Allocator Heap [0x7f2c640b4000]
mag=0xfefe0001 flg=0x5000003 fds=0x6 blksz=65536
blkdstbl=0x7f2c640b4010, iniblk=66560 maxblk=524288 numsegs=115
In-use num=65424 siz=4290117632, Freeable num=0 siz=0, Free num=1 siz=20512768


In our case max_map_count value is not enough and needs to investigate why the pl/sql processes requires such a large amount of memory. In this case adjusting PGA_AGGREGATE_TARGET or MEMORY_TARGET will not limit the size processes can grow and will not help.

To fix the issue increase the value of max_map_count. To find out current vale of max_map_count run below command

$cat /proc/sys/vm/max_map_count
65530

To change the value run below command
sysctl -w vm.max_map_count= <greater than 65530>

Follow below metalink note to change the value of vm.max_map_count.
Modification of "vm.max_map_count" on Oracle Linux 6 (Doc ID 2195159.1) 

Regards
Satishbabu Gunukula, Oracle ACE








Thursday, January 19, 2017

Enhance Tracing for SAP HANA SSO Issues

SAP HANA will not login and authentication failures and errors. To troubleshooting any login/authentication/sso issue user need to enable tracing

Normally I enable below tracing to get the required info

ALTER SYSTEM ALTER CONFIGURATION ('indexserver.ini', 'SYSTEM') set ('trace', 'authentication') = 'debug' with reconfigure;
ALTER SYSTEM ALTER CONFIGURATION ('indexserver.ini', 'SYSTEM') set ('trace', 'crypto') = 'debug' with reconfigure;
ALTER SYSTEM ALTER CONFIGURATION ('xsengine.ini', 'SYSTEM') set ('trace', 'authentication') = 'debug' with reconfigure;
ALTER SYSTEM ALTER CONFIGURATION ('xsengine.ini', 'SYSTEM') set ('trace', 'xssession') = 'debug' with reconfigure;
ALTER SYSTEM ALTER CONFIGURATION ('xsengine.ini', 'SYSTEM') set ('trace', 'xsauthentication') = 'debug' with reconfigure;

Reproduce the issue and collect the trace. After you collect all the necessary info you can run below command to disable the tracing.

ALTER SYSTEM ALTER CONFIGURATION ('indexserver.ini', 'SYSTEM') UNSET ('trace', 'authentication');
ALTER SYSTEM ALTER CONFIGURATION ('indexserver.ini', 'SYSTEM') UNSET ('trace', 'crypto');
ALTER SYSTEM ALTER CONFIGURATION ('xsengine.ini', 'SYSTEM') UNSET ('trace', 'authentication');
ALTER SYSTEM ALTER CONFIGURATION ('xsengine.ini', 'SYSTEM') UNSET ('trace', 'xssession');
ALTER SYSTEM ALTER CONFIGURATION ('xsengine.ini', 'SYSTEM') UNSET ('trace', 'xsauthentication');

Hope this helps,

Regards
Satishbabu Gunukula, Oracle ACE
http://oracleracexpert.com

Wednesday, February 17, 2016

How to enable and disable HANA Client Trace

Use hdbodbc_cons for tracing applications based on ODBC and hdbsqldbc_cons for tracing applications based on SQLDBC.

Use blow syntax

hdbodbc_cons.exe trace <level> <on/off>

Where levels are:
             api: tracing of ODBC api calls
             debug: detailed tracing of whole call stack
             sql: tracing of sql commands as well as output and parameter handling

Before you trun on trace, first configure the trace file name.
C:\Program Files\sap\hdbclient>hdbodbc_cons.exe config trace filename C:\temp\hdbodbc_trace.log

To turn on/off trace use below examples

Tracing of sql commands as well as output and parameter handling
          C:\Program Files\sap\hdbclient>hdbodbc_cons.exe trace sql on
          C:\Program Files\sap\hdbclient>hdbodbc_cons.exe trace sql off

Detailed tracing of whole call stack
          C:\Program Files\sap\hdbclient>hdbodbc_cons.exe trace debug on
          C:\Program Files\sap\hdbclient>hdbodbc_cons.exe trace debug off

Tracing of ODBC api calls
           C:\Program Files\sap\hdbclient>hdbodbc_cons.exe trace api on
           C:\Program Files\sap\hdbclient>hdbodbc_cons.exe trace api off

Reference:-
SAP Note: 1993254 - Collecting ODBC Trace
SAP Note: 1993251 - Collecting SQLDBC Trace

Regards,
Satishbabu Gunukula, Oracle ACE

Thursday, May 7, 2015

Terminating the instance due to error 16038 and unable to startup

I have encountered storage crash on one of the Database system. After storage is back online, not able to bring up the database. The database comes to mount stage and terminating the instance.

We are receiving below error.

ORA-03113: end-of-file on communication channel
Process ID: 5038
Session ID: 202 Serial number: 3


When I look into the alert.log I found below error messages

ARC2: Archival started
ARC1: Becoming the 'no FAL' ARCH
ARC2: Becoming the heartbeat ARCH
ARC3: Archival started
ARC0: STARTING ARCH PROCESSES COMPLETE
Tue Mar 10 10:24:10 2015
Incomplete read from log member '/oracle/ORAC/datafile/ORAC/onlinelog/ORAC_Redo1_.log'. Trying next member.
ARCH: All Archive destinations made inactive due to error 333
ARCH: Closing local archive destination LOG_ARCHIVE_DEST_1: '/oracle/ORAC/archiveORAC_ARCH_347548987.dbf' (error 333) (ORAC)
Committing creation of archivelog '/oracle/ORAC/archive/1_3541_854889367.dbf' (error 333)
Errors in file /oracle/diag/rdbms/orac/ORAC/trace/ORAC_ora_23498.trc:
ORA-16038: log 1 sequence# 3541 cannot be archived
ORA-00333: redo log read error block count
ORA-00312: online log 1 thread 1: '/oracle/ORAC/datafile/ORAC/onlinelog/ORAC_Redo1_.log'

USER (ospid: 23498): terminating the instance due to error 16038
Tue Mar 10 10:24:12 2015
System state dump requested by (instance=1, osid=23498), summary=[abnormal instance termination].
System State dumped to trace file /oracle/diag/rdbms/orac/ORAC/trace/ORAC_diag_23470.trc
Dumping diagnostic data in directory=[cdmp_20150310102412], requested by (instance=1, osid=23498), summary=[abnomal instance termination].
Instance terminated by USER, pid = 23498


By looking the error, I can see that one of the online redo is corrupted. Archive process cannot able to archive to disk. In order to bring up the database you need to clear the corrupted redo log.

From ALERT.LOG file, I am able to identify that group 1 redo log 1 corrupted. By using below command I have cleared the unarchived log file

SQL> alter database clear unarchived logfile ‘/oracle/ORAC/datafile/ORAC/onlinelog/ORAC_Redo1_.log'

If you find all multiplexed redo members are corrupted then you need to run below command to clear the GROUP.

SQL> alter database clear unarchived logfile group 1;

Now I am able to bring up the database. But note that you cannot able to bring the database back beyond corrupted point using old backups. Now you must take one FULL database backup.

Regards
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com
http://www.sqlserver-expert.com

ORA-16038: log 1 sequence# 3541 cannot be archived

I have encountered storage crash on one of the Database system. After storage is back online, not able to bring up the database. The database comes to mount stage, but going down.

We are receiving below error.

ORA-03113: end-of-file on communication channel
Process ID: 5038
Session ID: 202 Serial number: 3


When I look into the alert.log I found below error messages

ARC2: Archival started
ARC1: Becoming the 'no FAL' ARCH
ARC2: Becoming the heartbeat ARCH
ARC3: Archival started
ARC0: STARTING ARCH PROCESSES COMPLETE
Tue Mar 10 10:24:10 2015
Incomplete read from log member '/oracle/ORAC/datafile/ORAC/onlinelog/ORAC_Redo1_.log'. Trying next member.
ARCH: All Archive destinations made inactive due to error 333
ARCH: Closing local archive destination LOG_ARCHIVE_DEST_1: '/oracle/ORAC/archiveORAC_ARCH_347548987.dbf' (error 333) (ORAC)
Committing creation of archivelog '/oracle/ORAC/archive/1_3541_854889367.dbf' (error 333)
Errors in file /oracle/diag/rdbms/orac/ORAC/trace/ORAC_ora_23498.trc:
ORA-16038: log 1 sequence# 3541 cannot be archived
ORA-00333: redo log read error block count
ORA-00312: online log 1 thread 1: '/oracle/ORAC/datafile/ORAC/onlinelog/ORAC_Redo1_.log'

USER (ospid: 23498): terminating the instance due to error 16038
Tue Mar 10 10:24:12 2015
System state dump requested by (instance=1, osid=23498), summary=[abnormal instance termination].
System State dumped to trace file /oracle/diag/rdbms/orac/ORAC/trace/ORAC_diag_23470.trc
Dumping diagnostic data in directory=[cdmp_20150310102412], requested by (instance=1, osid=23498), summary=[abnomal instance termination].
Instance terminated by USER, pid = 23498


By looking the error, I can see that one of the online redo is corrupted. Archive process cannot able to archive to disk. In order to bring up the database you need to clear the corrupted redo log.

From ALERT.LOG file, I am able to identify that group 1 redo log 1 corrupted. By using below command I have cleared the unarchived log file

SQL> alter database clear unarchived logfile ‘/oracle/ORAC/datafile/ORAC/onlinelog/ORAC_Redo1_.log'

If you find all multiplexed redo members are corrupted then you need to run below command to clear the GROUP.

SQL> alter database clear unarchived logfile group 1;

Now I am able to bring up the database. But note that you cannot able to bring the database back beyond corrupted point using old backups. Now you must take one FULL database backup.

Regards
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com
http://www.sqlserver-expert.com

Tuesday, December 2, 2014

ORA-27300: OS system dependent operation:fork failed with status: 11

I have encounter below error on all the databases on a shared server.

Errors from alert_<SID>.log

Process startup failed, error stack:
Errors in file /u01/home/oracle/diag/rdbms/ORA1/ORA1/trace/ORA1_psp0_18712.trc:
ORA-27300: OS system dependent operation:fork failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
ORA-27302: failure occurred at: skgpspawn5
Mon Nov 10 22:00:10 2014
Process m000 died, see its trace file


I see that m000 process dies and oracle has problem forking more processes.

Trace file ORA1_ psp0_18712.trc contents

*** 2014-11-10 02:04:12.780
*** SESSION ID:(79.1) 2014-11-10 02:04:12.780
*** CLIENT ID:() 2014-11-10 02:04:12.780
*** SERVICE NAME:(SYS$BACKGROUND) 2014-11-10 02:04:12.780
*** MODULE NAME:() 2014-11-10 02:04:12.780
*** ACTION NAME:() 2014-11-10 02:04:12.780

Dump diagnostics for process W000 pid 9442 which did not start after 120 seconds:
(spawn_time:x376D2DDDA now:x376D4B38A diff:x1D5B0)

*** 2014-11-10 02:04:12.855
Process diagnostic dump for W000, OS id=9442
-------------------------------------------------------------------------------
*** 2014-11-10 02:04:29.700
loadavg : 368.16 219.45 107.21
Memory (Avail / Total) = 295.09M / 128955.82M
Swap (Avail / Total) = 0.00M / 24480.54M
skgpgcmdout: read() for cmd /bin/ps -elf | /bin/egrep 'PID | 9442' | /bin/grep -v grep timed out after 15.000 seconds
Stack:
ERROR: process 9442 is not alive
………
Killing process (ospid 21743): did not start after 120 seconds
... and the process is still alive after kill!
*** 2014-11-10 03:07:19.266
Killing process (ospid 23399): requester cancelled request
... and the process is still alive after kill!
Killing process (ospid 23780): requester cancelled request
*** 2014-11-10 03:12:33.255
.. and the process is still alive after kill!
*** 2014-11-10 03:13:33.536
Killed process 23780 is still alive after 60 seconds!
*** 2014-11-10 03:15:24.608
Killing process (ospid 23889): requester cancelled request
………………….

*** 2014-11-10 22:00:09.662
Process startup failed, error stack:
ORA-27300: OS system dependent operation:fork failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
ORA-27302: failure occurred at: skgpspawn5


It looks like the maximum number of PROCESSES allowed per user is low. Check the ulimit –a to see the values

$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 1031617
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 32800
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 2047
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited


I see that the max user process set to 2047 and when I check the total number of process it is reaching the limit. We need to increase the limit to fix the issue

For Linux
Increase the nproc in /etc/security/limits.conf file

For Solaris 10
Increase the project.max-lwps value

For HP and Solaris
Increase the maxuprc

For AIX
Increase maxuproc

I didn’t see the issue or error in the alert.log file.

Regards
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com

Friday, November 14, 2014

ORA-04030: out of process memory when trying to allocate 16328 bytes (koh-kghu call ,pmucalm coll)

User may encounter below error in 11g. First I would recommend running ADDM report and checking how the resources are utilized. If you see any undersize database configuration then increase.

ORA-04030: out of process memory when trying to allocate 16328 bytes (koh-kghu call ,pmucalm coll)
Incident details in: /u01/home/oracle/diag/rdbms/orac/ORAC/incident/incdir_17129/ORAC_ora_8372_i17129.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Non critical error ORA-48913 caught while writing to trace file "/u01/home/oracle/diag/rdbms/orac/ORAC/incident/incdir_17129/ORAC_ora_8372_i17129.trc"
Error message: ORA-48913: Writing into trace file failed, file size limit [5242880] reached
Writing to the above trace file is disabled for now on...

If your database has enough resources and if you still see the issue then you might need to check the trace file.

=======================================
PRIVATE MEMORY SUMMARY FOR THIS PROCESS
---------------------------------------
******************************************************
PRIVATE HEAP SUMMARY DUMP
4129 MB total:
4129 MB commented, 173 KB permanent
57 KB free (0 KB in empty extents),
2240 MB, 3 heaps: "koh-kghu call " 14 KB free held
1889 MB, 1 heap: "session heap " 4 KB free held
------------------------------------------------------


The trace file confirms that the process is limiting at 4GB. Change the upper limit either at OS level or Database level

Apply this change at OS level
1. Get the page count using below command and increase the limit
#more /proc/sys/vm/max_map_count

2. Increase the page count
#sysctl -w vm.max_map_count= 262144
or
#sysctl vm.max_map_count=262144

3. These changes will be revered on server reboot. To make changes permanent add the value in
$ vi /etc/sysctl.conf
or
$echo “vm.max_map_count = 262144” | tee -a /etc/sysctl.conf

4. You no need to reboot the server, you should reload the config file as root
$ sysctl –p

Apply this change at Database level
Adjust the realfree heap page size by setting below parameters in spfile or pfile and restart the database
_use_realfree_heap=TRUE
_realfree_heap_pagesize_hint = 262144

Note that the default realfree allocator pagesize is 64 kB (65536), so 64K entries take up 4GB. With 256kB (262144) pages, the limit goes to 16GB.

I hope this should resolve your issue.

Regards
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com