weblogic.jdbc.extensions.PoolDisabledSQLException

We have come across content pool suspended issue in WebLogic as pool got exhausted. All new connections failings and existing connections are running very slow. Which intern created issue in Agile PLM as it is using WebLogic.

weblogic.jdbc.extensions.PoolDisabledSQLException: weblogic.common.resourcepool.ResourceDisabledException: Pool AgileContentPool is Suspended, cannot allocate resources to applications..

        at weblogic.jdbc.common.internal.JDBCUtil.wrapAndThrowResourceException(JDBCUtil.java:265)

        at weblogic.jdbc.jts.Driver.newConnection(Driver.java:900)

        at weblogic.jdbc.jts.Driver.createLocalConnection(Driver.java:220)

        at weblogic.jdbc.jts.Driver.connect(Driver.java:170)

        at weblogic.jdbc.common.internal.RmiDataSource.getConnectionInternal(RmiDataSource.java:523)

        at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:516)

        at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:512)

<2019-09-10 10:38:56,366> <PCMHelperSessionBean_9xz6y2_Impl:ERROR> Session terminated...

To find the root causes examine the thread dumps while the error is occurring and pay attention to stuck thread.

If there are long running queries find out what it is running and users may see ORA-00020: Maximum number of processes reached.

Possible reasons :

1. Database running some custom or un-optimized long running queries
2. Application forgot to release the connection back to pool due to recent config changes, which can result connection leak.
3. Inactive connection timeout value may need to change to support system growth/load.

We have verified at DB level and everything working fine except more connections hanging around and not releasing. After investigation we found that most of the connections going though one server due to recent patch and didn’t update the Pcclient.jnlp, Jndiurl.properties..etc

After updating the config we see that user connections going trough all servers and connection load balance happening as expected and issue has been resolved.

Hope this post helps to resolve Agile Content pool suspended issue.

Thanks,
Satishbabu Gunukula, Oracle ACE

http://www.oracleracexpert.com

Oracle patch failed with “CheckActiveFilesAndExecutables” failed error

We are applying patch on Oracle 12c and we got Prerequisite check “CheckActiveFilesAndExecutables” failed error. By looking the error we can see that files are active and used by some active processes.

$opatch apply

.....

[Sep 6, 2018 6:56:10 PM] [INFO] Finish fuser command /sbin/fuser /oracle/product/12.1.0.2/dbhome_1/lib/libsrvm12.so at Thu Sep 06 18:56:10 PDT 2018
[Sep 6, 2018 6:56:10 PM] [SEVERE] OUI-67073:UtilSession failed: Prerequisite check "CheckActiveFilesAndExecutables" failed.
[Sep 6, 2018 6:56:10 PM] [INFO] Finishing UtilSession at Thu Sep 06 18:56:10 PDT 2018
[Sep 6, 2018 6:56:10 PM] [INFO] Log file location: /oracle/product/12.1.0.2/dbhome_1/cfgtoollogs/opatch/opatch2018-09-06_18-55-32PM_1.log
[Sep 6, 2018 6:56:10 PM] [INFO] Stack Description: java.lang.RuntimeException: Prerequisite check "CheckActiveFilesAndExecutables" failed.
at oracle.opatch.OPatchSessionHelper.runApplyPrereqs(OPatchSessionHelper.java:6670)
at oracle.opatch.opatchutil.NApply.legacy_process(NApply.java:988)
at oracle.opatch.opatchutil.NApply.legacy_process(NApply.java:363)
at oracle.opatch.opatchutil.NApply.process(NApply.java:343)
at oracle.opatch.opatchutil.OUSession.napply(OUSession.java:1105)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at oracle.opatch.UtilSession.process(UtilSession.java:359)
at oracle.opatch.OPatchSession.process(OPatchSession.java:2670)
at oracle.opatch.OPatch.process(OPatch.java:811)
at oracle.opatch.OPatch.main(OPatch.java:861)
Caused by: java.lang.RuntimeException: Prerequisite check "CheckActiveFilesAndExecutables" failed.
... 13 more
Caused by: oracle.opatch.PrereqFailedException: Prerequisite check "CheckActiveFilesAndExecutables" failed.
... 13 more

In the log file I see that below lib is active and used by some processes

[Sep 6, 2018 6:55:49 PM] [INFO] Following executables are active :
/oracle/product/12.1.0.2/dbhome_1/lib/libclntsh.so.12.1
[Sep 6, 2018 6:55:49 PM] [INFO] Prerequisite check "CheckActiveFilesAndExecutables" failed.
The details are:
Following executables are active :
/oracle/product/12.1.0.2/dbhome_1/lib/libclntsh.so.12.1
[Sep 6, 2018 6:55:49 PM] [INFO] Start fuser command /sbin/fuser /oracle/product/12.1.0.2/dbhome_1/bin/adrci at Thu Sep 06 18:55:49 PDT 2018

To find which process using below lib run below command and you will get the process id

$ /sbin/fuser /oracle/product/12.1.0.2/dbhome_1/lib/libclntsh.so.12.1

/oracle/product/12.1.0.2/dbhome_1/lib/libclntsh.so.12.1: 8631m

Find out the process that is active using below command

$ps –ef |grep 8631     

oracle     8631      1  0 Aug23 ?        00:02:22 /oracle/product/12.1.0.2/dbhome_1/bin/tnslsnr LISTENER_ORCL-inherit

Either you stop the processes or kill the process. Once the processes is stopped you can proceed with opatch again and it should be successful.

Note that after patch is successful check for “SEVERE” errors if any in the log file.

Thanks

Satishbabu Gunukula

http://www.oracleracexpert.com

SAP DBTech JDBC: [403]: internal error: Cannot get remote source objects: Unable to load driver manager

When trying to create “Remote source” to Microsoft SQL Server under Smart Data Access (SDA) using HANA received following error

SAP DBTech JDBC: [403]: internal error: Cannot get remote source objects: Unable to load driver manager 

By looking the error it looks like the driver does not exists or not installed properly.  To find out the driver install or not run below command, if you receive an error then its not installed.

#isql --help
SAP DBTech JDBC: [403]: internal error: Cannot get remote source objects: [unixODBC][Microsoft][ODBC Driver 11 for SQL Server][SQL Server]Login failed for user 'hanasdauser'. 

I have followed below steps to install the Drivers

1. Set the environment
# cd /hana/ms-odbc-driver/unixODBC-2.3.0
# export CPPFLAGS=”-DSIZEOF_LONG_INT=8”
# echo $CPPFLAGS
-DSIZEOF_LONG_INT=8

2. Configure
#./configure --prefix=/usr --libdir=/usr/lib64 --sysconfdir=/etc --enable-gui=no --enable-drivers=no --enable-iconv --with-iconv-char-enc=UTF8 --with-iconv-ucode-enc=UTF16LE

3. Run make utility
# make

4. Run make install
# make install

5. Test the driver is installed or not
# isql –help
unixODBC 2.3.0

After installing the driver I was able to create Remote Source in HANA under SDA.

Regards
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com

Database error : (CS) "Java Class not found in classpath : com.mysql.jdbc.Driver"(WIS 10901)

The users will see below error if they don’t configure JDBC connection properly to connect MYSQL database.

Database error : (CS) "Java Class not found in classpath : com.mysql.jdbc.Driver" . (WIS 10901)

Follow below troubleshooting steps

1. Locate jdbc.sbo file and make sure that you have added “ClassPath” with correct MySQL Connector jar file.

<ClassPath>
<Path>/home/boxi/ bobj/enterprise_xi40/dataAccess/connectionServer/jdbc/mysql-connector-java-5.1.25-bin.jar</Path>
</ClassPath>

2. Update environment variable CLASSPATH with correct jar file path.

3. Make sure that the MySQL Connector jar file is right version, means for 64bit O/S or app use 64bit jar file.

Common Error in ODBC:- 

1. The below error message is very common when using ODBC(.odbc.ini) connection

[01000][unixODBC][Driver Manager]Can't open lib '/home/boxi/bobj/enterprise_xi40/linux_x64/odbc/lib/libmyodbc5a.so' : /home/boxi/bobj/enterprise_xi40/linux_x64/odbc/lib/libmyodbc5a.so: wrong ELF class: ELFCLASS32

Cause: The file is not the right version

Possible Solution: The file is not 64-bits version of the libmyodbc5a.so, install the right version. Use 32bit version of the libmyodbc5a.so, if using 32bit.

Download MySQL Connector/ODBC

Regards
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com 

OALL8 is in an inconsistent state

"OALL8 is in an inconsistent state" is a generic exception that occurs in JDBC 10.1, 10.2, and 11.1. It indicates that an internal inconsistency has been detected in the JDBC connection but it does not provide information on what caused the inconsistency.

OALL8 is in an inconsistent state.
java.sql.SQLException: OALL8 is in an inconsistent state.
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:138)
at oracle.jdbc.driver.T4C8Oall.init(T4C8Oall.java:325)
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:170)
at oracle.jdbc.driver.T4CPreparedStatement.execute_for_describe(T4CPreparedStatement.java:503)
at oracle.jdbc.driver.OracleStatement.execute_maybe_describe(OracleStatement.java:1029)
at oracle.jdbc.driver.T4CPreparedStatement.execute_maybe_describe(T4CPreparedStatement.java:535)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1126)
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3001)
at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3043)
at oracle_jdbc_driver_T4CPreparedStatement_Proxy.executeQuery()
at com.agile.pc.cmserver.util.ClusterLock.acquireClusterLock(ClusterLock.java:61)
at com.agile.pc.cmserver.notification.NotifyTask.run(NotifyTask.java:413)
at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)
<2013-02-27 22:59:37,734> <NotifyTask:ERROR> notification task exception

You may also see a "Protocol Violation error, the thin driver throws this exception when it reads something from the RDBMS that it did not expect. This means that the protocol engine in the Thin driver and the protocol engine in the RDBMS are out of synch. There is no way to recover from this error. The connection is dead and You should try to close it, but that will probably fail too.

If you are able to reproduce the error, please file a TAR with Oracle Global Support by specifying the exact version numbers of the JDBC driver and the RDBMS.

Protocol violation
java.sql.SQLException: Protocol violation
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:138)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:175)
at oracle.jdbc.driver.DatabaseError.check_error(DatabaseError.java:898)
at oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:656)
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:185)
at oracle.jdbc.driver.T4CPreparedStatement.execute_for_describe(T4CPreparedStatement.java:503)
at oracle.jdbc.driver.OracleStatement.execute_maybe_describe(OracleStatement.java:1029)
at oracle.jdbc.driver.T4CPreparedStatement.execute_maybe_describeT4CPreparedStatement.
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1126)
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3001)
at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3043)
at oracle_jdbc_driver_T4CPreparedStatement_Proxy.executeQuery()
at com.agile.pc.cmserver.util.ClusterLock.acquireClusterLock(ClusterLock.java:61)
at com.agile.pc.cmserver.notification.NotifyTask.run(NotifyTask.java:199)
at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)

Possible Solution: It is very hard to troubleshoot and the simplest solution is to upgrade your JDBC Driver.

This exception no longer occurs in JDBC 11.2.

Other scenarios & Reference
1. Oracle Transportation Management (OTM) - Version 5.5.04.02 to 5.5.06.03
If you are facing the issue with OTM then check below metalink note for Cause and Solution
java.sql.SQLException: OALL8 is in an inconsistent state [ID 754144.1]

2. Master Note: Understanding the "OALL8 is in an Inconsistent State" Exception [ID 944692.1]

Regards
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com