Oracle Agile Application or Quick Search performance issue and how to resolve?

When you come across any performance issue on agile first, you need to verify schema integrity by running agile9_check.sql.

The agile9_check.sql it verifies Database Schema version with Agile Schema version, and it MUST match. Also, it validates schema objects against Agile schema version and report any errors or warnings.

You may see ERRORS/WARNINGS
ERROR: Missing or INVALID index (columns) XXXXXXXXX on table XXXXX.
ERROR: Missing or INVALID FTS CTX indexes XXXXXXX.
WARNING: ACTIVITY_BASELINE_XXXXX does not belong to AGILE DB. Please Drop the Table if not required.

In case of any Errors/Warnings you need we need to fix the issue. If you have created any custom objects you will see WARNING that object doesn't  below to Agile DB and you can ignore. 

Once user fix the schema integrity download listFTSInfo.sql and run the script as Agile user and it will generate listFTSInfo_xxxxxxxxxx.log file. You need to search for "029", where it says "List Index Percent of Fragmentation" where you can see the indexes that has fragmentation.

If you see any fragmentation, it will effect Quick search and stuck threads will be generated.

<[STUCK] ExecuteThread: '19' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "1,247" seconds working on the request "Http Request Information: weblogic.servlet.internal.ServletRequestImpl@2f2fb94[GET /Agile/PCMServlet]
", which is more than the configured time (StuckThreadMaxTime) of "1,200" seconds in "server-failure-trigger". Stack trace:
java.net.SocketInputStream.socketRead0(Native Method)

To resolve the issue, Log into sqlplus as agile database user, and run agile_ctx_recreate.sql found under %Oracle_base%\admin\{SID}\create\{agile_schema_username} to rebuild CTX indexes.

Oracle support recommends stopping Agile application, run agile_ctx_recreate.sql and Start the Application.

It is highly advisable to run complie_Invalid_objects.sql, agile9stats.sql to recompile invalid objects and collect stats to improve the performance.

Refer Oracle Agile support doc for Full Text search (FTS) Enablement, synchronization, Indexing and optimization.

Frequently Asked Questions on Agile Product Lifecycle Management (PLM) Full Text Search (FTS) Enablement, Synchronization, Indexing, and Optimization (Doc ID 1503311.1)

For Agile Stuck thread issue, refer below Oracle support Doc ID

Agile Managed Servers Goes to Warning State, Having Stuck Thread at com.agile.cs.query.QuerySessionBean (Doc ID 2919154.1)

Thanks & Regards

https://oracleracexpert.com, Oracle ACE

BEA-002936 maximum thread constraint or BEA-000337 [STUCK] ExecuteThread in WebLogic

I come across an issue and found below error message in server log file. As per oracle note Doc ID 1356278.1 this message is only informational and can be safely ignored. But our planning application crashed and we had to restart the services.

####<Sep 8, 2022 11:06:22 AM PDT> <Info> <WorkManager> <SCLWPA2188> <Planning0> <[ACTIVE] ExecuteThread: '29' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1662660382282> <BEA-002936> <maximum thread constraint ASYNC_REP_FLUSH_WM is reached>
####<Sep 8, 2022 11:08:39 AM PDT> <Info> <WorkManager> <SCLWPA2188> <Planning0> <[ACTIVE] ExecuteThread: '31' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1662660519981> <BEA-002936> <maximum thread constraint ASYNC_REP_FLUSH_WM is reached>

In many cases the error may clear after some time. After research I found that WebLogic Server associates some internal work managers whenever a JMS server is created and below are the parameters

weblogic.jms.<JMSServer_NAME>.AsyncPush
Min threads = 1, max threads = unlimited, high fair share

weblogic.jms.<JMSServer_NAME>.Limited
Min threads = 1, max threads = 8, normal fair share.

weblogic.jms.<JMSServer_NAME>.System
Min threads = 1, max threads = unlimited, high fair share.

The work managers will expand up to the max limit and will shrink back when the load returns to normal, however shrinking doesn’t start unless there are few idle state threads.

As you see the work managers have a limited thread count of 8, but this limit is internal and cannot be changed without contacting Oracle support.  

Users may also see STUCK thread issues

Sep 3, 2022 11:23:31 AM EDT> <Error> <WebLogicServer> <abcsystems.orass.com> <server-cc113sp> <[ACTIVE] ExecuteThread: '33' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <45443435326649> <BEA-000337> <[STUCK] ExecuteThread: 45 for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for “550" seconds working on the request "Workmanager: default, Version: 0, Scheduled=true, Started=true, Started time: 678511 ms

The root cause Thread gets stuck if it is continually working (not idle) for a set period of time. This time is called the StuckThreadMaxTime, and has a default value of 600 seconds. The stuck thread cannot complete its current work or accept any new work the server logs the message. Note that if all threads gets stuck then server health state will change to either “warning” or “critical”

The stuck thread errors may disappear after some time if you see these errors frequently then you should start tuning to avoid issues. Note that restart will clear these errors.

I would recommend you go to WebLogic configuration and finetune your parameters based upon the current settings such as “Stuck thread max time” and “Stuck Thread Timer Interval”

To fine tune the parameter go to weblogic console à Domain à Environment à Server à <Managed Server> à Configuration > Tuning

Make sure you apply changes and reboot the server for new changes to take into effect. You can also refer below WLS documentation and Metalink note for more details.

WLS documentation : https://docs.oracle.com/cd/E23943_01/apirefs.1111/e13952/taskhelp/tuning/TuningExecuteThreads.html
Metalink Note: Note 1302472.1 - Information about STUCK Threads Condition and Tuning in Weblogic Server

Thanks & Regards

http://oracleracexpert.com, Oracle ACE Pro

java.sql.SQLRecoverableException: IO Error: Connection reset

I have come across below error during Oracle Agile Managed Server deployment (Agile 9.3.3). When trying to startup the Managed server which is using Weblogic12c, it didn’t startup and found below error in logfile.

Your environment has been set.
AgileAuthenticationProviderImpl.initialize
log4j:WARN No appenders could be found for logger (com.agile.util.sql.OracleConnectionImpl).
log4j:WARN Please initialize the log4j system properly.
java.sql.SQLRecoverableException: IO Error: Connection reset
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:467)
at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:546)
at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:236)
at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:32)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:521)
at oracle.jdbc.pool.OracleDataSource.getPhysicalConnection(OracleDataSource.java:280)
at oracle.jdbc.pool.OracleDataSource.getConnection(OracleDataSource.java:207) at
oracle.jdbc.pool.OracleConnectionPoolDataSource.getPhysicalConnection(OracleConnectionPool Data Source.java:139) at oracle.jdbc.pool.OracleConnectionPoolDataSource.getPooledConnection (Oracle ConnectionPool Data Source.java:88) at........ com.bea.common.security.internal.service.LoginModuleWrapper$1.run(LoginModuleWrapper.java:110)
at java.security.AccessController.doPrivileged(Native Method) at com.bea.common.security.internal.service.LoginModuleWrapper.login(LoginModuleWrapper.java:106)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:762)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:203)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:690)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:688)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:687)
at javax.security.auth.login.LoginContext.login(LoginContext.java:595) at com.bea.common.security.internal.service.JAASLoginServiceImpl.login(JAASLoginServiceImpl.java:113)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606) at com.bea.common.security.internal.utils.Delegator$ProxyInvocationHandler.invoke(Delegator.java:57)
at com.sun.proxy.$Proxy13.login(Unknown Source) at weblogic.security.service.internal.WLSJAASLoginServiceImpl$ServiceImpl.login(WLSJAASLoginServiceImpl.java:89) at com.bea.common.security.internal.service.JAASAuthenticationServiceImpl.authenticate(JAASAuthenticationServiceImpl.java:82)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606) at com.bea.common.security.internal.utils.Delegator$ProxyInvocationHandler.invoke(Delegator.java:57)
at com.sun.proxy.$Proxy34.authenticate(Unknown Source) at weblogic.security.service.WLSJAASAuthenticationServiceWrapper.authenticate(WLSJAASAuthenticationServiceWrapper.java:40) at weblogic.security.service.PrincipalAuthenticator.authenticate(PrincipalAuthenticator.java:338) at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.doBootAuthorization(CommonSecurityServiceManagerDelegateImpl.
java:930) at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl. initialize(CommonSecurityServiceManagerDelegateImpl.java:1054) at
weblogic.security.service.SecurityServiceManager.initialize(SecurityServiceManager.java:873)
at weblogic.security.SecurityService.start(SecurityService.java:148)
at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
Caused by: java.net.SocketException: Connection reset
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:118)
at java.net.SocketOutputStream.write(SocketOutputStream.java:159)
at oracle.net.ns.DataPacket.send(DataPacket.java:210)
at oracle.net.ns.NetOutputStream.flush(NetOutputStream.java:230)
at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:312)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:260)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:185)
at oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket (T4CSocketInputStreamWrapper. java:124) at oracle.jdbc.driver.T4CSocketInputStream Wrapper.read(T4CSocketInputStreamWrapper.java:80)
at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1137)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:290)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:192)
at oracle.jdbc.driver.T4CTTIoauthenticate.doOSESSKEY(T4CTTIoauthenticate.java:404)
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:385)
... 59 more
java.sql.SQLRecoverableException: IO Error: Connection reset

You will also see below error in the log file, but these errors will mislead you

<Jun 19, 2014 2:13:44 PM PDT> <Critical> <Security> <BEA-090402> <Authentication denied: Boot identity not valid; The user name and/or password from the boot identity file (boot.properties) is not valid. The boot identity may have been changed since the boot identity file was created. Please edit and update the boot identity file with the proper values of username and password. The first time the updated boot identity file is used to start the server, these new values are encrypted.>

<Jun 19, 2014 2:13:44 PM PDT> <Critical> <WebLogicServer> <BEA-000386> <Server subsystem failed. Reason: weblogic.security.SecurityInitializationException: Authentication denied: Boot identity not valid; The user name and/or password from the boot identity file (boot.properties) is not valid. The boot identity may have been changed since the boot identity file was created. Please edit and update the boot identity file with the proper values of username and password. The first time the updated boot identity file is used to start the server, these new values are encrypted.
...(CommonSecurityServiceManagerDelegateImpl.java:960) at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initialize(CommonSecurityServiceManagerDelegateImpl.java:1054) at weblogic.security.service.SecurityServiceManager.initialize(SecurityServiceManager.java:873)
at weblogic.security.SecurityService.start(SecurityService.java:148)
at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
Truncated. see log file for complete stacktrace
Caused By: javax.security.auth.login.LoginException: java.lang.ExceptionInInitializerError
 at com.agile.admin.security.weblogic.WLSLoginModule.login
(WLSLoginModule.java:193) at com.bea.common.security.internal.service.LoginModuleWrapper$1.run(LoginModuleWrapper. java: 110 at java.security.AccessController.doPrivileged(Native Method)
at com.bea.common.security.internal.service.LoginModuleWrapper.login(LoginModuleWrapper .java:106) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)


Findings/troubleshooting

1. When trying to startup the Managed server the weblogic taking more time than expected

2. If you read the highlight error then you will understand what the exact cause for this error is “java.sql.SQLRecoverableException: IO Error: Connection reset”.

3. Found below warning in alert<sid>.log
WARNING: inbound connection timed out (ORA-3136)

4. The userid/password is correct

By looking the above findings it seems that weblogic taking more time and it was not able to connect to the database and timed out with ORA-3136 error. It looks like inbound connection timeout occurred.

Proposed Solution: I have changed the SQLNET.INBOUND_CONNECT_TIMEOUT from default (60 sec) value to 0(indefinite).

After changing the parameter the weblogic Instance comes online without any issues.

It looks like there is bug in Weblogic due to which it is not able to connect to database with in the specified time. Still troubleshooting to find the root cause, if I find anything will try to update in the post.

Regards,
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com

Exception in thread "Thread-1" java.lang.ExceptionInInitializerError

We are using the Weblogic 12.1.1with Oracle Agile 9.3.3. After applying the hot fix, when we try to start the application we received the below error message

Please upgrade your admin data schema to 9.3.385.2
Shutting down the server...

<May 30, 2014 1:15:23 PM PDT> <Alert> <WebLogicServer> <BEA-000396> <Server shutdown has been requested by <WLS Kernel>.>
Exception in thread "Thread-1" java.lang.ExceptionInInitializerError  at weblogic.wsee.runtime.JMSStoreRoutableIDMapper.shutdown(JMSStoreRoutableIDMapper.java:479)
at weblogic.wsee.runtime.WebServicesRuntime.shutdown(WebServicesRuntime.java:236)
at weblogic.wsee.runtime.WseeService.haltService(WseeService.java:65)
at weblogic.server.ServiceActivator.halt(ServiceActivator.java:113)
at weblogic.t3.srvr.ServerServicesManager.haltInternal(ServerServicesManager.java:504)
at weblogic.t3.srvr.ServerServicesManager.halt(ServerServicesManager.java:336)
at weblogic.t3.srvr.T3Srvr.shutdown(T3Srvr.java:1122)
at weblogic.t3.srvr.T3Srvr.forceShutdown(T3Srvr.java:1026)
at weblogic.t3.srvr.T3Srvr$2.run(T3Srvr.java:1039)
Caused by: java.lang.IllegalStateException: Shutdown in progress
at java.lang.ApplicationShutdownHooks.add(ApplicationShutdownHooks.java:66)
at java.lang.Runtime.addShutdownHook(Runtime.java:211)
at weblogic.wsee.persistence.LogicalStore.<clinit>(LogicalStore.java:159)

From the error it is clear that the admin data schema needs to be upgrade. That means the hot fix have Database component and that needs to run after the hot fix.

I found a Database script i.e. oracle_933HF30.sql and applied to upgrade the admin data schema.

Now the application comes online without any issues.

Regards
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com

OALL8 is in an inconsistent state

"OALL8 is in an inconsistent state" is a generic exception that occurs in JDBC 10.1, 10.2, and 11.1. It indicates that an internal inconsistency has been detected in the JDBC connection but it does not provide information on what caused the inconsistency.

OALL8 is in an inconsistent state.
java.sql.SQLException: OALL8 is in an inconsistent state.
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:138)
at oracle.jdbc.driver.T4C8Oall.init(T4C8Oall.java:325)
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:170)
at oracle.jdbc.driver.T4CPreparedStatement.execute_for_describe(T4CPreparedStatement.java:503)
at oracle.jdbc.driver.OracleStatement.execute_maybe_describe(OracleStatement.java:1029)
at oracle.jdbc.driver.T4CPreparedStatement.execute_maybe_describe(T4CPreparedStatement.java:535)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1126)
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3001)
at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3043)
at oracle_jdbc_driver_T4CPreparedStatement_Proxy.executeQuery()
at com.agile.pc.cmserver.util.ClusterLock.acquireClusterLock(ClusterLock.java:61)
at com.agile.pc.cmserver.notification.NotifyTask.run(NotifyTask.java:413)
at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)
<2013-02-27 22:59:37,734> <NotifyTask:ERROR> notification task exception

You may also see a "Protocol Violation error, the thin driver throws this exception when it reads something from the RDBMS that it did not expect. This means that the protocol engine in the Thin driver and the protocol engine in the RDBMS are out of synch. There is no way to recover from this error. The connection is dead and You should try to close it, but that will probably fail too.

If you are able to reproduce the error, please file a TAR with Oracle Global Support by specifying the exact version numbers of the JDBC driver and the RDBMS.

Protocol violation
java.sql.SQLException: Protocol violation
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:138)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:175)
at oracle.jdbc.driver.DatabaseError.check_error(DatabaseError.java:898)
at oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:656)
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:185)
at oracle.jdbc.driver.T4CPreparedStatement.execute_for_describe(T4CPreparedStatement.java:503)
at oracle.jdbc.driver.OracleStatement.execute_maybe_describe(OracleStatement.java:1029)
at oracle.jdbc.driver.T4CPreparedStatement.execute_maybe_describeT4CPreparedStatement.
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1126)
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3001)
at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3043)
at oracle_jdbc_driver_T4CPreparedStatement_Proxy.executeQuery()
at com.agile.pc.cmserver.util.ClusterLock.acquireClusterLock(ClusterLock.java:61)
at com.agile.pc.cmserver.notification.NotifyTask.run(NotifyTask.java:199)
at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)

Possible Solution: It is very hard to troubleshoot and the simplest solution is to upgrade your JDBC Driver.

This exception no longer occurs in JDBC 11.2.

Other scenarios & Reference
1. Oracle Transportation Management (OTM) - Version 5.5.04.02 to 5.5.06.03
If you are facing the issue with OTM then check below metalink note for Cause and Solution
java.sql.SQLException: OALL8 is in an inconsistent state [ID 754144.1]

2. Master Note: Understanding the "OALL8 is in an Inconsistent State" Exception [ID 944692.1]

Regards
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com

ORA-07445 exception encountered, ORA-04030 out of processes memrory

I have received below error on one of the 10gR2 database.

ORA-07445: exception encountered: core dump [kohfrem()+145] [SIGSEGV] [Address not mapped to object] [0x2B69FFFFFFF8] [] []
 
ORA-04030: out of process memory when trying to allocate 16408 bytes (koh-kghu sessi,pmucpkl kolctx)

If user or application running many jobs then the processes running the jobs may use an increasing amount of memory over time and the job queue process may fail with ORA-4030 errors.

In that case you might be hitting a bug 7715994 and refer below Metalink note
Bug 7715994 Memory leak / ORA-4030 when running many jobs
The bug has been fixed in 11.2.0.1. After spending many hours I found below workaround
Work around: When running large numbers of jobs, restart the session occasionally.

If user or application using UTL_FILE package then you see a constant growth of memory & the heapdumps suggest "koh-kghu sessi" heap is growing as a part of allocating chunks marked with "pmucalm coll", then you might be hitting the bug 7197637.

Refer below metalink note
Bug 7197637: MEMORY LEAK LEADING TO ORA-4030 (KOH-KGHU SESSI,PMUCALM COLL) USING UTL_FILE

The affected Oracle versions with this bug are 10.2.0.4 and 11.1.0.6 and it is fixed in 11gR2.
Workaround: Do not open too many files.

NOTE that users might expose Bug 9928290 due to regression exposed in the bug fix 7197637 and it is advised to apply both patches to fix the issue. But patch 9928290 is available from version 10.2.0.4.4

If your database running in <= 10.2.0.4.4 then you need to choose one of the below options.

1. Upgrade the database to 10.2.0.5 (using patch 8202632) and apply patches 9928290, 7197637
or
2. Upgrade the database to 10.2.0.4.4 PSU (using patch 9352164) and apply patches 9928290, 7197637

There are many bugs caused by this error. Check below metalink note for known bugs and possible actions in various versions

OERR: ORA 4030 "out of process memory when trying to allocate %s bytes (%s,%s)"

Please note that there are different errors across the various versions. I would recommend checking the Look-up Tool to find the Actual Cause and Possible solutions and contact Oracle Support.

ORA-600/ORA-7445/ORA-700 Error Look-up Tool [Article ID 153788.1]
Master Note for Diagnosing ORA-7445 and Related Core Dumps [Article ID 1092855.1]

Regards,
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com

Error while loading shared libraries: libdb.so.2 - After Migrating REHL4 to REHL5

We have migrated Oracle Agile application from RedHat Linux 4 to RedHat Linux 5 and while starting up the Agile application we have received following error.

$ opmnctl startall
opmnctl: starting opmn and all managed processes...
================================================================================
opmn id=agile:6201
    1 of 2 processes started.

ias-instance id=plm922.etna.corp.intusurg.com
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
--------------------------------------------------------------------------------
ias-component/process-type/process-set:
    HTTP_Server/HTTP_Server/HTTP_Server/

Error
--> Process (index=1,uid=6575556334,pid=75645)
    failed to start a managed process after the maximum retry limit
    Log:
     /oracle/product/10.1.3.1/OracleAS/opmn/logs//HTTP_Server~1.log

From HTTP Serve log, we have found that Apache service failed while coming up. We have found below error message

/oracle/product/10.1.3.1/OracleAS/Apache/Apache/bin/apachectl startssl: execing httpd
/oracle/product/10.1.3.1/OracleAS Apache/Apache/bin/httpd: error while loading shared libraries: libdb.so.2: cannot open shared object file: No such file or directory

There is no shared library/usr/lib/ libdb.so.2 and found that the soft link is missing.

You may see this error with applications that are using Oracle Application services after cloning or migrating from REHL 4 to REHL5.

To resolve the issue follow the steps
1. Stop the Oracle Application Service
2. Login as root and create the soft link
   # ln -s /usr/lib/libgdbm.so.2.0.0 /usr/lib/libdb.so.2
3. Start the Oracle Application Service.

Regards,
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com

Could Not Get DeploymentManager When Deploying or Undploying Agile

When deploying and un-deploying Oracle Agile application, users may receive error "Could not get DeploymentManager”.

Problem: You will receive below error when Agile unable to deploy/un-deploy using admin_client.jar utility

Failed at "Could not get DeploymentManager".

This is typically the result of an invalid deployer URI format being supplied, the target server not being in a started state or incorrect authentication details being supplied.

More information is available by enabling logging -- please see the Oracle Containers for J2EE Configuration and Administration Guide for details.

Cause 1:- The deployment command uses opmn request port 6003 by default. You will receive the error If the install used different opmn port (6004)

Solution:- Replace the port from 6003 to 6004 in DeployAgilePLM.sh or UndeployAgilePLM.sh script

deployer:oc4j:[rmis]:opmn://host[:opmnPort]/[iASInstanceName] /oc4jInstanceName

Cause 2:- If the server name changed from fully qualified name to short name or vice versa then admin_client.jar utility unable to deploy and you will receive the error.

Solution:- Replace fully qualified name to short name or vice versa in DeployAgilePLM.sh or UndeployAgilePLM.sh script and run the script.

In some cases, the application may hang with following error message when deploying the Agile Application. It is due to the version mismatch or Admin Data Schema Version not present

07/12/11 06:26:30 Notification ==>Initialize link ends...
07/12/11 06:26:30 Notification ==>Initialize integration begins...
07/12/11 06:26:30 Notification ==>Initialize integration ends...
07/12/11 06:26:30 Notification ==>Initialize JavaClient begins...
07/12/11 06:26:30 Notification ==>Initialize JavaClient ends...
07/12/11 06:26:30 Notification ==>Started application : Agile
07/12/11 06:26:30 Notification ==>Binding web application(s) to site default-web-site begins...
07/12/11 06:26:30 Notification ==>Binding application web-module for application Agile to site default-web-site under context root Agile
07/12/11 06:26:31 Notification ==>Initializing Servlet: com.agile.ui.pcm.InitServlet for web application application

You will see below message in default_group_home_default_group_1.log

07/12/11 13:26:32 ++++++++++++++++++++++++++++++++++++++++++++++++
07/12/11 13:26:32 + +
07/12/11 13:26:32 + Admin Data Schema Version NOT PRESENT!!!
07/12/11 13:26:32 + Admin Server Schema Version : 9.3.185.0
07/12/11 13:26:32 + +
07/12/11 13:26:32 ++++++++++++++++++++++++++++++++++++++++++++++++

Solution: Install the correct version or enter correct database information in agile.properties files, if version mismatch. Please refer Note 569229.1 for more information.

Please refer the link How to Deploy and Un-deploy Agile application.

Regards,
Satishbabu Gunukula
http://www.oracleracexpert.com

How to Deploy and Un-deploy Oracle Agile Application

Whenever you are applying the patches to Oracle Agile environment, you need to Un-deploy and deploy the application.

Before you apply the patch you must Un-deploy the Agile application. After successful installation of the patch you must deploy the agile application in order to take effect of the applied patch.

Un-Deploy Agile application:-Go to Agile home directory and run below command
$ cd $/agileDomain/bin
$./ UndeployAgilePLM.sh

Where password is the oc4jadmin user password

12/02/09 14:04:10 Notification ==>Application UnDeployer for Agile STARTS.
12/02/09 14:04:11 Notification ==>Removing all web binding(s) for application Agile from all web site(s)
12/02/09 14:04:19 Notification ==>Application UnDeployer for Agile COMPLETES.

Follow the steps to apply a Patch in Oracle Agile environment
1. Check PMN processes are running on the Agile Server.

2. Stop the OPMN processes
$ opmnctl stopall

3. Unzip the patch to a temporary directory
For Unix
$/tmp//Install_Patch.sh
For Windows
C:\temp\\Install_Patch.bat

You will get a message “Is the Agile Application already undeployed? (y/n) “. You must enter Y inorder to proceed further.

Ignore the message "Enter Passphrase for keystore:" and let the installation continue

When the installation is completed, the “INSTALLATION SUCCESSFUL” message will appear and press the return key to complete the installation.

4. Start OPMN processes
$ opmnctl startall

Deploy Agile application:-After successful installation of the patch deploy the Agile application

$ cd $/agileDomain/bin
$./DeployAgilePLM.sh

Where password is the oc4jadmin user password
12/02/09 14:06:50 Notification ==>Application Deployer for Agile STARTS.
………………………
……………………….
12/02/09 14:07:35 Notification ==>Binding web application(s) to site default-web-site ends...

12/02/09 14:07:35 Notification ==>Application Deployer for Agile COMPLETES. Operation time: 45434 msecs

Check the status of the agile application
$ opmnctl status

Make sure to check applied patch from application. If Deploy and Un-deploys fails with some reason then you will not see the applied patch in Application.

. Launch the application and login
. Click on help --> About Agile
. You should see the patch number under “updated versions”

Regards,
Satishbabu Gunukula
http://www.oracleracexpert.com