error in invoking target 'agent nmhs' of makefile

I come across a situation that servers & Operating systems are (O/S) are getting upgraded and we cannot perform in place upgrade for easy roll backing capability.

Also new O/S does not support the Database version.

We need to restore the Oracle database into unsupported Server version. We have copied the Database binaries and during the Database upgrade we got below error.

error in invoking target 'agent nmhs' of makefile

Workaround:
During upgrade or clone  you will get this error when relink is running

Here is the workaround 
ls $ORACLE_HOME/sysman/lib/ins_emagent.mk

Search for the line 
$(MK_EMAGENT_NMECTL)
Change it to:
$(MK_EMAGENT_NMECTL) -lnnz11

https://community.oracle.com/thread/1093616

Goto Database upgrade window and click on "Retry" and upgrade should run successfully.

Regards
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com

ORA-27300: OS system dependent operation:fork failed with status: 11

I have encounter below error on all the databases on a shared server.

Errors from alert_<SID>.log

Process startup failed, error stack:
Errors in file /u01/home/oracle/diag/rdbms/ORA1/ORA1/trace/ORA1_psp0_18712.trc:
ORA-27300: OS system dependent operation:fork failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
ORA-27302: failure occurred at: skgpspawn5
Mon Nov 10 22:00:10 2014
Process m000 died, see its trace file

I see that m000 process dies and oracle has problem forking more processes.

Trace file ORA1_ psp0_18712.trc contents

*** 2014-11-10 02:04:12.780
*** SESSION ID:(79.1) 2014-11-10 02:04:12.780
*** CLIENT ID:() 2014-11-10 02:04:12.780
*** SERVICE NAME:(SYS$BACKGROUND) 2014-11-10 02:04:12.780
*** MODULE NAME:() 2014-11-10 02:04:12.780
*** ACTION NAME:() 2014-11-10 02:04:12.780

Dump diagnostics for process W000 pid 9442 which did not start after 120 seconds:
(spawn_time:x376D2DDDA now:x376D4B38A diff:x1D5B0)

*** 2014-11-10 02:04:12.855
Process diagnostic dump for W000, OS id=9442
-------------------------------------------------------------------------------
*** 2014-11-10 02:04:29.700
loadavg : 368.16 219.45 107.21
Memory (Avail / Total) = 295.09M / 128955.82M
Swap (Avail / Total) = 0.00M / 24480.54M
skgpgcmdout: read() for cmd /bin/ps -elf | /bin/egrep 'PID | 9442' | /bin/grep -v grep timed out after 15.000 seconds
Stack:
ERROR: process 9442 is not alive
………
Killing process (ospid 21743): did not start after 120 seconds
... and the process is still alive after kill!
*** 2014-11-10 03:07:19.266
Killing process (ospid 23399): requester cancelled request
... and the process is still alive after kill!
Killing process (ospid 23780): requester cancelled request
*** 2014-11-10 03:12:33.255
.. and the process is still alive after kill!
*** 2014-11-10 03:13:33.536
Killed process 23780 is still alive after 60 seconds!
*** 2014-11-10 03:15:24.608
Killing process (ospid 23889): requester cancelled request
………………….

*** 2014-11-10 22:00:09.662
Process startup failed, error stack:
ORA-27300: OS system dependent operation:fork failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
ORA-27302: failure occurred at: skgpspawn5

It looks like the maximum number of PROCESSES allowed per user is low. Check the ulimit –a to see the values

$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 1031617
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 32800
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 2047
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

I see that the max user process set to 2047 and when I check the total number of process it is reaching the limit. We need to increase the limit to fix the issue

For Linux
Increase the nproc in /etc/security/limits.conf file

For Solaris 10
Increase the project.max-lwps value

For HP and Solaris
Increase the maxuprc

For AIX
Increase maxuproc

I didn’t see the issue or error in the alert.log file.

Regards
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com

Error while loading shared libraries: libdb.so.2 - After Migrating REHL4 to REHL5

We have migrated Oracle Agile application from RedHat Linux 4 to RedHat Linux 5 and while starting up the Agile application we have received following error.

$ opmnctl startall
opmnctl: starting opmn and all managed processes...
================================================================================
opmn id=agile:6201
    1 of 2 processes started.

ias-instance id=plm922.etna.corp.intusurg.com
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
--------------------------------------------------------------------------------
ias-component/process-type/process-set:
    HTTP_Server/HTTP_Server/HTTP_Server/

Error
--> Process (index=1,uid=6575556334,pid=75645)
    failed to start a managed process after the maximum retry limit
    Log:
     /oracle/product/10.1.3.1/OracleAS/opmn/logs//HTTP_Server~1.log

From HTTP Serve log, we have found that Apache service failed while coming up. We have found below error message

/oracle/product/10.1.3.1/OracleAS/Apache/Apache/bin/apachectl startssl: execing httpd
/oracle/product/10.1.3.1/OracleAS Apache/Apache/bin/httpd: error while loading shared libraries: libdb.so.2: cannot open shared object file: No such file or directory

There is no shared library/usr/lib/ libdb.so.2 and found that the soft link is missing.

You may see this error with applications that are using Oracle Application services after cloning or migrating from REHL 4 to REHL5.

To resolve the issue follow the steps
1. Stop the Oracle Application Service
2. Login as root and create the soft link
   # ln -s /usr/lib/libgdbm.so.2.0.0 /usr/lib/libdb.so.2
3. Start the Oracle Application Service.

Regards,
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com

Could Not Get DeploymentManager When Deploying or Undploying Agile

When deploying and un-deploying Oracle Agile application, users may receive error "Could not get DeploymentManager”.

Problem: You will receive below error when Agile unable to deploy/un-deploy using admin_client.jar utility

Failed at "Could not get DeploymentManager".

This is typically the result of an invalid deployer URI format being supplied, the target server not being in a started state or incorrect authentication details being supplied.

More information is available by enabling logging -- please see the Oracle Containers for J2EE Configuration and Administration Guide for details.

Cause 1:- The deployment command uses opmn request port 6003 by default. You will receive the error If the install used different opmn port (6004)

Solution:- Replace the port from 6003 to 6004 in DeployAgilePLM.sh or UndeployAgilePLM.sh script

deployer:oc4j:[rmis]:opmn://host[:opmnPort]/[iASInstanceName] /oc4jInstanceName

Cause 2:- If the server name changed from fully qualified name to short name or vice versa then admin_client.jar utility unable to deploy and you will receive the error.

Solution:- Replace fully qualified name to short name or vice versa in DeployAgilePLM.sh or UndeployAgilePLM.sh script and run the script.

In some cases, the application may hang with following error message when deploying the Agile Application. It is due to the version mismatch or Admin Data Schema Version not present

07/12/11 06:26:30 Notification ==>Initialize link ends...
07/12/11 06:26:30 Notification ==>Initialize integration begins...
07/12/11 06:26:30 Notification ==>Initialize integration ends...
07/12/11 06:26:30 Notification ==>Initialize JavaClient begins...
07/12/11 06:26:30 Notification ==>Initialize JavaClient ends...
07/12/11 06:26:30 Notification ==>Started application : Agile
07/12/11 06:26:30 Notification ==>Binding web application(s) to site default-web-site begins...
07/12/11 06:26:30 Notification ==>Binding application web-module for application Agile to site default-web-site under context root Agile
07/12/11 06:26:31 Notification ==>Initializing Servlet: com.agile.ui.pcm.InitServlet for web application application

You will see below message in default_group_home_default_group_1.log

07/12/11 13:26:32 ++++++++++++++++++++++++++++++++++++++++++++++++
07/12/11 13:26:32 + +
07/12/11 13:26:32 + Admin Data Schema Version NOT PRESENT!!!
07/12/11 13:26:32 + Admin Server Schema Version : 9.3.185.0
07/12/11 13:26:32 + +
07/12/11 13:26:32 ++++++++++++++++++++++++++++++++++++++++++++++++

Solution: Install the correct version or enter correct database information in agile.properties files, if version mismatch. Please refer Note 569229.1 for more information.

Please refer the link How to Deploy and Un-deploy Agile application.

Regards,
Satishbabu Gunukula
http://www.oracleracexpert.com

/etc/profile[57]: ulimit: pipe: is read only

You need to set the “session limits for Oracle user” before you install any Oracle Database product.

When you try to login into “oracle” user you will receive following error

test@server1$ su - oracle
/etc/profile[57]: ulimit: pipe: is read only

Check the ulimit using below command

test@server1$ulimit -a
address space limit (kbytes) (-M) unlimited
core file size (blocks) (-c) unlimited
cpu time (seconds) (-t) unlimited
data size (kbytes) (-d) unlimited
file size (blocks) (-f) unlimited
locks (-L) unlimited
locked address space (kbytes) (-l) 32
nofile (-n) 65536
nproc (-u) 16384
pipe buffer size (bytes) (-p) 4096
resident set size (kbytes) (-m) unlimited
socket buffer size (bytes) (-b) 4096
stack size (kbytes) (-s) 10240
threads (-T) not supported
process size (kbytes) (-v) unlimited

The syntax that was previously used to set the "Max user processes limitation" is changed from "ulimit -p" to "ulimit -u", which is causing the issue.

Replace the session limits in /etc/profile with below code to resolve the issue.

if [ $USER = "oracle" ]; then
if [ $SHELL = "/bin/ksh" ]; then
ulimit -u 16384
ulimit -n 65536
else
ulimit -u 16384 -n 65536
fi
fi

Regards
Satishbabu Gunukula
http://www.oracleracexpert.com

ORA-27090: Message 27090 not found; product=RDBMS; facility=ORA

If you notice any one of these errors in Alert.log during startup then the issue is related to “aio-max-nr" kernel limit. You will see this error in 10.2.0.4

ORA-27090: Message 27090 not found; product=RDBMS; facility=ORA

ORA-27090: UNABLE TO RESERVE KERNEL RESOURCES FOR ASYNCHRONOUS DISK I/O

ORA-27090: Message 27090 not found; product=RDBMS; facility=ORA
Linux-x86_64 Error: 4: Interrupted system call

You will encounter this issue when the "aio-max-nr" kernel limit is too low.

Here is how you can check the value
#/sbin/sysctl -a |grep aio

Solution : Change the "aio-max-nr" kernel limit as per Oracle recommendation
fs.aio-max-nr= 3145728

Regards,
Satishbabu Gunukula
http://www.oracleracexpert.com

Set DISPLAY variable & Enable access control

Before starting the Oracle Universal Installer, the DISPLAY environment must be set correctly for display terminal support. To set the DISPLAY environment run the following command where hostname is the hostname or IP address of the system where the X server is running:

$ export DISPLAY=hostname:0.0

It may be necessary to enable access control on the system where the X server is running, so that the clients can connect. The "xhost" command is used to set access controls. The xhost command must be run on the console of the system where the X server is running. It cannot be done remotely.

To enable access control so that clients from any host can connect to the X server, type the following command:
$ xhost +
access control disabled, clients can connect from any host

To enable access control only to certain clients run the following command
$ xhost +hostname
For ex:- $ xhost +server1
Server1 being added to access control list

Common Errors:-
1. Error: Can't open display
    Error: Couldn't find per display Information

    Solution: 1. Run xhost + to enable access control
                    2. Set your DISPLAY

Regards,
Satishbabu Gunukula
http://www.oracleracexpert.com/