Friday, December 19, 2014

ORA-27102: out of memory Linux-x86_64 Error: 28: No space left on device

When trying to start-up a database received below error on a REHL 5.x

ORA-27102: out of memory
Linux-x86_64 Error: 28: No space left on device 


By looking the error, you might think that it is a space issue. But note that the actual error is “Out of Memory”.

You can also check available memory on the server and you should see enough FREE memory.

$free -g
total used free shared buffers cached
Mem: 125 80 45 0 0 88
-/+ buffers/cache: 36 89
Swap: 23 2 21

The server has enough memory but out of shared memory pages that can be used system wide. You should increase the kernel.shmall value to fix the issue.

The default size for SHMALL in Red Hat Enterprise Linux 2.1, 3, 4 and 5 is 2097152. Refer My Oracle Support document ID 169706.1 for recommended values.

Once you change the value you no need to reboot the server, you should reload the system configuration file as root using below command

$ sysctl –p

Hope this should resolve your issue

Regards
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com

Tuesday, December 2, 2014

Recovery Manager(RMAN) New Features in Oracle Database 12c - Part2

Hi Everyone,

My articles published in Oracle Experts website media "Allthingsoracle.com" by RedGate

Please view the article using below link.

Oracle Database 12c – RMAN New Features: Part2

In this article I will cover:
  • Multisection Backup Improvements
  • Restoring and Recovering Files Over Network
  • Storage Snapshot Optimization
  • Active Database Duplication Improvements
This article will help all Oracle Community to understand new enhancements and additions in Recovery Manager (RMAN) and take advantage of new features for efficient backup & recovery.

I hope you will like the article and it will be helpful to you.

Please leave your valuable comments.

Regards,
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com

ORA-27300: OS system dependent operation:fork failed with status: 11

I have encounter below error on all the databases on a shared server.

Errors from alert_<SID>.log

Process startup failed, error stack:
Errors in file /u01/home/oracle/diag/rdbms/ORA1/ORA1/trace/ORA1_psp0_18712.trc:
ORA-27300: OS system dependent operation:fork failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
ORA-27302: failure occurred at: skgpspawn5
Mon Nov 10 22:00:10 2014
Process m000 died, see its trace file


I see that m000 process dies and oracle has problem forking more processes.

Trace file ORA1_ psp0_18712.trc contents

*** 2014-11-10 02:04:12.780
*** SESSION ID:(79.1) 2014-11-10 02:04:12.780
*** CLIENT ID:() 2014-11-10 02:04:12.780
*** SERVICE NAME:(SYS$BACKGROUND) 2014-11-10 02:04:12.780
*** MODULE NAME:() 2014-11-10 02:04:12.780
*** ACTION NAME:() 2014-11-10 02:04:12.780

Dump diagnostics for process W000 pid 9442 which did not start after 120 seconds:
(spawn_time:x376D2DDDA now:x376D4B38A diff:x1D5B0)

*** 2014-11-10 02:04:12.855
Process diagnostic dump for W000, OS id=9442
-------------------------------------------------------------------------------
*** 2014-11-10 02:04:29.700
loadavg : 368.16 219.45 107.21
Memory (Avail / Total) = 295.09M / 128955.82M
Swap (Avail / Total) = 0.00M / 24480.54M
skgpgcmdout: read() for cmd /bin/ps -elf | /bin/egrep 'PID | 9442' | /bin/grep -v grep timed out after 15.000 seconds
Stack:
ERROR: process 9442 is not alive
………
Killing process (ospid 21743): did not start after 120 seconds
... and the process is still alive after kill!
*** 2014-11-10 03:07:19.266
Killing process (ospid 23399): requester cancelled request
... and the process is still alive after kill!
Killing process (ospid 23780): requester cancelled request
*** 2014-11-10 03:12:33.255
.. and the process is still alive after kill!
*** 2014-11-10 03:13:33.536
Killed process 23780 is still alive after 60 seconds!
*** 2014-11-10 03:15:24.608
Killing process (ospid 23889): requester cancelled request
………………….

*** 2014-11-10 22:00:09.662
Process startup failed, error stack:
ORA-27300: OS system dependent operation:fork failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
ORA-27302: failure occurred at: skgpspawn5


It looks like the maximum number of PROCESSES allowed per user is low. Check the ulimit –a to see the values

$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 1031617
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 32800
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 2047
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited


I see that the max user process set to 2047 and when I check the total number of process it is reaching the limit. We need to increase the limit to fix the issue

For Linux
Increase the nproc in /etc/security/limits.conf file

For Solaris 10
Increase the project.max-lwps value

For HP and Solaris
Increase the maxuprc

For AIX
Increase maxuproc

I didn’t see the issue or error in the alert.log file.

Regards
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com

Friday, November 14, 2014

Oracle GoldenGate 12c New Features - Part1


Hi Everyone,

My articles published in Oracle Experts website media "Allthingsoracle.com" by RedGate

Please see the article using below link.

Oracle Golden Gate 12c New Features – Part 1

In this article I will cover:
  • Expanded heterogeneous Support
  • Multitenant Container Database (CDB) Support
  • Oracle Universal Installer (OUI) Support
  • Support for Public and Private Clouds
  • Integrated Replicat
This article will help all Oracle Community to understand Oracle GoldenGate 12c new features and how it helps in continuous availability, disaster tolerance and real-time data integration solutions that enable the management and movement of transactional data across the enterprise.

I hope you will like the article and it will be helpful to you.

Please leave your valuable comments.

Regards,
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com

ORA-04030: out of process memory when trying to allocate 16328 bytes (koh-kghu call ,pmucalm coll)

User may encounter below error in 11g. First I would recommend running ADDM report and checking how the resources are utilized. If you see any undersize database configuration then increase.

ORA-04030: out of process memory when trying to allocate 16328 bytes (koh-kghu call ,pmucalm coll)
Incident details in: /u01/home/oracle/diag/rdbms/orac/ORAC/incident/incdir_17129/ORAC_ora_8372_i17129.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Non critical error ORA-48913 caught while writing to trace file "/u01/home/oracle/diag/rdbms/orac/ORAC/incident/incdir_17129/ORAC_ora_8372_i17129.trc"
Error message: ORA-48913: Writing into trace file failed, file size limit [5242880] reached
Writing to the above trace file is disabled for now on...

If your database has enough resources and if you still see the issue then you might need to check the trace file.

=======================================
PRIVATE MEMORY SUMMARY FOR THIS PROCESS
---------------------------------------
******************************************************
PRIVATE HEAP SUMMARY DUMP
4129 MB total:
4129 MB commented, 173 KB permanent
57 KB free (0 KB in empty extents),
2240 MB, 3 heaps: "koh-kghu call " 14 KB free held
1889 MB, 1 heap: "session heap " 4 KB free held
------------------------------------------------------


The trace file confirms that the process is limiting at 4GB. Change the upper limit either at OS level or Database level

Apply this change at OS level
1. Get the page count using below command and increase the limit
#more /proc/sys/vm/max_map_count

2. Increase the page count
#sysctl -w vm.max_map_count= 262144
or
#sysctl vm.max_map_count=262144

3. These changes will be revered on server reboot. To make changes permanent add the value in
$ vi /etc/sysctl.conf
or
$echo “vm.max_map_count = 262144” | tee -a /etc/sysctl.conf

4. You no need to reboot the server, you should reload the config file as root
$ sysctl –p

Apply this change at Database level
Adjust the realfree heap page size by setting below parameters in spfile or pfile and restart the database
_use_realfree_heap=TRUE
_realfree_heap_pagesize_hint = 262144

Note that the default realfree allocator pagesize is 64 kB (65536), so 64K entries take up 4GB. With 256kB (262144) pages, the limit goes to 16GB.

I hope this should resolve your issue.

Regards
Satishbabu Gunukula, Oracle ACE
http://www.oracleracexpert.com