Thursday, May 7, 2015

Terminating the instance due to error 16038 and unable to startup

I have encountered storage crash on one of the Database system. After storage is back online, not able to bring up the database. The database comes to mount stage and terminating the instance.

We are receiving below error.

ORA-03113: end-of-file on communication channel
Process ID: 5038
Session ID: 202 Serial number: 3

When I look into the alert.log I found below error messages

ARC2: Archival started
ARC1: Becoming the 'no FAL' ARCH
ARC2: Becoming the heartbeat ARCH
ARC3: Archival started
Tue Mar 10 10:24:10 2015
Incomplete read from log member '/oracle/ORAC/datafile/ORAC/onlinelog/ORAC_Redo1_.log'. Trying next member.
ARCH: All Archive destinations made inactive due to error 333
ARCH: Closing local archive destination LOG_ARCHIVE_DEST_1: '/oracle/ORAC/archiveORAC_ARCH_347548987.dbf' (error 333) (ORAC)
Committing creation of archivelog '/oracle/ORAC/archive/1_3541_854889367.dbf' (error 333)
Errors in file /oracle/diag/rdbms/orac/ORAC/trace/ORAC_ora_23498.trc:
ORA-16038: log 1 sequence# 3541 cannot be archived
ORA-00333: redo log read error block count
ORA-00312: online log 1 thread 1: '/oracle/ORAC/datafile/ORAC/onlinelog/ORAC_Redo1_.log'

USER (ospid: 23498): terminating the instance due to error 16038
Tue Mar 10 10:24:12 2015
System state dump requested by (instance=1, osid=23498), summary=[abnormal instance termination].
System State dumped to trace file /oracle/diag/rdbms/orac/ORAC/trace/ORAC_diag_23470.trc
Dumping diagnostic data in directory=[cdmp_20150310102412], requested by (instance=1, osid=23498), summary=[abnomal instance termination].
Instance terminated by USER, pid = 23498

By looking the error, I can see that one of the online redo is corrupted. Archive process cannot able to archive to disk. In order to bring up the database you need to clear the corrupted redo log.

From ALERT.LOG file, I am able to identify that group 1 redo log 1 corrupted. By using below command I have cleared the unarchived log file

SQL> alter database clear unarchived logfile ‘/oracle/ORAC/datafile/ORAC/onlinelog/ORAC_Redo1_.log'

If you find all multiplexed redo members are corrupted then you need to run below command to clear the GROUP.

SQL> alter database clear unarchived logfile group 1;

Now I am able to bring up the database. But note that you cannot able to bring the database back beyond corrupted point using old backups. Now you must take one FULL database backup.

Satishbabu Gunukula, Oracle ACE


  1. The blog or and best that is extremely useful to keep I can share the ideas
    of the future as this is really what I was looking for, I am very comfortable and pleased to come here. Thank you very much.
    tanki online | 2048 game|

  2. شركه نقل عفش بمكه
    شركه نقل عفش بالطائف
    وفي منطقة المدينة المنورة تجدنا نمتلك افضل شركات نقل الاثاث مع الفك والتركيب وكذلك في ينبع وينبع البحر مثال عن تلك الشركات
    شركه نقل عفش بالمدينه المنوره
    شركه نقل اثاث بينبع
    وفي المنطقة الوسطي نقدم خدمات نقل العفش مع الفك والتركيب والتغليف ويتم النقل في سيارات مجهزة ومؤمنة لتحميل العفش وتفادي التلفيات في الرياض والقصيم والخرج مثل
    - شركات نقل الاثاث بالرياض
    - شركات نقل الاثاث بالقصيم
    - شركات نقل الاثاث بالخرج
    ايضا في المنطقة الشرقية نمتلك مجموعة فروع خاصة بنقل العفش في كلا من الرياض والدمام والخبر والقطيف والجبيل والاحساء والظهران مثل
    شركة نقل اثاث بالدمام
    شركة نقل اثاث بالخبر