ORA-16664: unable to receive the result from a database with ORA-12514

We have received below error when trying to enabling Data guard Broker configuration

Error: ORA-16664: unable to receive the result from a database
Dataguard broker log has below errors

Failed to connect to remote database sat. Error is ORA-12514
Failed to send message to member sat. Error code is ORA-12514.

To identify the issue review the confirmation

DGMGRL> SHOW CONFIGURATION;
Configuration - orcl_dgbroker_fsfconf
Protection Mode: MaxAvailability
Members:
orcl - Primary database
orcl_stdby - Physical standby database
Error: ORA-16664: unable to receive the result from a member

Fast-Start Failover: DISABLED

DGMGRL> show database verbose 'ORCL';
Database - ORCL
Role: PRIMARY
Intended State: TRANSPORT-ON
Instance(s):
ORCL
Properties:
DGConnectIdentifier = 'ORCL'
ObserverConnectIdentifier = ''
LogXptMode = 'SYNC'
RedoRoutes = ''
DelayMins = '0'
Binding = 'optional'
MaxFailure = '0'
MaxConnections = '1'
ReopenSecs = '300'
NetTimeout = '30'
RedoCompression = 'DISABLE'
LogShipping = 'ON'
PreferredApplyInstance = ''
ApplyInstanceTimeout = '0'
ApplyLagThreshold = '30'
TransportLagThreshold = '30'
TransportDisconnectedThreshold = '30'
ApplyParallel = 'AUTO'
ApplyInstances = '0'
StandbyFileManagement = 'AUTO'
ArchiveLagTarget = '0'
LogArchiveMaxProcesses = '4'
LogArchiveMinSucceedDest = '1'
DataGuardSyncLatency = '0'
DbFileNameConvert = ''
LogFileNameConvert = 'dummy, dummy'
FastStartFailoverTarget = ''
InconsistentProperties = '(monitor)'
InconsistentLogXptProps = '(monitor)'
SendQEntries = '(monitor)'
LogXptStatus = '(monitor)'
RecvQEntries = '(monitor)'
PreferredObserverHosts = ''
StaticConnectIdentifier = '(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=node1.oracleracexpert.com)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=ORCL_DGMGRL)(INSTANCE_NAME=ORCL)(SERVER=DEDICATED)))'
StandbyArchiveLocation = '/oraarch/ORCL'
AlternateLocation = ''
LogArchiveTrace = '0'
LogArchiveFormat = 'arch_ORCL_%r_%s_%t.arc'
TopWaitEvents = '(monitor)'

Log file locations:
Alert log : /oracle/diag/rdbms/ORCL/ORCL/trace/alert_ORCL.log
Data Guard Broker log : /oracle/diag/rdbms/ORCL/ORCL/trace/drcORCL.log

Database Status:
SUCCESS

DGMGRL> show database verbose 'ORCL_stdby';
Database - ORCL_stdby
Role: PHYSICAL STANDBY
Intended State: APPLY-ON
Transport Lag: (unknown)
Apply Lag: (unknown)
Average Apply Rate: (unknown)
Active Apply Rate: (unknown)
Maximum Apply Rate: (unknown)
Real Time Query: OFF
Instance(s):
ORCL
Properties:
DGConnectIdentifier = 'ORCL_stdby'
ObserverConnectIdentifier = ''
LogXptMode = 'SYNC'
RedoRoutes = ''
DelayMins = '0'
Binding = 'OPTIONAL'
MaxFailure = '0'
MaxConnections = '1'
ReopenSecs = '300'
NetTimeout = '30'
RedoCompression = 'DISABLE'
LogShipping = 'ON'
PreferredApplyInstance = ''
ApplyInstanceTimeout = '0'
ApplyLagThreshold = '30'
TransportLagThreshold = '30'
TransportDisconnectedThreshold = '30'
ApplyParallel = 'AUTO'
ApplyInstances = '0'
StandbyFileManagement = 'AUTO'
ArchiveLagTarget = '0'
LogArchiveMaxProcesses = '4'
LogArchiveMinSucceedDest = '1'
DataGuardSyncLatency = '0'
DbFileNameConvert = ''
LogFileNameConvert = 'dummy, dummy'
FastStartFailoverTarget = ''
InconsistentProperties = '(monitor)'
InconsistentLogXptProps = '(monitor)'
SendQEntries = '(monitor)'
LogXptStatus = '(monitor)'
RecvQEntries = '(monitor)'
PreferredObserverHosts = ''
StaticConnectIdentifier = '(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)HOST=node2.oracleracexpert.com)(PORT=1521))(CONNECT_DATA=SERVICE_NAME=ORCL_STDBY_DGMGRL)(INSTANCE_NAME=ORCL)SERVER=DEDICATED)))'
StandbyArchiveLocation = '/oraarch/ORCL'
AlternateLocation = ''
LogArchiveTrace = '0'
LogArchiveFormat = 'arch_ORCL_%r_%s_%t.arc'
TopWaitEvents = '(monitor)'

Log file locations:
(Failed to retrieve log file locations.)

Database Status:
DGM-17016: failed to retrieve status for database "ORCL_stdby"
ORA-16664: unable to receive the result from a member

Solution:

1. Makes sure you have Static listener entry has SID_DGMGRL

cat $ORACLE_HOME/network/admin/listener.ora

(SID_DESC =
(GLOBAL_DBNAME = ORCL_DGMGRL) <----------------add _DGMGRL
(ORACLE_HOME = /<oracle_home>)
(SID_NAME = ORCL)
)

2. Same way change the static entry of current primary also for feature role transition. For non-default port set the local listener on the respective database where ORA-12514 was thrown.

SQL>alter system set local_listener='(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=<hostIP>)(PORT=port)))';

Refer Below oracle notes for more scenarios
1. Causes and Solutions for DGMGRL ORA-16664 (Data Guard Broker) (Doc ID 2494260.1) 

Regards

Satishbabu Gunukula

http://www.oracleracexpert.com

RMAN-05001: auxiliary file name conflicts with a file used by the target database and RMAN-05501

The Duplicate db for standby failed with below error

RMAN-05501: aborting duplication of target database
RMAN-05001: auxiliary file name /oradata1/ORCL/system01.dbf conflicts with a file used by the target database
The duplicate script will look like below…

RMAN> run {
set until time "to_date('20018/12/08 14:00:00','yyyy/mm/dd HH24:MI:SS')";
allocate auxiliary channel aux1 type 'sbttape';
allocate auxiliary channel aux2 type 'sbttape';
duplicate target database for standby dorecover;
}

If primary and secondary has different directory structure then you will not see this issue. This is happening because the primary and secondary datafile directory structures are same.

To skip this check, use 'nofilenamecheck' clause. This will instructs rman not to check whether target database file names share the same name as auxiliary.

Now the script will look like below and you should not see this error.

RMAN> run {
set until time "to_date('20018/12/08 14:00:00','yyyy/mm/dd HH24:MI:SS')";
allocate auxiliary channel aux1 type 'sbttape';
allocate auxiliary channel aux2 type 'sbttape';
duplicate target database for standby dorecover nofilenamecheck;
}

Thanks

Satishbabu Gunukula, Oracle ACE

http://www.oracleracexpert.com

ORA-16053: DB_UNIQUE_NAME ORCL_STBY is not in the Data Guard Configuration with ORA-02097 error

The ORA-02097 is very generic error you need to check following error message to understand the issue.

ORA-02097: parameter cannot be modified because specified value is invalid
ORA-16053: DB_UNIQUE_NAME ORCL _STBY is not in the Data Guard Configuration
In my case I am trying to update DB_UNIQUE_NAME but it is failed, because the specified DB_UNIQUE_NAME is not in the Data Guard Configuration.

Before you update DB_UNIQUE_NAME parameter make sure LOG_ARCHIVE_CONFIG is enabled and it has valid DB_UNIQUE_NAME.

ALTER SYSTEM SET LOG_ARCHIVE_CONFIG='DG_CONFIG=(ORCL,ORCL_STDBY)'

You can find list of valid DB_UNIQUE_NAMEs in V$DATAGUARD_CONFIG view

Thanks,
Satishbabu Gunukula, Oracle ACE
http://oracleracexpert.com

ORA-00060: deadlock resolved & ORA-12012: error on auto execute of job

We have recently upgraded a dataset to 12c and started receiving below errors in alert.log

ORA-00060: deadlock resolved; details in file /home/oracle/diag/rdbms/ORCL/ORCL/trace/ORCL_j001_200521.trc
ORA-12012: error on auto execute of job "SYS"."ORA$AT_SA_SPC_SY_5723"
ORA-00060: deadlock detected while waiting for resource < ORA-06512: at "SYS.DBMS_SPACE", line 2741 < ORA-06512: at "SYS.DBMS_HEAT_MAP_INTERNAL", line 716 < ORA-06512: at "SYS.DBMS_HEAT_MAP_INTERNAL", line 1164 < ORA-06512: at "SYS.DBMS_HEAT_MAP", line 228 < ORA-06512: at "SYS.DBMS_SPACE", line 2747

After investigation found that there is a bug in 12.2 Release and Oracle working to it. You can refer below oracle support notes on this.

Bug 24687075 - SPACE ADVISOR TASKS/JOBS HITTING DEADLOCKS WITH GATHER DB STATS JOBS
In 12.2 Auto Space Advisor Job Fails With ORA-60 (Doc ID 2321020.1)

If you have setup alerts for ORA- errors then either you add exclusions to your script or you can run below command to disable the job Until Bug 24687075 gets fixed

SQL> EXEC DBMS_AUTO_TASK_ADMIN.DISABLE('AUTO SPACE ADVISOR',NULL, NULL);

Regards

Satishbabu Gunukula, Oracle ACE

https://oracleracexpert.com

ORA-04030: out of process memory when trying to allocate 4024 bytes (kxs-heap-b,kghsseg_1 : kokeibib

Normally users see ORA-04030 error with shortage of RAM or small PGA and kernel parameter config.

ORA-04030: out of process memory when trying to allocate 4024 bytes (kxs-heap-b,kghsseg_1 : kokeibib)
Non critical error ORA-48913 caught while writing to trace file "/u01/home/oracle/diag/rdbms/ORCL/ORCL/incident/incdir_8937/ORCL_ora_11258_i8937.trc"
Error message: ORA-48913: Writing into trace file failed, file size limit [5242880] reached
ORA-04030: out of process memory when trying to allocate 169040 bytes (pga heap,kgh stack)
ORA-04030: out of process memory when trying to allocate 4024 bytes (kxs-heap-b,kghsseg_1 : kokeibib)
Non critical error ORA-48913 caught while writing to trace file "/u01/home/oracle/diag/rdbms/ORCL/ORCL/incident/incdir_8938/ORCL_ora_11258_i8938.trc" Error message: ORA-48913: Writing into trace file failed, file size limit [5242880] reached

Users can refer metalink Note 233869.1 titled "Diagnosing and Resolving ORA-4030 errors"

By looking the error it is clear that “out of process memory”, where oracle cannot get the required RAM. But note that you need to analyze the trace to find out the root cause

In my case I found that one of the un-optimized PL/SQL job consuming the resources and caused the issue.

Current SQL Statement for this session (sql_id=gjn38xnrxcfbj) -----
BEGIN orcl_server_bom_script.all_level(:1 , :2 , :3 , :4 , :5 , :6 , :7 , :8 , :9 , :10 , :11 , :12 , :13 , :14 , :15 , :16 , :17 , :18 , :19 , :20 , :21 ) ; END;
57% pid 67: 4089 MB used of 4092 MB allocated <= CURRENT PROC
Dump of Real-Free Memory Allocator Heap [0x7f2c640b4000]
mag=0xfefe0001 flg=0x5000003 fds=0x6 blksz=65536
blkdstbl=0x7f2c640b4010, iniblk=66560 maxblk=524288 numsegs=115
In-use num=65424 siz=4290117632, Freeable num=0 siz=0, Free num=1 siz=20512768

In our case max_map_count value is not enough and needs to investigate why the pl/sql processes requires such a large amount of memory. In this case adjusting PGA_AGGREGATE_TARGET or MEMORY_TARGET will not limit the size processes can grow and will not help.

To fix the issue increase the value of max_map_count. To find out current vale of max_map_count run below command

$cat /proc/sys/vm/max_map_count
65530

To change the value run below command
sysctl -w vm.max_map_count= <greater than 65530>

Follow below metalink note to change the value of vm.max_map_count.
Modification of "vm.max_map_count" on Oracle Linux 6 (Doc ID 2195159.1) 

Regards

Satishbabu Gunukula, Oracle ACE

http://oracleracexpert.com