ORA-03114: not connected to ORACLE (DBD ERROR: OCIStmtExecute/Describe)

Cause:

This error occur in ASM when diskmon resource is down

This is exadata test setup and this error is not seen in a real production system

ASMCMD> lsdg
ORA-03114: not connected to ORACLE (DBD ERROR: OCIStmtExecute/Describe)
[oracle@exdbadm01 ~]$ asmcmd
Connected to an idle instance.

There is no detail with error code

[oracle@exdbadm01 ~]$ oerr ora 03114
03114, 00000, "not connected to ORACLE"
// *Cause:
// *Action:

Solution:

When i check the status of asm,it is up and running

[oracle@exdbadm01 log]$ srvctl status asm
ASM is running on exdbadm01

But still i am unable to view the diskgroup details under asmcmd

ASMCMD> lsdg
ASMCMD>

After checking the cluster resources,diskmon resource was down

[oracle@exdbadm01 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  OFFLINE      exdbadm01                STARTING
ora.FRA.dg
               ONLINE  OFFLINE      exdbadm01                STARTING
ora.LISTENER.lsnr
               ONLINE  ONLINE       exdbadm01
ora.asm
               ONLINE  ONLINE       exdbadm01                Started
ora.ons
               OFFLINE OFFLINE      exdbadm01
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
      1        ONLINE  ONLINE       exdbadm01
ora.diskmon
      1        OFFLINE OFFLINE
ora.evmd
      1        ONLINE  ONLINE       exdbadm01

When i checked the alertlog , i can see the error messages of diskmon not being loaded

01/23/21 19:34:27: Master Diskmon starting
skgzib_load_ib_symbols: libibmad lib file (/usr/lib64/libibmad.so) does not exists.

skgzib_ini: Fail to load infiniband symbols.

dskm_skgzib_ini: required inifiniband software is not found (libibmad, libibumad). SM Query will be disabled.

01/24/21 05:34:11: Master Diskmon starting
skgzib_load_ib_symbols: libibmad lib file (/usr/lib64/libibmad.so) does not exists.

skgzib_ini: Fail to load infiniband symbols.

dskm_skgzib_ini: required inifiniband software is not found (libibmad, libibumad). SM Query will be disabled.

01/25/21 18:38:08: Master Diskmon starting

Then i started the diskmon resource manually

[oracle@exdbadm01 ~]$ crsctl start res ora.diskmon
CRS-2672: Attempting to start 'ora.diskmon' on 'exdbadm01'
CRS-2676: Start of 'ora.diskmon' on 'exdbadm01' succeeded

Now i can able to view the diskgroups in asmcmd

ASMCMD> lsdg
State    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  EXTERN  N         512   4096  4194304      6144     5976                0            5976              0             N  DATA/
MOUNTED  EXTERN  N         512   4096  4194304      6144     5976                0            5976              0             N  FRA/

In alertlog ,there are heartbeat messages between ASM and storage cell using diskmon

2021-01-25 18:38:12.426: [ DISKMON][7776:552761664] dskm_process_msg5: received msg type KGZM_PING (0x0011)
2021-01-25 18:38:15.430: [ DISKMON][7776:552761664] dskm_process_msg5: received msg type KGZM_PING (0x0011)
2021-01-25 18:38:18.435: [ DISKMON][7776:552761664] dskm_process_msg5: received msg type KGZM_PING (0x0011)
2021-01-25 18:38:21.439: [ DISKMON][7776:552761664] dskm_process_msg5: received msg type KGZM_PING (0x0011)

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s