Skip to content

How to Restart Dead Dispatchers Manually

  • by
Theoretically, PMON will clean up dead process automatically, but it sometimes fails to do it in 9i databases like the error message in the alert log:
...
Thu Mar  3 14:12:50 2011
Errors in file /oracle/admin/ORCL/udump/orcl_ora_15969.trc:
ORA-00600: internal error code, arguments: [kjpcre2], [], [], [], [], [], [], []
Thu Mar  3 14:12:50 2011
Errors in file /oracle/admin/ORCL/udump/orcl_ora_15969.trc:
ORA-00603: ORACLE server session terminated by fatal error
ORA-00600: internal error code, arguments: [kjpcre2], [], [], [], [], [], [], []
...
Thu Mar 3 17:50:43 2012
Errors in file /oracle/admin/ORCL/udump/orcl_ora_15979.trc:
ORA-00600: internal error code, arguments: [kjpcre2], [], [], [], [], [], [], []
Thu Mar 3 17:50:43 2012
Errors in file /oracle/admin/ORCL/udump/orcl_ora_15979.trc:
ORA-00603: ORACLE server session terminated by fatal error
ORA-00600: internal error code, arguments: [kjpcre2], [], [], [], [], [], [], []
...

ORA-00600 is a general and internal error code which will show the arguments to indicate the causes of the problem. The argument "kjpcre2" indicates that PMON takes too long to clean up a dead process. But which processes are dead?
SQL> select inst_id, spid, username, program from gv$process where spid in ('15979','15969');

   INST_ID SPID         USERNAM PROGRAM
---------- ------------ ------- ---------------------------------------
         1 15969        oracle  oracle@ORCL (D000)
         1 15979        oracle  oracle@ORCL (D002)

You can see they are dispatchers. But, is there any message in the alert log when the two dispatchers dead? Yes, we found the message as below:
..
Thu Mar 3 13:11:08 2011
found dead dispatcher 'D002', pid = (48, 3)
...
Thu Mar 3 13:18:41 2011
found dead dispatcher 'D000', pid = (46, 3)
...

At last, we know the whole story, someone was trying to kill hung sessions on DB-level but failed, so he killed the related dispatchers on OS-level to force the other sessions go another circuits. But now, what conditions of the two dead dispatchers they are in?
SQL> select name, status, accept, spid from gv$dispatcher vd, gv$process vp where vd.inst_id =1 and vd.paddr = vp.addr order by name;

NAME STATUS           ACCEPT SPID
---- ---------------- ------ ------------
D000 REFUSE           NO     15969
D001 WAIT             YES    15973
D002 REFUSE           NO     15979
D003 WAIT             YES    15983
D004 WAIT             YES    15989

The dead processes are noted as "REFUSE" and "ACCEPT" is NO.

PMON in 9i has a bug which fails to clean up a dead process automatically under some conditions, you can do it manually to shutdown the dead processes explicitly, then PMON will restart the two processes automatically:
SQL> alter system shutdown immediate 'D000';
SQL> alter system shutdown immediate 'D002';

If PMON did not restart the dispatchers, you can re-enable dispathers by this:
SQL> ALTER SYSTEM SET DISPATCHERS="(ADDRESS=(PROTOCOL=TCP)(CONNECTIONS=500)(HOST=ORCL.EXAMPLE.COM))(DISPATCHERS=5)(SERVICE=ORCL)" SCOPE=MEMORY;Theoretically, PMON will clean up dead process automatically, but it sometimes fails to do it in 9i databases like the error message in the alert log:
...
Thu Mar  3 14:12:50 2011
Errors in file /oracle/admin/ORCL/udump/orcl_ora_15969.trc:
ORA-00600: internal error code, arguments: [kjpcre2], [], [], [], [], [], [], []
Thu Mar  3 14:12:50 2011
Errors in file /oracle/admin/ORCL/udump/orcl_ora_15969.trc:
ORA-00603: ORACLE server session terminated by fatal error
ORA-00600: internal error code, arguments: [kjpcre2], [], [], [], [], [], [], []
...
Thu Mar 3 17:50:43 2012
Errors in file /oracle/admin/primdb/udump/orcl_ora_15979.trc:
ORA-00600: internal error code, arguments: [kjpcre2], [], [], [], [], [], [], []
Thu Mar 3 17:50:43 2012
Errors in file /oracle/admin/primdb/udump/orcl_ora_15979.trc:
ORA-00603: ORACLE server session terminated by fatal error
ORA-00600: internal error code, arguments: [kjpcre2], [], [], [], [], [], [], []
...

ORA-00600 is a general and internal error code which will show the arguments to indicate the causes of the problem. The argument "kjpcre2" indicates that PMON takes too long to clean up a dead process. But which processes are dead?
SQL> select inst_id, spid, username, program from gv$process where spid in ('15979','15969');

   INST_ID SPID         USERNAM PROGRAM
---------- ------------ ------- ---------------------------------------
         1 15969        oracle  oracle@primdb (D000)
         1 15979        oracle  oracle@primdb (D002)

You can see they are dispatchers. But, is there any message in the alert log when the two dispatchers dead? Yes, we found the message as below:
..
Thu Mar 3 13:11:08 2011
found dead dispatcher 'D002', pid = (48, 3)
...
Thu Mar 3 13:18:41 2011
found dead dispatcher 'D000', pid = (46, 3)
...

At last, we know the whole story, someone was trying to kill hung sessions on DB-level but failed, so he killed the related dispatchers on OS-level to force the other sessions go another circuits. But now, what conditions of the two dead dispatchers they are in?
SQL> select name, status, accept, spid from gv$dispatcher vd, gv$process vp where vd.inst_id =1 and vd.paddr = vp.addr order by name;

NAME STATUS           ACCEPT SPID
---- ---------------- ------ ------------
D000 REFUSE           NO     15969
D001 WAIT             YES    15973
D002 REFUSE           NO     15979
D003 WAIT             YES    15983
D004 WAIT             YES    15989

The dead processes are noted as "REFUSE" and "ACCEPT" is NO.

PMON in 9i has a bug which fails to clean up a dead process automatically under some conditions, you can do it manually to shutdown the dead processes explicitly, then PMON will restart the two processes automatically:
SQL> alter system shutdown immediate 'D000';
SQL> alter system shutdown immediate 'D002';

If PMON did not restart the dispatchers, you can re-enable dispathers by this:
SQL> ALTER SYSTEM SET DISPATCHERS="(ADDRESS=(PROTOCOL=TCP)(CONNECTIONS=500)(HOST=ORCL.EXAMPLE.COM))(DISPATCHERS=5)(SERVICE=ORCL)" SCOPE=MEMORY;

Leave a Reply

Your email address will not be published.