12c: emctl start agent Fails or Hangs or Timeout or Connection Refused

EM 12c: emctl start agent Fails or Hangs or Timeout or Connection Refused Due to PKCS11
'security.pkcs11.P11SecureRandom.implNextBytes' Reported in Thread Dump 'Hang detected'
(Doc ID 1427773.1)

APPLIES TO:
Enterprise Manager Base Platform - Version 12.1.0.1.0 to 12.1.0.2.0 [Release 12.1]
Oracle Solaris on SPARC (64-bit)
SYMPTOMS
$AGENT_INST/bin/emctl start agent
fails to start or gets hung
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.
Starting agent ...........................................................................................
OR
$AGENT_INST/bin/emctl start agent
fails with following error
Oracle Enterprise Manager 12c Cloud Control 12.1.0.1.0
Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.
Agent status could not be determined. Check the agent process
Consult the log files in: /oracle/product/Agent_home/agent_inst/sysman/log
$AGENT_INST/bin/emctl status agent
Oracle Enterprise Manager 12c Cloud Control 12.1.0.1.0
Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Status agent Failure: Timed out after 300 seconds
/AGENT_INST/sysman/log/emagent.nohup reports th following message
---- Mon Jan 23 07:53:48 2012::8748::EMAgent exited at Mon Jan 23 07:53:48
2012 with signal 9 -----
----- Mon Jan 23 07:53:48 2012::8748::EMAgent either hung or in abnormal state.
----- Mon Jan 23 07:53:48 2012::8748::EMAgent will be restarted/thrashed.
----- Mon Jan 23 07:53:48 2012::8748::Restarting EMAgent.
..
----- Tue Jan 24 13:11:10 2012::27420::EMAgent exited at Tue Jan 24 13:11:10 2012 with signal 9 -----
----- Tue Jan 24 13:11:10 2012::27420::EMAgent has been forcibly killed.
7/31/2017 Document 1427773.1
https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-state=ykgsqkeuh_57&id=1427773.1 2/3
OR
/AGENT_INST/sysman/log/emagent.nohup reports th following message
----- Wed Mar 25 06:13:00 2015::24757::Hang detected for EMAgent : 18731 -----
----- Wed Mar 25 06:13:00 2015::24757::Debugging component EMAgent -----
----- Wed Mar 25 06:13:00 2015::generate first thread dump file for diagnosis -----
----- Wed Mar 25 06:30:43 2015::generate second thread dump file for diagnosis -----
----- Wed Mar 25 06:30:59 2015::generate Threads.18731lsof.1 for diagnosis -----
----- Attempting to kill EMAgent : 18731 -----
When agent is unresponsive or hung, thread dump (kill -3 <agent process id> or jstack) shows the PKCS11 thread remain
in RUNNABLE state
"GC.DPExecutor2.9
(CDProps:oracle_database.emrepos.hscdb4.hsc.usf.edu:decideIncludeDB)" prio=3
tid=0x0000000102875000 nid=0x4f runnable [0xffffffff6a4fd000]
java.lang.Thread.State: RUNNABLE
at sun.security.pkcs11.wrapper.PKCS11.C_GenerateRandom(Native Method)
at
sun.security.pkcs11.P11SecureRandom.implNextBytes(P11SecureRandom.java:166)
at
sun.security.pkcs11.P11SecureRandom.engineNextBytes(P11SecureRandom.java:118)
- locked <0xffffffff44056740> (a [B)
at java.security.SecureRandom.nextBytes(SecureRandom.java:433)
- locked <0xffffffff453860a0> (a java.security.SecureRandom)
Multiple JAVA processes gets started from <AGENT_HOME>/core/{version}/jdk/bin location
$ps -ef | grep agent
oracle 23991 23910 0 09:24:18 pts/1 0:00 grep agent
oracle 17235 1 0 15:50:44 ? 0:00 /u01/app/oracle/product/agent12c/core/12.1.0.2.0/jdk/bin/sparcv9/java -
Xmx128M
oracle 17173 1 0 15:50:41 ? 0:00 /u01/app/oracle/product/agent12c/core/12.1.0.2.0/jdk/bin/sparcv9/java -
Xmx128M
oracle 17025 1 0 15:50:33 ? 0:00 /u01/app/oracle/product/agent12c/core/12.1.0.2.0/jdk/bin/sparcv9/java -
Xmx128M
oracle 17181 1 0 15:50:42 ? 0:00 /u01/app/oracle/product/agent12c/core/12.1.0.2.0/jdk/bin/sparcv9/java -
Xmx128M
CAUSE
Bug 8223618 remove pkcs11 from java.security
using pkcs11 by java installs post-fork handlers in the child causing lot of hanging children processes and not cleaned up
Bug INTERNAL 16398691 - adding flags for solaris jvm options for emagent
Its due to Solaris side bug, described in http://bugs.java.com/view_bug.do?bug_id=6336770
SOLUTION
This issue is fixed in 12.1.0.3 Agent version.
Apply Patch 16398691 to the 12.1.0.2 agent home to address this issue.
There is also a workaround available as mentioned below
1. Backup and edit the file <agent_inst>/sysman/config/s_jvm_options.opt
7/31/2017 Document 1427773.1
https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-state=ykgsqkeuh_57&id=1427773.1 3/3
2. Add the following line:
-Dsun.security.pkcs11.enable-solaris=false
Save the changes.
3. Shutdown the Agent and remove any leftover processes:
$ agent_inst/bin/emctl stop agent
$ ps -ef | grep java | grep '<age nt based dir>'
$ ps -ef | grep perl
Kill if any JAVA /PERL pid are active from AGENT HOME/Base directory.
$ kill -9 <Process id>
4. Start the agent:
$ agent_inst/bin/emctl start agent
Time elapsed between Launch of Watchdog process and execing EMAgent is 89 secs --

Comments

Popular posts from this blog

11G ODI Temp usage