if OC4J (jvm) is out of memory, it means the OC4J is crash. We set a start parameter -XX:+HepDumpOnOutOfMemoryError for JVM ,it means it will generate hprof file automatically,when JVM is OOM . but sometimes ,the hprof can not be generated.
After checked yesterday's log, we didn't find OC4J "OOM" information, but we could find JVM full GC happened at that time. When JVM is doing full GC, the application would be hung and almost out of memory (but have not). That's why you feel the application was crashed but hprof didn't generate.
We couldn't wait the hprof generated then restart the OC4J, as the application has already been unavailability. Considering the same problem also happened at Asia's OC4J and hprof was generated, I suggest we could start the investigate from Asia's hprof. I've already upload hprof file on the server infrm0gat01.hosting.as:G:\DataSoft\zzhang03\java_pid2698.hprof. Please analyze it with us together.
ora10gas 12514 12249 0 Feb18 ? 00:18:18 /opt/oracle/as10g01/jdk/bin/java -Djava.net.preferIPv4Stack=true -server -Dcom.sun.management.jmxremote -Xmx512m -Xms512m -Djava.security.policy=/opt/oracle/as10g01/j2ee/OC4J_PROD02/config/java2.policy -Djava.awt.headless=true -Dhttp.webdir.enable=false -XX:MaxPermSize=128M -Dcom.decathlon.environment=PRODUCTION -DLOG_ROOT_PATH=/opt/applogs/OC4J_PROD02 -DHOST=proda1oas34.hosting.eu -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/opt/applogs/ -Dcom.sun.management.jmxremote.authenticate=true -Dcom.sun.management.jmxremote.access.file=/opt/appdata/jmx/jmxremote.access -Dcom.sun.management.jmxremote.password.file=/opt/appdata/jmx/jmx_OAS_remote.password -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=50005 -Dmqjazzconfig=/opt/mqjazz/mqmapp/config/flux.properties -XX:+PrintGCDetails -Xloggc:/opt/applogs/OC4J_PROD02/gc.log -Doracle.ons.oraclehome=/opt/oracle/as10g01 -Doracle.home=/opt/oracle/as10g01 -Doracle.ons.oracleconfighome=/opt/oracle/as10g01 -Doracle.ons.clustername=default -Doracle.ons.instancename=as10g01.proda1oas34.hosting.eu -Dopmn.compatible=904 -Doracle.ons.indexid=OC4J_PROD02.OC4J_PROD02.1 -Doracle.ons.numprocs=1 -Doracle.ons.uid=58678077 -Doracle.oc4j.groupname=OC4J_PROD02 -Doracle.oc4j.instancename=OC4J_PROD02 -Doracle.oc4j.islandname=OC4J_PROD02 -Doracle.opmn.routingid=g_rt_id -DOPMN=true -jar oc4j.jar -config /opt/oracle/as10g01/j2ee/OC4J_PROD02/config/server.xml -properties -userThreads -ports default-web-site:ajp:12502,rmi:12402,rmis:12702,jms:12602
cdj2ee 上一层 --> log --> grep pid * --> 可以看到是哪个 OC4J
/opt/oracle/as10g01/opmn/logs
[root@proda1oas34 logs]# grep pid17684 *
OC4J_PRODC01~1.log:Dumping heap to /opt/applogs/java_pid17684.hprof ...
*/5 * * * * find /opt/applogs/ -name "*.hprof" -exec chmod go+wr {} \;
https://community.oracle.com/thread/2525590
I could detect that the main leak suspect is:weblogic.servlet.internal.session.ReplicatedSessionData.
The problem is caused by an
application that has a wrong timeout configuration:0
For that reason, sessions never die; I changed
this parameter and the problem was solved.
Possible memory leak for long-running continuous instances?
http://stackoverflow.com/questions/14313129/java-memory-leak-while-using-axis2-and-was-7
We saw memory growth and sockets left open on WebSphere 6.1 with Axis 2 1.4 in the past. It's been a long time, but my notes suggest it might be worth considering an upgrade to at least Axis 2 1.5.1 to fix this bug with the open sockets and also to ensure you're not creating new objects repeatedly where a singleton exists (e.g. the Service object).
https://issues.apache.org/jira/browse/AXIS2-2883