Memory / SWAP Usage issue
Memory Usage: 97% (W:0, C:0) -- Swap Usage: 61% (W:60, C:0)

Work-around:

Top检查系统资源

找出最消耗MEM的进程

Define APP accourding to PID

Restart WebLogic instance to free cache

[hnie@lgtca1ows05 ~]$ top

top - 08:46:59 up 176 days, 9:24, 1 user, load average: 0.47, 0.68, 0.72

Tasks: 173 total, 2 running, 171 sleeping, 0 stopped, 0 zombie

Cpu(s): 3.1%us, 0.4%sy, 0.0%ni, 96.1%id, 0.1%wa, 0.1%hi, 0.2%si, 0.0%st

Mem: 20549164k total, 19596584k used, 952580k free, 210104k buffers

Swap: 4192956k total, 1942032k used, 2250924k free, 2425068k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

20970 oraclew 18 0 1439m 1.1g 47m S 7.0 5.5 10:30.13 java

11572 oraclew 18 0 1756m 1.0g 47m S 3.0 5.3 5:44.66 java

17884 oraclew 17 0 1358m 837m 25m S 1.0 4.2 3577:01 java

18474 oraclew 18 0 1819m 1.2g 52m S 1.0 5.9 4:56.58 java

10418 apache 17 0 774m 9892 2256 S 0.7 0.0 0:49.52 httpd.worker

10419 apache 17 0 902m 9840 2256 S 0.7 0.0 0:49.81 httpd.worker

21013 lp 18 0 86372 6752 3116 R 0.7 0.0 0:00.02 gs

20785 hnie 15 0 10860 1084 760 R 0.3 0.0 0:00.06 top

21006 lp 18 0 79324 3328 1444 S 0.3 0.0 0:00.01 gstoraster

31046 oraclew 16 0 1431m 998m 47m S 0.3 5.0 5:08.72 java

31659 oraclew 16 0 990m 549m 19m S 0.3 2.7 410:32.34 java

1 root 15 0 10348 580 548 S 0.0 0.0 0:53.96 init

2 root RT -5 0 0 0 S 0.0 0.0 2:06.95 migration/0

3 root 34 19 0 0 0 S 0.0 0.0 0:00.86 ksoftirqd/0

4 root RT -5 0 0 0 S 0.0 0.0 2:24.05 migration/1

5 root 34 19 0 0 0 S 0.0 0.0 0:02.17 ksoftirqd/1

6 root RT -5 0 0 0 S 0.0 0.0 1:46.95 migration/2

7 root 34 19 0 0 0 S 0.0 0.0 0:02.21 ksoftirqd/2

8 root RT -5 0 0 0 S 0.0 0.0 1:40.55 migration/3

9 root 34 19 0 0 0 S 0.0 0.0 0:02.15 ksoftirqd/3

10 root RT -5 0 0 0 S 0.0 0.0 1:38.30 migration/4

11 root 34 19 0 0 0 S 0.0 0.0 0:02.18 ksoftirqd/4

12 root RT -5 0 0 0 S 0.0 0.0 1:34.29 migration/5

13 root 34 19 0 0 0 S 0.0 0.0 0:02.17 ksoftirqd/5

14 root 10 -5 0 0 0 S 0.0 0.0 0:22.93 events/0

[hnie@lgtca1ows05 ~]$ free -m

total used free shared buffers cached

Mem: 20067 19144 923 0 205 2368

-/+ buffers/cache: 16570 3496

Swap: 4094 1896 2198

[hnie@lgtca1ows05 ~]$ ps -ef| grep 20970

oraclew 20970 5839 2 02:37 ? 00:10:33 /u01/app/weblogic/jvm/jdk1.6.0_24/jre/bin/java -Dweblogic.Name=TW0043_PART02 -Dbea.home=/u01/app/weblogic/wls11g -Djava.security.policy=/u01/app/weblogic/wls11g/weblogic1034/server/lib/weblogic.policy -Dweblogic.management.server=http://lgtca1ows05:9000 -Djava.library.path="/u01/app/weblogic/jvm/jdk1.6.0_24/jre/lib/amd64/server:/u01/app/weblogic/jvm/jdk1.6.0_24/jre/lib/amd64:/u01/app/weblogic/jvm/jdk1.6.0_24/jre/../lib/amd64:/u01/app/weblogic/wls11g/patch_wls1034/profiles/default/native:/u01/app/weblogic/wls11g/patch_wls1034/profiles/default/native:/u01/app/weblogic/wls11g/weblogic1034/server/native/linux/x86_64:/u01/app/weblogic/wls11g/weblogic1034/server/native/linux/x86_64/oci920_8:/u01/app/weblogic/wls11g/weblogic1034/server/native/linux/x86_64:/u01/app/weblogic/wls11g/weblogic1034/server/native/linux/x86_64/oci920_8:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib" -Djava.class.path=/u01/app/weblogic/wls11g/patch_wls1034/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/u01/app/weblogic/jvm/jdk1.6.0_24/lib/tools.jar:/u01/app/weblogic/wls11g/weblogic1034/server/lib/weblogic_sp.jar:/u01/app/weblogic/wls11g/weblogic1034/server/lib/weblogic.jar:/u01/app/weblogic/wls11g/modules/features/weblogic.server.modules_10.3.4.0.jar:/u01/app/weblogic/wls11g/weblogic1034/server/lib/webservices.jar:/u01/app/weblogic/wls11g/modules/org.apache.ant_1.7.1/lib/ant-all.jar:/u01/app/weblogic/wls11g/modules/net.sf.antcontrib_1.1.0.0_1-0b2/lib/ant-contrib.jar::/opt/dksharedlibs/commons-logging-1.0.4.jar:/opt/dksharedlibs/log4j-1.2.14.jar:/u01/app/weblogic/wls11g/weblogic1034/server/lib/weblogic.jar:/opt/dksharedlibs/fwk-startup-weblo-3.0.jar:/opt/dksharedlibs/log4jFwk3Initializer-1.0.0.jar:/u01/app/weblogic/wls11g/patch_wls1034/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/u01/app/weblogic/jvm/jdk1.6.0_24/lib/tools.jar:/u01/app/weblogic/wls11g/weblogic1034/server/lib/weblogic_sp.jar:/u01/app/weblogic/wls11g/weblogic1034/server/lib/weblogic.jar:/u01/app/weblogic/wls11g/modules/features/weblogic.server.modules_10.3.4.0.jar:/u01/app/weblogic/wls11g/weblogic1034/server/lib/webservices.jar:/u01/app/weblogic/wls11g/modules/org.apache.ant_1.7.1/lib/ant-all.jar:/u01/app/weblogic/wls11g/modules/net.sf.antcontrib_1.1.0.0_1-0b2/lib/ant-contrib.jar::/u01/app/weblogic/wls11g -Dweblogic.system.BootIdentityFile=/u01/app/weblogic/node/lgtca1ows05/servers/TW0043_PART02/data/nodemanager/boot.properties -Dweblogic.nodemanager.ServiceEnabled=true -Dweblogic.security.SSL.ignoreHostnameVerification=false -Dweblogic.ReverseDNSAllowed=false -Xverify:none -DserverName=TW0043_PART02 -DLOG_ROOT_PATH=/opt/applogs/TW0043_PART02 -Dweblogic.Stdout=/u01/app/weblogic/logs/TW0043_PART02/TW0043_PART02.log -Dweblogic.Stderr=/u01/app/weblogic/logs/TW0043_PART02/TW0043_PART02.err -Dcom.decathlon.environment=PRODUCTION -Xms512m -Xmx512m -XX:MaxPermSize=384m -XX:+HeapDumpOnOutOfMemoryError -XX:OnOutOfMemoryError=/u01/app/weblogic/apputils/manageHPROF.sh -XX:HeapDumpPath=/opt/applogs/HEAPDUMP -Dcom.oxit.twist.config=/opt/appdata/twist-config/twist_0043.properties -DSODA_BASE_FOLDER=/opt/appdata/soda -DSODA_JNDI_PROVIDER_URL=file:///opt/appdata/soda/jndi/ -DSODA_ZONE_ID=EUROPE -DSODA_DISABLED=true -DHOST=lgtca1ows05 -XX:+PrintGCDetails -Xloggc:/u01/app/weblogic/logs/TW0043_PART02/gc.log weblogic.Server

hnie 21562 20741 0 08:48 pts/0 00:00:00 grep 20970

[hnie@lgtca1ows05 ~]$ sudo su -

[root@lgtca1ows05 ~]# su - oraclew

[oraclew@lgtca1ows05 ~]$ manage_application.sh list

Application | Target | Info

--------------------------+--------------------------------+------------------

twist-admin_0035 | TW0035_PART02 | /opt/appdata/deploy/twist-admin_0035.ear (ear)

twist-picking_0035 | TW0035_PART02 | /opt/appdata/deploy/twist-picking_0035.ear (ear)

twist-parcel_0035 | TW0035_PART02 | /opt/appdata/deploy/twist-parcel_0035.ear (ear)

twist-anomaly_0035 | TW0035_PART01 | /opt/appdata/deploy/twist-anomaly_0035.ear (ear)

twist-shipment_0035 | TW0035_PART01 | /opt/appdata/deploy/twist-shipment_0035.ear (ear)

twist-visu_0035 | TW0035_PART01 | /opt/appdata/deploy/twist-visu_0035.ear (ear)

twist-stream-listener_0035 | TW0035_FLOW | /opt/appdata/deploy/twist-stream-listener_0035.ear (ear)

twist-picking_0058 | TW0058_PART02 | /opt/appdata/deploy/twist-picking_0058.ear (ear)

cas | TW0035_PART02,TW0058_PART02,TW0011_PART02,TW0085_PART02 | /opt/appdata/deploy/cas.ear (ear)

twist-admin_0043 | TW0043_PART02 | /opt/appdata/deploy/twist-admin_0043.ear (ear)

twist-picking_0043 | TW0043_PART02 | /opt/appdata/deploy/twist-picking_0043.ear (ear)

twist-parcel_0043 | TW0043_PART02 | /opt/appdata/deploy/twist-parcel_0043.ear (ear)

twist-visu_0011 | TW0011_PART01 | /opt/appdata/deploy/twist-visu_0011.ear (ear)

twist-anomaly_0011 | TW0011_PART01 | /opt/appdata/deploy/twist-anomaly_0011.ear (ear)

twist-shipment_0011 | TW0011_PART01 | /opt/appdata/deploy/twist-shipment_0011.ear (ear)

twist-parcel_0011 | TW0011_PART02 | /opt/appdata/deploy/twist-parcel_0011.ear (ear)

twist-admin_0011 | TW0011_PART02 | /opt/appdata/deploy/twist-admin_0011.ear (ear)

twist-picking_0011 | TW0011_PART02 | /opt/appdata/deploy/twist-picking_0011.ear (ear)

twist-visu_0043 | TW0043_PART01 | /opt/appdata/deploy/twist-visu_0043.ear (ear)

twist-shipment_0043 | TW0043_PART01 | /opt/appdata/deploy/twist-shipment_0043.ear (ear)

twist-anomaly_0043 | TW0043_PART01 | /opt/appdata/deploy/twist-anomaly_0043.ear (ear)

twist-stream-listener_0043 | TW0043_FLOW | /opt/appdata/deploy/twist-stream-listener_0043.ear (ear)

twist-parcel_0058 | TW0058_PART02 | /opt/appdata/deploy/twist-parcel_0058.ear (ear)

twist-anomaly_0058 | TW0058_PART01 | /opt/appdata/deploy/twist-anomaly_0058.ear (ear)

twist-shipment_0058 | TW0058_PART01 | /opt/appdata/deploy/twist-shipment_0058.ear (ear)

twist-visu_0058 | TW0058_PART01 | /opt/appdata/deploy/twist-visu_0058.ear (ear)

twist-stream-listener_0058 | TW0058_FLOW | /opt/appdata/deploy/twist-stream-listener_0058.ear (ear)

twist-admin_0058 | TW0058_PART02 | /opt/appdata/deploy/twist-admin_0058.ear (ear)

twist-shipperbox_0058 | TW0058_PART01 | /opt/appdata/deploy/twist-shipperbox_0058.ear (ear)

twist-stream-listener_0011 | TW0011_FLOW | /opt/appdata/deploy/twist-stream-listener_0011.ear (ear)

twist-returns | RETURN | /opt/appdata/deploy/twist-returns.ear (ear)

twist-shipperbox_0011 | TW0011_PART01 | /opt/appdata/deploy/twist-shipperbox_0011.ear (ear)

twist-shipperbox_0043 | TW0043_PART01 | /opt/appdata/deploy/twist-shipperbox_0043.ear (ear)

cas_0043 | TW0043_PART02 | /opt/appdata/deploy/cas_0043.ear (ear)

twist-shipperbox_0035 | TW0035_PART01 | /opt/appdata/deploy/twist-shipperbox_0035.ear (ear)

twist-parcel_0085 | TW0085_PART02 | /opt/appdata/deploy/twist-parcel_0085.ear (ear)

twist-picking_0085 | TW0085_PART02 | /opt/appdata/deploy/twist-picking_0085.ear (ear)

twist-admin_0085 | TW0085_PART02 | /opt/appdata/deploy/twist-admin_0085.ear (ear)

twist-anomaly_0085 | TW0085_PART01 | /opt/appdata/deploy/twist-anomaly_0085.ear (ear)

twist-shipment_0085 | TW0085_PART01 | /opt/appdata/deploy/twist-shipment_0085.ear (ear)

twist-visu_0085 | TW0085_PART01 | /opt/appdata/deploy/twist-visu_0085.ear (ear)

twist-stream-listener_0085 | TW0085_FLOW | /opt/appdata/deploy/twist-stream-listener_0085.ear (ear)

twist-shipperbox_0085 | TW0085_PART01 | /opt/appdata/deploy/twist-shipperbox_0085.ear (ear)

重启某个应用
Weblogic
[oraclew@lgtca1ows05 ~]$ wlsctl

Manage WebLogic instance

Usage

wlsctl stop/start/restart

[stop/start/restart]_all

[stop/start/restart]_domain

[stop/start/restart]_node

[stop/start/restart]_console

status

[node/console]_status

Stop/start/restart or get status of a weblogic instance/node/console

When start/stop/restarting a WebLogic instance, you can use the following options:

-h --help print this message.

--console connect to the console instead of node manager.

Use to force refresh after an instance modification.

--refresh Alias of --console

--max-parallel-start Max instance to start in parallel (0 = no limit, default 3)

[oraclew@lgtca1ows05 ~]$ wlsctl stop TW0043_PART02

[oraclew@lgtca1ows05 ~]$ wlsctl start TW0043_PART02

OC4J

[ora10gas@oxita1oas19 ~]$ ps -ef | grep 14023

ora10gas 14023 14853 0 Jan16 ? 00:04:30 /opt/oracle/as10g01/jdk/bin/java -server -Dcom.sun.management.jmxremote -Xmx1024m -Xms1024m -Djava.net.preferIPv4Stack=true -Djava.security.policy=/opt/oracle/as10g01/j2ee/OC4J_CONCEPT01/config/java2.policy -Djava.awt.headless=true -Dhttp.webdir.enable=false -XX:MaxPermSize=512M-Dcom.decathlon.environment=PRODUCTION -DLOG_ROOT_PATH=/opt/applogs/OC4J_CONCEPT01 -DHOST=oxita1oas19.dktetrix.net -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/opt/applogs/ -Dcom.sun.management.jmxremote.authenticate=true -Dcom.sun.management.jmxremote.access.file=/opt/appdata/jmx/jmxremote.access -Dcom.sun.management.jmxremote.password.file=/opt/appdata/jmx/jmx_OAS_remote.password -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=50006 -Dmqjazzconfig=/opt/mqjazz/mqmapp/config/flux.properties -Dcom.webmethods.jms.clientIDSharing=true -Doracle.ons.oraclehome=/opt/oracle/as10g01 -Doracle.home=/opt/oracle/as10g01 -Doracle.ons.oracleconfighome=/opt/oracle/as10g01 -Doracle.ons.clustername=default -Doracle.ons.instancename=as10g01.oxita1oas19.dktetrix.net -Dopmn.compatible=904 -Doracle.ons.indexid=OC4J_CONCEPT01.OC4J_CONCEPT01.1 -Doracle.ons.numprocs=1 -Doracle.ons.uid=264570916 -Doracle.oc4j.groupname=OC4J_CONCEPT01 -Doracle.oc4j.instancename=OC4J_CONCEPT01 -Doracle.oc4j.islandname=OC4J_CONCEPT01 -Doracle.opmn.routingid=g_rt_id -DOPMN=true -jar oc4j.jar -config /opt/oracle/as10g01/j2ee/OC4J_CONCEPT01/config/server.xml -properties -userThreads -ports default-web-site:ajp:12503,rmi:12404,rmis:12704,jms:12604

ora10gas 17657 9320 0 07:25 pts/0 00:00:00 grep --color 14023

oc4jrestart OC4J_CONCEPT01

------------------------------------------------------------------------------------------------------

在Linux下查看内存我们一般用free命令：
[root@scs-2 tmp]# free
total       used       free     shared    buffers     cached
Mem:       3266180    3250004      16176          0     110652    2668236
-/+ buffers/cache:     471116    2795064
Swap:      2048276      80160    1968116

下面是对Linux查看内存命令中这些数值的解释：
total:总计物理内存的大小。
used:已使用多大。
free:可用有多少。
Shared:多个进程共享的内存总额。
Buffers/cached:磁盘缓存的大小。
第三行(-/+ buffers/cached):
used:已使用多大。
free:可用有多少。
第四行就不多解释了。
区别：第二行(mem)的used/free与第三行(-/+ buffers/cache) used/free的区别。这两个的区别在于使用的角度来看，第一行是从OS的角度来看，因为对于OS，buffers/cached 都是属于被使用，所以他的可用内存是16176KB,已用内存是3250004KB,其中包括，内核（OS）使用+Application(X, oracle,etc)使用的+buffers+cached.
第三行所指的是从应用程序角度来看，对于应用程序来说，buffers/cached 是等于可用的，因为buffer/cached是为了提高文件读取的性能，当应用程序需在用到内存的时候，buffer/cached会很快地被回收。
所以从应用程序的角度来说，可用内存=系统free memory+buffers+cached。
如上例：
2795064=16176+110652+2668236

接下来解释什么时候内存会被交换，以及按什么方交换。当可用内存少于额定值的时候，就会开会进行交换。
Linux查看内存命令时如何看额定值：
cat /proc/meminfo
[root@scs-2 tmp]# cat /proc/meminfo
MemTotal:      3266180 kB
MemFree:         17456 kB
Buffers:        111328 kB
Cached:        2664024 kB
SwapCached:          0 kB
Active:         467236 kB
Inactive:      2644928 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:      3266180 kB
LowFree:         17456 kB
SwapTotal:     2048276 kB
SwapFree:      1968116 kB
Dirty: 8 kB
Writeback:           0 kB
Mapped:         345360 kB
Slab:           112344 kB
Committed_AS:   535292 kB
PageTables:       2340 kB
VmallocTotal: 536870911 kB
VmallocUsed:    272696 kB
VmallocChunk: 536598175 kB
HugePages_Total:     0
HugePages_Free:      0
Hugepagesize:     2048 kB

用free -m查看的结果：
[root@scs-2 tmp]# free -m
total       used       free     shared    buffers     cached
Mem:          3189       3173         16          0        107       2605
-/+ buffers/cache:        460       2729
Swap:         2000         78       1921

查看/proc/kcore文件的大小（内存镜像）：
[root@scs-2 tmp]# ll -h /proc/kcore
-r-------- 1 root root 4.1G Jun 12 12:04 /proc/kcore

备注：
占用内存的测量
测量一个进程占用了多少内存，linux为我们提供了一个很方便的方法，/proc目录为我们提供了所有的信息，实际上top等工具也通过这里来获取相应的信息。
/proc/meminfo 机器的内存使用信息
/proc/pid/maps pid为进程号，显示当前进程所占用的虚拟地址。
/proc/pid/statm 进程所占用的内存
[root@localhost ~]# cat /proc/self/statm
654 57 44 0 0 334 0

Linux查看内存命令的输出解释
CPU 以及CPU0。。。的每行的每个参数意思（以第一行为例）为：

参数解释 /proc//status
Size (pages) 任务虚拟地址空间的大小 VmSize/4
Resident(pages) 应用程序正在使用的物理内存的大小 VmRSS/4
Shared(pages) 共享页数 \
Trs(pages) 程序所拥有的可执行虚拟内存的大小 VmExe/4
Lrs(pages) 被映像到任务的虚拟内存空间的库的大小 VmLib/4
Drs(pages) 程序数据段和用户态的栈的大小（VmData+ VmStk ）4

dt(pages) 04

查看机器可用内存
/proc/28248/>free
total used free shared buffers cached
Mem: 1023788 926400 97388 0 134668 503688
-/+ buffers/cache: 288044 735744
Swap: 1959920 89608 1870312

我们通过free命令查看机器空闲内存时，会发现free的值很小。这主要是因为，在linux中有这么一种思想，内存不用白不用，因此它尽可能的cache和buffer一些数据，以方便下次使用。但实际上这些内存也是可以立刻拿来使用的。

所以空闲内存=free+buffers+cached=total-used

High memory & SWAP cache,重启应用服务器