Troubleshooting high application server CPU or memory utilization

High application server CPU or memory utilization is visible in JVM CPU utilization or JVM memory utilization graphs when viewed in an external monitoring tool, such as JConsole or Java VisualVM.

This issue is typically sporadic in nature and may not be consistently reproducible.

Typical causes

High application server CPU or memory utilization is typically caused by a running batch job that is resource intensive, excessive garbage collection, or a looping thread.

Information to collect

Collect the following information to assist with troubleshooting:

  • Screenshots from the relevant dashboard visualization
  • Java thread dumps. Dumps must be collected while the problem occurs
  • Diagnostic output from top, ps, and vmstat commands
  • Application server logs
  • Heap dumps if a memory leak is suspected

Troubleshooting

Correlate the increased resource utilization with any running batch processes or activity in the application server logs. If no obvious reason can be identified, collect additional diagnostic information as soon as possible including:
Note: If the JVM recovered, crashed, or was restarted, then you cannot capture diagnostic information without reproducing the issue.
  • The output of top -bH -d 1 -n 60 -p <pid>
  • In parallel, take three or more Java core thread dumps 20 seconds apart
  • Output of ps and vmstat commands
  • Also, capture verbose garbage collection logs for the entire day.

Examine verbose garbage collection logs and look for signs of excessive garbage collection, for example, the overhead is much higher than normal.

Correlate with Session Count statistics and consider whether:

  • The elevated CPU and memory utilization is causing user sessions to become load balanced on other nodes.
  • The user traffic is responsible for the additional load.
  • There is an impact on the end-user experience.
If your environment uses WebSphere Application Server (WAS), consider that:
  • WebSphere’s IBM JDK employs gencon for the default garbage collection policy. In this case, it is normal for the JVM heap utilization to reach approximately 90% before a full garbage collection is triggered.
  • You may need to install the IBM support assistant tool to evaluate thread dumps and garbage collection logs.
  • You may require IBM MustGather scripts to collect further data to resolve for performance issues. Scripts are located here:http://www-01.ibm.com/support/docview.wss?uid=swg21115785.