Performance

Monitoring
(focused on Java SE)

11
Topics
• What information to monitor
• What tools to use
• What level of the software stack to monitor
> Operating system level
> JVM level
> Application level
• What to monitor is covered per component at a
given level
> Example. At OS level, monitor CPU usage

2
Tools For Monitoring
• Definition: An act of non-intrusively collecting or
observing performance data from an operating or
running application.
• What to monitor and where?
> Operating system: cpu utilization (kernel & user), network
i/o, disk i/o, memory, processes and kernel (locks).
> JVM: garbage collection frequency and duration, heap
usage, threads, lock contention, cpu usage
> Application: throughput, responsiveness

3
CPU
Monitoring Tools
at the OS-Level
Tools For Monitoring : OS Level
• cpu
> vmstat (Solaris & Linux)
> mpstat (Solaris)
> prstat (Solaris)
> top (Linux, prefer prstat on Solaris)
> Task Manager (Windows)
> Performance Monitor (Windows)
> xosview (Linux)
> cpubar (Solaris – Performance Tools CD)
> iobar (Solaris – Performance Tools CD)
> dtrace (Solaris)
5
Tools For Monitoring : vmstat

6
vmstat
• Virtual Memory Statistics
• reports virtual memory statistics of process, virtual
memory, disk, trap, and CPU activity.
• cpu (breakdown of percentage usage of CPU time.
On multiprocessors this is an a verage across all
processors.)
> us - user time
> sy - system time
> id - idle time

7
Tools For Monitoring : mpstat

8
mpstat
• Multi-processor status
• Reveals the individual CPU utilization on multi-
processor
• Each row of the table represents the activity of one
processor.
• The first table summarizes all activity since boot
• Each subsequent table summarizes activity for the
preceding interval.
• All values are rates (events per second) unless
otherwise noted.
9
mpstat
• icsw involuntary context switches
• migr thread migrations (to another processor)
• usr percent user time
• sys percent system time

10
Tools For Monitoring : prstat

11
Tools For Monitoring : prstat -Lm

12
vmstat
• CPU The percentage of recent CPU time used by
the process.
• VCX The number of voluntary context switches.
• ICX The number of involuntary context switches.

13
Tools For Monitoring : cpubar

• Available on Solaris Performance Tools 3.0 CD, or

download from:
http://mediacast.sun.com/share/stefanschneider/PerformanceCD3.0.tar.gz

14
CPU monitoring : What to look for
• Using HotSpot's jps command you can find the
process ids of all running Java processes on your
machine.
• Using Solaris prstat -Lm, or prstat -Lmp <pid> to
locate the LWP id(s) consuming the most cpu (usr
or sys) and using HotSpot's jstack you can find the
executing threads taking the cpu (usr and sys) time.
> LWPID is in the far right column of prstat.
> Look for jstack's corresponding 'tid', reported in hex.

15
CPU monitoring : What to look for
• Idle cpu
> On multi-threaded applications and multi-core systems,
idle cpu can be an indicator of an application's inability to
scale.
> Combination of high sys or kernel CPU utilization and
idle CPU could indicate shared resource contention as
the scalability blocker.
> Applicable to all operating systems, i.e. Windows, Linux
and Solaris

16
CPU monitoring : What to look for

17
CPU monitoring : What to look for

18
Networking I/O
Monitoring
at OS Level
Tools For Monitoring : OS Level
• network i/o
> netstat (Solaris & Linux)
> Performance Monitor (Windows)
> dtrace (Solaris)
> nicstat (Solaris – Performance Tools CD)
> tcptop (Dtrace Toolkit)
• Data of interest
> network utilization, established connections,

20
Network i/o : What to look for
• tcptop can show per process tcp stats

• Here we can see 'rcp' is generating about 115 kb of

traffic
21
Tools For Monitoring : nicstat

• Notice wAvs, write average size, during four intervals is

about 1420 bytes, the MTU size.
22
Disk I/O
Monitoring
at OS Level
Tools For Monitoring : OS Level
• disk i/o
> iostat (Solaris & Linux)
> Performance Monitor (Windows)
> dtrace (Solaris)
> iobar (Solaris – Performance Tools CD)
> iotop (Solaris – Performance Tools CD)
• Data of interest
> number of disk accesses, latency, average latencies

24
Tools For Monitoring : iostat, iobar,
iotop
• iostat reports per disk, text output
• iobar reports per disk, gui output
• iotop reports per process statistics, text output
• Data of interest
> number of disk accesses, latency, average latencies

25
Tools For Monitoring : iotop example

• iotop reporting at a 5 second interval

• DISKTIME reported in microseconds
• CMD find is keeping disk cmdk0 busy almost 60%
of time during the 5 second interval

26
disk i/o : What to look for
• Disk cache
> Why not enable disk cache?
> What's the risk?
> On Solaris x86, disk cache may be disabled by default.
Linux & Windows systems usually have it enabled.
> Disk cache being disabled depends on the Sun branded
model and how recent the model.

27
Memory
Monitoring
at OS Level
Tools For Monitoring : OS Level
• memory
> vmstat (Solaris & Linux)
> prstat (Solaris)
> top (Linux – prefer prstat on Solaris)
> Performance Monitor (Windows)
> dtrace (Solaris)
> cpubar (Solaris – Performance Tools CD)
> meminfo (Solaris – Performance Tools CD)
• Data of interest
> paging, memory usage

29
Tools For Monitoring : vmstat

30
Tools For Monitoring : cpubar

• Data of interest
> p/s - pages per second, sr - scan rate
> Watch for high scan rate, or increasing trend. Low scan
is ok if they are infrequent.

31
memory utilization : What to look for

• Why is swapping bad for a Java application?

• Anyone volunteers want to explain?

32
memory utilization : What to look for
• How do you fix the swapping problem?
> Smaller Java heap sizes
> Add physical memory
> Reduce number of applications running on the machine
> Any one, or any combination of the above will help

33
Processes
Monitoring
at OS Level
Tools For Monitoring : OS Level
• processes
> ps (Solaris & Linux)
> vmstat (Solaris & Linux)
> mpstat (Solaris)
> prstat (Solaris)
> Performance Monitor (Windows)
> top (Linux – prefer prstat on Solaris)
> dtrace (Solaris)
• Data of interest
> footprint size, number of threads, thread state, cpu
usage, runtime stack, context switches
35
Tools For Monitoring : prstat -Lm

• Data of interest
> number of threads, cpu usage, locks, context switches

36
Tools For Monitoring : mpstat

• Data of interest
> context switches, lock contention

37
processes : What to look for
• Why footprint size, number of threads, thread state,
lock contention and context switching are important
to monitor?
• What does lock contention and/or context switching
look like on Solaris?
• How can you find the lock or locks causing
problems?
• How can you address the thread context switching
problem?

38
Kernel
Monitoring
at OS Level
Tools For Monitoring : OS Level
• kernel
> vmstat (Linux & Solaris)
> mpstat (Solaris)
> lockstat & plockstat (Solaris)
> Performance Monitor (Windows)
> dtrace (Solaris)
> intrstat (Solaris)
• Data of interest
> kernel cpu utilization, locks, system calls, interrupts,
migrations, run queue depth

40
Tools For Monitoring : vmstat

• Data of interest
> kernel cpu utilization, run queue depth
41
Tools For Monitoring : mpstat

• Data of interest
> kernel cpu utilization, locks, system calls, interrupts,
migrations
42
Tools For Monitoring : prstat -Lm

• Data of interest
> kernel cpu utilization, locks, system calls, involuntary
context switches
43
kernel : What to look for
• Why are high sys / kernel cpu, run queue depth,
lock contention, migrations and context switching
are important to monitor?
> Discussion
• What do they indicate when each is observed?
> Discussion
• How to do you address each of these problems?
> Discussion

44
Monitoring Tools
at the JVM-Level
Tools For Monitoring: JVM
• Garbage Collection
> -verbose:gc
> -XX:+PrintGCTimeStamps
> -XX:+PrintGCDetails
> -XX:+PrintGCApplicationStoppedTime
> -XX:+PrintGCApplicationConcurrentTime
> jstat, jps
> Jconsole
> VisualVM
> VisualGC
> dtrace (HotSpot JDK 6 contains samples)
46
Tools For Monitoring: JVM
• Garbage Collection Data of Interest
> Frequency and duration of collections
> Java heap usage
> Number of application threads
> Lock contention
> CPU usage

47
Tools For Monitoring : -verbose:gc
• -verbose:gc
> [GC 1884K->1299K(5056K), 0.0031820 secs]
• With -XX:+PrintGCTimeStamps
> 3.791: [GC 1884K->1299K(5056K), 0.0031820 secs]

• Data of interest
> frequency and duration, heap usage
• Explain what pattern(s) indicate potential problems.
> Quick demo

48
Tools For Monitoring : GCDetails
• -XX:+PrintGCDetails
• [GC [DefNew: 490K->64K(960K), 0.0032800 secs] 5470K->5151K(7884K),
0.0033270 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
• [Full GC (System) [Tenured: 5087K->5151K(6924K), 0.0971070 secs] 6047K-
>5151K(7884K), [Perm : 11178K->11178K(16384K)], 0.0972120 secs] [Times:
user=0.10 sys=0.01, real=0.10 secs]

• Data of interest
> frequency and duration, heap usage

49
Tools For Monitoring : pause time
• -XX:+PrintGCApplicationStoppedTime
• -XX:+PrintGCApplicationConcurrentTime
• Helpful when tuning pause time sensitive applications
• Useful for identifying odd pause time issues when
combined with gc timestamps and gc duration.

50
Tools For Monitoring : jps
• jps
> included in the HotSpot JDK.
> command line utility to find running java processes.
> jps [-q] [-mlvV] [<hostid> where <hostid> =
<hostname>[:<port>]

51
Tools For Monitoring : jstat
• jstat
> included in the HotSpot JDK.
> command line utility.
> jstat -<option> [-t] [-h<lines>] <vmid> [<internal>
[<count>]]
> Garbage collection option(s):
– -gc, -gccapacity, -gccause, -gcnew, -gcnewcapacity, -gcold,
-gcoldcapacity, -gcpermcapacity, -gcutil

52
Tools For Monitoring : jstat
• Beware: When using the concurrent collector
(CMS), jstat reports two full gc events per CMS
cycle, obviously misleading.
• But, young generation stats are accurate with CMS.

53
Tools For Monitoring : jconsole
• jconsole
> Monitoring and management GUI console.
> Included in the HotSpot JDK.
> Attach local or remote.
> Monitor internals of a target JVM.
> Monitor multiple JVMs.
> Explain what patterns indicate potential problems.

54
Tools For Monitoring : jconsole
• Endless observability
> MBean support for
– JVM memory usage by memory pool / spaces
– Class loading, JIT compilation, garbage collector, runtime,
threading and logging
– Thread monitor contention
> Graphical view of heap memory, threads, cpu usage and
class loading

55
Tools For Monitoring : VisualVM
• VisualVM
> Open source project at https://visualvm.dev.java.net
> Integrates several existing JDK software tools,
lightweight memory and CPU profiling capabilities.
– JConsole
– Subset of NetBeans Profiler
> Includes performance analysis and troubleshooting
abilities.
– Thread deadlock detection
– Thread monitor contention

56
Tools For Monitoring : VisualVM
• VisualVM
> Can be further extended with specific functionality for
target application via additional plug-in or extending an
existing plug-in.
– Possibilities include; GlassFish performance monitoring plug-in,
JavaDB performance monitoring plug-in and external vendors
such as WebSphere performance monitoring plug-in.
> Plugins, enhancements and updates delivered through
VisualVM plug-in center.

57
Tools For Monitoring : VisualVM
• VisualVM
> Explain what patterns indicate potential performance
issues.

58
Tools For Monitoring : VisualGC
• VisualGC
> Standalone GUI or VisualVM plug-in.
> Not included in HotSpot JDK. Separate download.
> Visually observe garbage collection behavior. (A picture
is worth a thousand words).
> Also includes classloading and JIT compilation
information.

59
Tools For Monitoring : JVM
• JIT Compilation
> jstat
> Jconsole
> VisualVM
> VisualGC
> -XX:+PrintCompilation (can be intrusive)
> -XX:+LogCompilation (can be intrusive)
> DTrace (HotSpot JDK 6 contains samples)
• Data of interest
> frequency, duration, possible opt / de-opt cycles, failed
compilations
60
Tools For Monitoring : JIT
• -XX:+PrintCompilation
• 1 java.util.Properties$LineReader::readLine (452 bytes)
• 2 java.lang.String::hashCode (60 bytes)
• 3 java.lang.String::equals (88 bytes)
• 3 made not entrant (2) java.lang.String::equals (88 bytes)
• 4 java.lang.Object::<init> (1 bytes)
• 5 java.lang.String::indexOf (151 bytes)

• Data of interest
> frequency, duration, possible opt / de-opt cycles, (explain
the patterns which indicate trouble)

61
Tools For Monitoring : JIT
• -XX:+LogCompilation
> Beware, it can be intrusive
• Will probably need someone from JIT compiler team
to analyze it.
• Data of interest
> frequency, duration, possible opt / de-opt cycles

62
Tools For Monitoring : JIT
• Using .hotspot_compiler file
• When & why to use it?
> JIT compiler in an endless loop attempting a “heroic”
optimization which will not converge
> JIT compiler in a de-optimization – re-optimization cycle
> JIT compiler producing 'bad' code resulting in a core
dump or other severe problem

63
Tools For Monitoring : JIT
• Using .hotspot_compiler file, continued ...
• What is the format?
> exclude A/B/C/D methodName where
> A.B.C.D is the fully qualified package and class name
and methodName is the method name.
> To exclude java.util.HashMap.clear(), specify:
– exclude java/util/HashMap clear

64
Monitoring Tools
at the Application-Level
Tools For Monitoring : Application
• Application throughput and / or responsiveness
> JConsole (application Mbeans)
> Extend VisualVM with a plug-in to gather Java
application data of interest and monitor the application
with VisualVM
> Application log
> Specialized DTrace scripts
• Data of interest
> critical application information and instrumentation

66
Performance
Monitoring
(focused on Java SE)

67
67