JMX Monitoring in a hurry

Have used this 2 times within 1 month on 2 different clients with 2 different application servers, so it must be worth sharing.

Both times I had a need to monitor some values available to me from JMX, and then chart those results afterwards. And in both cases having to make requests to the SysAdmins to setup the monitoring in their normal tools would have taken days/weeks. And this needed to be running by that afternoon…you know how it is.

First case was when we were handling a set of load testing for a client whom we host. The applications where hosted on JBoss EAP and the developers needed to monitor the some of the core features (infinispan caches, datasources, jms) as well as some custom MBeans they had developed.

Second case was an AEM6 application where we wanted to watch some basics like the thread count, heap usage, cpu load while we were running JMeter tests.

What does it do?

It basically output a raw text looking like this:

2015-02-27 16:00:11.736 jboss.as:subsystem=ejb3,thread-pool=default.queueSize 0
2015-02-27 16:00:11.739 jboss.as:subsystem=ejb3,thread-pool=default.rejectedCount 0
2015-02-27 16:00:11.742 jboss.as:subsystem=ejb3,thread-pool=default.completedTaskCount 53588
2015-02-27 16:00:11.744 jboss.as:subsystem=ejb3,thread-pool=default.currentThreadCount 10
2015-02-27 16:00:11.747 jboss.as:subsystem=ejb3,thread-pool=default.maxThreads 10

This is a TAB separated output with the format:

Timestamp <TAB> MBeanObjectName.AttributeName <TAB> AttributeValue

I have it outputing to STDOUT because that allows me to use rotatelogs that comes with Apache HTTPD to break the logs out into 4hr blocks easily.

This is output format is obviously a really easy format to work with, so as a separate process I have a Perl script that takes these raw logs and turns them into some nice and basic Google Charts that can be viewed locally.

jmxclient-basic

A quick change around and a tiny bit of jQuery allowed generation of JSON files with the data, that could be loaded into a parent page. This allowed multiple data points to be graphed together:

jmxclient-json

 

At the moment the JMXClient can only read attributes, but for AEM I wanted to log the workflow count during an Author performance test. Luckily with AEM JMX is also exposed over HTTP.

So a simple shell script with curl and I was able to output the workflow count into the same format as the JMXClient output and append it to the logs, allowing the use of the same scripts to generate some graphs:

jmxclient-workflowcount

Configuration

The MBeans attributes to watch is done via a TAB separated configuration file in this format:

MBeanObjectName <TAB> AttributeName1 <TAB> AttributeName2 <TAB> ...

Like this one I was using for AEM:

java.lang:type=Threading ThreadCount PeakThreadCount
java.lang:type=Memory HeapMemoryUsage NonHeapMemoryUsage
java.lang:type=Memory HeapMemoryUsage NonHeapMemoryUsage
java.lang:type=OperatingSystem ProcessCpuLoad SystemCpuLoad
Q>org.apache.jackrabbit.oak:name="Consolidated Cache statistics",type="ConsolidatedCacheStats",id=* CacheStats

The line starting with “Q>” tells the JMXClient to perform a query to get the real MBeanObject name. This is useful if that can change. In the example above the id is an integer that might be different between servers. On the JBoss configurations I also used to remove the need to hardcode any application versions under the deployment sub area where they would normally be my-app-1.0.3.war for example

Q>jboss.as:deployment=my-app-*,subsystem=ejb3,stateless-session-bean=* poolCurrentSize waitTime executionTime peakConcurrentInvocations
Q>jboss.as:deployment=my-app-*,subsystem=ejb3,message-driven-bean=* poolCurrentSize waitTime executionTime peakConcurrentInvocations

Running JMXClient

Running the client is straightforward:

java -jar jmxclient.jar type \
  ip port jmx_username jmx_password \
  config-file.tsv sleep_time \
  refresh_mbean_names_every \
  perform_gc_logging perform_thread_logging_every
  • type – used to define the JMX Connection String. Only known value is “jboss”, for none JBoss monitoring it uses standard connection (supply any value)
  • ip – IP of the JMX server
  • port – Port of the JMX server
  • jmx_username / jmx_password – connection details
  • config-file.tsv – the MBean attribute configuration file
  • sleep_time – Stop for this number of seconds between monitoring requests
  • refresh_mbeans_names_every – For performance we only perform a refresh of queried MBeans every X loop iterations. Multiply by sleep time to get real time.
  • perform_gc_logging – Log Garbage Collection notifications
  • perform_thread_logging_every – Log Thread State counts every X loop iterations

For example

java -jar jmxclient.jar jboss \
  127.0.0.1 9998 admin admin \
  config-cluster.tsv \
  60 10 true  10 > jmx-log.txt

Creating the Charts

Then once you have your log output it becomes a simple process of running the perl script to generate the charts.

cat jmx-log.txt | perl split-charts.pl destination-dir

I have a shell script wrapper that performs a bit of clean up etc around it, but that is essentially it.

The source for this will be up on GitHub shortly (once I remove a few customer specific configs etc).

Hope this helps someone!