Have used this 2 times within 1 month on 2 different clients with 2 different application servers, so it must be worth sharing.
Both times I had a need to monitor some values available to me from JMX, and then chart those results afterwards. And in both cases having to make requests to the SysAdmins to setup the monitoring in their normal tools would have taken days/weeks. And this needed to be running by that afternoon…you know how it is.
First case was when we were handling a set of load testing for a client whom we host. The applications where hosted on JBoss EAP and the developers needed to monitor the some of the core features (infinispan caches, datasources, jms) as well as some custom MBeans they had developed.
Second case was an AEM6 application where we wanted to watch some basics like the thread count, heap usage, cpu load while we were running JMeter tests.
What does it do?
It basically output a raw text looking like this:
2015-02-27 16:00:11.736 jboss.as:subsystem=ejb3,thread-pool=default.queueSize 0 2015-02-27 16:00:11.739 jboss.as:subsystem=ejb3,thread-pool=default.rejectedCount 0 2015-02-27 16:00:11.742 jboss.as:subsystem=ejb3,thread-pool=default.completedTaskCount 53588 2015-02-27 16:00:11.744 jboss.as:subsystem=ejb3,thread-pool=default.currentThreadCount 10 2015-02-27 16:00:11.747 jboss.as:subsystem=ejb3,thread-pool=default.maxThreads 10
This is a TAB separated output with the format:
Timestamp <TAB> MBeanObjectName.AttributeName <TAB> AttributeValue
I have it outputing to STDOUT because that allows me to use
rotatelogs that comes with Apache HTTPD to break the logs out into 4hr blocks easily.
This is output format is obviously a really easy format to work with, so as a separate process I have a Perl script that takes these raw logs and turns them into some nice and basic Google Charts that can be viewed locally.
A quick change around and a tiny bit of jQuery allowed generation of JSON files with the data, that could be loaded into a parent page. This allowed multiple data points to be graphed together:
At the moment the JMXClient can only read attributes, but for AEM I wanted to log the workflow count during an Author performance test. Luckily with AEM JMX is also exposed over HTTP.
So a simple shell script with curl and I was able to output the workflow count into the same format as the JMXClient output and append it to the logs, allowing the use of the same scripts to generate some graphs:
The MBeans attributes to watch is done via a TAB separated configuration file in this format:
MBeanObjectName <TAB> AttributeName1 <TAB> AttributeName2 <TAB> ...
Like this one I was using for AEM:
java.lang:type=Threading ThreadCount PeakThreadCount java.lang:type=Memory HeapMemoryUsage NonHeapMemoryUsage java.lang:type=Memory HeapMemoryUsage NonHeapMemoryUsage java.lang:type=OperatingSystem ProcessCpuLoad SystemCpuLoad Q>org.apache.jackrabbit.oak:name="Consolidated Cache statistics",type="ConsolidatedCacheStats",id=* CacheStats
The line starting with “Q>” tells the JMXClient to perform a query to get the real MBeanObject name. This is useful if that can change. In the example above the id is an integer that might be different between servers. On the JBoss configurations I also used to remove the need to hardcode any application versions under the
deployment sub area where they would normally be
my-app-1.0.3.war for example
Q>jboss.as:deployment=my-app-*,subsystem=ejb3,stateless-session-bean=* poolCurrentSize waitTime executionTime peakConcurrentInvocations Q>jboss.as:deployment=my-app-*,subsystem=ejb3,message-driven-bean=* poolCurrentSize waitTime executionTime peakConcurrentInvocations
Running the client is straightforward:
java -jar jmxclient.jar type \ ip port jmx_username jmx_password \ config-file.tsv sleep_time \ refresh_mbean_names_every \ perform_gc_logging perform_thread_logging_every
- type – used to define the JMX Connection String. Only known value is “jboss”, for none JBoss monitoring it uses standard connection (supply any value)
- ip – IP of the JMX server
- port – Port of the JMX server
- jmx_username / jmx_password – connection details
- config-file.tsv – the MBean attribute configuration file
- sleep_time – Stop for this number of seconds between monitoring requests
- refresh_mbeans_names_every – For performance we only perform a refresh of queried MBeans every X loop iterations. Multiply by sleep time to get real time.
- perform_gc_logging – Log Garbage Collection notifications
- perform_thread_logging_every – Log Thread State counts every X loop iterations
java -jar jmxclient.jar jboss \ 127.0.0.1 9998 admin admin \ config-cluster.tsv \ 60 10 true 10 > jmx-log.txt
Creating the Charts
Then once you have your log output it becomes a simple process of running the perl script to generate the charts.
cat jmx-log.txt | perl split-charts.pl destination-dir
I have a shell script wrapper that performs a bit of clean up etc around it, but that is essentially it.
The source for this will be up on GitHub shortly (once I remove a few customer specific configs etc).
Hope this helps someone!