Using JQ to extract info from AEM

JQ is the JSON Query tool. Useful for parsing JSON from the command line – and combined with cURL makes the default .json extensions for AEM scriptable.

The syntax is explained well the manual but it does take some getting used to. Let’s start with a simple example before looking at a script with AEM

Using the Network tools in Chrome and Firefox you can “Copy as HAR” or “Save as HAR”

2016-03-23 21_48_20-save-as-har

Then when you have that file you can use jq to extract all the requested URLs on the page:

$ jq '.log.entries[] | .request.url' www.tbscg.com.har | head
"http://www.tbscg.com/"
"http://www.tbscg.com/resources/themes/tbscg-theme/css/tbscg-pack.min.css"
"http://www.tbscg.com/dam/jcr:3773445b-6f79-457b-b15a-b605a9b1eaf0/TBSCG_Logo_Horizontal(white).2015-10-29-10-22-35.png"
"http://www.tbscg.com/dam/jcr:e87afa96-d276-4462-a94b-8cab9b51ae0d/Doosan.2015-08-05-18-11-13.2016-01-28-10-36-27.png"
"http://www.tbscg.com/dam/jcr:35e6cd9d-32d2-4d36-8858-62c5665b13ae/uni_20Southampton.2015-08-05-18-11-06.2016-01-28-10-36-32.png"
"http://www.tbscg.com/dam/jcr:c8ee450c-1dfd-49b0-bd46-8f0c730adc5d/Electrabel_GDF_Suez.2015-08-05-18-11-12.2016-01-28-10-36-27.png"
"http://www.tbscg.com/dam/jcr:a68d2861-5905-449b-b3aa-a72348a6185b/American%20Airlines.2016-01-28-10-36-23.png"
"http://www.tbscg.com/dam/jcr:582b3e19-ead1-4be3-871b-52826d30d15d/Renfe.2015-08-05-18-11-09.2016-01-28-10-36-30.png"
"http://www.tbscg.com/dam/jcr:413a7dea-245a-4dc2-98eb-676e2233833a/Philips.2016-01-28-10-36-24.png"
"http://www.tbscg.com/dam/jcr:3c539ce7-57ad-4043-83e4-97395bec2424/Nationwide.2015-08-05-18-11-10.2016-01-28-10-36-29.png"

OK, great. So useful, but not super exciting.

The other day I had some problems with the user profile synchronisation in an AEM6.1 with AEM Communities (using Sling Distribution to sync the user profiles). Essentially I had seen some inconsistencies between a couple of the publish nodes and I wanted to check that out.

All the accounts I was interested in where stored under sub-folders of /home/users/community (in the randomly associated list).

Because of the 10k+ users I was not able to do the infamous “-1” json depth selector. So I had to walk the tree

http://localhost:4502/home/users/community.1.json

{
  "jcr:primaryType": "rep:AuthorizableFolder",
  "jcr:mixinTypes": [
    "rep:AccessControllable",
    "mix:lockable"
  ],
  "jcr:createdBy": "admin",
  "jcr:created": "Wed Jul 29 2015 15:24:55 GMT+0200",
  "k": {
    "jcr:primaryType": "rep:AuthorizableFolder",
    "jcr:mixinTypes": ["mix:lockable"]
    },
  "p": {
    "jcr:primaryType": "rep:AuthorizableFolder",
    "jcr:mixinTypes": ["mix:lockable"]
    },
 "rep:policy": {"jcr:primaryType": "rep:ACL"}, 
....
}

This was saved as aem-users.json

$ jq -r 'to_entries[] | select(.value["jcr:primaryType"]? == "rep:AuthorizableFolder") | .key' < aem-users.json
k
6
p
....

So let’s break this down. Think of it like the command line. JQ treats part of the filter (the part in single quotes above) as a separate command that takes whole JSON objects. We start with the to_entries[]

to_entries takes a object and creates an array of objects with the 2 properties “key” and “value”

$ jq -r 'to_entries' < aem-users.json  | head -n30
[
  {
    "key": "jcr:primaryType",
    "value": "rep:AuthorizableFolder"
  },
  {
    "key": "jcr:mixinTypes",
    "value": [
      "rep:AccessControllable",
      "mix:lockable"
    ]
  },
  {
    "key": "jcr:createdBy",
    "value": "admin"
  },
  {
    "key": "jcr:created",
    "value": "Wed Jul 29 2015 15:24:55 GMT+0200"
  },
  {
    "key": "k",
    "value": {
      "jcr:primaryType": "rep:AuthorizableFolder",
      "jcr:mixinTypes": [
        "mix:lockable"
      ]
    }
  },
  {

adding the square brackets converts the array into a stream of objects instead (notice the lack of a comma between the objects)

$ jq -r 'to_entries[]' < aem-users.json | head -n30
{
 "key": "jcr:primaryType",
 "value": "rep:AuthorizableFolder"
}
{
 "key": "jcr:mixinTypes",
 "value": [
 "rep:AccessControllable",
 "mix:lockable"
 ]
}
{
 "key": "jcr:createdBy",
 "value": "admin"
}
{
 "key": "jcr:created",
 "value": "Wed Jul 29 2015 15:24:55 GMT+0200"
}
{
 "key": "k",
 "value": {
 "jcr:primaryType": "rep:AuthorizableFolder",
 "jcr:mixinTypes": [
 "mix:lockable"
 ]
 }
}

And with a stream of objects we can use select to filter only the objects we want (in this case “rep:AuthorizableFolder”)
The ? at the end of .value[“jcr:primaryType”]? means that the property does not have to exist

$ jq -r 'to_entries[] | select(.value["jcr:primaryType"]? == "rep:AuthorizableFolder")' < aem-users.json  | head -n30
{
  "key": "k",
  "value": {
    "jcr:primaryType": "rep:AuthorizableFolder",
    "jcr:mixinTypes": [
      "mix:lockable"
    ]
  }
}
{
  "key": "6",
  "value": {
    "jcr:primaryType": "rep:AuthorizableFolder"
  }
}
{
  "key": "p",
  "value": {
    "jcr:primaryType": "rep:AuthorizableFolder"
  }
}
{
  "key": "x",
  "value": {
    "jcr:primaryType": "rep:AuthorizableFolder"
  }
}
....

Then after that we only want the value of the key from the original object, so we extract the key property:

$ jq -r 'to_entries[] | select(.value["jcr:primaryType"]? == "rep:AuthorizableFolder") | .key' < aem-users.json
k
6
p
....

From there I took each line ($D below) and ran another curl to get the contents of the folder (knowing that the users are max 1 level deep):

$ curl -ks -u "$AEMUSER" "$HOST/home/users/community/$D.1.json" > aem-users-$D.json
$ # from now we will use the first $D in the list "k"
$ jq '.' <  aem-users-k.json | head -n 20
{
  "jcr:primaryType": "rep:AuthorizableFolder",
  "jcr:mixinTypes": [
    "mix:lockable"
  ],
  "j_ZpYfJvXt6Pe6G_EtTd": {
    "jcr:primaryType": "rep:User",
    "jcr:mixinTypes": [
      "rep:AccessControllable"
    ],
    "jcr:createdBy": "admin",
    "rep:password": "{SHA-256}dac40c901dac879d-1000-9ac833605d2152dab160a88634f3dd4fb698edeed3c96eb197e8c4a4f8907677",
    "jcr:created": "Fri Feb 05 2016 05:21:15 GMT+0100",
    "rep:principalName": "kelly.brett",
    "jcr:uuid": "302413f2-604d-30e0-8c15-17f53c9fdff3",
    "rep:authorizableId": "kelly.brett"
  },
  "kv7kjXcI-d3JD-xdQqXo": {
    "jcr:primaryType": "rep:User",
    "jcr:mixinTypes": [

Note: $ jq '.' will just pretty-print your JSON.

From here we can produce a list of JSON Objects with just the path and username like this:

$ D=k
$ jq "to_entries[] | select(.value[\"jcr:primaryType\"]? == \"rep:User\") | {path: (\"/home/users/community/$D/\" + .key), username: .value[\"rep:authorizableId\"]}" <  aem-users-k.json | head -n 20
{
  "path": "/home/users/community/k/j_ZpYfJvXt6Pe6G_EtTd",
  "username": "kelly.brett"
}
{
  "path": "/home/users/community/k/kv7kjXcI-d3JD-xdQqXo",
  "username": "k.j.001"
}
{
  "path": "/home/users/community/k/s8O1ee7bVhNXP3j51j7r",
  "username": "kepner.christopher"
}
{
  "path": "/home/users/community/k/F-Nv0PITaFIvl5gE4FSF",
  "username": "kamran.muhammad"
}
{
  "path": "/home/users/community/k/JzsrDYmGePItpv46X2Wq",
  "username": "kostadinov.miro"
}

Hope this intro to JQ helps. There is a lot of options for JQ, but the easiest way to handle it is to just build up your filter one part at at time, like I did with the middle example.

AEM6 and Archiva Servlet gotcha

I am working on a project that is using AEM6 and an AEM FeaturePack for Adobe Campaign. As this is a FP, the APIs are not available through the normal public repositories. I needed to get the project to build so the solution is to use the Archive Servlet. This worked great and the project compiled and installed into AEM fine.

I then needed to use HTTPClient to make some external webservice calls, so looked at what was exposed by the Archiva servlet, found that AEM already uses 4.3.3, added the dependency, mvn clean install, code compiled, happy days.

Until I went to install it into AEM. Then I was getting weird bundle dependency issues:

21.04.2015 16:56:53.374 *INFO* [OsgiInstallerImpl] org.apache.sling.installer.core.impl.tasks.RestartActiveBundlesTask Unable to start bundle com.xyz.test-bundle [396] : Unresolved constraint in bundle com.st.test-bundle [396]: Unable to resolve 396.9: missing requirement [396.9] osgi.wiring.package; (osgi.wiring.package=com.carrotsearch.hppc)

CarrotSearch? Very strange. A quick look at the MANIFEST.MF revealed a huge number of dependencies now on my bundle:

$ cat ./bundle/target/classes/META-INF/MANIFEST.MF
Manifest-Version: 1.0
Bnd-LastModified: 1429628261535
Build-Jdk: 1.7.0_67
Built-By: brobertson
Bundle-ClassPath: .,httpcore-4.3.2.jar,httpclient-4.3.2.jar
Bundle-Description: Maven Multimodule project.
Bundle-ManifestVersion: 2
Bundle-Name: ST Test Integration Bundle
Bundle-SymbolicName: com.st.test-bundle
Bundle-Version: 0.1.1.SNAPSHOT
Created-By: Apache Maven Bundle Plugin
Embed-Dependency: httpcore;httpclient;
Embedded-Artifacts: httpcore-4.3.2.jar;g="org.apache.httpcomponents";a="
 httpcore";v="4.3.2",httpclient-4.3.2.jar;g="org.apache.httpcomponents";
 a="httpclient";v="4.3.2"
Export-Package: com.st.test;uses:="org.apache.sling.api.servlets,javax.s
 ervlet,org.apache.sling.api";version="0.1.1.SNAPSHOT"
Import-Package: com.carrotsearch.hppc,com.carrotsearch.hppc.cursors,com.
 google.common.base,com.google.common.cache,com.google.common.collect,co
 m.google.common.io,com.googlecode.concurrentlinkedhashmap,com.sun.manag
 ement,com.vividsolutions.jts.algorithm,com.vividsolutions.jts.geom,com.
 vividsolutions.jts.io,com.vividsolutions.jts.operation.union,com.vivids
 olutions.jts.operation.valid,com.vividsolutions.jts.simplify,com.vivids
 olutions.jts.util,javax.annotation,javax.crypto,javax.crypto.spec,javax
 .management,javax.management.openmbean,javax.management.remote,javax.na
 ming,javax.net,javax.net.ssl,javax.script,javax.security.auth,javax.sec
 urity.auth.callback,javax.security.auth.kerberos,javax.security.auth.lo
 gin,javax.security.auth.spi,javax.security.auth.x500,javax.security.sas
 l,javax.servlet,javax.servlet.http,javax.xml.namespace,javax.xml.parser
 s,javax.xml.stream,javax.xml.transform,javax.xml.transform.dom,javax.xm
 l.transform.sax,javax.xml.transform.stream,javax.xml.xpath,jline,org.ap
 ache.commons.codec.binary;version="[1.6,2)",org.apache.commons.fileuplo
 ad,org.apache.commons.fileupload.disk,org.apache.commons.fileupload.ser
 vlet,org.apache.commons.io,org.apache.commons.io.input,org.apache.commo
 ns.logging;version="[1.1,2)",org.apache.hadoop.conf,org.apache.hadoop.f
 s,org.apache.hadoop.fs.permission,org.apache.hadoop.ipc,org.apache.hado
 op.metrics,org.apache.hadoop.metrics.jvm,org.apache.hadoop.security,org
 .apache.hadoop.util,org.apache.jackrabbit.oak.api;version="[1.0,2)",org
 .apache.jackrabbit.oak.commons;version="[1.0,2)",org.apache.jackrabbit.
 oak.plugins.index;version="[1.0,2)",org.apache.jackrabbit.oak.plugins.i
 ndex.aggregate;version="[1.0,2)",org.apache.jackrabbit.oak.query.fullte
 xt;version="[1.0,2)",org.apache.jackrabbit.oak.spi.commit;version="[1.0
 ,2)",org.apache.jackrabbit.oak.spi.lifecycle;version="[1.0,2)",org.apac
 he.jackrabbit.oak.spi.query;version="[1.0,2)",org.apache.jackrabbit.oak
 .spi.state;version="[1.0,2)",org.apache.log4j,org.apache.log4j.jmx,org.
 apache.log4j.spi,org.apache.lucene.expressions,org.apache.lucene.expres
 sions.js,org.apache.lucene.index.memory,org.apache.regexp,org.apache.sl
 ing.api;version="[2.3,3)",org.apache.sling.api.servlets;version="[2.1,3
 )",org.eclipse.jetty.server,org.eclipse.jetty.server.bio,org.eclipse.je
 tty.server.handler,org.eclipse.jetty.server.nio,org.eclipse.jetty.serve
 r.session,org.eclipse.jetty.server.ssl,org.eclipse.jetty.servlet,org.ec
 lipse.jetty.util.component,org.eclipse.jetty.util.log,org.eclipse.jetty
 .util.ssl,org.eclipse.jetty.util.thread,org.ietf.jgss,org.jboss.netty.b
 ootstrap,org.jboss.netty.buffer,org.jboss.netty.channel,org.jboss.netty
 .channel.group,org.jboss.netty.channel.socket.nio,org.joda.time,org.jod
 a.time.format,org.osgi.service.component;version="[1.1,2)",org.restlet,
 org.restlet.data,org.restlet.representation,org.restlet.resource,org.re
 stlet.routing,org.slf4j;version="[1.5,2)",org.slf4j.impl,org.w3c.dom,or
 g.xml.sax,org.xml.sax.ext,org.xml.sax.helpers
Service-Component: OSGI-INF/serviceComponents.xml
Tool: Bnd-1.50.0

And when I removed the HTTPClient dependency, they all disappeared.

What I found was that in my ~/.m2/repository/org/apache/httpcomponents/httpclient/4.3.3 the JAR was suspiciously large at over 10MB. And when I looked at the MANIFEST.MF of this file:

Manifest-Version: 1.0
Bnd-LastModified: 1399600663141
Build-Jdk: 1.7.0_40
Built-By: jzitting
Bundle-Category: oak
Bundle-Description: Oak Solr OSGi support
Bundle-DocURL: http://jackrabbit.apache.org/oak/
Bundle-License: http://www.apache.org/licenses/LICENSE-2.0.txt
Bundle-ManifestVersion: 2
Bundle-Name: Oak Solr OSGi
Bundle-SymbolicName: org.apache.jackrabbit.oak-solr-osgi
Bundle-Vendor: The Apache Software Foundation
Bundle-Version: 1.0.0
Created-By: Apache Maven Bundle Plugin
Embed-Dependency: *;scope=runtime;inline=true
Import-Package: org.apache.lucene.expressions;resolution:=optional,org.a
 pache.lucene.expressions.js;resolution:=optional,org.apache.lucene.inde
 x.memory;resolution:=optional,com.carrotsearch.hppc;resolution:=optiona
 l,com.carrotsearch.hppc.cursors;resolution:=optional,com.googlecode.con
 currentlinkedhashmap;resolution:=optional,com.vividsolutions.jts.algori
 thm;resolution:=optional,com.vividsolutions.jts.geom;resolution:=option
 al,com.vividsolutions.jts.io;resolution:=optional,com.vividsolutions.jt
 s.operation.union;resolution:=optional,com.vividsolutions.jts.operation
 .valid;resolution:=optional,com.vividsolutions.jts.simplify;resolution:
 =optional,com.vividsolutions.jts.util;resolution:=optional,com.sun.mana
 gement;resolution:=optional,jline;resolution:=optional,org.apache.hadoo
 p.conf;resolution:=optional,org.apache.hadoop.fs;resolution:=optional,o
 rg.apache.hadoop.fs.permission;resolution:=optional,org.apache.hadoop.i
 pc;resolution:=optional,org.apache.hadoop.metrics;resolution:=optional,
 org.apache.hadoop.metrics.jvm;resolution:=optional,org.apache.hadoop.se
 curity;resolution:=optional,org.apache.hadoop.util;resolution:=optional
 ,org.apache.regexp;resolution:=optional,org.apache.log4j;resolution:=op
 tional,org.apache.log4j.jmx;resolution:=optional,org.apache.log4j.spi;r
 esolution:=optional,org.jboss.netty.bootstrap;resolution:=optional;vers
 ion="[3.2,4)",org.jboss.netty.buffer;resolution:=optional;version="[3.2
 ,4)",org.jboss.netty.channel;resolution:=optional;version="[3.2,4)",org
 .jboss.netty.channel.group;resolution:=optional;version="[3.2,4)",org.j
 boss.netty.channel.socket.nio;resolution:=optional;version="[3.2,4)",or
 g.restlet;resolution:=optional,org.restlet.data;resolution:=optional,or
 g.restlet.representation;resolution:=optional,org.restlet.resource;reso
 lution:=optional,org.restlet.routing;resolution:=optional,org.joda.time
 ;resolution:=optional;version="[2.2,3)",org.joda.time.format;resolution
 :=optional;version="[2.2,3)",org.eclipse.jetty.server;resolution:=optio
 nal;version="[8.1,9)",org.eclipse.jetty.server.bio;resolution:=optional
 ;version="[8.1,9)",org.eclipse.jetty.server.handler;resolution:=optiona
 l;version="[8.1,9)",org.eclipse.jetty.server.nio;resolution:=optional;v
 ersion="[8.1,9)",org.eclipse.jetty.server.session;resolution:=optional;
 version="[8.1,9)",org.eclipse.jetty.server.ssl;resolution:=optional;ver
 sion="[8.1,9)",org.eclipse.jetty.servlet;resolution:=optional;version="
 [8.1,9)",org.eclipse.jetty.util.component;resolution:=optional;version=
 "[8.1,9)",org.eclipse.jetty.util.log;resolution:=optional;version="[8.1
 ,9)",org.eclipse.jetty.util.ssl;resolution:=optional;version="[8.1,9)",
 org.eclipse.jetty.util.thread;resolution:=optional;version="[8.1,9)",ja
 vax.servlet;resolution:=optional;version="[2.6,3)",javax.servlet.http;r
 esolution:=optional;version="[2.6,3)",com.google.common.base;version="[
 15.0,16)",com.google.common.cache;version="[15.0,16)",com.google.common
 .collect;version="[15.0,16)",com.google.common.io;version="[15.0,16)",j
 avax.annotation,javax.crypto,javax.crypto.spec,javax.management,javax.m
 anagement.openmbean,javax.management.remote,javax.naming,javax.net,java
 x.net.ssl,javax.script,javax.security.auth,javax.security.auth.callback
 ,javax.security.auth.kerberos,javax.security.auth.login,javax.security.
 auth.spi,javax.security.auth.x500,javax.security.sasl,javax.xml.namespa
 ce,javax.xml.parsers,javax.xml.stream,javax.xml.transform,javax.xml.tra
 nsform.dom,javax.xml.transform.sax,javax.xml.transform.stream,javax.xml
 .xpath,org.apache.commons.codec.binary;version="[1.5,2)",org.apache.com
 mons.fileupload;version="[1.2,2)",org.apache.commons.fileupload.disk;ve
 rsion="[1.2,2)",org.apache.commons.fileupload.servlet;version="[1.2,2)"
 ,org.apache.commons.io;version="[1.4,2)",org.apache.commons.io.input;ve
 rsion="[1.4,2)",org.apache.commons.logging;version="[1.1,2)",org.apache
 .jackrabbit.oak.api;version="[1.0,2)",org.apache.jackrabbit.oak.commons
 ;version="[1.0,2)",org.apache.jackrabbit.oak.plugins.index;version="[1.
 0,2)",org.apache.jackrabbit.oak.plugins.index.aggregate;version="[1.0,2
 )",org.apache.jackrabbit.oak.query.fulltext;version="[1.0,2)",org.apach
 e.jackrabbit.oak.spi.commit;version="[1.0,2)",org.apache.jackrabbit.oak
 .spi.lifecycle;version="[1.0,2)",org.apache.jackrabbit.oak.spi.query;ve
 rsion="[1.0,2)",org.apache.jackrabbit.oak.spi.state;version="[1.0,2)",o
 rg.ietf.jgss,org.osgi.service.component;version="[1.1,2)";resolution:=o
 ptional,org.slf4j;version="[1.7,2)",org.slf4j.impl;version="[1.6,2)",or
 g.w3c.dom,org.xml.sax,org.xml.sax.ext,org.xml.sax.helpers
Service-Component: OSGI-INF/org.apache.jackrabbit.oak.plugins.index.solr
 .osgi.SolrQueryIndexProviderService.xml,OSGI-INF/org.apache.jackrabbit.
 oak.plugins.index.solr.osgi.SolrServerProviderService.xml,OSGI-INF/org.
 apache.jackrabbit.oak.plugins.index.solr.osgi.SolrIndexEditorProviderSe
 rvice.xml,OSGI-INF/org.apache.jackrabbit.oak.plugins.index.solr.osgi.Em
 beddedSolrServerConfigurationProvider.xml,OSGI-INF/org.apache.jackrabbi
 t.oak.plugins.index.solr.osgi.RemoteSolrServerConfigurationProvider.xml
 ,OSGI-INF/org.apache.jackrabbit.oak.plugins.index.solr.osgi.OakSolrConf
 igurationProviderService.xml
Tool: Bnd-2.1.0.20130426-122213

So in the end I moved the Archiva servlet to the last Repository in the Repositories list to make this work.

Hope this helps someone!

JMX Monitoring in a hurry

Have used this 2 times within 1 month on 2 different clients with 2 different application servers, so it must be worth sharing.

Both times I had a need to monitor some values available to me from JMX, and then chart those results afterwards. And in both cases having to make requests to the SysAdmins to setup the monitoring in their normal tools would have taken days/weeks. And this needed to be running by that afternoon…you know how it is.

First case was when we were handling a set of load testing for a client whom we host. The applications where hosted on JBoss EAP and the developers needed to monitor the some of the core features (infinispan caches, datasources, jms) as well as some custom MBeans they had developed.

Second case was an AEM6 application where we wanted to watch some basics like the thread count, heap usage, cpu load while we were running JMeter tests.

What does it do?

It basically output a raw text looking like this:

2015-02-27 16:00:11.736 jboss.as:subsystem=ejb3,thread-pool=default.queueSize 0
2015-02-27 16:00:11.739 jboss.as:subsystem=ejb3,thread-pool=default.rejectedCount 0
2015-02-27 16:00:11.742 jboss.as:subsystem=ejb3,thread-pool=default.completedTaskCount 53588
2015-02-27 16:00:11.744 jboss.as:subsystem=ejb3,thread-pool=default.currentThreadCount 10
2015-02-27 16:00:11.747 jboss.as:subsystem=ejb3,thread-pool=default.maxThreads 10

This is a TAB separated output with the format:

Timestamp <TAB> MBeanObjectName.AttributeName <TAB> AttributeValue

I have it outputing to STDOUT because that allows me to use rotatelogs that comes with Apache HTTPD to break the logs out into 4hr blocks easily.

This is output format is obviously a really easy format to work with, so as a separate process I have a Perl script that takes these raw logs and turns them into some nice and basic Google Charts that can be viewed locally.

jmxclient-basic

A quick change around and a tiny bit of jQuery allowed generation of JSON files with the data, that could be loaded into a parent page. This allowed multiple data points to be graphed together:

jmxclient-json

 

At the moment the JMXClient can only read attributes, but for AEM I wanted to log the workflow count during an Author performance test. Luckily with AEM JMX is also exposed over HTTP.

So a simple shell script with curl and I was able to output the workflow count into the same format as the JMXClient output and append it to the logs, allowing the use of the same scripts to generate some graphs:

jmxclient-workflowcount

Configuration

The MBeans attributes to watch is done via a TAB separated configuration file in this format:

MBeanObjectName <TAB> AttributeName1 <TAB> AttributeName2 <TAB> ...

Like this one I was using for AEM:

java.lang:type=Threading ThreadCount PeakThreadCount
java.lang:type=Memory HeapMemoryUsage NonHeapMemoryUsage
java.lang:type=Memory HeapMemoryUsage NonHeapMemoryUsage
java.lang:type=OperatingSystem ProcessCpuLoad SystemCpuLoad
Q>org.apache.jackrabbit.oak:name="Consolidated Cache statistics",type="ConsolidatedCacheStats",id=* CacheStats

The line starting with “Q>” tells the JMXClient to perform a query to get the real MBeanObject name. This is useful if that can change. In the example above the id is an integer that might be different between servers. On the JBoss configurations I also used to remove the need to hardcode any application versions under the deployment sub area where they would normally be my-app-1.0.3.war for example

Q>jboss.as:deployment=my-app-*,subsystem=ejb3,stateless-session-bean=* poolCurrentSize waitTime executionTime peakConcurrentInvocations
Q>jboss.as:deployment=my-app-*,subsystem=ejb3,message-driven-bean=* poolCurrentSize waitTime executionTime peakConcurrentInvocations

Running JMXClient

Running the client is straightforward:

java -jar jmxclient.jar type \
  ip port jmx_username jmx_password \
  config-file.tsv sleep_time \
  refresh_mbean_names_every \
  perform_gc_logging perform_thread_logging_every
  • type – used to define the JMX Connection String. Only known value is “jboss”, for none JBoss monitoring it uses standard connection (supply any value)
  • ip – IP of the JMX server
  • port – Port of the JMX server
  • jmx_username / jmx_password – connection details
  • config-file.tsv – the MBean attribute configuration file
  • sleep_time – Stop for this number of seconds between monitoring requests
  • refresh_mbeans_names_every – For performance we only perform a refresh of queried MBeans every X loop iterations. Multiply by sleep time to get real time.
  • perform_gc_logging – Log Garbage Collection notifications
  • perform_thread_logging_every – Log Thread State counts every X loop iterations

For example

java -jar jmxclient.jar jboss \
  127.0.0.1 9998 admin admin \
  config-cluster.tsv \
  60 10 true  10 > jmx-log.txt

Creating the Charts

Then once you have your log output it becomes a simple process of running the perl script to generate the charts.

cat jmx-log.txt | perl split-charts.pl destination-dir

I have a shell script wrapper that performs a bit of clean up etc around it, but that is essentially it.

The source for this will be up on GitHub shortly (once I remove a few customer specific configs etc).

Hope this helps someone!

Sightly Optimisations

I recently performed an audit for a client of their AEM development and deployment. Part of that audit covered their AEM page renderers and components. They had started their development with AEM 6 which allowed them to use the new Sightly interpretor for their components. The following is some items I picked up that are generic enough to share.

Sightly != JSP

It may sound obvious, but Sightly is not JSP. This means a S(l)ightly different mindset is required when developing these files. This mostly revolves around the use of control structures that are such a definite part of Java in JSP, but are more elegant in Sightly.

data-sly-unwrap as control structures

In this audit I repeatedly saw the use of an extra <div> just to wrap a conditional. This is basically taking the logic from a JSP script and converting it to Sightly. Imagine the following code in Sightly:

<div data-sly-test="${myUseObject.showIFrame}" data-sly-unwrap>
  <IFRAME id="XXX" class="hide" src=""></IFRAME>
</div>

You can imagine this as straight JSP:

<% if (myUseObject.showIFrame) { %>
  <IFRAME id="XXX" class="hide" src=""></IFRAME>
<% } %>

when it could be written like this:

<IFRAME
  data-sly-test="${myUseObject.showIFrame}" 
  id="XXX" class="hide" src=""></IFRAME>

A more extreme example is this:

<a class="sitemap-item" href="${item.path}.html">
    <div data-sly-test="${item.path == currentPage.path}" data-sly-unwrap>
        <b class="orange" alt="Current Page">
    </div>
    <div data-sly-test="${item.navigationTitle}" data-sly-unwrap>
        ${item.navigationTitle}
    </div>
    <div data-sly-test="${!item.navigationTitle}" data-sly-unwrap>
        ${item.title}
    </div>
    <div data-sly-test="${item.path == currentPage.path}" data-sly-unwrap>
        </b>
    </div>
</a>

Which could be written like this:

<a class="sitemap-item1" href="${item.path}.html">
  <b 
    data-sly-unwrap="${item.path != currentPage.path}" 
    data-sly-text="${item.navigationTitle || item.title}"
    class="orange" alt="Current Page">
  </b>
</a>

Here using the data-sly-unwrap with a conditional test, the <b> tag is output only if the page is the current page.

Looping

In the following snippet you can see that extra <div>‘s have been used again as a control structure, for outputting a standard unordered list.

<div 
  data-sly-test="${parent.listChildren}" 
  data-sly-list.child="${parent.listChildren}" 
  data-sly-unwrap>
    <div 
      data-sly-test="${!child.hideInNav}"
      data-sly-unwrap>
      <ul>
        <li>
          ..
        </li>
      </ul>
    </div>
</div>

Assuming that having an individual <ul> for each list item was a mistake, the code can be optimised like this:

<ul data-sly-list.child="${parent.listChildren}">
  <li data-sly-test="${!child.hideInNav}">
    ..
  </li>
</ul>

Inline conditionals

Using inline conditionals can help simply your code:

<div data-sly-test="${myUseClass.getVertical}">
  <input type="hidden" id="verticalTab" value="true" name="verticalTab"/>
</div>
<div data-sly-test="!${myUseClass.getVertical}">
  <input type="hidden" id="verticalTab" value="false" name="verticalTab"/>
</div>

like this:

<input
  type="hidden"
  id="verticalTab"
  value="${myUseClass.getVertical ? 'true' : 'false'}"
  name="verticalTab" />

Or as a more extreme example:

<ul>
  <div data-sly-list="${myUseClass.tabs}" data-sly-unwrap>
    <div data-sly-test="${!myUseClass.getVertical}" data-sly-unwrap>
      <li style="width:${myUseClass.tabWidth @ context='styleToken'}%">
        <a href='#tabs-${itemList.index @ context="scriptString"}'>
          ${item.label @ context='html'}
        </a>
      </li>
    </div>
    <div data-sly-test="${myUseClass.getVertical}" data-sly-unwrap>
      <li>
        <a href='#tabs-${itemList.index @ context="scriptString"}' >
          ${item.label @ context='html'}
        </a>
      </li>
    </div>
  </div>
</ul>

To this (using ‘join’ operator on an array to concatenate the string):

<ul data-sly-list="${myUseClass.tabs}">
  <li 
    style="${myUseClass.vertical ? [] : ['width', myUseClass.tabWidth] @ join=': ', context='attribute'}">
    <a href='#tabs-${itemList.index @ context="scriptString"}'>
      ${item.label @ context='html'}
    </a>
  </li>
</ul>

Pass WCMUse Objects to templates

The client was following best practice by using templates for their repetitive code, but in some cases I noticed they were passing a WCMUse class name through to the template so that it could be instantiated with a data-sly-use. However in all cases I could find this WCMUse class was already instantiated in the Sightly file calling the template – resulting in two instantiations of the object.

For example

<!-- Component Sightly file -->
<div
  data-sly-use.formTpl="../formTemplates.html"
  data-sly-use.formElem="com.tbscg.sightly.test.TextBox">
  <div
    class="form_text_label"
    data-sly-call="${formTpl.printLabel @ elemClass='com.tbscg.sightly.test.TextBox'}">
  </div>
   ...
</div>

<!-- formTemplates.html -->
<template data-sly-template.printLabel="${@ elemClass}">
  <div
    data-sly-use.formElem="${elemClass}"
    data-sly-test="${formElem.hideTitle}"
    class="leftcol">
    <div class="label">
      <label
        class="text-light"
        for="${formElem.id}"
        data-sly-text="${'{0} {1} :' @ format=[formElem.title, (formElem.required) ? '*' : '']}">
      </label>
    </div>
  </div>
</template>

could be improved like this:

<!-- Component Sightly file -->
<div
  data-sly-use.formTpl="../formTemplates.html"
  data-sly-use.formElem="com.tbscg.sightly.test.TextBox">
  <div
    class="form_text_label"
    data-sly-call="${formTpl.printLabel @ formElem=formElem}">
  </div>
   ...
</div>

<!-- formTemplates.html -->
<template data-sly-template.printLabel="${@ formElem}">
  <div
    data-sly-test="${formElem.hideTitle}"
    class="leftcol">
    <div class="label">
      <label
        class="text-light"
        for="${formElem.id}"
        data-sly-text="${'{0} {1} :' @ format=[formElem.title, (formElem.required) ? '*' : '']}">
      </label>
    </div>
  </div>
</template>

Sightly Resources

The original Intro to Sightly posts from Feike: Part 1, Part 2, Part 3, Part 4 & Part 5

Official Sightly AEM Documentation

Thanks for reading, I hope this helps in your AEM6 development. If you have any questions, please post them in the comments below.