HOWTO: Leveraging Jenkins to Distribute Work

In GUI testing research, experiments are expensive. To have validity, they should be large; but the cost of executing a single GUI test is relatively high, as it requires an isolated "desktop" environment. In the past few months, I have used Jenkins to make the most of my hardware when running GUI testing experiments. Here I'll briefly explain my approach and share some code.

Github URL: http://github.com/bryantrobbins/jenkinsutils
Gist of example usage: https://gist.github.com/bryantrobbins/9872943

The basic approach I apply to my problems is this:

• Break my work down into logical units that stand alone. I call these work units.
• Write a parameterized Jenkins job that accomplishes one work unit in its necessary environment. I call this job the "single" job (S). In my case, these jobs usually take advantage of SSH Slaves and Xvnc sessions, so that I can run several isolated jobs on the same machine.
• Write a second Jenkins job that fetches all work units and schedules S jobs for each. I call this job the batch job (B).

I accomplish the task of scheduling S jobs via the Jenkins Remote API and a simple Java class leveraging HTTPClient. You can read more about the Remote API and Authentication with the Remote API on the Jenkins wiki.

The code is available on Github. You'll need Git to check out and Maven to build.

[ Please note that this code is using basic HTTP authentication, which is not the best idea (you have to pass user and pass over the wire, etc.). Use at your own risk, of course. ]

You may notice that the JenkinsClient class in that code has no main(...) method, simply submitJob(...). The method isn't static either, so we'd be required to create a JenkinsClient object somewhere in order to use this code as part of the B job. In my work, I use simple Groovy scripts for this purpose.

[If you haven't heard of or used Groovy, it's a scripting language whose syntax/std lib is a superset of Java. So, we can write Java code in a script instead of a main method in a jar with a Manifest, etc. As a bonus, Jenkins is also very compatible with Groovy, having a Groovy admin console for executing Jenkins Java API calls directly against a running server.]

Here is a sample bit of Groovy for the B script which kicks off S jobs, available as a Gist.

The Jenkins Utils I have aren't perfect. They use basic auth, they are asynchronous (don't wait for jobs to return), they don't support https out of the box, etc. Please feel free to contribute to the little Github project if this code is useful to you and you would like to see it expanded.