Parallel Ant allows you to automagically execute your ant build in parallel. There is very little set up to do and, provided your dependencies are all declared correctly, you don’t need to modify your ant script at all.
- Obtaining Parallel Ant
- How it works
- A warning about dependencies
- The “pre-phase”
- Running Parallel Ant
Parallel Ant is hosted on Github: https://github.com/codeaholics/parallel-ant.
The latest download is at: https://github.com/downloads/codeaholics/parallel-ant/parallel-ant-0.9-beta-dist.zip
When you execute Ant, it analyses the dependencies in your build file to decide what order to execute your targets. Parallel Ant makes use of this information to decide which targets it can execute in parallel. The logic is actually quite simple. Every time a target finishes, Parallel Ant looks at all the remaining targets, and any targets whose dependencies have completed is now eligible for execution itself.
As an example, let’s assume you have the following very simple ant script (the tasks within the targets have been elided for clarity):
<target name="dist" depends="jar, javadoc"/> <target name="jar" depends="compile"/> <target name="compile"/> <target name="javadoc"/>
The dependencies between these targets can be viewed as follows:
dist ------> jar ------> compile \------> javadoc
It’s pretty clear to see that both compile and javadoc can be run at the same time, but nothing else can, so Parallel Ant queue both of these targets.
Now, let’s say compile finishes. Parallel Ant re-examines the dependency graph, and finds that jar is now eligible because all of its dependencies have completed. So jar is queued up.
Next, javadoc finishes. This time, Parallel Ant does nothing. The dist target is not eligible to run because not all of its dependencies are complete. Specifically, jar is still running.
Finally, jar finishes, and dist can be scheduled.
By taking this approach of re-examining the dependency graph as each target completes, Parallel Ant can ensure that it makes the best use of the available CPU. Naturally, the benefit to be gained from parallelising your build will depend on the targets and tasks you have. The simple example above probably won’t gain much. But if you’re compiling multiple independent modules, doing a GWT compile, uploading binaries to a repository, etc. then you stand to gain quite a lot.
It’s quite common to come across builds with badly declared dependencies. Take this example:
dist ------> compile \------> jar
Parallel Ant is going to try and run compile and jar at the same time, with disasterous consequences. This build kind-of works because ant generally runs dependencies in the order they’re declared, so it will run compile before jar. But the jar target clearly depends on the output of the compile target, so it really should be declared as such. Right now, if you did ant clean jar, this build would fail because there would be no binaries to jar.
In short, if you can’t take every single target in your build and successfully run ant clean followed by ant your-target then you probably have your dependencies set-up incorrectly and Parallel Ant is likely going to hurt you!
That said, be wary of declaring too many dependencies on things you really don’t depend on!
Suppose you have the following dependencies:
ci ------> clean \------> build
You might imagine that this is the sort of target you would set up for your continuous integration environment – ensure you always do a clean before doing a build.
The problem here is as above: Parallel Ant is going to try and run clean and build at the same time. This time, however, you can’t fix it by putting a dependency from build to clean. (Well, that would fix it, but you probably don’t want every single non-CI build to be clean, right?)
For these cases, Parallel Ant supports a “pre-phase”. The pre-phase is a set of targets which are run before any others, but without having to specify dependencies. Specifically, when Parallel Ant is looking for new targets to schedule, if it finds any pre-phase targets, or there are any pre-phase targets queued or running, then it won’t schedule any regular targets. Put another way, no regular targets will get scheduled until all eligible pre-phase targets have been discovered, queued and completed. This last point is important – you don’t want your build target to get queued until you know for sure that your clean target has finished.
The list of pre-phase targets is configured as follows:
<target name="pant:pre-phase" depends="clean"/>
There is no way to pass meta-data to a custom ant executor via the script, so we declare a target with a special naming convention (pant:pre-phase). Parallel Ant will never execute this target, and it will not allow you to specify it as a dependency of another target. In this target’s depends attribute, we list all the targets which we want included in the pre-phase.
Rules around pre-phase targets:
- The pant:pre-phase target cannot have any tasks in it
- Any targets which are defined as pre-phase targets (in this case, clean) are allowed to have their own dependencies, but those dependencies must also be pre-phase targets
The latter rule allows you to have, say, the clean target depend on a setup-props target. Both targets should be marked as pre-phase targets, because you surely want setup-props to run before you try and use any of the properties. In this case, you will also cause setup-props to run before clean (because of the dependency between them).
There is a shell script (pant) included in the distribution that should work on Linux/other Unixes. If you only speak Windows or else you want to run Parallel Ant by hand or via another tool such as your CI, take a look at the last line of the pant script.
Running from the pant script
The script takes the following arguments:
- -h – Help
- -l path – A path to the directory containing the Parallel Ant JAR. If you don’t specify this, it will look for the JAR in the same directory as the pant script.
- -t nThreads – The number of threads to use during the build. If you don’t specify this, it defaults to the number of CPUs your machine has.
pant -l /usr/local/lib -t 4 clean build dist
Running directly or from another tool
Run ant with the following parameters:
- -lib path/to/parallel-ant.jar – This is the path to the JAR, including the JAR file name. (Note this is different to the script, where you just give the path, and the script assume the JAR is called parallel-ant.jar)
org.codeaholics.tools.build.pant.ParallelExecutor – This tells ant to use the custom Parallel Ant executor.
- -Dpant.threads=n – This is optional. If you don’t specify it, Parallel Ant will default to 2. (Note this is different to the script, which defaults to the number of CPUs.)
- -logger org.codeaholics.tools.build.pant.ParallelExecutorLogger – This is optional. See later.
- …any normal ant parameters, e.g. target names…
When multiple threads are running, and tasks are writing log output, following the build can get very tricky. Parallel Ant comes with a custom logger which changes the log fomat from:
[task] Message from task
[target/task] Message from task running under target
This makes things a bit easier to follow, but if you’re getting lost debugging build issues, I recommend switching back to plain, boring, old, single-threaded ant for a bit. And remembering my warning above about dependencies!