Parallel Ant

Parallel Ant allows you to automagically execute your ant build in parallel. There is very little set up to do and, provided your dependencies are all declared correctly, you don’t need to modify your ant script at all.

Contents

  1. Obtaining Parallel Ant
  2. How it works
  3. A warning about dependencies
  4. The “pre-phase”
  5. Running Parallel Ant
  6. Logging

Obtaining Parallel Ant

Parallel Ant is hosted on Github: https://github.com/codeaholics/parallel-ant.

The latest download is at: https://github.com/downloads/codeaholics/parallel-ant/parallel-ant-0.9-beta-dist.zip

How it works

When you execute Ant, it analyses the dependencies in your build file to decide what order to execute your targets. Parallel Ant makes use of this information to decide which targets it can execute in parallel. The logic is actually quite simple. Every time a target finishes, Parallel Ant looks at all the remaining targets, and any targets whose dependencies have completed is now eligible for execution itself.

As an example, let’s assume you have the following very simple ant script (the tasks within the targets have been elided for clarity):

<target name="dist" depends="jar, javadoc"/>

<target name="jar" depends="compile"/>

<target name="compile"/>

<target name="javadoc"/>

The dependencies between these targets can be viewed as follows:

dist ------> jar ------> compile
    \------> javadoc

It’s pretty clear to see that both compile and javadoc can be run at the same time, but nothing else can, so Parallel Ant queue both of these targets.

Now, let’s say compile finishes. Parallel Ant re-examines the dependency graph, and finds that jar is now eligible because all of its dependencies have completed. So jar is queued up.

Next, javadoc finishes. This time, Parallel Ant does nothing. The dist target is not eligible to run because not all of its dependencies are complete. Specifically, jar is still running.

Finally, jar finishes, and dist can be scheduled.

By taking this approach of re-examining the dependency graph as each target completes, Parallel Ant can ensure that it makes the best use of the available CPU. Naturally, the benefit to be gained from parallelising your build will depend on the targets and tasks you have. The simple example above probably won’t gain much. But if you’re compiling multiple independent modules, doing a GWT compile, uploading binaries to a repository, etc. then you stand to gain quite a lot.

A warning about dependencies

It’s quite common to come across builds with badly declared dependencies. Take this example:

dist ------> compile
    \------> jar

Parallel Ant is going to try and run compile and jar at the same time, with disasterous consequences. This build kind-of works because ant generally runs dependencies in the order they’re declared, so it will run compile before jar. But the jar target clearly depends on the output of the compile target, so it really should be declared as such. Right now, if you did ant clean jar, this build would fail because there would be no binaries to jar.

In short, if you can’t take every single target in your build and successfully run ant clean followed by ant your-target then you probably have your dependencies set-up incorrectly and Parallel Ant is likely going to hurt you!

That said, be wary of declaring too many dependencies on things you really don’t depend on!

The “pre-phase”

Suppose you have the following dependencies:

ci ------> clean
  \------> build

You might imagine that this is the sort of target you would set up for your continuous integration environment – ensure you always do a clean before doing a build.

The problem here is as above: Parallel Ant is going to try and run clean and build at the same time. This time, however, you can’t fix it by putting a dependency from build to clean. (Well, that would fix it, but you probably don’t want every single non-CI build to be clean, right?)

For these cases, Parallel Ant supports a “pre-phase”. The pre-phase is a set of targets which are run before any others, but without having to specify dependencies. Specifically, when Parallel Ant is looking for new targets to schedule, if it finds any pre-phase targets, or there are any pre-phase targets queued or running, then it won’t schedule any regular targets. Put another way, no regular targets will get scheduled until all eligible pre-phase targets have been discovered, queued and completed. This last point is important – you don’t want your build target to get queued until you know for sure that your clean target has finished.

The list of pre-phase targets is configured as follows:

<target name="pant:pre-phase" depends="clean"/>

There is no way to pass meta-data to a custom ant executor via the script, so we declare a target with a special naming convention (pant:pre-phase). Parallel Ant will never execute this target, and it will not allow you to specify it as a dependency of another target. In this target’s depends attribute, we list all the targets which we want included in the pre-phase.

Rules around pre-phase targets:

  • The pant:pre-phase target cannot have any tasks in it
  • Any targets which are defined as pre-phase targets (in this case, clean) are allowed to have their own dependencies, but those dependencies must also be pre-phase targets

The latter rule allows you to have, say, the clean target depend on a setup-props target. Both targets should be marked as pre-phase targets, because you surely want setup-props to run before you try and use any of the properties. In this case, you will also cause setup-props to run before clean (because of the dependency between them).

Running Parallel Ant

There is a shell script (pant) included in the distribution that should work on Linux/other Unixes. If you only speak Windows or else you want to run Parallel Ant by hand or via another tool such as your CI, take a look at the last line of the pant script.

Running from the pant script

The script takes the following arguments:

  • -h – Help
  • -l path – A path to the directory containing the Parallel Ant JAR. If you don’t specify this, it will look for the JAR in the same directory as the pant script.
  • -t nThreads – The number of threads to use during the build. If you don’t specify this, it defaults to the number of CPUs your machine has.

Example:

pant -l /usr/local/lib -t 4 clean build dist

Running directly or from another tool

Run ant with the following parameters:

  • -lib path/to/parallel-ant.jar – This is the path to the JAR, including the JAR file name. (Note this is different to the script, where you just give the path, and the script assume the JAR is called parallel-ant.jar)
  • -Dant.executor.class=
    org.codeaholics.tools.build.pant.ParallelExecutor
    – This tells ant to use the custom Parallel Ant executor.
  • -Dpant.threads=n – This is optional. If you don’t specify it, Parallel Ant will default to 2. (Note this is different to the script, which defaults to the number of CPUs.)
  • -logger org.codeaholics.tools.build.pant.ParallelExecutorLogger – This is optional. See later.
  • …any normal ant parameters, e.g. target names…

Logging

When multiple threads are running, and tasks are writing log output, following the build can get very tricky. Parallel Ant comes with a custom logger which changes the log fomat from:

    [task] Message from task

to:

    [target/task] Message from task running under target

This makes things a bit easier to follow, but if you’re getting lost debugging build issues, I recommend switching back to plain, boring, old, single-threaded ant for a bit. And remembering my warning above about dependencies!

9 Responses to Parallel Ant

  1. Pingback: Parallel Ant « codeaholics.org | Anthill | Scoop.it

  2. gringo says:

    I tried to integrate it in eclipse, so I can start parallel ant builds from the eclipse gui, but I failed miserably. Can you give any hints on how to do that?

  3. Danny says:

    Hi. I never do ant builds from within Eclipse (I don’t know why – I just don’t), so I’ve never tried this. I can see how it would be tricky. You may be able to edit the Ant Builder used by your project to include the appropriate command line options, but I suspect not. Another (slightly dirty) approach would be to write a second ant script which had the same targets as your main ant script, but within each target used the <ant> task with appropriate options to delegate to your main build script. You could even wrap that up as a macro. But it’s not going to be pretty. I’ve raised 2 issues in the issue tracker about this: https://github.com/codeaholics/parallel-ant/issues/2 and https://github.com/codeaholics/parallel-ant/issues/3

  4. drivehappy says:

    You mention the warning about dependencies being correct. What about the case where no-dependency/private targets are used to decrease build times when re-building a particular target. This article: http://stackoverflow.com/questions/374673/does-ant-offer-a-way-to-bypass-dependency mentions what I’m talking about. Is there a particular way to flag these no-dependency targets to be ignored when attempting to build directly via parallel ant?

  5. Danny says:

    I fundamentally disagree with the accepted answer there. I can see the benefit, but I believe using an “if” or “unless” attribute is a better solution. In the case of the original poster’s question, if you did “ant clean” followed by “ant do-tests”, the build would fail because there was no compiled code to test. Running the tests depends on having compiled code, and it should say so.

  6. drivehappy says:

    I understand, and I think the both sides are valid. I think the largest problem with an if/unless is that you would need to know all the dependency properties to flag to not be built, which could be many.

    Unfortunately, I’m working with a rather large codebase where such a change is probably not feasible given the downsides and time. Given this restriction, do you have a suggestion on how this might be worked around in parallel-ant?

    I’m thinking along the lines of using an Ant property that would be set if running with parallel-ant, and this property could be checked via an unless on the “do-” target. This is still quite a bit of manual work adding this, but more desirable.

  7. Dave says:

    Two questions.

    1. What version of Java does your logger depend on? I’m compiling with JDK 1.5 and get an error with the -logger option stating:
    The specified logger class org.codeaholics.tools.build.pant.ParallelExecutorLogger could not be used because Class org.codeaholics.tools.build.pant.ParallelExecutorLogger could not be loaded because of an invalid dependency.

    2. I’m running with default logging, but cannot tell that I’m getting any interleaving of output, so how do I tell if pant is actually spawning off things in parallel? Is there another debug option that will give me more detailed info about what it is doing?

  8. Raju says:

    Hi,
    I’m running two different ant tasks under parallel-ant, I’m getting mixed log. but i want to see output log for both in defferent log files.

    Can you suggest.

    • Danny says:

      Hi. It’s been a while since I looked at this project. If the custom logger is not providing enough separation for you, you may have to write your own custom logger or modify the existing one, I’m afraid. I “borrowed” this one from another project (see the comment in the source file), so I can’t even provide much of a pointer in terms of modifying it. Another approach might be to pipe the output into a file and use other tools (easier if you’re on a Unix variant, I guess) to separate the output, but I understand that’s not ideal.

      It’s nice to hear somebody is using this successfully! :-)

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>