The Worker API provides the ability to break up the execution of a task action into discrete units of work and then to execute that work concurrently and asynchronously. This allows Gradle to fully utilize the resources available and complete builds faster. This section will walk you through the process of converting an existing custom task to use the Worker API.

This section assumes that you understand the basics of writing Gradle custom tasks. For more information on that topic, consult the section on custom tasks.

You’ll start by creating a custom task class that generates MD5 hashes for a configurable set of files. Then, you’ll convert this custom task to use the Worker API. Then we’ll explore running the task with different levels of isolation. In the process, you’ll learn about the basics of the Worker API and the capabilities it provides.

Create a custom task class

First, you’ll need to create a custom task that generates MD5 hashes of a configurable set of files.

In a new directory, create a buildSrc/build.gradle(.kts) file.

buildSrc/build.gradle
repositories {
    jcenter()
}

dependencies {
    implementation 'commons-io:commons-io:2.5'
    implementation 'commons-codec:commons-codec:1.9' (1)
}
buildSrc/build.gradle.kts
repositories {
    jcenter()
}

dependencies {
    implementation("commons-io:commons-io:2.5")
    implementation("commons-codec:commons-codec:1.9") (1)
}
1 Your custom task class will use Apache Commons Codec to generate MD5 hashes.
If you are not familiar with buildSrc, this is a special directory that allows you to define and build custom classes that should be available for use in your build script. See the section on organizing build logic for further information.

Now, create a custom task class in your buildSrc/src/main/java directory. You should name this class CreateMD5.

buildSrc/src/main/java/CreateMD5.java
import org.apache.commons.codec.digest.DigestUtils;
import org.apache.commons.io.FileUtils;
import org.gradle.api.file.DirectoryProperty;
import org.gradle.api.file.RegularFile;
import org.gradle.api.provider.Provider;
import org.gradle.api.tasks.OutputDirectory;
import org.gradle.api.tasks.SourceTask;
import org.gradle.api.tasks.TaskAction;
import org.gradle.workers.WorkerExecutor;

import java.io.File;
import java.io.FileInputStream;
import java.io.InputStream;

abstract public class CreateMD5 extends SourceTask { (1)

    @OutputDirectory
    abstract public DirectoryProperty getDestinationDirectory(); (2)

    @TaskAction
    public void createHashes() {
        for (File sourceFile : getSource().getFiles()) { (3)
            try {
                InputStream stream = new FileInputStream(sourceFile);
                System.out.println("Generating MD5 for " + sourceFile.getName() + "...");
                // Artificially make this task slower.
                Thread.sleep(3000); (4)
                Provider<RegularFile> md5File = getDestinationDirectory().file(sourceFile.getName() + ".md5");  (5)
                FileUtils.writeStringToFile(md5File.get().getAsFile(), DigestUtils.md5Hex(stream), (String) null);
            } catch (Exception e) {
                throw new RuntimeException(e);
            }
        }
    }
}
1 SourceTask is a convenience type for tasks that operate on a set of source files.
2 The output of the task will go into a configured directory.
3 The task iterates over all of the files defined as "source files" and creates an MD5 hash of each.
4 Insert an artificial sleep to simulate hashing a large file (the sample files won’t be that large).
5 The MD5 hash of each file is written to the output directory into a file of the same name with an "md5" extension.

Next, create a build.gradle(.kts) that registers your new CreateMD5 task.

build.gradle
plugins { id 'base' } (1)

tasks.register("md5", CreateMD5) {
    destinationDirectory = project.layout.buildDirectory.dir("md5") (2)
    source(project.layout.projectDirectory.file('src')) (3)
}
build.gradle.kts
plugins { id("base") } (1)

tasks.register<CreateMD5>("md5") {
    destinationDirectory.set(project.layout.buildDirectory.dir("md5")) (2)
    source(project.layout.projectDirectory.file("src")) (3)
}
1 Apply the base plugin so that you’ll have a clean task to use to remove the output.
2 MD5 hash files will be written to build/md5.
3 This task will generate MD5 hash files for every file in the src directory.

Now, you’ll need some source to generate MD5 hashes from. Create 3 files in the src directory:

src/einstein.txt
Intellectual growth should commence at birth and cease only at death.
src/feynman.txt
I was born not knowing and have had only a little time to change that here and there.
src/oppenheimer.txt
No man should escape our universities without knowing how little he knows.

At this point, you can give your task a try:

$ gradle md5

You should see output similar to:

> Task :md5
Generating MD5 for einstein.txt...
Generating MD5 for feynman.txt...
Generating MD5 for oppenheimer.txt...

BUILD SUCCESSFUL in 0s
1 actionable task: 1 executed

In the build/md5 directory, you should now see corresponding files with an md5 extension containing MD5 hashes of the files from the src directory. Notice that the task takes at least 9 seconds to run because it hashes each file one at a time (i.e. 3 files at ~3 seconds a piece).

Converting to the Worker API

Although this task processes each file in sequence, the processing of each file is independent of any other file. It would be really nice if this work was done in parallel and could take advantage of multiple processors. This is where the Worker API can help.

First, you’ll need to define an interface that represents the parameters of each unit of work and extends org.gradle.workers.WorkParameters. For the generation of MD5 hash files, the unit of work will require two parameters: the file to be hashed and the file to write the hash to. There is no need to create a concrete implementation, though, because Gradle will generate one for us at runtime.

buildSrc/src/main/java/MD5WorkParameters.java
import org.gradle.api.file.RegularFileProperty;
import org.gradle.workers.WorkParameters;

public interface MD5WorkParameters extends WorkParameters {
    RegularFileProperty getSourceFile(); (1)
    RegularFileProperty getMD5File();
}
1 Use Property objects to represent the source and MD5 hash files.

Second, you’ll need to refactor the part of your custom task that does the work for each individual file into a separate class. This class is your "unit of work" implementation and it should be an abstract class that extends org.gradle.workers.WorkAction.

buildSrc/src/main/java/GenerateMD5.java
import org.apache.commons.codec.digest.DigestUtils;
import org.apache.commons.io.FileUtils;
import org.gradle.workers.WorkAction;

import java.io.File;
import java.io.FileInputStream;
import java.io.InputStream;

public abstract class GenerateMD5 implements WorkAction<MD5WorkParameters> { (1)
    @Override
    public void execute() {
        try {
            File sourceFile = getParameters().getSourceFile().getAsFile().get();
            File md5File = getParameters().getMD5File().getAsFile().get();
            InputStream stream = new FileInputStream(sourceFile);
            System.out.println("Generating MD5 for " + sourceFile.getName() + "...");
            // Artificially make this task slower.
            Thread.sleep(3000);
            FileUtils.writeStringToFile(md5File, DigestUtils.md5Hex(stream), (String) null);
        } catch (Exception e) {
            throw new RuntimeException(e);
        }
    }
}
1 Do not implement the getParameters() method - Gradle will inject this at runtime.

Now, you should change your custom task class to submit work to the {api-reference}org/gradle/workers/WorkerExecutor.html[WorkerExecutor] instead of doing the work itself.

buildSrc/src/main/java/CreateMD5.java
import org.gradle.api.Action;
import org.gradle.api.file.RegularFile;
import org.gradle.api.provider.Provider;
import org.gradle.api.tasks.*;
import org.gradle.workers.*;
import org.gradle.api.file.DirectoryProperty;

import javax.inject.Inject;
import java.io.File;

abstract public class CreateMD5 extends SourceTask {

    @OutputDirectory
    abstract public DirectoryProperty getDestinationDirectory();

    @Inject
    abstract public WorkerExecutor getWorkerExecutor(); (1)

    @TaskAction
    public void createHashes() {
        WorkQueue workQueue = getWorkerExecutor().noIsolation(); (2)

        for (File sourceFile : getSource().getFiles()) {
            Provider<RegularFile> md5File = getDestinationDirectory().file(sourceFile.getName() + ".md5");
            workQueue.submit(GenerateMD5.class, parameters -> { (3)
                parameters.getSourceFile().set(sourceFile);
                parameters.getMD5File().set(md5File);
            });
        }
    }
}
1 You’ll need to have the WorkerExecutor service in order to submit your work. Create a abstract getter methods annotated javax.inject.Inject and Gradle will inject the service at runtime when the task is created.
2 Before submitting work, you’ll need to get a WorkQueue object with the desired isolation mode. We’ll talk more about isolation modes later.
3 When submitting the unit of work, specify the unit of work implementation, in this case GenerateMD5 and configure its parameters.

At this point, you should be able to try your task again.

$ gradle clean md5

> Task :md5
Generating MD5 for einstein.txt...
Generating MD5 for feynman.txt...
Generating MD5 for oppenheimer.txt...

BUILD SUCCESSFUL in 0s
1 actionable task: 1 executed

The results should look the same as before, although the MD5 hash files may be generated in a different order due to the fact that the units of work are executed in parallel. One thing you should notice, however, is that the task runs much faster. This is because the Worker API executes the MD5 calculation for each file in parallel rather than in sequence.

Changing the isolation mode

The isolation mode controls how strongly Gradle will isolate items of work from each other as well as from the rest of the Gradle runtime. There are three methods on WorkerExecutor that control this: noIsolation(), classLoaderIsolation() and processIsolation(). The noIsolation() mode is the lowest level of isolation and will prevent a unit of work from changing the project state. This is the fastest isolation mode because it requires the least overhead to set up the work item to execute, so you’ll probably want to use this for simple cases. However, it will use a single shared classloader for all units of work. This means that each unit of work can potentially affect one another through static class state. It also means that every unit of work uses the same version of libraries that are on the buildscript classpath. If you wanted the user to be able to configure the task to run with a different (but compatible) version of the Apache Commons Codec library, you would need to use a different isolation mode.

First, you’ll want to change the dependency in buildSrc/build.gradle to be compileOnly. This tells Gradle that it should use this dependency when building the classes, but should not put it on the build script classpath.

buildSrc/build.gradle
repositories {
    jcenter()
}

dependencies {
    implementation 'commons-io:commons-io:2.5'
    compileOnly 'commons-codec:commons-codec:1.9'
}
buildSrc/build.gradle.kts
repositories {
    jcenter()
}

dependencies {
    implementation("commons-io:commons-io:2.5")
    compileOnly("commons-codec:commons-codec:1.9")
}

Next, you’ll want to change the CreateMD5 task to allow the user to configure the version of the codec library that they want to use. It’ll resolve the appropriate version of the library at runtime and configure the workers to use this version. The classLoaderIsolation() method tells Gradle to run this work in a thread with an isolated classloader.

buildSrc/src/main/java/CreateMD5.java
import org.gradle.api.Action;
import org.gradle.api.file.ConfigurableFileCollection;
import org.gradle.api.file.DirectoryProperty;
import org.gradle.api.file.RegularFile;
import org.gradle.api.provider.Provider;
import org.gradle.api.tasks.*;
import org.gradle.process.JavaForkOptions;
import org.gradle.workers.*;

import javax.inject.Inject;
import java.io.File;
import java.util.Set;

abstract public class CreateMD5 extends SourceTask {

    @InputFiles
    abstract public ConfigurableFileCollection getCodecClasspath(); (1)

    @OutputDirectory
    abstract public DirectoryProperty getDestinationDirectory();

    @Inject
    abstract public WorkerExecutor getWorkerExecutor();

    @TaskAction
    public void createHashes() {
        WorkQueue workQueue = getWorkerExecutor().classLoaderIsolation(workerSpec -> {
            workerSpec.getClasspath().from(getCodecClasspath()); (2)
        });

        for (File sourceFile : getSource().getFiles()) {
            Provider<RegularFile> md5File = getDestinationDirectory().file(sourceFile.getName() + ".md5");
            workQueue.submit(GenerateMD5.class, parameters -> {
                parameters.getSourceFile().set(sourceFile);
                parameters.getMD5File().set(md5File);
            });
        }
    }
}
1 Expose an input property for the codec library classpath.
2 Configure the classpath on the ClassLoaderWorkerSpec when creating the work queue.

Next, you’ll need to configure your build so that it has a repository to look up the codec version at task execution time. We’ll also create a dependency to resolve our codec library from this repository.

build.gradle
plugins { id 'base' }

repositories {
    jcenter() (1)
}

configurations.create('codec') { (2)
    attributes {
        attribute(Usage.USAGE_ATTRIBUTE, objects.named(Usage, Usage.JAVA_RUNTIME))
    }
    visible = false
    canBeConsumed = false
    canBeResolved = true
}

dependencies {
    codec 'commons-codec:commons-codec:1.10' (3)
}

tasks.register('md5', CreateMD5) {
    codecClasspath.from(configurations.codec) (4)
    destinationDirectory = project.layout.buildDirectory.dir('md5')
    source(project.layout.projectDirectory.file('src'))
}
build.gradle.kts
plugins { id("base") }

repositories {
    jcenter() (1)
}

val codec = configurations.create("codec") { (2)
    attributes {
        attribute(Usage.USAGE_ATTRIBUTE, objects.named(Usage.JAVA_RUNTIME))
    }
    isVisible = false
    isCanBeConsumed = false
    isCanBeResolved = true
}

dependencies {
    codec("commons-codec:commons-codec:1.10") (3)
}

tasks.register<CreateMD5>("md5") {
    codecClasspath.from(codec) (4)
    destinationDirectory.set(project.layout.buildDirectory.dir("md5"))
    source(project.layout.projectDirectory.file("src"))
}
1 Add a repository to resolve the codec library - this can be a different repository than the one used to build the CreateMD5 task class.
2 Add a configuration to resolve our codec library version.
3 Configure an alternate, compatible version of Apache Commons Codec.
4 Configure the md5 task to use the configuration as its classpath. Note that the configuration will not be resolved until the task is actually executed.

Now, if you run your task, it should work as expected using the configured version of the codec library:

$ gradle clean md5

> Task :md5
Generating MD5 for einstein.txt...
Generating MD5 for feynman.txt...
Generating MD5 for oppenheimer.txt...

BUILD SUCCESSFUL in 0s
1 actionable task: 1 executed

Creating a Worker Daemon

Sometimes it is desirable to create even further isolation when executing items of work. For instance, external libraries may rely on certain system properties to be set which may conflict between work items. Or a library might not be compatible with the version of JDK that Gradle is running with and may need to be run with a different version. The Worker API can accommodate this using the processIsolation() method that causes the work to execute in a separate "worker daemon". These worker daemon processes will persist across builds and can be reused during subsequent builds. If system resources get low, however, Gradle will stop any unused worker daemons.

To utilize a worker daemon, simply use the processIsolation() method when creating the WorkQueue. You may also want to configure custom settings for the new process.

buildSrc/src/main/java/CreateMD5.java
import org.gradle.api.Action;
import org.gradle.api.file.ConfigurableFileCollection;
import org.gradle.api.file.DirectoryProperty;
import org.gradle.api.file.RegularFile;
import org.gradle.api.provider.Provider;
import org.gradle.api.tasks.*;
import org.gradle.process.JavaForkOptions;
import org.gradle.workers.*;

import javax.inject.Inject;
import java.io.File;
import java.util.Set;

abstract public class CreateMD5 extends SourceTask {

    @InputFiles
    abstract public ConfigurableFileCollection getCodecClasspath(); (1)

    @OutputDirectory
    abstract public DirectoryProperty getDestinationDirectory();

    @Inject
    abstract public WorkerExecutor getWorkerExecutor();

    @TaskAction
    public void createHashes() {
        (1)
        WorkQueue workQueue = getWorkerExecutor().processIsolation(workerSpec -> {
            workerSpec.getClasspath().from(getCodecClasspath());
            workerSpec.forkOptions(options -> {
                options.setMaxHeapSize("64m"); (2)
            });
        });

        for (File sourceFile : getSource().getFiles()) {
            Provider<RegularFile> md5File = getDestinationDirectory().file(sourceFile.getName() + ".md5");
            workQueue.submit(GenerateMD5.class, parameters -> {
                parameters.getSourceFile().set(sourceFile);
                parameters.getMD5File().set(md5File);
            });
        }
    }
}
1 Change the isolation mode to PROCESS.
2 Set up the {api-reference}org/gradle/process/JavaForkOptions.html[JavaForkOptions] for the new process.

Now, you should be able to run your task, and it will work as expected but using worker daemons instead:

$ gradle clean md5

> Task :md5
Generating MD5 for einstein.txt...
Generating MD5 for feynman.txt...
Generating MD5 for oppenheimer.txt...

BUILD SUCCESSFUL in 0s
1 actionable task: 1 executed

Note that the execution time may be somewhat high. This is because Gradle has to start a new process for each worker daemon, which is expensive. However, if you run your task again, you’ll see that it runs much faster. This is because the worker daemon(s) started during the initial build have persisted and are available for use immediately during subsequent builds.

$ gradle clean md5

> Task :md5
Generating MD5 for einstein.txt...
Generating MD5 for feynman.txt...
Generating MD5 for oppenheimer.txt...

BUILD SUCCESSFUL in 0s
1 actionable task: 1 executed