Testing in Java & JVM projects

Testing on the JVM is a rich subject matter. There are many different testing libraries and frameworks, as well as many different types of test. All need to be part of the build, whether they are executed frequently or infrequently. This chapter is dedicated to explaining how Gradle handles differing requirements between and within builds, with significant coverage of how it integrates with the two most common testing frameworks: JUnit and TestNG.

Test execution

Gradle executes tests in a separate ('forked') JVM, isolated from the main build process. This prevents classpath pollution and excessive memory consumption for the build process. It also allows you to run the tests with different JVM arguments than the build is using.

You can control how the test process is launched via several properties on the Test task, including the following:

maxParallelForks — default: 1

You can run your tests in parallel by setting this property to a value greater than 1. This may make your test suites complete faster, particularly if you run them on a multi-core CPU. When using parallel test execution, make sure your tests are properly isolated from one another. Tests that interact with the filesystem are particularly prone to conflict, causing intermittent test failures.

Your tests can distinguish between parallel test processes by using the value of the org.gradle.test.worker property, which is unique for each process. You can use this for anything you want, but it’s particularly useful for filenames and other resource identifiers to prevent the kind of conflict we just mentioned.

forkEvery - default: 0 (no maximum)

This property specifies the maximum number of test classes that Gradle should run on a test process before its disposed of and a fresh one created. This is mainly used as a way to manage leaky tests or frameworks that have static state that can’t be cleared or reset between tests.

Warning: a low value (other than 0) can severely hurt the performance of the tests

ignoreFailures — default: false

If this property is true, Gradle will continue with the project’s build once the tests have completed, even if some of them have failed. Note that, by default, the Test task always executes every test that it detects, irrespective of this setting.

failFast —  (since Gradle 4.6) default: false

Set this to true if you want the build to fail and finish as soon as one of your tests fails. This can save a lot of time when you have a long-running test suite and is particularly useful when running the build on continuous integration servers. When a build fails before all tests have run, the test reports only include the results of the tests that have completed, successfully or not.

You can also enable this behavior by using the --fail-fast command line option.

testLogging — default: not set

This property represents a set of options that control which test events are logged and at what level. You can also configure other logging behavior via this property. See TestLoggingContainer for more detail.

See Test for details on all the available configuration options.

The test process can exit unexpectedly if configured incorrectly. For instance, if the Java executable does not exist or an invalid JVM argument is provided, the test process will fail to start. Similarly, if a test makes programmatic changes to the test process, this can also cause unexpected failures.

For example, issues may occur if a SecurityManager is modified in a test because Gradle’s internal messaging depends on reflection and socket communication, which may be disrupted if the permissions on the security manager change. In this particular case, you should restore the original SecurityManager after the test so that the gradle test worker process can continue to function.

Debugging when running tests

On the few occasions that you want to debug your code while the tests are running, it can be helpful if you can attach a debugger at that point. You can either set the Test.getDebug() property to true or use the --debug-jvm command line option.

When debugging for tests is enabled, Gradle will start the test process suspended and listening on port 5005.

Test filtering

It’s a common requirement to run subsets of a test suite, such as when you’re fixing a bug or developing a new test case. Gradle provides two mechanisms to do this:

  • Filtering (the preferred option)

  • Test inclusion/exclusion

Filtering supersedes the inclusion/exclusion mechanism, but you may still come across the latter in the wild.

With Gradle’s test filtering you can select tests to run based on:

  • A fully-qualified class name or fully qualified method name, e.g. org.gradle.SomeTest, org.gradle.SomeTest.someMethod

  • A simple class name or method name if the pattern starts with an upper-case letter, e.g. SomeTest, SomeTest.someMethod (since Gradle 4.7)

  • '*' wildcard matching

You can enable filtering either in the build script or via the --tests command-line option. Here’s an example of some filters that are applied every time the build runs:

Example: Filtering tests in the build script

build.gradle

test {
    filter {
        //include specific method in any of the tests
        includeTestsMatching "*UiCheck"

        //include all tests from package
        includeTestsMatching "org.gradle.internal.*"

        //include all integration tests
        includeTestsMatching "*IntegTest"
    }
}

For more details and examples of declaring filters in the build script, please see the TestFilter reference.

The command-line option is especially useful to execute a single test method. When you use --tests, be aware that the inclusions declared in the build script are still honored. It is also possible to supply multiple --tests options, all of whose patterns will take effect. The following sections have several examples of using the command-line option.

Not all test frameworks play well with filtering. Some advanced, synthetic tests may not be fully compatible. However, the vast majority of tests and use cases work perfectly well with Gradle’s filtering mechanism.

The following two sections look at the specific cases of simple class/method names and fully-qualified names.

Simple name pattern

Since 4.7, Gradle has treated a pattern starting with an uppercase letter as a simple class name, or a class name + method name. For example, the following command lines run either all or exactly one of the tests in the SomeTestClass test case, regardless of what package it’s in:

# Executes all tests in SomeTestClass
gradle test --tests SomeTestClass

# Executes a single specified test in SomeTestClass
gradle test --tests SomeTestClass.someSpecificMethod

gradle test --tests SomeTestClass.*someMethod*

Fully-qualified name pattern

Prior to 4.7 or if the pattern doesn’t start with an uppercase letter, Gradle treats the pattern as fully-qualified. So if you want to use the test class name irrespective of its package, you would use --tests *.SomeTestClass. Here are some more examples:

# specific class
gradle test --tests org.gradle.SomeTestClass

# specific class and method
gradle test --tests org.gradle.SomeTestClass.someSpecificMethod

# method name containing spaces
gradle test --tests "org.gradle.SomeTestClass.some method containing spaces"

# all classes at specific package (recursively)
gradle test --tests 'all.in.specific.package*'

# specific method at specific package (recursively)
gradle test --tests 'all.in.specific.package*.someSpecificMethod'

gradle test --tests '*IntegTest'

gradle test --tests '*IntegTest*ui*'

gradle test --tests '*ParameterizedTest.foo*'

# the second iteration of a parameterized test
gradle test --tests '*ParameterizedTest.*[2]'

Note that the wildcard '*' has no special understanding of the '.' package separator. It’s purely text based. So --tests *.SomeTestClass will match any package, regardless of its 'depth'.

You can also combine filters defined at the command line with continuous build to re-execute a subset of tests immediately after every change to a production or test source file. The following executes all tests in the 'com.mypackage.foo' package or subpackages whenever a change triggers the tests to run:

gradle test --continuous --tests "com.mypackage.foo.*"

Single test execution via System Properties

This mechanism has been superseded by 'Test Filtering', described above. We only include it in case you encounter it in online forums and blogs.

Test inclusions/exclusions are a file-based — as opposed to a class name-based — mechanism for selecting tests to run. It’s activated when you use the -DtaskName.single=<pattern> option on the command line, e.g. -Dtest.single=MyTest.

Test reporting

The Test task generates the following results by default:

  • An HTML test report

  • XML test results in a format compatible with the Ant JUnit report task — one that is supported by many other tools, such as CI servers

  • An efficient binary format of the results used by the Test task to generate the other formats

In most cases, you’ll work with the standard HTML report, which automatically includes the results from all your Test tasks, even the ones you explicitly add to the build yourself. For example, if you add a Test task for integration tests, the report will include the results of both the unit tests and the integration tests if both tasks are run.

Unlike with many of the testing configuration options, there are several project-level convention properties that affect the test reports. For example, you can change the destination of the test results and reports like so:

Example: Changing the default test report and results directories

build.gradle

reporting.baseDir = "my-reports"
testResultsDirName = "$buildDir/my-test-results"

task showDirs {
    doLast {
        logger.quiet(rootDir.toPath().relativize(project.reportsDir.toPath()).toString())
        logger.quiet(rootDir.toPath().relativize(project.testResultsDir.toPath()).toString())
    }
}

Output of gradle -q showDirs

> gradle -q showDirs
my-reports
build/my-test-results

Follow the link to the convention properties for more details.

There is also a standalone TestReport task type that you can use to generate a custom HTML test report. All it requires are a value for destinationDir and the test results you want included in the report. Here is a sample which generates a combined report for the unit tests from all subprojects:

Example: Creating a unit test report for subprojects

build.gradle

subprojects {
    apply plugin: 'java'

    // Disable the test report for the individual test task
    test {
        reports.html.enabled = false
    }
}

task testReport(type: TestReport) {
    destinationDir = file("$buildDir/reports/allTests")
    // Include the results from the `test` task in all subprojects
    reportOn subprojects*.test
}

You should note that the TestReport type combines the results from multiple test tasks and needs to aggregate the results of individual test classes. This means that if a given test class is executed by multiple test tasks, then the test report will include executions of that class, but it can be hard to distinguish individual executions of that class and their output.

Test detection

By default, Gradle will run all tests that it detects, which it does by inspecting the compiled test classes. This detection uses different criteria depending on the test framework used.

For JUnit, Gradle scans for both JUnit 3 and 4 test classes. A class is considered to be a JUnit test if it:

  • Ultimately inherits from TestCase or GroovyTestCase

  • Is annotated with @RunWith

  • Contains a method annotated with @Test or a super class does

For TestNG, Gradle scans for methods annotated with @Test.

Note that abstract classes are not executed. In addition, be aware that Gradle scans up the inheritance tree into jar files on the test classpath. So if those JARs contain test classes, they will also be run.

If you don’t want to use test class detection, you can disable it by setting the scanForTestClasses property on Test to false. When you do that, the test task uses only the includes and excludes properties to find test classes.

If scanForTestClasses is false and no include or exclude patterns are specified, Gradle defaults to running any class that matches the patterns **/*Tests.class and **/*Test.class, excluding those that match **/Abstract*.class.

With JUnit Platform, only includes and excludes are used to filter test classes — scanForTestClasses has no effect.

Test grouping

JUnit, JUnit Platform and TestNG allow sophisticated groupings of test methods.

JUnit 4.8 introduced the concept of categories for grouping JUnit 4 tests classes and methods.[13] Test.useJUnit(org.gradle.api.Action) allows you to specify the JUnit categories you want to include and exclude. For example, the following configuration includes tests in CategoryA and excludes those in CategoryB for the test task:

Example: JUnit Categories

build.gradle

test {
    useJUnit {
        includeCategories 'org.gradle.junit.CategoryA'
        excludeCategories 'org.gradle.junit.CategoryB'
    }
}

JUnit Platform introduced tagging to replace categories. You can specify the included/excluded tags via Test.useJUnitPlatform(org.gradle.api.Action), as follows:

Example: JUnit Platform Tags

build.gradle

test {
    useJUnitPlatform {
        includeTags 'fast'
        excludeTags 'slow'
    }
}

The TestNG framework uses the concept of test groups for a similar effect.[14] You can configure which test groups to include or exclude during the test execution via the Test.useTestNG(org.gradle.api.Action) setting, as seen here:

Example: Grouping TestNG tests

build.gradle

test {
    useTestNG {
        excludeGroups 'integrationTests'
        includeGroups 'unitTests'
    }
}

Using JUnit 5

JUnit 5 is the latest version of the well-known JUnit test framework. Unlike its predecessor, JUnit 5 is modularized and composed of several modules:

JUnit 5 = JUnit Platform + JUnit Jupiter + JUnit Vintage

The JUnit Platform serves as a foundation for launching testing frameworks on the JVM. JUnit Jupiter is the combination of the new programming model and extension model for writing tests and extensions in JUnit 5. JUnit Vintage provides a TestEngine for running JUnit 3 and JUnit 4 based tests on the platform.

The following code enables JUnit Platform support in build.gradle:

Example: Enabling JUnit Platform to run your tests

build.gradle

test {
    useJUnitPlatform()
}

See Test.useJUnitPlatform() for more details.

There are some known limitations of using JUnit 5 with Gradle, for example that tests in static nested classes won’t be discovered and classes are still displayed by their class name instead of @DisplayName. These will be fixed in future version of Gradle. If you find more, please tell us at https://github.com/gradle/gradle/issues/new

Compiling and executing JUnit Jupiter tests

To enable JUnit Jupiter support in Gradle, all you need to do is add the following dependencies:

Example: JUnit Jupiter dependencies

build.gradle

dependencies {
    testImplementation 'org.junit.jupiter:junit-jupiter-api:5.1.0'
    testRuntimeOnly 'org.junit.jupiter:junit-jupiter-engine:5.1.0'
}

You can then put your test cases into src/test/java as normal and execute them with gradle test.

Executing legacy tests with JUnit Vintage

If you want to run JUnit 3/4 tests on JUnit Platform, or even mix them with Jupiter tests, you should add extra JUnit Vintage Engine dependencies:

Example: JUnit Vintage dependencies

build.gradle

dependencies {
    testImplementation 'org.junit.jupiter:junit-jupiter-api:5.1.0'
    testRuntimeOnly 'org.junit.jupiter:junit-jupiter-engine:5.1.0'
    testCompileOnly 'junit:junit:4.12'
    testRuntimeOnly 'org.junit.vintage:junit-vintage-engine:5.1.0'
}

In this way, you can use gradle test to test JUnit 3/4 tests on JUnit Platform, without the need to rewrite them.

A sample of mixed tests can be found at samples/testing/junitplatform/mix in the '-all' distribution of Gradle.

Filtering test engine

JUnit Platform allows you to use different test engines. JUnit currently provides two TestEngine implementations out of the box: junit-jupiter-engine and junit-vintage-engine. You can also write and plug in your own TestEngine implementation as documented here.

By default, all test engines on the test runtime classpath will be used. To control specific test engine implementations explicitly, you can add the following setting to your build script:

Example: Filter specific engines

build.gradle

test {
    useJUnitPlatform {
        includeEngines 'junit-vintage'
        // excludeEngines 'junit-jupiter'
    }
}

A test engine filtering sample can be found at samples/testing/junitplatform/engine in the '-all' distribution of Gradle.

Test execution order in TestNG

TestNG allows explicit control of the execution order of tests when you use a testng.xml file. Without such a file — or an equivalent one configured by TestNGOptions.getSuiteXmlBuilder() — you can’t specify the test execution order. However, what you can do is control whether all aspects of a test — including its associated @BeforeXXX and @AfterXXX methods, such as those annotated with @Before/AfterClass and @Before/AfterMethod — are executed before the next test starts. You do this by setting the TestNGOptions.getPreserveOrder() property to true. If you set it to false, you may encounter scenarios in which the execution order is something like: TestA.doBeforeClass()TestB.doBeforeClass()TestA tests.

While preserving the order of tests is the default behavior when directly working with testng.xml files, the TestNG API that is used by Gradle’s TestNG integration executes tests in unpredictable order by default.[15] The ability to preserve test execution order was introduced with TestNG version 5.14.5. Setting the preserveOrder property to true for an older TestNG version will cause the build to fail.

Example: Preserving order of TestNG tests

build.gradle

test {
    useTestNG {
        preserveOrder true
    }
}

The groupByInstance property controls whether tests should be grouped by instance rather than by class. The TestNG documentation explains the difference in more detail, but essentially, if you have a test method A() that depends on B(), grouping by instance ensures that each A-B pairing, e.g. B(1)-A(1), is executed before the next pairing. With group by class, all B() methods are run and then all A() ones.

Note that you typically only have more than one instance of a test if you’re using a data provider to parameterize it. Also, grouping tests by instances was introduced with TestNG version 6.1. Setting the groupByInstances property to true for an older TestNG version will cause the build to fail.

Example: Grouping TestNG tests by instances

build.gradle

test {
    useTestNG {
        groupByInstances true
    }
}

TestNG parameterized methods and reporting

TestNG supports parameterizing test methods, allowing a particular test method to be executed multiple times with different inputs. Gradle includes the parameter values in its reporting of the test method execution.

Given a parameterized test method named aTestMethod that takes two parameters, it will be reported with the name aTestMethod(toStringValueOfParam1, toStringValueOfParam2). This makes it easy to identify the parameter values for a particular iteration.

Configuring integration tests

A common requirement for projects is to incorporate integration tests in one form or another. Their aim is to verify that the various parts of the project are working together properly. This often means that they require special execution setup and dependencies compared to unit tests.

The simplest way to add integration tests to your build is by taking these steps:

  1. Create a new source set for them

  2. Add the required dependencies to the appropriate configurations for that source set

  3. Configure the compilation and runtime classpaths for that source set

You may also need to perform some additional configuration depending on what form the integration tests take. We will discuss those as we go.

Here’s a practical example that implements the above steps in a build script:

Example: Setting up working integration tests

build.gradle

sourceSets {
    intTest {
        java.srcDir file('src/intTest/java')
        resources.srcDir file('src/intTest/resources')
        compileClasspath += sourceSets.main.output + configurations.testRuntime
        runtimeClasspath += output + compileClasspath
    }
}

configurations {
    intTestImplementation.extendsFrom implementation
}

dependencies {
    intTestImplementation 'junit:junit:4.12'
}

This will set up a new source set called intTest that automatically creates:

  • intTestImplementation, intTestCompileOnly, intTestRuntimeOnly configurations (and a few others that are less commonly needed)

  • A compileIntTestJava task that will compile all the source files under src/intTest/java

The example also does the following:

  • Adds the production classes from the main source set to the compilation and runtime classpaths of the integration tests — sourceSets.main.output is a file collection of all the directories containing compiled production classes and resources

  • Makes the intTestImplementation configuration extend from implementation, which means that all the declared dependencies of the production code also become dependencies of the integration tests

Another common step is to attach all the unit test dependencies to the integration tests as well — via intTestImplementation.extendsFrom testImplementation — but that only makes sense if the integration tests require all the dependencies that the unit tests have.

There are a couple of other facets of the example you should take note of:

  • += allows you to append paths and collections of pAths to compileClasspath and runtimeClasspath instead of overwriting them

  • If you want to use the convention-based configurations, such as intTestImplementation, you must declare the dependencies after the new source set

Creating and configuring a source set automatically sets up the compilation stage, but it does nothing with respect to running the integration tests. So the last piece of the puzzle is a custom test task that uses the information from the new source set to configure its runtime classpath and the test classes:

Example: Defining a working integration test task

build.gradle

task integrationTest(type: Test) {
    description = 'Runs integration tests.'
    group = 'verification'

    testClassesDirs = sourceSets.intTest.output.classesDirs
    classpath = sourceSets.intTest.runtimeClasspath
    mustRunAfter test
}

check.dependsOn integrationTest

Again, we’re accessing a source set to get the relevant information, i.e. where the compiled test classes are — the testClassesDir property — and what needs to be on the classpath when running them — classpath.



[13] The JUnit wiki contains a detailed description on how to work with JUnit categories: https://github.com/junit-team/junit/wiki/Categories.

[14] The TestNG documentation contains more details about test groups: http://testng.org/doc/documentation-main.html#test-groups.

[15] The TestNG documentation contains more details about test ordering when working with testng.xml files: http://testng.org/doc/documentation-main.html#testng-xml.