Testing in the native ecosystem is a rich subject matter. There are many different testing libraries and frameworks, as well as many different types of test. All need to be part of the build, whether they are executed frequently or infrequently. This chapter is dedicated to explaining how Gradle handles differing requirements between and within builds, with significant coverage of how it integrates with XCTest on both macOS and Linux.

It explains: - Ways to control how the tests are run (Test execution) - How to select specific tests to run (Test filtering) - What test reports are generated and how to influence the process (Test reporting) - How Gradle finds tests to run (Test detection)

But first, we look at the basics of native testing in Gradle.

The basics

Gradle supports deep integration with XCTest testing framework for the Swift language and revolves around the XCTest task type. This runs a collection of test cases using the Xcode XCTest on macOS or the open source Swift core library alternative on Linux and collates the results. You can then turn those results into a report via an instance of the TestReport task type.

In order to operate, the XCTest task type requires three pieces of information: - Where to find the built testable bundle (on macOS) or executable (on Linux) (property: XCTest.getTestInstalledDirectory()) - The run script for executing the bundle or executable (property: XCTest.getRunScriptFile()) - The working directory to execution the bundle or executable (property: XCTest.getWorkingDirectory())

When you’re using the XCTest Plugin you will automatically get the following: - A dedicated xctest extension of type SwiftXCTestSuite for configuring test component and its variants - A xcTest task of type XCTest that runs those unit tests - A testable bundle or executable linked with the main component’s object files

The test plugins configure the required pieces of information appropriately. In addition, they attach the xcTest or run task to the check lifecycle task. It also create the testImplementation dependency configuration. Dependencies that are only needed for test compilation, linking and runtime may be added to this configuration. The xctest script block behave similarly to a application or library script block.

The XCTest task has many configuration options. We cover a significant number of them in the rest of the chapter.

Test execution

Gradle executes tests in a separate (‘forked’) process.

You can control how the test process is launched via several properties on the XCTest task, including the following:

ignoreFailures - default: false

If this property is true, Gradle will continue with the project’s build once the tests have completed, even if some of them have failed. Note that, by default, both task type always executes every test that it detects, irrespective of this setting.

testLogging - default: not set

This property represents a set of options that control which test events are logged and at what level. You can also configure other logging behavior via this property. Set TestLoggingContainer for more detail.

See XCTest for details on all the available configuration options.

Test filtering

It’s a common requirement to run subsets of a test suite, such as when you’re fixing a bug or developing a new test case. Gradle provides filtering to do this. You can select tests to run based on:

  • A simple class name or method name, e.g. SomeTest, SomeTest.someMethod

  • ‘*’ wildcard matching

You can enable filtering either in the build script or via the --tests command-line option. Here’s an example of some filters that are applied every time the build runs:

Example 1. Filter tests on every build
build.gradle.kts
xctest {
    binaries.configureEach {
        runTask.get().filter.includeTestsMatching("SomeIntegTest.*") // or `"Testing.SomeIntegTest.*"` on macOS
    }
}
build.gradle
xctest {
    binaries.configureEach {
        runTask.get().configure {
            // include all tests from test class
            filter.includeTestsMatching "SomeIntegTest.*" // or `"Testing.SomeIntegTest.*"` on macOS
        }
    }
}

For more details and examples of declaring filters in the build script, please see the TestFilter reference.

The command-line option is especially useful to execute a single test method. It is also possible to supply multiple --tests options, all of whose patterns will take effect. The following sections have several examples of using command-line option.

The test filtering only support XCTest compatible filters at the moment. It means the same filter will differ between macOS and Linux. On macOS, the bundle base name needs to be prepended to the filter, e.g. TestBundle.SomeTest, TestBundle.SomeTest.someMethod See the Simple name pattern section below for more information about valid filtering pattern.

The following section looks at the specific cases of simple class/method names.

Simple name pattern

Gradle support simple class name, or a class name + method name test filtering. For example, the following command lines run either all or exactly one of the tests in the SomeTestClass test case:

# Executes all tests in SomeTestClass
gradle xcTest --tests SomeTestClass
# or `gradle xcTest --tests TestBundle.SomeTestClass` on macOS

# Executes a single specified test in SomeTestClass
gradle xcTest --tests TestBundle.SomeTestClass.someSpecificMethod
# or `gradle xcTest --tests TestBundle.SomeTestClass.someSpecificMethod` on macOS

You can also combine filters defined at the command line with continuous build to re-execute a subset of tests immediately after every change to a production or test source file. The following executes all tests in the ‘SomeTestClass’ test class whenever a change triggers the tests to run:

gradle test --continuous --tests SomeTestClass

Test reporting

The XCTest task generates the following results by default:

  • An HTML test report

  • XML test results in a format compatible with the Ant JUnit report task - one that is supported by many other tools, such as CI servers

  • An efficient binary format of the results used by the XCTest task to generate the other formats

In most cases, you’ll work with the standard HTML report, which automatically includes the result from your XCTest tasks.

There is also a standalone TestReport task type that you can use to generate a custom HTML test report. All it requires are a value for destinationDir and the test results you want included in the report. Here is a sample which generates a combined report for the unit tests from all subprojects:

Example 2. Combine test reports from all subprojects
buildSrc/src/main/kotlin/myproject.xctest-conventions.gradle.kts
plugins {
    id("xctest")
}

extensions.configure<SwiftXCTestSuite>() {
    binaries.configureEach {
        // Disable the test report for the individual test task
        runTask.get().reports.html.required = false
    }
}

configurations.create("binaryTestResultsElements") {
    isCanBeResolved = false
    isCanBeConsumed = true
    attributes {
        attribute(Category.CATEGORY_ATTRIBUTE, objects.named(Category.DOCUMENTATION))
        attribute(DocsType.DOCS_TYPE_ATTRIBUTE, objects.named("test-report-data"))
    }
    tasks.withType<XCTest>() {
        outgoing.artifact(binaryResultsDirectory)
    }
}
build.gradle.kts
plugins {
    `reporting-base`
}

val testReportData by configurations.creating {
    isCanBeConsumed = false
    attributes {
        attribute(Category.CATEGORY_ATTRIBUTE, objects.named(Category.DOCUMENTATION))
        attribute(DocsType.DOCS_TYPE_ATTRIBUTE, objects.named("test-report-data"))
    }
}

dependencies {
    testReportData(project(":core"))
    testReportData(project(":util"))
}

tasks.register<TestReport>("testReport") {
    destinationDirectory = reporting.baseDirectory.dir("allTests")
    // Use test results from testReportData configuration
    testResults.from(testReportData)
}
buildSrc/src/main/groovy/myproject.xctest-conventions.gradle
plugins {
    id 'xctest'
}

xctest {
    binaries.configureEach {
        runTask.get().configure {
            // Disable the test report for the individual test task
            reports.html.required = false
        }
    }
}

// Share the test report data to be aggregated for the whole project
configurations {
    binaryTestResultsElements {
        canBeResolved = false
        attributes {
            attribute(Category.CATEGORY_ATTRIBUTE, objects.named(Category, Category.DOCUMENTATION))
            attribute(DocsType.DOCS_TYPE_ATTRIBUTE, objects.named(DocsType, 'test-report-data'))
        }
        tasks.withType(XCTest).configureEach {
            outgoing.artifact(it.binaryResultsDirectory)
        }
    }
}
build.gradle
// A resolvable configuration to collect test reports data
plugins {
    id 'reporting-base'
}

configurations {
    testReportData {
        canBeConsumed = false
        attributes {
            attribute(Category.CATEGORY_ATTRIBUTE, objects.named(Category, Category.DOCUMENTATION))
            attribute(DocsType.DOCS_TYPE_ATTRIBUTE, objects.named(DocsType, 'test-report-data'))
        }
    }
}

dependencies {
    testReportData project(':core')
    testReportData project(':util')
}

tasks.register('testReport', TestReport) {
    destinationDirectory = reporting.baseDirectory.dir('allTests')
    // Use test results from testReportData configuration
    testResults.from(configurations.testReportData)
}

In this example, we use a convention plugin myproject.xctest-conventions to expose the test results from a project to Gradle’s variant aware dependency management engine.

The plugin declares a consumable binaryTestResultsElements configuration that represents the binary test results of the test task. In the aggregation project’s build file, we declare the testReportData configuration and depend on all of the projects that we want to aggregate the results from. Gradle will automatically select the binary test result variant from each of the subprojects instead of the project’s jar file. Lastly, we add a testReport task that aggregates the test results from the testResultsDirs property, which contains all of the binary test results resolved from the testReportData configuration.

You should note that the TestReport type combines the results from multiple test tasks and needs to aggregate the results of individual test classes. This means that it a given test class is executed by multiple test tasks, then the test report will include executions of that class, but it can be hard to distinguish individual executions of that class and their output.