Testing in Swift projects
The Swift testing support is not compatible with the configuration cache. |
Testing in the native ecosystem is a rich subject matter. There are many different testing libraries and frameworks, as well as many different types of test. All need to be part of the build, whether they are executed frequently or infrequently. This chapter is dedicated to explaining how Gradle handles differing requirements between and within builds, with significant coverage of how it integrates with XCTest on both macOS and Linux.
It explains: - Ways to control how the tests are run (Test execution) - How to select specific tests to run (Test filtering) - What test reports are generated and how to influence the process (Test reporting) - How Gradle finds tests to run (Test detection)
But first, we look at the basics of native testing in Gradle.
The basics
Gradle supports deep integration with XCTest testing framework for the Swift language and revolves around the XCTest task type. This runs a collection of test cases using the Xcode XCTest on macOS or the open source Swift core library alternative on Linux and collates the results. You can then turn those results into a report via an instance of the TestReport task type.
In order to operate, the XCTest task type requires three pieces of information: - Where to find the built testable bundle (on macOS) or executable (on Linux) (property: XCTest.getTestInstalledDirectory()) - The run script for executing the bundle or executable (property: XCTest.getRunScriptFile()) - The working directory to execution the bundle or executable (property: XCTest.getWorkingDirectory())
When you’re using the XCTest Plugin you will automatically get the following:
- A dedicated xctest
extension of type SwiftXCTestSuite for configuring test component and its variants
- A xcTest
task of type XCTest that runs those unit tests
- A testable bundle or executable linked with the main component’s object files
The test plugins configure the required pieces of information appropriately.
In addition, they attach the xcTest
or run
task to the check
lifecycle task.
It also create the testImplementation
dependency configuration.
Dependencies that are only needed for test compilation, linking and runtime may be added to this configuration.
The xctest
script block behave similarly to a application
or library
script block.
The XCTest task has many configuration options. We cover a significant number of them in the rest of the chapter.
Test execution
Gradle executes tests in a separate (‘forked’) process.
You can control how the test process is launched via several properties on the XCTest task, including the following:
ignoreFailures
- default: false-
If this property is
true
, Gradle will continue with the project’s build once the tests have completed, even if some of them have failed. Note that, by default, both task type always executes every test that it detects, irrespective of this setting. testLogging
- default: not set-
This property represents a set of options that control which test events are logged and at what level. You can also configure other logging behavior via this property. Set TestLoggingContainer for more detail.
See XCTest for details on all the available configuration options.
Test filtering
It’s a common requirement to run subsets of a test suite, such as when you’re fixing a bug or developing a new test case. Gradle provides filtering to do this. You can select tests to run based on:
-
A simple class name or method name, e.g.
SomeTest
,SomeTest.someMethod
-
‘*’ wildcard matching
You can enable filtering either in the build script or via the --tests
command-line option.
Here’s an example of some filters that are applied every time the build runs:
xctest {
binaries.configureEach {
runTask.get().filter.includeTestsMatching("SomeIntegTest.*") // or `"Testing.SomeIntegTest.*"` on macOS
}
}
xctest {
binaries.configureEach {
runTask.get().configure {
// include all tests from test class
filter.includeTestsMatching "SomeIntegTest.*" // or `"Testing.SomeIntegTest.*"` on macOS
}
}
}
For more details and examples of declaring filters in the build script, please see the TestFilter reference.
The command-line option is especially useful to execute a single test method.
It is also possible to supply multiple --tests
options, all of whose patterns will take effect.
The following sections have several examples of using command-line option.
The test filtering only support XCTest compatible filters at the moment.
It means the same filter will differ between macOS and Linux.
On macOS, the bundle base name needs to be prepended to the filter, e.g. TestBundle.SomeTest , TestBundle.SomeTest.someMethod
See the Simple name pattern section below for more information about valid filtering pattern.
|
The following section looks at the specific cases of simple class/method names.
Simple name pattern
Gradle support simple class name, or a class name + method name test filtering.
For example, the following command lines run either all or exactly one of the tests in the SomeTestClass
test case:
# Executes all tests in SomeTestClass
gradle xcTest --tests SomeTestClass
# or `gradle xcTest --tests TestBundle.SomeTestClass` on macOS
# Executes a single specified test in SomeTestClass
gradle xcTest --tests TestBundle.SomeTestClass.someSpecificMethod
# or `gradle xcTest --tests TestBundle.SomeTestClass.someSpecificMethod` on macOS
You can also combine filters defined at the command line with continuous build to re-execute a subset of tests immediately after every change to a production or test source file. The following executes all tests in the ‘SomeTestClass’ test class whenever a change triggers the tests to run:
gradle test --continuous --tests SomeTestClass
Test reporting
The XCTest task generates the following results by default:
-
An HTML test report
-
XML test results in a format compatible with the Ant JUnit report task - one that is supported by many other tools, such as CI servers
-
An efficient binary format of the results used by the
XCTest
task to generate the other formats
In most cases, you’ll work with the standard HTML report, which automatically includes the result from your XCTest
tasks.
There is also a standalone TestReport task type that you can use to generate a custom HTML test report.
All it requires are a value for destinationDir
and the test results you want included in the report.
Here is a sample which generates a combined report for the unit tests from all subprojects:
plugins {
id("xctest")
}
extensions.configure<SwiftXCTestSuite>() {
binaries.configureEach {
// Disable the test report for the individual test task
runTask.get().reports.html.required = false
}
}
configurations.create("binaryTestResultsElements") {
isCanBeResolved = false
isCanBeConsumed = true
attributes {
attribute(Category.CATEGORY_ATTRIBUTE, objects.named(Category.DOCUMENTATION))
attribute(DocsType.DOCS_TYPE_ATTRIBUTE, objects.named("test-report-data"))
}
tasks.withType<XCTest>() {
outgoing.artifact(binaryResultsDirectory)
}
}
plugins {
`reporting-base`
}
val testReportData by configurations.creating {
isCanBeConsumed = false
attributes {
attribute(Category.CATEGORY_ATTRIBUTE, objects.named(Category.DOCUMENTATION))
attribute(DocsType.DOCS_TYPE_ATTRIBUTE, objects.named("test-report-data"))
}
}
dependencies {
testReportData(project(":core"))
testReportData(project(":util"))
}
tasks.register<TestReport>("testReport") {
destinationDirectory = reporting.baseDirectory.dir("allTests")
// Use test results from testReportData configuration
testResults.from(testReportData)
}
plugins {
id 'xctest'
}
xctest {
binaries.configureEach {
runTask.get().configure {
// Disable the test report for the individual test task
reports.html.required = false
}
}
}
// Share the test report data to be aggregated for the whole project
configurations {
binaryTestResultsElements {
canBeResolved = false
attributes {
attribute(Category.CATEGORY_ATTRIBUTE, objects.named(Category, Category.DOCUMENTATION))
attribute(DocsType.DOCS_TYPE_ATTRIBUTE, objects.named(DocsType, 'test-report-data'))
}
tasks.withType(XCTest).configureEach {
outgoing.artifact(it.binaryResultsDirectory)
}
}
}
// A resolvable configuration to collect test reports data
plugins {
id 'reporting-base'
}
configurations {
testReportData {
canBeConsumed = false
attributes {
attribute(Category.CATEGORY_ATTRIBUTE, objects.named(Category, Category.DOCUMENTATION))
attribute(DocsType.DOCS_TYPE_ATTRIBUTE, objects.named(DocsType, 'test-report-data'))
}
}
}
dependencies {
testReportData project(':core')
testReportData project(':util')
}
tasks.register('testReport', TestReport) {
destinationDirectory = reporting.baseDirectory.dir('allTests')
// Use test results from testReportData configuration
testResults.from(configurations.testReportData)
}
In this example, we use a convention plugin myproject.xctest-conventions
to expose the test results from a project to Gradle’s variant aware dependency management engine.
The plugin declares a consumable binaryTestResultsElements
configuration that represents the binary test results of the test
task.
In the aggregation project’s build file, we declare the testReportData
configuration and depend on all of the projects that we want to aggregate the results from. Gradle will automatically select the binary test result variant from each of the subprojects instead of the project’s jar file.
Lastly, we add a testReport
task that aggregates the test results from the testResultsDirs
property, which contains all of the binary test results resolved from the testReportData
configuration.
You should note that the TestReport type combines the results from multiple test tasks and needs to aggregate the results of individual test classes. This means that it a given test class is executed by multiple test tasks, then the test report will include executions of that class, but it can be hard to distinguish individual executions of that class and their output.