Testing Gradle plugins
Testing plays a crucial role in the development process by ensuring reliable and high-quality software. This principle applies to build code, including Gradle plugins.
The sample project
This section revolves around a sample project called the "URL verifier plugin".
This plugin creates a task named verifyUrl
that checks whether a given URL can be resolved via HTTP GET.
The end user can provide the URL via an extension named verification
.
The following build script assumes that the plugin JAR file has been published to a binary repository. The script demonstrates how to apply the plugin to the project and configure its exposed extension:
plugins {
id("org.myorg.url-verifier") (1)
}
verification {
url = "https://www.google.com/" (2)
}
plugins {
id 'org.myorg.url-verifier' (1)
}
verification {
url = 'https://www.google.com/' (2)
}
1 | Applies the plugin to the project |
2 | Configures the URL to be verified through the exposed extension |
Executing the verifyUrl
task renders a success message if the HTTP GET call to the configured URL returns with a 200 response code:
$ gradle verifyUrl
> Task :verifyUrl
Successfully resolved URL 'https://www.google.com/'
BUILD SUCCESSFUL in 0s
5 actionable tasks: 5 executed
Before diving into the code, let’s first revisit the different types of tests and the tooling that supports implementing them.
The importance of testing
Testing is a crucial part of the software development life cycle, ensuring that software functions correctly and meets quality standards before release. Automated testing allows developers to refactor and improve code with confidence.
The testing pyramid
- Manual Testing
-
While manual testing is straightforward, it is error-prone and requires human effort. For Gradle plugins, manual testing involves using the plugin in a build script.
- Automated Testing
-
Automated testing includes unit, integration, and functional testing.
The testing pyramid introduced by Mike Cohen in his book Succeeding with Agile: Software Development Using Scrum describes three types of automated tests:
-
Unit Testing: Verifies the smallest units of code, typically methods, in isolation. It uses Stubs or Mocks to isolate code from external dependencies.
-
Integration Testing: Validates that multiple units or components work together.
-
Functional Testing: Tests the system from the end user’s perspective, ensuring correct functionality. End-to-end tests for Gradle plugins simulate a build, apply the plugin, and execute specific tasks to verify functionality.
Tooling support
Testing Gradle plugins, both manually and automatically, is simplified with the appropriate tools. The table below provides a summary of each testing approach. You can choose any test framework you’re comfortable with.
For detailed explanations and code examples, refer to the specific sections below:
Test type | Tooling support |
---|---|
Any JVM-based test framework |
|
Any JVM-based test framework |
|
Any JVM-based test framework and Gradle TestKit |
Setting up manual tests
The composite builds feature of Gradle makes it easy to test a plugin manually. The standalone plugin project and the consuming project can be combined into a single unit, making it straightforward to try out or debug changes without re-publishing the binary file:
. ├── include-plugin-build (1) │ ├── build.gradle │ └── settings.gradle └── url-verifier-plugin (2) ├── build.gradle ├── settings.gradle └── src
1 | Consuming project that includes the plugin project |
2 | The plugin project |
There are two ways to include a plugin project in a consuming project:
-
By using the command line option
--include-build
. -
By using the method
includeBuild
insettings.gradle
.
The following code snippet demonstrates the use of the settings file:
pluginManagement {
includeBuild("../url-verifier-plugin")
}
pluginManagement {
includeBuild '../url-verifier-plugin'
}
The command line output of the verifyUrl
task from the project include-plugin-build
looks exactly the same as shown in the introduction, except that it now executes as part of a composite build.
Manual testing has its place in the development process, but it is not a replacement for automated testing.
Setting up automated tests
Setting up a suite of tests early on is crucial to the success of your plugin. Automated tests become an invaluable safety net when upgrading the plugin to a new Gradle version or enhancing/refactoring the code.
Organizing test source code
We recommend implementing a good distribution of unit, integration, and functional tests to cover the most important use cases. Separating the source code for each test type automatically results in a project that is more maintainable and manageable.
By default, the Java project creates a convention for organizing unit tests in the directory src/test/java
.
Additionally, if you apply the Groovy plugin, source code under the directory src/test/groovy
is considered for compilation (with the same standard for Kotlin under the directory src/test/kotlin
).
Consequently, source code directories for other test types should follow a similar pattern:
. └── src ├── functionalTest │ └── groovy (1) ├── integrationTest │ └── groovy (2) ├── main │ ├── java (3) └── test └── groovy (4)
1 | Source directory containing functional tests |
2 | Source directory containing integration tests |
3 | Source directory containing production source code |
4 | Source directory containing unit tests |
The directories src/integrationTest/groovy and src/functionalTest/groovy are not based on an existing standard convention for Gradle projects.
You are free to choose any project layout that works best for you.
|
You can configure the source directories for compilation and test execution.
The Test Suite plugin provides a DSL and API to model multiple groups of automated tests into test suites in JVM-based projects. You can also rely on third-party plugins for convenience, such as the Nebula Facet plugin or the TestSets plugin.
Modeling test types
A new configuration DSL for modeling the below integrationTest suite is available via the incubating JVM Test Suite plugin.
|
In Gradle, source code directories are represented using the concept of source sets. A source set is configured to point to one or more directories containing source code. When you define a source set, Gradle automatically sets up compilation tasks for the specified directories.
A pre-configured source set can be created with one line of build script code. The source set automatically registers configurations to define dependencies for the sources of the source set:
// Define a source set named 'test' for test sources
sourceSets {
test {
java {
srcDirs = ['src/test/java']
}
}
}
// Specify a test implementation dependency on JUnit
dependencies {
testImplementation 'junit:junit:4.12'
}
We use that to define an integrationTestImplementation
dependency to the project itself, which represents the "main" variant of our project (i.e., the compiled plugin code):
val integrationTest by sourceSets.creating
dependencies {
"integrationTestImplementation"(project)
}
def integrationTest = sourceSets.create("integrationTest")
dependencies {
integrationTestImplementation(project)
}
Source sets are responsible for compiling source code, but they do not deal with executing the bytecode. For test execution, a corresponding task of type Test needs to be established. The following setup shows the execution of integration tests, referencing the classes and runtime classpath of the integration test source set:
val integrationTestTask = tasks.register<Test>("integrationTest") {
description = "Runs the integration tests."
group = "verification"
testClassesDirs = integrationTest.output.classesDirs
classpath = integrationTest.runtimeClasspath
mustRunAfter(tasks.test)
}
tasks.check {
dependsOn(integrationTestTask)
}
def integrationTestTask = tasks.register("integrationTest", Test) {
description = 'Runs the integration tests.'
group = "verification"
testClassesDirs = integrationTest.output.classesDirs
classpath = integrationTest.runtimeClasspath
mustRunAfter(tasks.named('test'))
}
tasks.named('check') {
dependsOn(integrationTestTask)
}
Configuring a test framework
Gradle does not dictate the use of a specific test framework. Popular choices include JUnit, TestNG and Spock. Once you choose an option, you have to add its dependency to the compile classpath for your tests.
The following code snippet shows how to use Spock for implementing tests:
repositories {
mavenCentral()
}
dependencies {
testImplementation(platform("org.spockframework:spock-bom:2.2-groovy-3.0"))
testImplementation("org.spockframework:spock-core")
testRuntimeOnly("org.junit.platform:junit-platform-launcher")
"integrationTestImplementation"(platform("org.spockframework:spock-bom:2.2-groovy-3.0"))
"integrationTestImplementation"("org.spockframework:spock-core")
"integrationTestRuntimeOnly"("org.junit.platform:junit-platform-launcher")
"functionalTestImplementation"(platform("org.spockframework:spock-bom:2.2-groovy-3.0"))
"functionalTestImplementation"("org.spockframework:spock-core")
"functionalTestRuntimeOnly"("org.junit.platform:junit-platform-launcher")
}
tasks.withType<Test>().configureEach {
// Using JUnitPlatform for running tests
useJUnitPlatform()
}
repositories {
mavenCentral()
}
dependencies {
testImplementation platform("org.spockframework:spock-bom:2.2-groovy-3.0")
testImplementation 'org.spockframework:spock-core'
testRuntimeOnly 'org.junit.platform:junit-platform-launcher'
integrationTestImplementation platform("org.spockframework:spock-bom:2.2-groovy-3.0")
integrationTestImplementation 'org.spockframework:spock-core'
integrationTestRuntimeOnly 'org.junit.platform:junit-platform-launcher'
functionalTestImplementation platform("org.spockframework:spock-bom:2.2-groovy-3.0")
functionalTestImplementation 'org.spockframework:spock-core'
functionalTestRuntimeOnly 'org.junit.platform:junit-platform-launcher'
}
tasks.withType(Test).configureEach {
// Using JUnitPlatform for running tests
useJUnitPlatform()
}
Spock is a Groovy-based BDD test framework that even includes APIs for creating Stubs and Mocks. The Gradle team prefers Spock over other options for its expressiveness and conciseness. |
Implementing automated tests
This section discusses representative implementation examples for unit, integration, and functional tests. All test classes are based on the use of Spock, though it should be relatively easy to adapt the code to a different test framework.
Implementing unit tests
The URL verifier plugin emits HTTP GET calls to check if a URL can be resolved successfully.
The method DefaultHttpCaller.get(String)
is responsible for calling a given URL and returns an instance of type HttpResponse
. HttpResponse
is a POJO containing information about the HTTP response code and message:
package org.myorg.http;
public class HttpResponse {
private int code;
private String message;
public HttpResponse(int code, String message) {
this.code = code;
this.message = message;
}
public int getCode() {
return code;
}
public String getMessage() {
return message;
}
@Override
public String toString() {
return "HTTP " + code + ", Reason: " + message;
}
}
The class HttpResponse
represents a good candidate for a unit test.
It does not reach out to any other classes nor does it use the Gradle API.
package org.myorg.http
import spock.lang.Specification
class HttpResponseTest extends Specification {
private static final int OK_HTTP_CODE = 200
private static final String OK_HTTP_MESSAGE = 'OK'
def "can access information"() {
when:
def httpResponse = new HttpResponse(OK_HTTP_CODE, OK_HTTP_MESSAGE)
then:
httpResponse.code == OK_HTTP_CODE
httpResponse.message == OK_HTTP_MESSAGE
}
def "can get String representation"() {
when:
def httpResponse = new HttpResponse(OK_HTTP_CODE, OK_HTTP_MESSAGE)
then:
httpResponse.toString() == "HTTP $OK_HTTP_CODE, Reason: $OK_HTTP_MESSAGE"
}
}
When writing unit tests, it’s important to test boundary conditions and various forms of invalid input. Try to extract as much logic as possible from classes that use the Gradle API to make it testable as unit tests. It will result in maintainable code and faster test execution. |
You can use the ProjectBuilder class to create Project instances to use when you test your plugin implementation.
public class GreetingPluginTest {
@Test
public void greeterPluginAddsGreetingTaskToProject() {
Project project = ProjectBuilder.builder().build();
project.getPluginManager().apply("org.example.greeting");
assertTrue(project.getTasks().getByName("hello") instanceof GreetingTask);
}
}
Implementing integration tests
Let’s look at a class that reaches out to another system, the piece of code that emits the HTTP calls.
At the time of executing a test for the class DefaultHttpCaller
, the runtime environment needs to be able to reach out to the internet:
package org.myorg.http;
import java.io.IOException;
import java.net.HttpURLConnection;
import java.net.URI;
import java.net.URISyntaxException;
public class DefaultHttpCaller implements HttpCaller {
@Override
public HttpResponse get(String url) {
try {
HttpURLConnection connection = (HttpURLConnection) new URI(url).toURL().openConnection();
connection.setConnectTimeout(5000);
connection.setRequestMethod("GET");
connection.connect();
int code = connection.getResponseCode();
String message = connection.getResponseMessage();
return new HttpResponse(code, message);
} catch (IOException e) {
throw new HttpCallException(String.format("Failed to call URL '%s' via HTTP GET", url), e);
} catch (URISyntaxException e) {
throw new RuntimeException(e);
}
}
}
Implementing an integration test for DefaultHttpCaller
doesn’t look much different from the unit test shown in the previous section:
package org.myorg.http
import spock.lang.Specification
import spock.lang.Subject
class DefaultHttpCallerIntegrationTest extends Specification {
@Subject HttpCaller httpCaller = new DefaultHttpCaller()
def "can make successful HTTP GET call"() {
when:
def httpResponse = httpCaller.get('https://www.google.com/')
then:
httpResponse.code == 200
httpResponse.message == 'OK'
}
def "throws exception when calling unknown host via HTTP GET"() {
when:
httpCaller.get('https://www.wedonotknowyou123.com/')
then:
def t = thrown(HttpCallException)
t.message == "Failed to call URL 'https://www.wedonotknowyou123.com/' via HTTP GET"
t.cause instanceof UnknownHostException
}
}
Implementing functional tests
Functional tests verify the correctness of the plugin end-to-end.
In practice, this means applying, configuring, and executing the functionality of the plugin implementation.
The UrlVerifierPlugin
class exposes an extension and a task instance that uses the URL value configured by the end user:
package org.myorg;
import org.gradle.api.Plugin;
import org.gradle.api.Project;
import org.myorg.tasks.UrlVerify;
public class UrlVerifierPlugin implements Plugin<Project> {
@Override
public void apply(Project project) {
UrlVerifierExtension extension = project.getExtensions().create("verification", UrlVerifierExtension.class);
UrlVerify verifyUrlTask = project.getTasks().create("verifyUrl", UrlVerify.class);
verifyUrlTask.getUrl().set(extension.getUrl());
}
}
Every Gradle plugin project should apply the plugin development plugin to reduce boilerplate code. By applying the plugin development plugin, the test source set is preconfigured for the use with TestKit. If we want to use a custom source set for functional tests and leave the default test source set for only unit tests, we can configure the plugin development plugin to look for TestKit tests elsewhere.
gradlePlugin {
testSourceSets(functionalTest)
}
gradlePlugin {
testSourceSets(sourceSets.functionalTest)
}
Functional tests for Gradle plugins use an instance of GradleRunner
to execute the build under test.
GradleRunner
is an API provided by TestKit, which internally uses the Tooling API to execute the build.
The following example applies the plugin to the build script under test, configures the extension and executes the build with the task verifyUrl
.
Please see the TestKit documentation to get more familiar with the functionality of TestKit.
package org.myorg
import org.gradle.testkit.runner.GradleRunner
import spock.lang.Specification
import spock.lang.TempDir
import static org.gradle.testkit.runner.TaskOutcome.SUCCESS
class UrlVerifierPluginFunctionalTest extends Specification {
@TempDir File testProjectDir
File buildFile
def setup() {
buildFile = new File(testProjectDir, 'build.gradle')
buildFile << """
plugins {
id 'org.myorg.url-verifier'
}
"""
}
def "can successfully configure URL through extension and verify it"() {
buildFile << """
verification {
url = 'https://www.google.com/'
}
"""
when:
def result = GradleRunner.create()
.withProjectDir(testProjectDir)
.withArguments('verifyUrl')
.withPluginClasspath()
.build()
then:
result.output.contains("Successfully resolved URL 'https://www.google.com/'")
result.task(":verifyUrl").outcome == SUCCESS
}
}
IDE integration
TestKit determines the plugin classpath by running a specific Gradle task.
You will need to execute the assemble
task to initially generate the plugin classpath or to reflect changes to it even when running TestKit-based functional tests from the IDE.
Some IDEs provide a convenience option to delegate the "test classpath generation and execution" to the build. In IntelliJ, you can find this option under Preferences… > Build, Execution, Deployment > Build Tools > Gradle > Runner > Delegate IDE build/run actions to Gradle.