Creating Test Stages With JUnit

Over the last few years, the prominent effectiveness of Continuous Integration (CI) and Continuous Delivery (CD) have cemented their places in the software distribution toolbox and are likely here to stay. One aspect of this move away from previous integration and delivery schemes is the reduction of the build, package, test, and delivery steps into stages that can be chained together into a pipeline. This allows for standalone portions of a build to be completed in serial or in parallel, with the completion of each step leading to the execution of subsequent steps.

This not only allows us to separate certain portions of a build from one another, but it also allows us to run more nimble parts of the build without waiting for more lengthy portions to complete. In particular, unit tests are often executed first, while integration, acceptance, performance, and stress tests are often executed later in a pipeline. This allows us to quickly know if the fast unit tests—which provide us a basic, yet essential coverage of isolated parts of our system—failed and stop all subsequent, long-running stages (i.e. fail-fast) which would otherwise waste time and resources.

In order to apply the efficiency and versatility of staged builds, we first need to break our build into stages that can be run independently of one another. In this article, we will look at how to break the execution of JUnit tests into categories, allowing us to execute unit and integration tests separately. This article includes information on how to execute these stages in both Maven and Gradle and will explore how to divide our tests into categories in both JUnit 4 and JUnit 5. All of the source code and build scripts for this article can be found in the following repositories:

• Maven & Gradle Builds with Junit 4
• Maven & Gradle Builds with Junit 5

Creating an Example Project

In order to demonstrate how to create test stages, we will first create a simple example project that includes objects that we can test in isolation using unit tests, as well as more complex objects, such as a Representation State Transfer (REST) controller, that will require integration tests to fully cover. This distinction between unit and integration tests will eventually lead to the development of two test stages that can be executed independently of one another.

Note that this project uses Spring 5 WebFlux to create the REST controller, but it is not required that the reader fully understand Spring 5 or even the mechanism used to test a Spring 5 controller. Instead, any desired framework can be used, but conceptually, it should be understood that we are creating a REST controller that requires more than simple unit tests to exercise. The details of the REST controller and its associated integration tests will be explained at a high-level, but the interested reader should refer to the Guide to Spring 5 WebFlux and the Spring 5 WebClient and WebTestClient Tutorial with Examples.

Our project will follow the standard design of creating domain objects (one in our case: Foo), creating a repository object to manage the persistence of our domain objects (FooRepository), and creating a REST controller to allow clients to access our domain objects over Hypertext Transfer Protocol (HTTP) (FooController). Since this is a simple project, we will create an in-memory repository object, rather than a repository backed by persistent storage, such as a database or file system.

Being that we only need to store an ID for our domain objects, our Foo domain class ends up being trivially simple:

public class Foo {

    private long id;

    public Foo() {}

    public Foo(long id) {
        this.id = id;
    }

    public void setId(long id) {
        this.id = id;
    }

    public long getId() {
        return id;
    }
}

With our domain class established, we can now implement our repository class:

@Service
public class FooRepository {

    private final List<Foo> foos = new ArrayList<>();

    public void save(Foo foo) {
        foos.add(foo);
    }

    public Optional<Foo> findById(long id) {
        return foos.stream()
            .filter(foo -> foo.getId() == id)
            .findFirst();
    }

    public List<Foo> findAll() {
        return foos;
    }
}

The @Service annotation informs the Spring framework that this class is a candidate to be autowired into another object using the Spring Dependency Injection (DI) framework. This will be important when we create our REST controller later. This class stores its Foo objects in a List and has three methods that allow a client to add a new Foo object to the repository, find an existing Foo object by its ID, or obtain a List of all existing Foo objects.

Lastly, with our domain and repository classes implemented, we create our REST controller:

@RestController
@RequestMapping("/foo")
public class FooController {

    @Autowired
    private FooRepository repository;

    @GetMapping("/{id}")
    public Mono<Foo> findById(@PathVariable Long id) {
        return repository.findById(id)
            .map(Mono::just)
            .orElseThrow();
    }

    @ResponseStatus(HttpStatus.NOT_FOUND)
    @ExceptionHandler(NoSuchElementException.class)
    public void handleNotFound() {}
}

The @RestController annotation informs the Spring framework that this controller should be included as a part of the HTTP server that is started by Spring when the application is executed. The @RequestMapping annotation informs Spring that we wish for this controller to have the path /foo (i.e. http://localhost:8080/foo). The @Autowired annotation instructs Spring to inject an object of the FooRepository class that we previously created, allowing us to access Foo domain objects within our controller. The findById method accepts the desired ID as a path variable and to find a Foo object in the FooRepository with a matching ID. If one is found, it is converted to a Mono object and returned or else a NoSuchElementException is thrown (using the orElseThrow method).

The handleNotFound method is responsible for handling this exception (using the @ExceptionHandler annotation to inform Spring of its ability to handle a NoSuchElementException) and converting it to an HTTP 404 response (as denoted by the @ResponseStatus annotation). While the three classes we have created are relatively simple, they require some nuance when deciding which level of testing is appropriate.

Adding Unit and Integration Tests

Since our domain object is trivially simple, and for the sake of conciseness, we will forgo adding JUnit tests to exercise its functionality. Instead, we will create a set of unit tests for our FooRepository to ensure that it functions as intended. Since our FooController class has interconnected dependencies with our FooRepository and the Spring WebFlux framework, we will create a set of integration tests to exercise its functionality.

Two important points to note: (1) although there is no syntactic difference between a JUnit unit test and a JUnit integration test (JUnit simply provides a mechanism for executing automated tests, regardless of the nature of the tests themselves), we will make a logical separation between unit and integration tests using annotations; and (2) both unit and integration tests should be created prior to the implementation of the classes they exercise, in line with the practices of Test-Driven Development (TDD). For the purposes of brevity, we did not walk through the process of creating a test and then implementing a small portion of code to get that test to pass. Instead, the tests in the following section can be thought of as the final result after the test suite and desired classes were fully implemented. In practice, though, these tests would be created using TDD.

JUnit 4

In order to differentiate the unit tests from integration tests, we will use JUnit 4 Categories. In the case of our unit tests, we will not define a category and instead treat them as the default set of tests that will be executed whenever a build is run (unless explicitly restricted from executing). This results in the following unit test class for our FooRepository:

public class FooRepositoryTest {

    private FooRepository repository;

    @Before
    public void setUp() {
        repository = new FooRepository();
    }

    @Test
    public void noExistingFoosResultsInFooNotFoundById() {
        assertTrue(repository.findAll().isEmpty());
        assertFalse(repository.findById(1).isPresent());
    }

    @Test
    public void existingFooNotMatchingIdResultsInFooNotFoundById() {

        long desiredId = 1;
        long differentId = 2;

        repository.save(new Foo(differentId));

        assertFalse(repository.findById(desiredId).isPresent());
    }

    @Test
    public void existingFooMatchingIdResultsInFooFoundById() {

        Foo foo = new Foo(1);

        repository.save(foo);

        Optional<Foo> result = repository.findById(foo.getId());

        assertTrue(result.isPresent());
        assertEquals(foo, result.get());
    }
}

We are simply exercising three cases in which a desired Foo object may be found by its ID: (1) no Foo objects exists, which results in no Foo object found by any ID (i.e. return an empty Optional), (2) a Foo object with a different ID exists, which results in no Foo object being found by the desired ID, and (3) there exists a Foo object with the desired ID, which results in the desired Foo object being returned (i.e. an Optional populated with the desired Foo object).

For the integration test that exercises our FooController, we need to create a marker interface that is used by the JUnit @Category annotation to categorize our test class. We will aptly name this interface IntegrationTest:

public interface IntegrationTest {}

We will also need to create a Spring configuration that informs Spring how it should inject our desired dependencies and where possible injection candidates should be found:

@Configuration
@ComponentScan("com.dzone.albanoj2.example.junitstages")
public class ControllerTestConfig {}

In this case, the @Configuration annotation informs Spring that this class should be used a configuration class and the @ComponentScan annotation states that Spring should look for injection candidates in the com.dzone.albanoj2.example.junitstages package or any of its sub-packages. Note that this package name will change based on the package structure of a particular project. Using this configuration, we can define our test cases for FooController:

@WebFluxTest
@RunWith(SpringRunner.class)
@Category(IntegrationTest.class)
@ContextConfiguration(classes = ControllerTestConfig.class)
public class FooControllerTest {

    @Autowired
    private WebTestClient webClient;

    @MockBean
    private FooRepository repository;

    @Test
    public void findByIdWithNoFoosResultsIn404() {

        webClient.get()
            .uri("/foo/{id}", 1)
            .accept(MediaType.APPLICATION_JSON)
            .exchange()
            .expectStatus().isNotFound();
    }

    @Test
    public void findByIdWithFooWithDifferentIdResultsInFooNotFound() {

        long desiredId = 1;
        long differentId = 2;
        Foo foo = new Foo(differentId);

        doReturn(Optional.of(foo)).when(repository).findById(eq(differentId));

        webClient.get()
            .uri("/foo/{id}", desiredId)
            .accept(MediaType.APPLICATION_JSON)
            .exchange()
            .expectStatus().isNotFound();
    }

    @Test
    public void findByIdWithMatchingFoosResultsInFooFound() {

        long id = 1;
        Foo foo = new Foo(id);

        doReturn(Optional.of(foo)).when(repository).findById(eq(id));

        webClient.get()
            .uri("/foo/{id}", id)
            .accept(MediaType.APPLICATION_JSON)
            .exchange()
            .expectStatus().isOk()
            .expectBody(Foo.class);
    }
}

The @WebFluxTest configures the test class to execute using the default Spring WebFlux configuration (i.e. how the test class should be bootstrapped, configuring the WebTestClient, etc.), the @RunsWith annotation instructs JUnit to execute this test class using the SpringRunner test runner, the @Category annotation categorizes the test cases using the marker interface we previously created, and the @ContextConfiguration annotation instructs Spring to use the ControllerTestConfig class we previously created.

The @MockBean annotation allows for a mock FooRepository object to be injected into both the test class and the FooController class. This allows us to define the expected outputs for the FooRepository class when running our test cases, which in turn allows us to control the output received by the FooController when calling methods on the FooRepository that was autowired into it. Each of the tests cases simply executes an HTTP GET request using the injected WebTestClient object and inspects the results. Note that the doReturn calls are Mockito methods that configure the mock FooRepository object to return a certain value when that method is called (see the Mockito class documentation for more information).

JUnit 5

Creating the equivalent test classes using JUnit 5 is similar to the steps laid out in the previous section, but there are some important differences. Chiefly among these differences that the @Category annotation has been replaced with the @Tag annotation. The @Tag annotation allows test classes to be categorized using a String value rather than a marker interface, but that introduces another problem. For each of the test classes (suppose we had 10 integration test classes), we would need to repeat the exact same String value when tagging each class (i.e. we would need to annotate every integration test class with @Tag("integration") and hope that we did not mis-type "integration" or define some constant and ensure we used the same constant everywhere). Note that any String value can be used (i.e. integration does not represent any special value in JUnit).

Fortunately, JUnit 5 can process meta-annotations, which allows us to create an @IntegrationTest annotation that can be used in place of @Tag("integration"):

@Target({ ElementType.TYPE, ElementType.METHOD })
@Retention(RetentionPolicy.RUNTIME)
@Tag("integration")
public @interface IntegrationTest {}

This means that when JUnit 5 sees a test class annotated with @IntegrationTest, it processes that class as if it were annotated with @Tag("integration") instead. With this annotation defined, we can create our test classes. First, we define the FooRepository unit test class in the same way we did for JUnit 4, replacing JUnit 4 annotations with JUnit 5 annotations:

public class FooRepositoryTest {

    private FooRepository repository;

    @BeforeEach
    public void setUp() {
        repository = new FooRepository();
    }

    @Test
    public void noExistingFoosResultsInFooNotFoundById() {
        assertTrue(repository.findAll().isEmpty());
        assertFalse(repository.findById(1).isPresent());
    }

    @Test
    public void existingFooNotMatchingIdResultsInFooNotFoundById() {

        long desiredId = 1;
        long differentId = 2;

        repository.save(new Foo(differentId));

        assertFalse(repository.findById(desiredId).isPresent());
    }

    @Test
    public void existingFooMatchingIdResultsInFooFoundById() {

        Foo foo = new Foo(1);

        repository.save(foo);

        Optional<Foo> result = repository.findById(foo.getId());

        assertTrue(result.isPresent());
        assertEquals(foo, result.get());
    }
}

Next, we define our Spring configuration class in exactly the same manner as JUnit 4:

@Configuration
@ComponentScan("com.dzone.albanoj2.example.junitstages")
public class ControllerTestConfig {}

Lastly, we define the integration test class for our FooController in a slightly different manner:

@WebFluxTest
@IntegrationTest
@ContextConfiguration(classes = ControllerTestConfig.class)
public class FooControllerTest {

    @Autowired
    private WebTestClient webClient;

    @MockBean
    private FooRepository repository;

    @Test
    public void findByIdWithNoFoosResultsIn404() {

        webClient.get()
            .uri("/foo/{id}", 1)
            .accept(MediaType.APPLICATION_JSON)
            .exchange()
            .expectStatus().isNotFound();
    }

    @Test
    public void findByIdWithFooWithDifferentIdResultsInFooNotFound() {

        long desiredId = 1;
        long differentId = 2;
        Foo foo = new Foo(differentId);

        doReturn(Optional.of(foo)).when(repository).findById(eq(differentId));

        webClient.get()
            .uri("/foo/{id}", desiredId)
            .accept(MediaType.APPLICATION_JSON)
            .exchange()
            .expectStatus().isNotFound();
    }

    @Test
    public void findByIdWithMatchingFoosResultsInFooFound() {

        long id = 1;
        Foo foo = new Foo(id);

        doReturn(Optional.of(foo)).when(repository).findById(eq(id));

        webClient.get()
            .uri("/foo/{id}", id)
            .accept(MediaType.APPLICATION_JSON)
            .exchange()
            .expectStatus().isOk()
            .expectBody(Foo.class);
    }
}

A majority of the test class is identical to the JUnit example, but there are two important differences to note: (1) we annotate our test class with @IntegrationTest rather than @Category, and (2) we can remove the @RunWith annotation, as the default-configured runner supplied by JUnit 5 is sufficient. Apart from these two differences, the rest of the test code itself remains unchanged. Regardless of which JUnit version is used, we are now ready to configure our build scripts to execute only certain tests, depending on the build command executed.

Configuring the Build Scripts

In recent years, both Maven and Gradle have been the main players in the Java build ecosystem. While there are a number of similarities between Maven and Gradle, the process of executing test cases based on their annotations is very different. The following sections will walk through the process of configuring either Maven or Gradle to execute unit and integration tests separately in either JUnit 4 or JUnit 5.

Maven

With Maven, we can separate the execution of unit and integration tests using build profiles. Build profiles allow us to change the configuration of a Maven build depending on the selected profile. In our case, we will configure Maven to execute our unit tests whenever a build is performed with the default profile (i.e. no build profile is supplied) and execute only our integration tests (no unit tests) when the integration build profile is supplied. Note that the build profile can be named any valid profile name (not just integration) and that integration does not represent any special value in Maven.

Using the build profile mechanism, by default, we will configure the maven-surefire-plugin, which executes our JUnit tests, to exclude any integration tests, denoted by the <excludedGroups> element. If the integration build profile is selected, we will override this default maven-surefire-plugin configuration and include only integration tests (utilizing the @Category or @Tag annotations, depending on the JUnit version used) using the <groups> element. By specifying a value for <groups>, all other groups will be excluded (see Using JUnit Categories).

The specifics of how to do this vary by the JUnit version we use.

JUnit 4

Using JUnit 4, we have to exclude or include groups using the fully-qualified class name of the class supplied to @Category. In our case, that means com.dzone.albanoj2.example.junitstages.rest.IntegrationTest. Simply add the following build plugin to the project pom.xml file:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <configuration>
        <excludedGroups>com.dzone.albanoj2.example.junitstages.rest.IntegrationTest</excludedGroups>
        <argLine>--illegal-access=permit</argLine>
    </configuration>
</plugin>

The <excludedGroups> element instructs Surefire to run any tests that do not have a category of com.dzone.albanoj2.example.junitstages.rest.IntegrationTest. Note that this has been verified to work with Surefire plugin version 2.1.1.RELEASE and mileage may vary with older versions. Also note that the <argLine>--illegal-access=permit</argLine> is needed when compiling with Java 9 or higher and is not specific to having different test stages (i.e. it can be safely removed if compiling with Java 8 or lower).

Next, add the following build profile to the pom.xml file:

<profile>
    <id>integration</id>
    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <configuration>
                    <groups>com.dzone.albanoj2.example.junitstages.rest.IntegrationTest</groups>
                    <excludedGroups>none</excludedGroups>
                </configuration>
            </plugin>
        </plugins>
    </build>
</profile>

Note that <id> element corresponds to the name of our build profile. The <groups> element instructs Surefire to include only those test classes that have a category of com.dzone.albanoj2.example.junitstages.rest.IntegrationTest (all others will be excluded). We include a dummy value (none) for <excludedGroups> to overwrite the <excludedGroups>com.dzone.albanoj2.example.junitstages.rest.IntegrationTest</excludedGroups>element that specified as the default Surefire plugin configuration. Note that if <excludedGroups> is not explicitly supplied in our build profile, the <excludedGroups> from our Surefire plugin configuration will be used instead, which would both exclude and include the same group.

Note that none is not a special value, but rather, a category class name that is unlikely to ever be used. If, hypothetically, there was a class with a fully-qualified class name of none on the classpath for our project, we could run into trouble.

JUnit 5

The configuration and thought process is identical for JUnit 5, but instead of using the fully-qualified class name, we use the String value we provided to the @Tag annotation (i.e. integration). For the Surefire plugin, we add the following to our pom.xml file:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <configuration>
        <excludedGroups>integration</excludedGroups>
        <argLine>--illegal-access=permit</argLine>
    </configuration>
</plugin>

For the build profile, we add the following to the pom.xml file:

<profile>
    <id>integration</id>
    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <configuration>
                    <groups>integration</groups>
                    <excludedGroups>none</excludedGroups>
                </configuration>
            </plugin>
        </plugins>
    </build>
</profile>

Note that the profile ID and the group name do not have to match, but in this case, they do.

Running the Test Stages

Regardless of the JUnit version used, the unit tests can be executed using the following command (note that the clean phase can be excluded if a clean build is not needed):

mvn clean test

Likewise, the integration tests can be executed by specifying the integration build profile using the following command (again, clean is optional):

mvn clean test -P integration

Note that any phase that includes the test phase will also execute either the unit tests or integration tests (if a build profile is supplied). For example, mvn clean install will execute the unit tests while mvn clean install -P integration will likewise execute the integration tests.

Gradle

Although the thought process is similar for Gradle, the mechanism we use is different. Instead of using build profiles and overriding the default Surefire configuration, we define two tasks: One for unit tests and another for integration tests.

JUnit 4

In the case of JUnit 4, we will define the following tasks in the build.gradle file:

test {

    useJUnit {
        excludeCategories 'com.dzone.albanoj2.example.junitstages.rest.IntegrationTest'
    }

    testLogging {
        events 'PASSED', 'FAILED', 'SKIPPED'
    }
}

task integrationTest(type: Test) {

    useJUnit {
        includeCategories 'com.dzone.albanoj2.example.junitstages.rest.IntegrationTest'
    }

    check.dependsOn it
    shouldRunAfter test

    testLogging {
        events 'PASSED', 'FAILED', 'SKIPPED'
    }
}

In both the test and integrationTest tasks (we override the test task already defined by Gradle), we simply specify which categories to include or exclude (includeCateories and excludeCategories, respectively) using the fully-qualified class name of the class we supplied to the @Category annotation (see this Stack Overflow post). We also include testLogging so that we can see the results of each of the tests that run, but this is not required (and the test case output is omitted by default if no testLogging element is supplied). In this case, we configure both tasks to print test logging output for the PASSED, FAILED, and SKIPPED events.

Note that in the integrationTest task, we also create an ordering for the Gradle build steps. Using the check.dependsOn it statement, we tell Gradle that the integrationTest task should be run whenever the check task is run. Similarly, using the shouldRunAfter test statement, we also instruct Gradle to execute the integrationTest task after the test task if both tasks are run in the same Gradle build.

JUnit 5

The build configuration for JUnit 5 is nearly identical, but instead of fully-qualified class names, we use the String-based tag names instead (i.e. integration) and replace includeCategories and excludeCategories with includeTags and excludeTags, respectively:

test {

    useJUnitPlatform {
        excludeTags 'integration'
    }

    testLogging {
        events 'PASSED', 'FAILED', 'SKIPPED'
    }
}

task integrationTest(type: Test) {

    useJUnitPlatform {
        includeTags 'integration'
    }

    check.dependsOn it
    shouldRunAfter test

    testLogging {
        events 'PASSED', 'FAILED', 'SKIPPED'
    }
}

Running the Test Stags

Unlike our Maven case, which overrides the default Surefire plugin configuration, the unit and integration test tasks are independent tasks, which allows them to be run separately or both in the same build. To run the unit tests only, execute the following command (the clean task can be removed if a clean build is not needed):

gradle clean test

To execute the integration tests only, execute the following command (again, clean is optional):

gradle clean integrationTest

To run both the unit and integration tests, execute the following command (clean is optional):

gradle clean test integrationTest

Note that if another task includes the test or integrationTest tasks, the unit or integration tests, respectively, will run. For example, running the following command will also execute both the unit and integration tests:

gradle clean check

Conclusion

With the advent of CI and CD, building software in stages has become the norm among many of the most successful software products. Staged, independent build steps, and particularly staged tests, allow for pipelines to fail-fast without wasting time and resources executing long-running tests on a build that has already failed. Not only does this introduce independence and flexibility in the build process, but it also allows for pipelined stages to be executed in parallel, dramatically reducing build times and providing engineering and operations with more fine-grained control over how builds are executed.