Testing Java modules with exportable test fixtures

[ppkarwasz]

In the Log4j project for each published artifact we have 4 sets of sources:

1. the main sources for the artifact itself,
2. test fixtures composed mostly of JUnit 4 rules and JUnit 5 extensions that are also consumed by other artifacts. These test fixtures depend on the main sources,
3. some white-box tests (hard to tell if they are unit or integration tests) that require package-level access to 1 and consume 2,
4. some black-box tests that we can put in a separate module and that use 1 and 2

How would you perform testing in this case to detect problems in JPMS encapsulation?

Currently we run all the tests on the classpath, mainly because:

• 3 must be in the same module as 1 (for package access),
• we can not patch module 1 with 3, since the latter depends on 2, which in turn depends on 1. So we get a nice cycle of dependencies.

I don't mind the classpath tests, since Unit tests should probably break encapsulation, but:

• we use Maven (which limits us to two output folders per module) so test fixtures bloat the number of our Maven modules: we have a log4j-api module that only contains 1 and a log4j-api-test module that contains 2 as main sources and 3/4 as test sources,
• we are not testing encapsulation problems, like wrongly declared optional dependencies. If an optional dependency does not have a static modifier, testing on a classpath without that dep will still succeed.

Since JUnit has probably one of the most evolved set of unit and integration tests, do you have any suggestions on how to solve this conundrum? Should we have an additional test-only Maven module for JPMS integration tests? How do you solve this kind of problems in your case?

[sormuras]

Hi Piotr,

with "Log4j project" you are talking about (testing) those artifacts, right?

org.apache.logging.log4j=https://repo.maven.apache.org/maven2/org/apache/logging/log4j/log4j-api/3.0.0-alpha1/log4j-api-3.0.0-alpha1.jar
org.apache.logging.log4j.api.kotlin=https://repo.maven.apache.org/maven2/org/apache/logging/log4j/log4j-api-kotlin/1.3.0/log4j-api-kotlin-1.3.0.jar
org.apache.logging.log4j.api.scala=https://repo.maven.apache.org/maven2/org/apache/logging/log4j/log4j-api-scala_3/13.0.0/log4j-api-scala_3-13.0.0.jar
org.apache.logging.log4j.changelog=https://repo.maven.apache.org/maven2/org/apache/logging/log4j/log4j-changelog/0.5.0/log4j-changelog-0.5.0.jar
org.apache.logging.log4j.core=https://repo.maven.apache.org/maven2/org/apache/logging/log4j/log4j-core/3.0.0-alpha1/log4j-core-3.0.0-alpha1.jar
org.apache.logging.log4j.core.test=https://repo.maven.apache.org/maven2/org/apache/logging/log4j/log4j-core-test/3.0.0-alpha1/log4j-core-test-3.0.0-alpha1.jar
org.apache.logging.log4j.jpl=https://repo.maven.apache.org/maven2/org/apache/logging/log4j/log4j-jpl/3.0.0-alpha1/log4j-jpl-3.0.0-alpha1.jar
org.apache.logging.log4j.jul=https://repo.maven.apache.org/maven2/org/apache/logging/log4j/log4j-jul/3.0.0-alpha1/log4j-jul-3.0.0-alpha1.jar
org.apache.logging.log4j.layout.template.json=https://repo.maven.apache.org/maven2/org/apache/logging/log4j/log4j-layout-template-json/3.0.0-alpha1/log4j-layout-template-json-3.0.0-alpha1.jar
org.apache.logging.log4j.plugins=https://repo.maven.apache.org/maven2/org/apache/logging/log4j/log4j-plugins/3.0.0-alpha1/log4j-plugins-3.0.0-alpha1.jar
org.apache.logging.log4j.plugins.test=https://repo.maven.apache.org/maven2/org/apache/logging/log4j/log4j-plugins-test/3.0.0-alpha1/log4j-plugins-test-3.0.0-alpha1.jar
org.apache.logging.log4j.script=https://repo.maven.apache.org/maven2/org/apache/logging/log4j/log4j-script/3.0.0-alpha1/log4j-script-3.0.0-alpha1.jar
org.apache.logging.log4j.test=https://repo.maven.apache.org/maven2/org/apache/logging/log4j/log4j-api-test/3.0.0-alpha1/log4j-api-test-3.0.0-alpha1.jar
org.apache.logging.log4j.tojul=https://repo.maven.apache.org/maven2/org/apache/logging/log4j/log4j-to-jul/3.0.0-alpha1/log4j-to-jul-3.0.0-alpha1.jar

[ppkarwasz]

Yes, exactly.
Right now each module that contains test fixtures must be split into log4j-<module> and log4j-<module>-test: https://github.com/apache/logging-log4j2/tree/main
The unit tests end up in the *-test module, although this does not feel right.

[sormuras]

... and not a single module-info.java file can be found in the entire repository. (-:

[ppkarwasz]

We recently switched to bnd-maven-plugin and replaced our old manually written module-info.java files, with those automatically generated by BND.

This has some advantages: there is a smaller chance we forget something and IDEs don't go crazy if multiple module descriptors are present (especially Eclipse). However this introduces a problem with optional deps, which are not handled automatically.

[sormuras]

I wrote a blog about modular testing and most parts still apply today: https://sormuras.github.io/blog/2018-09-11-testing-in-the-modular-world

Most parts? I don't like the module-info.test approach as much as I did back then. Today, I prefer dedicated module-info.java files in test-only modules. So, yes to your question:

Should we have an additional test-only Maven module for JPMS integration tests?

An integration test should be as close as possible to what your real users do. They would write something like:

module org.example.app {
  requires org.apache.logging.log4j;
}

Thus, a test could read:

module test.org.apache.logging.log4j {

  requires org.apache.logging.log4j; // module under test
  // ... here be more modules under test
  
  requires test.fixtures; // contains rules and extensions required for testing
  
  requires org.junit.jupiter; // for @Test and friends
  requires org.assertj.core; // for its fluent assertion API
}

[ppkarwasz]

Thanks @sormuras,

Yes, I already read your blog entry, which is basically one of the only references on JPMS testing. However I was unable to apply it to our more evolved case.

[sormuras]

How do you solve this kind of problems in your case?

First, we prevent some problems to show up by sticking with official language and JDK support:

• Use module-info.java syntax to define the API of JUnit's modules.
• Use javac and jar tooling with --release 9 to compile and package them.

Then, we have an integration test compiling and running all example from the User Guide on the module path: https://github.com/junit-team/junit5/blob/main/platform-tooling-support-tests/src/test/java/platform/tooling/support/tests/ModularUserGuideTests.java

Additionally, the junit5-modular-world project from the samples repository acts as another "modular customer" scenario.