Maven Cobertura and JaCoCo Plugins – What Are Your Rock Bottom, Minimum Standards?

When it comes to test driven development, the best line I’ve heard is the following.

Your clients are not a distributed test environment

These wise words where uttered by my Spring Core lecturer while covering the difference between unit and integration tests in Spring parlance. On the note of unit and integration tests, after working on a vast array of projects, it has dawned on me, with some sadness, that not a single project, organization or team I’ve been on has had non negotiable standards when it comes to code coverage. Of course, most projects have had unit testing as a part of a checklist, but not a single one has made a lasting effort to both measure and enforce a minimum goal in terms of code coverage.

In this post we take a look at potential rock bottom configurations for the free cobertura-maven-plugin in particular and also visit the jacoco-maven-plugin. Finally we encounter lacking JDK 8 support and start considering paying for a commercial plugin.

Before delving into code coverage tooling it’s worth asking why it matters, to whom, and what it means. So, what does code coverage mean? Why should software engineers care? Why should project managers care? Why should project sponsors care? If a consultancy (vendor) is doing the development, why should this organisation care? These are not questions we’ll delve into here in depth other than noting that coverage reports help detect code that had not been adequately tested by automated test suites.

No Standards? Introduce a Lower Quality Gate

So what to do in a world of little to no standards? In my mind the answer is to set one’s own personal standards, starting with defining what rock bottom is. This is a personal professional line in the sand. Its also a great question, when considering joining a project or organization, to ask of your prospective employer. The question would be what unit, integration and system test code coverage standards the organization has and then crucially how they are enforced and made visible to all concerned.

In terms of motivating the need to minimum standards, the term quality gate seems apt. On a given project, even personal project, one would have two gates, the lower gate would be enabled by default and builds will fail on developer machines if this minimum standard is not met, a CI server would also independently verify using the minimum standard. If this lower quality gate has not been met, the project manager or development manager should know about it.

The Plugins

Lets move onto the plugins. The cobertura-maven-plugin is used to report on and check your unit test code coverage using the Cobertura code coverage utility. So we’ll first check if all tests are passing and then check to make sure our standards have been met. Once we move onto the integration test phase, where our beans and infrastructure is tested in concert, the jacoco-maven-plugin will report on and check our integration test code coverage. 

The importance of performing both unit testing (individual classes) and integration testing (incorporating a container such as the Spring context) cannot be overstated. Both plugins and so both types of testing must be done in a given project and this stands to reason: we want coverage standards for individual classes as well as cooperating runtime services and ordinarily we only proceed to the latter once the former has succeeded as per the Maven Build Lifecycle.

Rock Bottom – Our Lower Quality Gate

It stands to reason that there some should be some correlation between the application domain and the amount of effort one will invest in unit and integration testing. When it comes to rock bottom however the application domain is irrelevant since it represents our bare minimum standard that is domain agnostic.

In terms of the merits of a rock bottom configuration for Cobertura and JaCoCo, the following IBM developerWorks sourced statement supports such an approach.

The main thing to understand about coverage reports is that they’re best used to expose code that hasn’t been adequately tested.

Cobertura

Defining a minimum standard when it comes to Cobertura, as it turns out, takes some effort when one considers the array of options one has to consider. For example the configuration below is the usage example provided on the official plugin page.

      <plugin>
        <groupId>org.codehaus.mojo</groupId>
        <artifactId>cobertura-maven-plugin</artifactId>
        <version>2.6</version>
        <configuration>
          <check>
            <!-- Min branch coverage rate per class. 0 to 100. -->
            <branchRate>85</branchRate>
            <!-- Min line coverage rate per class. 0 to 100. -->
            <lineRate>85</lineRate>
            <haltOnFailure>true</haltOnFailure>
            <!-- Min branch coverage rate for project as a whole. -->
            <totalBranchRate>85</totalBranchRate>
            <!-- Min line coverage rate for project as a whole. -->
            <totalLineRate>85</totalLineRate>
            <!-- Min line coverage rate per package. -->
            <packageLineRate>85</packageLineRate>
            <!-- Min branch coverage rate per package. -->
            <packageBranchRate>85</packageBranchRate>
            <regexes>
              <!-- Package specific settings. -->
              <regex>
                <pattern>com.example.reallyimportant.*</pattern>
                <branchRate>90</branchRate>
                <lineRate>80</lineRate>
              </regex>
              <regex>
                <pattern>com.example.boringcode.*</pattern>
                <branchRate>40</branchRate>
                <lineRate>30</lineRate>
              </regex>
            </regexes>
          </check>
        </configuration>
        <executions>
          <execution>
            <goals>
              <goal>clean</goal>
              <goal>check</goal>
            </goals>
          </execution>
        </executions>
      </plugin>

The first question that comes to mind when it comes to the above is what the configuration means in the first place. The main concept we need is the difference between the line rate and branch rate, which has been neatly explained here. So, a potential starting point would be a 50% line coverage rate on a project wide basis as a rock bottom configuration with branch coverage excluded. Naturally we will halt on failure as a rule since this is our bare minimum standard and not necessarily what we will aspire to achieve.

			<plugin>
        		<groupId>org.codehaus.mojo</groupId>
        		<artifactId>cobertura-maven-plugin</artifactId>
        		<version>2.5.2</version>
        		<configuration>
        			<instrumentedDirectory>target/cobertura/instrumented-classes</instrumentedDirectory>
          			<outputDirectory>target/cobertura/report</outputDirectory>
          			<check>
            			<haltOnFailure>true</haltOnFailure>
            			<totalLineRate>50</totalLineRate>
          			</check>
        		</configuration>
        		<executions>
          			 <execution>
                        <id>cobertura-clean</id>
                        <phase>clean</phase>
                        <goals>
                            <goal>clean</goal>
                        </goals>
                    </execution>
                    <execution>
                        <id>cobertura-instrument</id>
                        <phase>process-classes</phase>
                        <goals>
                            <goal>instrument</goal>
                        </goals>
                    </execution>
                    <execution>
                        <id>cobertura-verify</id>
                        <phase>verify</phase>
                        <goals>
                            <goal>check</goal>
                        </goals>
                    </execution>
        		</executions>
      		</plugin>

JaCoCo

When using JaCoCo to generate code coverage reports both that jacoco-maven-plugin and maven-failsafe-plugin must be configured as per this excellent resource.

<plugin>
    <groupId>org.jacoco</groupId>
    <artifactId>jacoco-maven-plugin</artifactId>
    <version>0.6.3.201306030806</version>
    <executions>
        <!-- The Executions required by unit tests are omitted. -->
        <!--
            Prepares the property pointing to the JaCoCo runtime agent which
            is passed as VM argument when Maven the Failsafe plugin is executed.
        -->
        <execution>
            <id>pre-integration-test</id>
            <phase>pre-integration-test</phase>
            <goals>
                <goal>prepare-agent</goal>
            </goals>
            <configuration>
                <!-- Sets the path to the file which contains the execution data. -->
                <destFile>${project.build.directory}/coverage-reports/jacoco-it.exec</destFile>
                <!--
                    Sets the name of the property containing the settings
                    for JaCoCo runtime agent.
                -->
                <propertyName>failsafeArgLine</propertyName>
            </configuration>
        </execution>
        <!--
            Ensures that the code coverage report for integration tests after
            integration tests have been run.
        -->
        <execution>
            <id>post-integration-test</id>
            <phase>post-integration-test</phase>
            <goals>
                <goal>report</goal>
            </goals>
            <configuration>
                <!-- Sets the path to the file which contains the execution data. -->
                <dataFile>${project.build.directory}/coverage-reports/jacoco-it.exec</dataFile>
                <!-- Sets the output directory for the code coverage report. -->
                <outputDirectory>${project.reporting.outputDirectory}/jacoco-it</outputDirectory>
            </configuration>
        </execution>
    </executions>
</plugin>

JDK 8 Support Lacking – Time To Look At Altassian Clover

While producing this post I had to abondon using both cited plugins and to start looking at Altassian Clover since the two cited free plugins do not support JDK 8 at present but Altassian Clover does. The latter does come with a $300 price tag, and that should be fine, it worth spending money on good development tools.

cobertura-maven-plugin issue log

Issue 1: cobertura-maven-plugin 2.6 gave incessant error, downgraded to 2.5.2 which made the error go away. Did not have the time for analysing the reasons for the failure.

Issue 2: Tests would not run with the mvn clean verify command, got incessant exceptions and bytecode on the console with the reason being “Expected stackmap frame at this location.” As it turns out this is was due to JDK 8 not being supported. Downgrading to JDK 7 was not an option for me, neither was spending time on understanding subtle new behaviours of JDK 7 .

Advertisements

WS-Security Basic Policy Definition And Client Testing In Java With JAX-WS

If you’ve defined a wsdl and need to introduce the most basic security policy, generate the client side code and then run some integration tests then this post may help you.

The focus here is on getting it all working, not the most efficient solution, and neither the most secure. We choose a UsernameToken with plain text password and work that into the wsdl, then generate code with the Maven cxf-codegen-plugin, then find a way to add the missing security headers and finally write an integration test.

In terms of context, the steps below where followed in a project where a web service was implemented, and already thoroughly tested using the help of jetty and the maven-soapui-plugin with the latter executing tests during the integration-test phase.

Step 1: Modify The WSDL

The pretinent parts of the this wsdl are shown below. Note the wsp:PolicyReference below the wsdl:service element.

<?xml version="1.0" encoding="UTF-8"?>
<wsdl:definitions
xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/"
xmlns:sch="http://vouchertool.com/vouchserv/schemas"
xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/"
xmlns:tns="http://vouchertool.com/vouchserv/definitions"
targetNamespace="http://vouchertool.com/vouchserv/definitions"
xmlns:wsp="http://www.w3.org/ns/ws-policy"
xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd"
xmlns:sp="http://docs.oasis-open.org/ws-sx/ws-securitypolicy/200702">
<wsdl:types xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/">
...
<wsdl:service name="VoucherServiceService" xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/">
  <wsp:PolicyReference xmlns:wsp="http://schemas.xmlsoap.org/ws/2004/09/policy" URI="#VouchUsernameToken"/>
  <wsdl:port binding="tns:VoucherServiceSoap11" name="VoucherServiceSoap11" xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/">
    <soap:address location="http://vouchertool.com:80/vouchserv/" xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/"/>
  </wsdl:port>
</wsdl:service>

<wsp:Policy wsu:Id="VouchUsernameToken">
  <sp:SupportingTokens>
    <wsp:Policy>
      <sp:UsernameToken
sp:IncludeToken="http://docs.oasis-open.org/ws-sx/ws-securitypolicy/200702/IncludeToken/AlwaysToRecipient">
        <wsp:Policy/>
      </sp:UsernameToken>
    </wsp:Policy>
  </sp:SupportingTokens>
</wsp:Policy>
 

Step2: Code Generation In Our Maven Pom Using cxf-codegen-plugin

<plugin>
  <groupId>org.apache.cxf</groupId>
  <artifactId>cxf-codegen-plugin</artifactId>
  <version>${cxf.version}</version>
  <executions>
    <execution>
      <id>generate-sources</id>
      <phase>generate-sources</phase>
      <configuration>
        <sourceRoot>${basedir}/src/main/java/</sourceRoot>
        <wsdlOptions>
          <wsdlOption>
            <wsdl>${basedir}/src/main/webapp/WEB-INF/wsdl/vouchserv.wsdl</wsdl>
            <extraargs>
              <extraarg>-verbose</extraarg>
            </extraargs>
          </wsdlOption>
        </wsdlOptions>
      </configuration>
      <goals>
        <goal>wsdl2java</goal>
      </goals>
    </execution>
  </executions>
</plugin>

Step 3: Add A HeaderHandler and HeaderHandlerResolver

This post shows how to add the security headers. It works, try it. You’ll need some solution since the code generated by cxf will not provide any means of adding the headers, which is naturally suboptimal.

Step 4: Test It All Using An Integration Test

public class VoucherServiceIT {

@Test
public void test() {
  URL url = null;
  try {
    url = new URL("http://localhost:8080/vouchserv/vouchserv.wsdl");
  } catch (MalformedURLException e) {
   throw new RuntimeException(e);
  }

  VoucherServiceService voucherServiceService = new VoucherServiceService(url);
  HeaderHandlerResolver handlerResolver = new HeaderHandlerResolver();
  voucherServiceService.setHandlerResolver(handlerResolver);
  VoucherService voucherService = voucherServiceService.getVoucherServiceSoap11();
  RegisterRequest registerRequest = new RegisterRequest();
  RegisterResponse registerResponse = voucherService.register(registerRequest);
}

}

Spring Web Services Tomcat Compatibility

In this post we attempt to help you choose the appropriate version of Tomcat when using Spring Web Services.

As context, if you’ve developed a web service, using Spring Web Services, chances are you’ve blissfully been using the Maven Jetty Plugin in your project during the development phase. When you’re getting ready to deploy to Tomcat however, you may need to know what the Servlet/JSP Spec version supported by Spring Web Services is since this is how the Tomcat Which Version page guides users in their decision making process.

Here are steps that could be used to determine the relevant Servlet/JSP Spec version and hence Tomcat version:

  1. Run mvn dependency:tree
  2. Look for the org.springframework.ws:spring-ws-core line and then for the version number of its org.springframework:spring-webmvc line, in our case, we have 3.1.0.RELEASE as the version
  3. Now look at the dependencies of the version of spring-webmvc and in turn it’s org.apache.tomcat:tomcat-servlet-api dependency, in our case, we find that we find that we are depedant on tomcat-servlet-api version 7.0.8
  4. Finally, have a look at the MANIFEST of the above mentioned jar, and here you will find the relevant Servlet Spec, in our case we find: Specification-Title : Java API for Servlets, Specification-Version : 3.0
  5. Choose Tomcat 7 as advised.

 

org.apache.tomcat

Spring JUnit Ehcache: Hibernate Second-Level Caching: Another unnamed CacheManager already exists in the same VM.

If you are quite simply trying to introduce Hibernate second level caching into your Spring based application using Ehcache, you may end up banging your head against this wall when running your Junit tests: Another unnamed CacheManager already exists in the same VM.

Even worse, you may get sidetracked by posts such as this one which has to do with Spring 3.1 caching.

It may be that you have decided to included a version of Ehcache > 2.5, if that is the case, then before wasting more time on this, like I did, downgrade to Ehcache 2.4.7:

BEFORE

<dependency>
<groupId>net.sf.ehcache</groupId>
<artifactId>ehcache-core</artifactId>
<version>2.5.2</version>
</dependency>

AFTER

<dependency>
<groupId>net.sf.ehcache</groupId>
<artifactId>ehcache-core</artifactId>
<version>2.4.7</version>
</dependency>

One more tip, to get started with you ehcache.xml config, copy the defaults in ehcache-failsafe.xml and go from there, dump the file in your resources folder in your project and consult the Ehcache Hibernate Second-Level Cache Documentation.

Automated deployment with Maven and friends

I’m busy with an automated deployment configuration task and stumbled upon the following excellent presentation by John Ferguson Smart from Wakaleo Consulting: Automated deployment with Maven and friends – Going the whole nine yards

If I can add some context to labelling the presentation as excellent. In essence it represents what I regard as best practices. Setting up the mentioned infrastructure takes time, but WILL save time and money in the long run. Here are some other reasons why I advocate the use of the practices outlined in the presentation:

  • I’ve had the misfortune of a three month stint as a human continuous integration / deployment “Jenkins” in the early days of a large scale enterprise project, this has left me scarred and obsessed with automated deployment and subsequent testing.
  • I’ve seen Liquibase replicated many times, at considerable expense.
  • If you don’t set it all up early in a project, you will pay for it with wasted time, one way or the other.