09 May 2014 Concordion Integration With Jenkins
We recently introduced acceptance test driven development (ATDD) at a client. The idea was for the product owners, developers, and testers to work as a team to come up with the acceptance criteria for user stories before development begins. We adopted this approach as an attempt to increase a shared understanding of user stories, as well as a shared agreement on the definition of ‘done’.
As we introduced more functionality to the application, it became evident that more and more effort had to be put into regression testing prior to a software release. The application has to be supported on mobile (iOS and Android) and desktop (Safari, Chrome, Firefox, IE8 and above). Our team of 2 to 3 testers would spend up to 3 days testing for regressions! In an attempt to reduce the need for talented testers to carry out monkey work, the team began to push for a focus on automated browser-level testing.
In this post I will demonstrate the acceptance testing framework we used and describe how the test suite can be tied to a Jenkins build (with a beautiful results page!).
Introducing Concordion
An important requirement for our chosen acceptance testing framework was readability. This was because we wanted to eventually move towards having the product owners, who are non-technical, write and own the acceptance criteria.
Concordion advertises itself as similar to Cucumber, but with a focus on readability. Test Specifications (as Concordion calls them) are written in natural language and documented in HTML files. This was advantageous for us, because historically non-technical members of the team found it challenging to write in a structured language – for instance the ‘Given-When-Then’ approach. Several members of the team had experience in Cucumber, and we hoped that this knowledge carried over to Concordion.
Another requirement was for these automated acceptance tests to run as part of our Jenkins build after the application has been deployed into our system test environment. An advantage of this is fast feedback of changes to the application. Breaking changes can be identified shortly after they have been committed, instead of several weeks later during regression testing.
Let’s look at some code!
My example is a simple Java project based on The Concordion kickstart tutorial, which is an excellent resource for learning Concordion. I recommend giving this tutorial a read at some stage.
In the example, we will be testing the home page of the Shine blog (ie, the site that you’re reading right now).
The first step is to write the test specification, which will cover an acceptance criterion. Here is an example:
<p>
When I visit the Shine blog, there will be a heading that reads
'<span concordion:assertEquals="getBlogHeading()">
The Shine Blog
</span>'.
</p>
As you can see, the test specification is easy for a non-technical person to read.
The function getBlogHeading() is defined in HomeFixture.java:
@RunWith(ConcordionRunner.class)
public class HomeFixture extends BaseFixture {
public String getBlogHeading() {
Home home = new Home(driver);
return home.getBlogHeading();
}
}
Note that we are using the PageObject pattern to write our tests. HomeFixture can be run as a simple JUnit test. This test starts up PhantomJS, loads up http://blog.shinetech.com, and finally checks to see if the page contains a heading that reads ‘The Shine Blog’. The console should show the path to a test result page, which loads in the browser and looks similar to this:
Let’s pretend there is a requirement to change the heading to ‘The New Shine Blog’. In an ATDD world, we would first change the test specification before changing the heading on the page. Therefore, the test specification becomes the following:
<p>
When I visit the Shine blog, there will be a heading that reads
'<span concordion:assertEquals="getBlogHeading()">
The New Shine Blog
</span>'.
</p>
If we run the Concordion test now, the test will fail, as evident in the screenshot below:
Great! The test failed – as expected. We can figure out what is wrong by using a debugger as we normally would with most JUnit tests. With browser-level tests however, it can be nice to be able to see the actual application. Fortunately, Concordion provides a simple mechanism to capture screenshots.
Screenshots!
Concordion comes with a screenshot extension that adds screenshots to the test results page. This is incredibly simple to get going using the @Extension annotation. Note that the following class is inherited by the HomeFixture from before:
public class BaseFixture {
protected static WebDriver driver = new PhantomJSDriver();
@Extension
public ConcordionExtension extension =
new ScreenshotExtension().
setScreenshotTaker(new SeleniumScreenshotTaker(driver)).
setScreenshotOnAssertionSuccess(true).setMaxWidth(400);
}
Now that this is set up, an image pop-up will be shown when you mouseover a test result in the results page (screenshot in image below is truncated to only show relevant area):
Now a developer can go into the source code of the blog and change the heading of the blog from ‘The Shine Blog’ to ‘The New Shine Blog’.
Jenkins Integration
In order to ensure fast feedback of code changes, the aim now was to get this test suite onto Jenkins and have it run at the end of our build. We needed to ensure that each Jenkins build has a test result page associated with it. This could be achieved using the HTML Publisher Plugin.
However, we faced a problem. As the test suite grew, we began having more and more files. At our client, we have one test specification file for each area of the application: registration, authentication, user profile, etc. The HTML Publisher Plugin would only publish files that are specified explicitly. We needed to ensure that we would not have to reconfigure the plugin each time a test specification was added or changed.
The solution was to have a main test specification file that triggers other test specifications. This is how the source of the main test specification file would look in our example.
<div class='testcontent'>
<li>
<a concordion:run="concordion" href="blog/Home.html">
Home
</a>
</li>
<li>
<a concordion:run="concordion" href="blog/AboutUs.html">
About Us
</a>
</li>
</div>
A blank JUnit test class needs to be written to trigger these tests.
@RunWith(ConcordionRunner.class)
public class Overview {
}
When the Overview test class is run, it triggers the tests specified in Home.html and AboutUs.html. The output of this is an Overview.html test result page, which tells you whether any tests under Home.html and/or AboutUs.html have failed. You can click into each of the two links and drill down to the individual tests that have failed.
This Overview.html result page can be saved in a reports/ directory using a build tool like Ant. Here is a snippet of our Ant target for running the acceptance test suite and generating a results page:
<target name="run-acceptance-tests">
<copy todir="${acceptance.reports.dir}">
<fileset dir="test/specs/fixtures">
<include name="Overview.html"/>
<include name="**/*.html"/>
</fileset>
</copy>
<run-tests test.reports.dir="${acceptance.reports.dir}">
<batchtest todir="${acceptance.reports.dir}/xml">
<fileset dir="${acceptance.test.classes.dir}">
<include name="**/Overview.class"/>
</fileset>
</batchtest>
</run-tests>
<check-tests test.reports.dir="${acceptance.reports.dir}"/>
</target>
Let’s get this test results overview page into Jenkins!
You can set up the HTML Publisher Plugin to publish the reports/Overview.html file as follows:
You will now have a link to the test results overview page called ‘Concordion Results’ in the left hand panel of Jenkins for each build you run.
Conclusion
In my ideal world, these automated acceptance tests document the requirements for the application, and eventually become a major part of the regression test suite. With the majority of the application tested in an automated fashion, humans can then test at a whole new level. They can use their wits to test for usability and actively try to break the application, instead of spending their time going through the same boring test cases again and again.
No Comments