Integration of TeamCity 6.x and Fitnesse

Team city is wonderful CI and build management platform, which combines great capabilities with really simple configuration. I seriously love it for the way it is designed, configured and managed and would recommend it to anybody who cares about quality and wants to have fast feedback on build/test/integration problems. FitNess is “The fully integrated standalone wiki, and acceptance testing framework”, which I, personally, do not like, but testers (at least ones I know) say it is good. I still do not believe them, but have to live with it :)

Integration problem is divided nicely into two pieces - one is fighting with FitNesse, another is integration of battle outcome into the TeamCity. So, lets start with FitNesse first.

FitNess is not easily-integratable framework for a few reasons. The first problem is test URL & Fixtures’ path usually are not configurable via parameters. Fortunately, there is quite good solution, which allows to pass variable to FitNesse using “-D” java argument. Another issue is that it runs just as HTTP server, there is no other way to execute tests, other from running the server. Luckily it can run it and after execution stop it and such behaviour can be managed via command line. And the last issue is report produced by Fitnesse. It has to be converted into something widely-acceptable, e.g. into JUnit report.

Now let sort all these issues out. The first step is updating test scripts to make them accept test URL and path to fixtures from command line. Assuming that there is just one property defined for URL and just one reference to fixture path, all which need to be done is Wiki page to contain something like this:

!define TEST_SERVER_URL {${testserver.host_port}}
!path ${testserver.fixture_jar}

Note another set of curly brackets around value of TEST_SERVER_URL variable, it shouldn’t be missed. After these values are defined, it is possible to pass them via command line:

$JAVA_HOME/bin/java -classpath <path_to_fixtures_jar> -Dtestserver.fixture_jar=<path_to_fixtures_jar> -DDtestserver.host_port=<test_server_url> fitnesseMain.FitNesseMain -p <fitnesse_port> -d <path_to_test_suite> -c <suite_url>\&format=xml > <report_temp_file>

where:
path_to_fixtures_jar - is the path to jar with fixtures. It is referenced twice, because it is assumed that it has both FitNesse distribution and fixtures inside it. For me, it just looks like the most convenient way to package it.
test_server_url - URL to the server being tested. That will be the value of TEST_SERVER_URL. Be careful here - as many other tools written in cowboy style dev technique, it hardly have any logs and when you will try to run test on bad (not connectable) URL or with invalid or unavailable port number, it will hang forever.
fitnesse_port - now this is weired, but is required by FitNesse. It requires port to run. Just choose something random, which is not likely to be used by something else. I have no idea why, but '0' port (which should just select first available port) didn’t work for me, FitNesse just hanged without any messages, as it loves to do.
path_to_test_suite - path to the folder with test suite.
suite_url - that’s the path of the suite without host and port. To identify it, run FitNesse, goto the page with Suite and hover mouse over “Suite” link on the left hand side of FitNesse page with the suite. Do not miss ‘format=xml’ suffix.
report_temp_file - is just a test file with report which will be generated by FitNesse.

Now when we have report, we need to convert it into the JUnit format to allow TeamCity to understand it. Fist, it has to be cleaned-up, because contains not just XML, but also some other irrelevant stuff. I was was lazy here and done it simply with following command:

grep ".*<.*>" $TEMP_FILE_TXT > $TEMP_FILE_XML"

then is more interesting part. That report has to be converted into JUnit XML and that is done by applying following XSLT:

<?xml version="1.0" encoding="ISO-8859-1"?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:template match="/">
  <xsl:element name="testsuite">
    <xsl:attribute name="tests">
      <xsl:value-of select="sum(testResults/finalCounts/*)" />
    </xsl:attribute>
    <xsl:attribute name="failures">
      <xsl:value-of select="testResults/finalCounts/wrong" />
    </xsl:attribute>
    <xsl:attribute name="disabled">
      <xsl:value-of select="testResults/finalCounts/ignores" />
    </xsl:attribute>
    <xsl:attribute name="errors">
      <xsl:value-of select="testResults/finalCounts/exceptions" />
    </xsl:attribute>
    <xsl:attribute name="name">AcceptanceTests</xsl:attribute>
  <xsl:for-each select="testResults/result">
    <xsl:element name="testcase">
      <xsl:attribute name="classname">
        <xsl:value-of select="/testResults/rootPath" />
      </xsl:attribute>
      <xsl:attribute name="name">
        <xsl:value-of select="relativePageName" />
      </xsl:attribute>
      <xsl:choose>
        <xsl:when test="counts/exceptions > 0">
          <xsl:element name="error">
            <xsl:attribute name="message">
              <xsl:value-of select="counts/exceptions" />
              <xsl:text> exceptions thrown</xsl:text>
              <xsl:if test="counts/wrong > 0">
                <xsl:text> and </xsl:text>
                <xsl:value-of select="counts/wrong" />
                <xsl:text> assertions failed</xsl:text>
              </xsl:if>
            </xsl:attribute>
          </xsl:element>
        </xsl:when>
        <xsl:when test="counts/wrong > 0">
          <xsl:element name="failure">
            <xsl:attribute name="message">
              <xsl:value-of select="counts/wrong" />
              <xsl:text> assertions failed</xsl:text>
            </xsl:attribute>
          </xsl:element>
        </xsl:when>
      </xsl:choose>
    </xsl:element>
  </xsl:for-each>
  </xsl:element>
</xsl:template>
</xsl:stylesheet>

to execute conversion we can use standard tool, provided by JDK. Surprisingly, found that not many people know about it, but it can be very handy. Here is snippet which allows to execute XSLT transformation in command line:

$JAVA_HOME/bin/java com.sun.org.apache.xalan.internal.xsltc.cmdline.Compile fitnesse2junit.xslt
$JAVA_HOME/bin/java com.sun.org.apache.xalan.internal.xsltc.cmdline.Transform <report_temp_file> fitnesse2junit > funct_test_res.xml

The result is going to be XML test report in JUnit format.

That’s all with FitNesse. Now it’s TeamCity’s turn. It is really up to developer how to do that, I will just give a brief overview and some flavour of what has to be done:

• Create new build task for execution of the script which runs FitNesse. That script should be just a collection of command lines provided above.
• FitNesse requires fixtures and tests, so these have to be retrieved from VCS or linked as artifact dependencies from the other build. Either way these files will available for script which runs FitNesse and can be added into classpath, etc.
• I would recommend to put such properties like 'test_server_url' into 'Environment Variables' in 'Build Parameters'. Then they can be accessible by scripts.
• To see test execution results, add a Feature which will analyse XML with JUnit report. That is easy and there is already build-in Feature type in TeamCity which supports JUnit. That functionality is available from “Build Step/Add build feature”.