A few days ago I decided to download the recently released Vaadin TestBench 2.0 and give it a try. Vaadin TestBench is a tool for automated UI testing of Vaadin applications and it’s built on top of widely popular Selenium with extensions to make it work seamlessly with Vaadin.
The add-on is distributed as a zip file so first I had to unzip the downloaded package. The package contains four directories and a jar file.
The documentation directory contains the TestBench manual as a PDF file. The example directory contains an example test that I'll be running in this blog post. In the subdirectories of the grid are all the files needed to run the TestBench Hub Server and the TestBench Remote Control Nodes. The jar file is needed to run the tests and finally the vaadin-testbench-recorder directory contains the Firefox add-on for recording test scripts.
The TestBench grid could be distributed across multiple servers but in this case I want to keep things simple and I'll be running everything on a single MacBook Pro. So next I started up the Hub by first opening the /grid/hub directory. I simply accepted all the default configuration values (the grid_configuration.yml file) and started the Hub by running the hub.sh shell script. I verified that the Hub is running by opening the http://localhost:4444/console page on my browser. The page displays all configured environments and lists any available or currently active Remote Controls. There aren't any Remote Controls running, so perhaps I should start one to get some tests running.
As I said, to keep things simple I want run everything on a single laptop. So next I'll go to the /grid/remote-control directory. The default configuration is suitable for a Windows machine, but as I'm running on Mac OS X, I'll have to edit some of the configuration parameters. To do that I opened up the rc_configuration.xml file and commented out all the "winxp" prefixed environments and in turn enabled the "osx" prefixed ones. Now when I run the rc.sh shell script the remote control should appear on the Grid Hub Console page under the Available Remote Controls (see the screenshot).
Now that I have an available Remote Control, I can start running the tests. The example directory contains an example test script in the testscripts subdirectory and an Ant script (test.xml) for converting the test script into a JUnit test case and executing the test. The test.xml is again tailored for a Windows environment, so it needs some minor modifications. The property named "browsers" must be changed, so I replaced the value "winxp-ie8,winxp-firefox3" with "osx-safari4" to run the test on a Safari 4 browser in OS X. All the other configuration options are suitable for the test out of the box.
Next I can run the test by executing the Ant script (ant -f test.xml). As the Remote Control is running on my local machine I now see the Safari browser opening up, navigating automatically to the Vaadin demo site and executing the predefined steps. But what is this? The test failed. What's going on here?
run-tests:
[junit] Testsuite: demo_vaadin_com.osx_safari4.demo_vaadin_com
[junit] Tests run: 1, Failures: 1, Errors: 0, Time elapsed: 28.18 sec
[junit]
[junit] Testcase: testosx_safari4(demo_vaadin_com.osx_safari4.demo_vaadin_com): FAILED
[junit] Test was missing reference images.
[junit] junit.framework.AssertionFailedError: Test was missing reference images.
[junit] at demo_vaadin_com.osx_safari4.demo_vaadin_com.internal_demo_vaadin_com(demo_vaadin_com.java:81)
[junit] at demo_vaadin_com.osx_safari4.demo_vaadin_com.testosx_safari4(demo_vaadin_com.java:20)
[junit] at com.thoughtworks.selenium.SeleneseTestCase.runBare(SeleneseTestCase.java:212)
[junit]
[junit]
[junit] Test demo_vaadin_com.osx_safari4.demo_vaadin_com FAILED
After reading the reported failure I realized that the test takes also a screenshot of the application but it has no reference image to compare it to and therefore the test fails. The taken screenshot causing the test to fail is stored in the screenshots/errors directory. Now we can use the taken screenshot as the reference for further testing, so I simply move the PNG file to the screenshots/reference directory to use it as the reference image.
Now I can run the test again and this time the test passes and everything worked like a charm. In a next blog post I'll show you how to record and run your own tests using the TestBench Recorder add-on for Firefox. Stay tuned!