Overview
Selenium helps to automate web applications for testing purposes. The wide browser support and open source licensing makes Selenium very popular for testing Javascript applications and also a core technology for other automation testing tools. Vaadin Testbench is a testing tool for mainly Vaadin applications. It is based on Selenium and provides additional features and helper methods for Vaadin application testing. Testbench is a commercial product, and we are frequently asked, if Testbench is worth paying for. In this blog post I will compare the usage of Testbench and Selenium, which hopefully will help to understand better whether you need to use Testbench or not.
Client server communication
Vaadin is a stateful framework, meaning that an event on the client side may affect the state of the application on the server side. Selenium is a testing tool on the client side, so it does not know about Vaadin specific features, like client-server communication. When an event happens on the client side, it will notify the server side through RPC. If the RPC call causes a change in the shared state, it will send those changes to the client. Because of a network delay or long time code execution on the server side, there might be a delay between the client side event and the change in the UI. In these circumstances, the client should wait for the server side code to execute, because it might affect the next client side instruction. It is a typical situation in Selenium that you need to add a delay between one instruction and the next. A simple Vaadin button click test in Selenium may look like this:
@Test
public void testWithoutTestbench() {
driver.get("http://myhost.com/app");
driver.manage().timeouts()
.implicitlyWait(5, TimeUnit.SECONDS);
final WebElement button = findButtonByCaption("Click");
button.click();
final List<WebElement> elements =
driver.findElements(By.id("labelid"));
if (elements.isEmpty()) {
throw new RuntimeException("No Label found");
}
driver.manage().timeouts()
.implicitlyWait(5, TimeUnit.SECONDS);
final String value = elements.get(0).getText();
Assert.assertEquals("Clicked", value);
}
public WebElement findButtonByCaption(String caption) {
final List<WebElement> buttons =
driver.findElements(By.className("v-button"));
for (final WebElement button : buttons) {
if (button.getText().equals(caption)) {
return button;
}
}
return null;
}
Relying on a delay may cause a working test fail because the delay might be too small. After facing such problems several times, developers start adding huge delays, which will increase the test time execution significantly. Testbench suspends client-side execution while server-side code is being executed and resumes client-side code when it gets a response from the server side. When using Testbench, a developer can be sure that a component’s client and server side states are always synchronized, so the same button click test in Testbench looks like this:
@Test
public void testClick() {
getDriver().get("http://myhost.com/app");
ButtonElement button =
$(ButtonElement.class).caption("Click").first();
button.click();
LabelElement label = $(LabelElement.class).first();
Assert.assertEquals(label.getText(), "Clicked");
}
Easier Vaadin API
Selenium operates on the DOM of the Web page and provides basic methods of Web elements like "click" or "sendKeys". For example, to find a value in the Vaadin table when using Selenium, you need to understand how the Vaadin component is built and prepare a selector to find a specific row and cell in it, which is often error prone even for advanced front-end developers. TestBench provides an alternative selector variant with a greater abstraction for Vaadin components. This makes it easier and faster to write your tests and also protects your test scripts from potential changes made to the client side implementation of Vaadin components. Using the previous example to get a row or cell of the Table by index you need two lines of code:
TableElement table = $(TableElement.class).first(); String value = table.getRow(0).getCell(1).getValue();
If you end up in a situation where Testbench lacks helper methods, you can write the test or a helper method with plain Selenium. Usually in Selenium tests, elements are picked from the page by identifiers. Adding ids to all elements can be quite annoying, especially if you add ids only for testing purposes. When using Testbench you can search elements by class or caption, which makes navigating on the page easier.
Parallel Testing
TestBench4 introduced a "ParallelTest" class which is a thread pool and executes tests in separate threads in parallel. The amount of threads can be changed by calling Parameters.setMaxThreads() method. Selenium does not provide a ready made solution for starting those tests in parallel, but developers may use a maven surefire plugin or JUnit ParallelComputer class.
Screen comparison
Since version 2.0.0 TestBench has an API for comparing screenshots. This feature was introduced to help users to test the visual appearance of the application. When styling your application, it is very easy to break one part of it while fixing another. Manual UI testing is a bad option, because you have to check all the visuals on all the browsers you are supporting. Automatic screenshot comparison can solve this problem. The comparison is done between a reference screenshot and a current screenshot. If there are differences in the images, these parts are highlighted in the test results, so it is easier for a tester to find out what was the problem with the test. A TestBench test can also be configured to automatically take screenshots of the failing tests by adding a screenshotOnFailue rule:
@Rule public ScreenshotOnFailureRule screenshotOnFailureRule = new ScreenshotOnFailureRule(this, true);
Selenium does not provide a ready made solution for screenshot comparison, so you need to use an external tool like im4java or ImageMagick.
Conclusions
To summarize, TestBench tests for Vaadin applications require less boilerplate code, which makes them easier to write and maintain. Testbench is based on Selenium, and you can use Selenium to test your Vaadin applications, but it will probably cost you more time and effort. This all depends on the size of your application and the number of tests you are going to write and maintain. For a handful of tests you’ll be happy enough with Selenium, for sure. But when you start writing more test code you’ll find the Testbench investment worth every penny. For the next steps and for looking into this deeper, you can find complete examples of button click tests written with Vaadin and Selenium here. Happy testing.