Taking and Comparing Screenshots

Screenshot Parameters
Taking Screenshots on Failure
Taking Screenshots for Comparison
Practices for Handling Screenshots
Known Compatibility Problems

You can take and compare screenshots with reference screenshots taken earlier. If there are differences, you can fail the test case.

Screenshot Parameters

The screenshot configuration parameters are defined with static methods in the com.vaadin.testbench.Parameters class.

screenshotErrorDirectory(default: null): Defines the directory where screenshots for failed tests or comparisons are stored.
screenshotReferenceDirectory(default: null): Defines the directory where the reference images for screenshot comparison are stored.
screenshotComparisonTolerance(default: 0.01): Screen comparison is usually not done with exact pixel values, because rendering in browser often has some tiny inconsistencies. Also image compression may cause small artifacts.
screenshotComparisonCursorDetection(default: false): Some field component get a blinking cursor when they have the focus. The cursor can cause unnecessary failures depending on whether the blink happens to make the cursor visible or invisible when taking a screenshot. This parameter enables cursor detection that tries to minimize these failures.
maxScreenshotRetries(default: 2): Sometimes a screenshot comparison may fail because the screen rendering has not yet finished, or there is a blinking cursor that is different from the reference screenshot. For these reasons, Vaadin TestBench retries the screenshot comparison for a number of times defined with this parameter.
screenshotRetryDelay(default: 500): Delay in milliseconds for making a screenshot retry when a comparison fails.

For example:

@Before
public void setUp() throws Exception {
    Parameters.setScreenshotErrorDirectory(
        "screenshots/errors");
    Parameters.setScreenshotReferenceDirectory(
        "screenshots/reference");
    Parameters.setMaxScreenshotRetries(2);
    Parameters.setScreenshotComparisonTolerance(1.0);
    Parameters.setScreenshotRetryDelay(10);
    Parameters.setScreenshotComparisonCursorDetection(true);
    Parameters.setCaptureScreenshotOnFailure(true);
}

Taking Screenshots on Failure

Vaadin TestBench can take screenshots automatically when a test fails. To enable the feature, you need to include the ScreenshotOnFailureRule JUnit rule with a member variable annotated with @Rule in the test case as follows:

@Rule
public ScreenshotOnFailureRule screenshotOnFailureRule =
    new ScreenshotOnFailureRule(this, true);

Notice that you must not call quit() for the driver in the @After method, as that would close the driver before the rule takes the screenshot.

The screenshots are written to the error directory defined with the screenshotErrorDirectory parameter. You can configure it in the test case setup as follows:

@Before
public void setUp() throws Exception {
    Parameters.setScreenshotErrorDirectory("screenshots/errors");
    ...
}

Taking Screenshots for Comparison

Vaadin TestBench allows taking screenshots of the web browser window with the compareScreen() command in the TestBenchCommands interface. The method has a number of variants.

The compareScreen([classname]#File)# takes a File object pointing to the reference image. In this case, a possible error image is written to the error directory with the same file name. You can get a file object to a reference image with the static ImageFileUtil.getReferenceScreenshotFile() helper method.

assertTrue("Screenshots differ",
           testBench(driver).compareScreen(
               ImageFileUtil.getReferenceScreenshotFile(
                   "myshot.png")));

The compareScreen([classname]#String)# takes a base name of the screenshot. It is appended with browser identifier and the file extension.

assertTrue(testBench(driver).compareScreen("tooltip"));

The compareScreen([classname]#BufferedImage,String)# allows keeping the reference image in memory. An error image is written to a file with a name determined from the base name given as the second parameter.

Screenshots taken with the compareScreen() method are compared to a reference image stored in the reference image folder. If differences are found (or the reference image is missing), the comparison method returns false and stores the screenshot in the error folder. It also generates an HTML file that highlights the differing regions.

Screenshot Comparison Error Images

Screenshots with errors are written to the error folder, which is defined with the screenshotErrorDirectory parameter described in Screenshot Parameters.

For example, the error caused by a missing reference image could be written to screenshot/errors/tooltip_firefox_12.0.png. The image is shown in A screenshot taken by a test run.

A screenshot taken by a test run

Screenshots cover the visible page area in the browser. The size of the browser is therefore relevant for screenshot comparison. The browser is normally sized with a predefined default size. You can set the size of the browser window in a couple of ways. You can set the size of the browser window with, for example, driver.manage().window().setSize(new Dimension(1024, 768)); in the @Before method. The size includes any browser chrome, so the actual screenshot size will be smaller. To set the actual view area, you can use TestBenchCommands.resizeViewPortTo(1024, 768).

Reference Images

Reference images are expected to be found in the reference image folder, as defined with the screenshotReferenceDirectory parameter described in Screenshot Parameters. To create a reference image, just copy a screenshot from the errors/ directory to the reference/ directory.

For example:

$ cp screenshot/errors/tooltip_firefox_12.0.png screenshot/reference/

Now, when the proper reference image exists, rerunning the test outputs success:

$ java …
JUnit version 4.5
.
Time: 18.222

OK (1 test)

Masking Screenshots

You can make masked screenshot comparison with reference images that have non-opaque regions. Non-opaque pixels in the reference image, that is, ones with less than 1.0 value in the alpha channel, are ignored in the screenshot comparison.

Please see the ScreenshotITCase.java example in the TestBench Demo for an example of using masked screenshots. The example/Screenshot_Comparison_Tests.pdf document describes how to enable the example and how to create the screenshot masks in an image editor.

Visualization of Differences in Screenshots with Highlighting

Vaadin TestBench supports advanced difference visualization between a captured screenshot and the reference image. A difference report is written to a HTML file that has the same name as the failed screenshot, but with .html suffix. The reports are written to the same errors/ folder as the screenshots from the failed tests.

The differences in the images are highlighted with blue rectangles. Moving the mouse pointer over a square shows the difference area as it appears in the reference image. Clicking the image switches the entire view to the reference image and back. Text " Image for this run" is displayed in the top-left corner of the screenshot to distinguish it from the reference image.

The reference image and a highlighted error image shows a difference report with one difference between the visualized screenshot (bottom) and the reference image (top).

The reference image and a highlighted error image

Practices for Handling Screenshots

Access to the screenshot reference image directory should be arranged so that a developer who can view the results can copy the valid images to the reference directory. One possibility is to store the reference images in a version control system and check-out them to the reference/ directory.

A build system or a continuous integration system can be configured to automatically collect and store the screenshots as build artifacts.

Known Compatibility Problems

Screenshots when running Internet Explorer 9 in Compatibility Mode

:: Internet Explorer prior to version 9 adds a two-pixel border around the content area. Version 9 no longer does this and as a result screenshots taken using Internet Explorer 9 running in compatibility mode (IE7/IE8) will include the two pixel border, contrary to what the older versions of Internet Explorer do.

Docs