Visual regression tests with JLineup

With visual regression tests, one can check two versions of a website for visual differences. Those tests are, for example, useful to check a website on a test server before and after deploying changes, to only move the changes to the live server when no differences are found.

There are various solutions to run such tests, here I will show you JLineup. JLineup can be used as a command-line tool or as a web service. I only tried the CLI version and am not going to describe the web service possibility.

Installation of JLineup CLI

The GitHub repo contains a file with installation and usage instructions for the CLI version of JLineup. Because it is a Java archive, the server needs to have at least Java 8 installed. On Ubuntu 20.04 we could install Java in form of OpenJDK with the following command:

sudo apt-get install openjdk-14-jre 
Code language: Bash (bash)

Installing the CLI itself happens with just downloading the .jar file:

wget https://repo1.maven.org/maven2/de/otto/jlineup-cli/4.3.4/jlineup-cli-4.3.4.jar -O jlineup.jar
Code language: Bash (bash)

To be able to run visual regression tests, we need a browser, too. I decided to use Firefox, JLineup also supports Chrome and Chromium. To install Firefox on Ubuntu, we run the following command:

sudo apt-get install --no-install-recommends firefox
Code language: JavaScript (javascript)

The --no-install-recommends param prevents the installation of GNOME dependencies.

Configuration

Configuration of JLineup is done in a JSON file, the tool by default looks for a lineup.json in the current directory when no other file is specified via the --config parameter.

The tool comes with many configuration options that are described in the repo in detail. For example, you can define multiple URLs to check, test in different viewport dimensions, and set a time in seconds that should be waited before taking the screenshot after loading the page.

An easy example for testing a site looks like the following:

{
  "urls": {
    "https://time.gov/": {
      "paths": [
        ""
      ],
      "max-diff": 0.0,
      "devices" : [{
        "width" : 800,
        "height" : 600
      }],
      "max-scroll-height": 100000,
      "wait-after-page-load": 5,
      "wait-after-scroll": 0
    }
  },
  "browser": "firefox-headless"
}
Code language: JSON / JSON with Comments (json)

Using this configuration, JLineup checks the time.gov site with Firefox in headless mode and a window width of 800x600 pixels. The max height that is scrolled for taking screenshots is 100.000 pixels and after loading the page, the tool waits 5 seconds before taking a screenshot, after scrolling there is no pause.

The defined window size also specifies the screenshot size, because JLineup does not take one screenshot of the whole site but multiple in the specified size. It visits the page, makes a screenshot, scrolls down, makes a screenshot, scrolls down, and so on.

Run tests

To run a test with the configuration from above, we need the following command to run the »before« step:

java -jar jlineup.jar --step before
Code language: CSS (css)

The command line shows a log of what JLineup does, for example, »Browsing to https://time.gov/ with window size 800x600«.

Let’s imagine our changes were deployt now, and we run the »after« step and comparison:

java -jar jlineup.jar --step after
Code language: CSS (css)

time.gov changes every second, so the command should finish with »JLineup finished. There was a difference between before and after. Return code is 1.«. In background, JLineup created a report directory that contains, among other files, the screenshot files and a report in JSON and one in HTML format.

The result in HTML format looks like the following example when an error is found:

The report shows information about window size, scroll position, and the difference between before and after in percent, and the before and after screenshots. If there is a difference between them, you also get an image with the difference.

And that’s it. As described here, one could use JLineup with GitHub actions, for example, to make sure, a change does not have unwanted side effects.