Visual Regression Testing: How to Test Dynamic Content with PhantomCSS

#Testing | Posted

Visual regression testing is an incredibly useful and growing area of testing in the software development sphere. Launching off Micah Godbolt’s recent blog post, CSS Testing with PhantomCSS, PhantomJS, CasperJS and Grunt, we worked with the Department of Energy to expand their current suite of tests to include visual regression testing. Then we went one step farther by automating the process with continuous integration with Grunt and Jenkins.

This blog post represents part 1 of a 3 part blog series on visual regression testing. Today I'll dive into how we visually tested the Department of Energy platform without interference from dynamic content.

The Department of Energy platform has long had a suite of Behat tests that run prior to every deployment, enabling our combined team to test key functionality throughout the site. While a great asset to the platform, Behat unfortunately doesn't address cosmetic changes. This could be anything from minor font-size changes to major contextual changes. However, with the integration of the PhantomCSS library, we have been able to supplement Behat with a new suite of visual regression tests. These new tests allow us to visually compare pages or elements of a page against baseline screenshots, confirming that no visual changes have occurred.

Because the platform includes over 60 offices, each with their own subset of constantly updated pages and customizations, we were forced to take a unique approach regarding the granularity of our tests. Typically, you’d use PhantomCSS to target only specific elements within a site. But taking snapshots of individual elements whose content could change anytime would give us unreliable baselines, so we instead used PhantomCSS to uniquely test our layouts and select global theming components. To avoid a windfall of false-positives, we needed a mechanism to stash dynamic user content and temporary styles from our screenshots.


First, we identified the dynamic content and targeted it.

[xhtml]<div class="block-2 last field-item even">
  <div class="block-bean-eere-bbnp---homepage-get-inspi-0-wrapper">
    <div id="block-bean-eere-bbnp-homepage-get-inspi-0" class="block block-bean bean-imagelink block-bean-eere-bbnp---homepage-get-inspi-0 clearfix">
      <h4 class="block-title">Watch Videos</h4>
      <div class="content"></div>

Then, prior to every test of the UI, we have CasperJS evaluate several snippets of jQuery.

 this.evaluate(function() {
   jQuery('#page-wrapper .block').css('background', 'gray').css('border', '1px solid black');
   jQuery('.block-bean > *').css('opacity', '0');
   jQuery('.block-bean').css('height', '200px');
.then(function() {
  phantomcss.screenshot('#page-wrapper', <test-name>);

We first hide the dynamic content within our pages. This includes text, images, videos, and so on. Doing so ensures swapping out one piece of content for another will not cause our tests to fail. Note that we are only hiding the content, not removing it so we don't alter the structure of the page. Next, we set fixed heights to each parent element so adding or removing a piece of content within each container will not cause our tests to fail from the container growing or shrinking. This leaves us with only fixed outlines of the containers on each page, allowing us to test the layout and structure of pages within the platform.

Now, within a Drupal context, we are able to test how many blocks are present on a page and if regions of our contexts are properly laid out. Furthermore, a user simply adding a paragraph to an article or replacing one image with another won’t cause our tests to fail, falsely indicating the platform has broken. Below are two examples of baseline screenshots that our test suite generates.

Screen-Shot-2015-04-16-at-3.40.20-PM-279x300Screen-Shot-2015-04-15-at-5.17.50-PM-162x300 (1)


Testing Static Elements

For a few significant and constant elements on the platform, we tested them as a whole, instead of concealing the theming and content. Such elements include the header, the navigation, and the footer. For these elements we simply target the element’s ID and let PhantomJS do it’s thing.


What happens when a test fails?

When a test fails the screenshot diff is shown to us, highlighting any and all changes. For example, when the Department of Energy’s footer is altered the following test result is generated.


When a block is mistakenly removed from a page we receive the following failing test result.


This was a great first step. After creating these tests for many of the pages on the Department of Energy platform we were able to properly test the layout of our pages - but it left us wanting more. We needed an improved workflow to automatically run our visual regression tests after pull requests were merged to development, after every deployment to staging and production, and locally as desired. We needed an update to the open-source grunt-phantomcss module and the introduction of Jenkins, both of which will be covered in subsequent blog posts.

Subscribe to the Phase2 mailing list to learn when the next post in the visual regression test series goes live!