C2LC-261: Add full browser tests.

Description

There are a number of cases in which JSDom does not mimic a browser faithfully enough for us to test our work. For example, as we recently discovered, JSDom lacks much of the WebAudio API. There are also things that are much easier to test faithfully in a full browser, such as focus management and running third-party accessibility checks.

I was talking with a colleague about testing a different project, and started reading up on webdriver.io, which seems like a reasonable candidate for us to explore if we find that we’re ready for full browser tests.

Unlike the reference Webdriver Javascript implementation, Webdriver.io has a built-in understanding of and ability to easily interact with React components. Webdriver.io has the option to run synchronously, which is nice for writing straightforward browser tests.

Issue Links

Comments

  • Tony Atkins [RtF] commented 2021-01-25T05:00:23.608-0500

    I read up a bit to see how feasible/involved this would be. Specifically, I wanted to see how we would keep track of test code coverage, as one of the key goals here is to test uncovered branches for things like audio and TTS that are mocked in our Enzyme tests.

    I ran across this article, which outlines an approach that’s very familiar from other work I’ve done:

    1. Create an instrumented build using the Babel istanbul plugin.
    2. Run one or more tests, collecting coverage data along the way.
    3. Create a combined report with the coverage data from our Enzyme tests and the Webdriver tests.


    The one bit I’m missing from the blog is how to ensure that the app is only built with instrumentation for these tests. I would start by going through their example for my own knowledge. Webdriver.io has its own learning curve, but they provide some very handy deep integration with React, so I’d expect this to be pretty palatable. The combined reporting stuff is trivial at this point, I’ve done it for years integrating Node and Testem tests.

    All told I’d say a week of learning and sketching to get this done, and it’s a good one to chip away at in the background while waiting for input from others.

  • Tony Atkins [RtF] commented 2021-01-26T04:24:32.373-0500

    In a related conversation about the broader use of functional browser tests yesterday, I investigate a few topics which helped confirm my inclination to use webdriver.io for this work. First, there are a few solutions that control a range of actual browsers, notably:

    1. The javascript reference implementation of the WebDriver API (npm package: selenium-webdriver).
    2. webdriver.io, which can use a range of technologies, including the WebDriver API.
    3. cypress.io, which uses a different approach.

    I have long experience with the first. The previous stable release stopped working (at least in the way I used it) for FireFox years ago, and the next release has been in alpha state for years. Although it could probably do what we need, it doesn’t seem like a healthy foundation for a new suite of tests.

    I looked into webdriver.io and cypress.io, and there are a few key points that suggest to me that webdriver.io is the way to proceed. First, cypress.io lacks support for Safari, which by itself is a deal-breaker in my opinion.

    I also considered the types of interaction we want to simulate. Although it’s possible to test drag and drop in cypress.io, we’d have to write a lot of our own code to accomplish it. By comparison, webdriver.io provides the ability to easily drop on a particular target element, or on a region of the screen.

    The keyboard support in cypress.io involves sending commands directly to an element’s handlers. While we use this strategy currently, longer term it limits us in properly testing keyboard navigation. By comparison, webdriver.io lets you send keys to its browser instance instead of a single element. It uses the same underlying mechanisms as the WebDriver API, and I have used this to simply test things like making use of a “skip to content” link.

    With cypress.io, you would have to locate the link using a selector, send its key handler an enter key, and then check focus. By comparison, with the WebDriver API and webdriver.io, you just send the browser “tab” and “enter”, which also confirms that the “skip to content” link comes into focus when a “tab” is pressed.

    In summary, as we want to test keyboard navigation and drag and drop in particular, webdriver.io seems like a strong candidate.