What was missing

Playwright for JavaScript has toHaveScreenshot(): one line, and the test compares the page with a baseline. Playwright for Python has nothing like it. When I started, the options were thin: a couple of small plugins on top of pixel-by-pixel comparison, or a home-grown wrapper that every team writes from scratch.

And pixel-by-pixel comparison breaks in two ways.

The first: the test goes red although the page did not change. A font rendered half a pixel differently, CI ran on another machine. The team puts up with it for a week, then someone adds automatic baseline updates to the pipeline. Visual tests still exist on paper, but in practice they are switched off.

The second: to stop the red, people set a threshold like "98% similar". And the test stops seeing small things. But a UI breaks exactly in small things: the wrong checkbox state, the wrong button colour, a border off by one pixel.

I have seen this first-hand. In one project a screenshot suite had been green for six months. It compared the whole frame with a 0.98 threshold. And all that time it was missing a real bug. More on that below.

What I wanted was simple:

• pip install, one function in the test, baselines in the repository;
• the test does not fail when the page is merely redrawn a little differently;
• the test does fail on a small real change;
• and most importantly, it says in words what changed: "button recoloured", "block moved by one pixel", not "3.2% mismatch".

That is how VisTest started.

First I built the wrong thing

Let me admit it right away: the first version was not a library but a platform. A server, a review web UI, run history, a mouse recorder for tests, single sign-on. I thought the product was all of that, and image comparison was a detail inside that already worked.

Then I counted. The comparison itself was about a tenth of the code. The other nine tenths served an engine I had never tested on real screenshots, only on pictures drawn by my own code.

I realised it when I tried to plug the tool into someone else's project. You cannot pip-install a platform into someone else's tests. So VisTest narrowed down to what I should have started with: a library and a pytest plugin. The team server did not go away, but it became a separate product for later.

First contact with a real project

The pilot was an internal Playwright project with 11 screenshot tests. There was one piece of good news, and it mattered: adoption came down to replacing a single wrapper method. The tests themselves did not change by a single line.

Then, within the first few minutes, we found more problems than in the whole development so far. The three most telling ones:

One difference killed the whole run. The fifth test found a real mismatch, and pytest crashed entirely with an internal error. The other six tests never ran, and there was no report. A tool meant to help find regressions buried the run at the exact moment it found one. Now the rule is simple: a configuration error is loud and immediate; a failure in an optional part is a warning, and the tests carry on.

The numbers in the message did not add up. "1.44% of the frame changed", while the regions it named added up to dozens of times less. A person reading a CI log cannot reconcile numbers like that, and stops trusting both. Now the area is broken into parts that always add up to the total.

The engine erased text. To remove noise, the change mask was cleaned with a morphological filter, and along with the noise it threw away everything thinner than five pixels. Which is almost all text. On one pair of images, 97% of a real change disappeared before the engine even started looking at it. I changed the order of operations and gave noise its own explanations. Now every discarded difference is labelled with the rule that discarded it.

None of these bugs would have been caught by my own tests, and by then I had more than a thousand of them. Your own tests check what you thought of. Someone else's project checks what nobody thought of.

What it was all for

On the second run, VisTest found a real bug in that very suite that had been green for six months. The state of a filter panel was carrying over from one page to another when it should not have. The difference sat on 35-pixel checkboxes. A whole-frame comparison with a 0.98 threshold could never see it: 35 pixels on a full-screen screenshot barely move the overall "similarity".

One finding across eleven tests is not statistics, and I do not conclude from it that "VisTest is better". But it shows the problem is real: a whole class of changes slips past an ordinary check.

Where the project is now

VisTest is on PyPI as an alpha, open source under the Apache-2.0 licence.

pip install "vistest[browser]"
playwright install chromium

from vistest import expect_screenshot

def test_landing(page):
    page.goto("https://example.com")
    expect_screenshot(page, "landing.png")

The first run is red: there is no baseline yet. pytest --vistest-update writes the baselines; you look at them and commit them. From then on the test compares, and if something changed, it explains what and where.

There are no comparison numbers against other tools here, on purpose. The next posts are about exactly that: how I measured my engine, how I fooled myself with nice-looking results, and how I then lost to Playwright on real browser screenshots. That story is more useful than any victory table.

If you write tests in Python and have also been looking for a toHaveScreenshot replacement, give VisTest a try and tell me what broke. Someone else's project, as it turned out, is the best test.

Next in the series…

Repository: github.com/marshlw/vistest