1447366 - Some Selenium Tests are Failing with Latest Nightly Running in Headless Mode

Status: RESOLVED FIXED

(Firefox :: Headless, defect)

Description

[Barry Chen]

Attachment: failed-tests-trace.log

There are a few Selenium tests in the Firefox Screenshots repo.  Three of them started to consistently fail around 2017-03-17.  However, they only fail when Firefox is running in headless mode (or in Xvfb).  These tests use the selenium-webdriver npm module, which in turn drives Firefox with geckodriver.

mozregression gave me the following:

Last good revision: b61bf51fd0c2f788ce98c96d7f205cc4d5c48424
First bad revision: ad5d0d34bddb427b2b89d218d7761c472a64428a
Pushlog: https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=b61bf51fd0c2f788ce98c96d7f205cc4d5c48424&tochange=ad5d0d34bddb427b2b89d218d7761c472a64428a

I've attached two trace logs.  One with the three failing tests, and the other with all the tests passing (using an older build of Nightly).  The logs start to deviate at the following lines:

Failed: 273
Passed: 265

Failed: 1689
Passed: 569

Failed: 1777
Passed: 747

Comment 1

[Barry Chen]

Attachment: passed-tests-trace.log

Comment 2

[Brendan Dahl [:bdahl]]

Are these tests available somewhere so that I can download and run them?

Comment 3

[Barry Chen]

Yes.  I created https://github.com/chenba/screenshots/tree/run-all-selenium-tests.  Currently those three tests are disabled in trunk so that the Screenshots CI isn't failing all the time, and that branch has a change to allow the tests to run without running the Screenshots server locally.  All the tests are in test/test.js in the repo.

git clone --depth 1 [e-mail]:chenba/screenshots.git -b run-all-selenium-tests
cd screenshots
npm install
MOZ_HEADLESS=1 SCREENSHOTS_BACKEND=https://screenshots.dev.mozaws.net npm run test:selenium

That should work.  The tests should pass with this build: https://archive.mozilla.org/pub/firefox/nightly/2018/03/2018-03-17-10-03-37-mozilla-central/.

Comment 4

[Barry Chen]

Those tests are passing again in Nightly.  Closing this bug.