1448086 - permafailing webdriver and webreftests testing backgrounds with Webrender when Gecko 61 merges to Beta on 2018-04-26

Status: RESOLVED FIXED

(Core :: Graphics: WebRender, defect)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

central-as-beta simulation: https://treeherder.mozilla.org/#/jobs?repo=try&revision=811e2ea438a88cb9e407f4ab66387bc2c18d0f35&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=runnable&group_state=expanded&selectedJob=169644852

This central-as-beta simulation has webreftests on QuantumRender failing, e.g. css/css-backgrounds/background-clip-color-repaint.html 

Looks like a regression from bug 1439960 (tests passed successfully yesterday).

Log: https://treeherder.mozilla.org/logviewer.html#?job_id=169644852&repo=try

TEST-UNEXPECTED-FAIL | /css/css-backgrounds/background-clip-color-repaint.html | Testing http://web-platform.test:8000/css/css-backgrounds/background-clip-color-repaint.html == http://web-platform.test:8000/css/css-backgrounds/background-clip-color-repaint-ref.html

Comment 1

[Kartikaya Gupta (email:[e-mail])]

I'll take this, we shouldn't even be running QR tests on beta. I probably forgot something in the taskcluster configs somewhere.

Comment 2

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

They aren't running on beta, just in the central-as-beta simulation (it's difficult to get the same tasks across trees). Until now they were passing.

Comment 3

[Kartikaya Gupta (email:[e-mail])]

Until bug 1425589 landed yesterday they shouldn't have been running on any QR test platform anywhere. If you have a link to an older simulation we can check.

Comment 4

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

They indeed didn't run. Yesterday's early beta sim: https://treeherder.mozilla.org/#/jobs?repo=try&revision=1f35b55fcbc3bd4d511fdcc6e82ea76def0b956c&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=runnable&filter-resultStatus=success&filter-searchStr=quantumrender

Comment 5

[Kartikaya Gupta (email:[e-mail])]

Ok, so there's like three problems here.

One is that I legitimately forgot to disable the tests on beta. When I simulate the taskgraph using a parameters.yml from the beta tree, I see the web-platform-tests showing up there. That's the first problem.

The second is that on beta I *also* see QR reftests, jsreftests, and mochitests. Which means I forgot to disable it for those suites as well - you can see them running uselessly on beta at [1]. They're running "with webrender enabled" but webrender isn't even built on beta, so they're just applying the webrender test expectations on a regular beta build. They *should* be failing, but out of sheer luck they're not. The mochitests are passing on beta are because they pass on central too (there's no mochitests which are annotated as fails-if webrender, although there are some that are skip-if webrender). And the reftest harness checks for webrender at runtime, so the expectations that it uses are the same as for the non-webrender.

Anyway, I have a patch to fix those two problems, will post it in a sec.

The third problem is what you said in comment 2 - the central-as-beta simulation doesn't run the same jobs as actually get run on beta. My patch to disable the tests on beta doesn't disable them in your simulation, so you'll just have to ignore the failures in your simulations, because the tests won't actually run on real beta.

[1] https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&revision=998a57bd05ada832da0cd78e2db628699afee80e&filter-searchStr=qr&group_state=expanded

Comment 6

[Kartikaya Gupta (email:[e-mail])]

Attachment: Bug 1448086 - Ensure QR tests are disabled on beta and project branches.

This updates the existing entries to use the 'trunk' shorthand notation,
and adds missing entries for mochitests, reftests, jsreftests, and the
three categories of web-platform-tests.

Review commit: https://reviewboard.mozilla.org/r/230268/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/230268/

Comment 7

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Comment on attachment 8961501 [details]
Bug 1448086 - Ensure QR tests are disabled on beta and project branches.

https://reviewboard.mozilla.org/r/230268/#review235874

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/e4304fffb4f0
Ensure QR tests are disabled on beta and project branches. r=aryx

Comment 9

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/mozilla-central/rev/e4304fffb4f0

Comment 11

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-esr60/rev/fba041cdcc2e

Comment 12

[Dustin J. Mitchell [:dustin] (he/him)]

https://tools.taskcluster.net/groups/LQ5-KlfCTwmsZbuj65LHvw/tasks/LQ5-KlfCTwmsZbuj65LHvw/runs/0/logs/public%2Flogs%2Flive.log

looks like that broke nightlies on that branch

Comment 13

[Kartikaya Gupta (email:[e-mail])]

It seems to be more likely a result of one of the other commits in this push: https://treeherder.mozilla.org/#/jobs?repo=mozilla-esr60&revision=1d67d6de4783c0683c87a924d51cc9bece88f6c0