Closed Bug 1448086 Opened 2 years ago Closed 2 years ago
permafailing webdriver and webreftests testing backgrounds with Webrender when Gecko 61 merges to Beta on 2018-04-26
59 bytes, text/x-review-board-request
central-as-beta simulation: https://treeherder.mozilla.org/#/jobs?repo=try&revision=811e2ea438a88cb9e407f4ab66387bc2c18d0f35&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=runnable&group_state=expanded&selectedJob=169644852 This central-as-beta simulation has webreftests on QuantumRender failing, e.g. css/css-backgrounds/background-clip-color-repaint.html Looks like a regression from bug 1439960 (tests passed successfully yesterday). Log: https://treeherder.mozilla.org/logviewer.html#?job_id=169644852&repo=try TEST-UNEXPECTED-FAIL | /css/css-backgrounds/background-clip-color-repaint.html | Testing http://web-platform.test:8000/css/css-backgrounds/background-clip-color-repaint.html == http://web-platform.test:8000/css/css-backgrounds/background-clip-color-repaint-ref.html
I'll take this, we shouldn't even be running QR tests on beta. I probably forgot something in the taskcluster configs somewhere.
Assignee: nobody → bugmail
Component: Graphics: Layers → Graphics: WebRender
They aren't running on beta, just in the central-as-beta simulation (it's difficult to get the same tasks across trees). Until now they were passing.
Until bug 1425589 landed yesterday they shouldn't have been running on any QR test platform anywhere. If you have a link to an older simulation we can check.
They indeed didn't run. Yesterday's early beta sim: https://treeherder.mozilla.org/#/jobs?repo=try&revision=1f35b55fcbc3bd4d511fdcc6e82ea76def0b956c&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=runnable&filter-resultStatus=success&filter-searchStr=quantumrender
Ok, so there's like three problems here. One is that I legitimately forgot to disable the tests on beta. When I simulate the taskgraph using a parameters.yml from the beta tree, I see the web-platform-tests showing up there. That's the first problem. The second is that on beta I *also* see QR reftests, jsreftests, and mochitests. Which means I forgot to disable it for those suites as well - you can see them running uselessly on beta at . They're running "with webrender enabled" but webrender isn't even built on beta, so they're just applying the webrender test expectations on a regular beta build. They *should* be failing, but out of sheer luck they're not. The mochitests are passing on beta are because they pass on central too (there's no mochitests which are annotated as fails-if webrender, although there are some that are skip-if webrender). And the reftest harness checks for webrender at runtime, so the expectations that it uses are the same as for the non-webrender. Anyway, I have a patch to fix those two problems, will post it in a sec. The third problem is what you said in comment 2 - the central-as-beta simulation doesn't run the same jobs as actually get run on beta. My patch to disable the tests on beta doesn't disable them in your simulation, so you'll just have to ignore the failures in your simulations, because the tests won't actually run on real beta.  https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&revision=998a57bd05ada832da0cd78e2db628699afee80e&filter-searchStr=qr&group_state=expanded
Comment on attachment 8961501 [details] Bug 1448086 - Ensure QR tests are disabled on beta and project branches. https://reviewboard.mozilla.org/r/230268/#review235874
Attachment #8961501 - Flags: review?(aryx.bugmail) → review+
Pushed by email@example.com: https://hg.mozilla.org/integration/autoland/rev/e4304fffb4f0 Ensure QR tests are disabled on beta and project branches. r=aryx
Duplicate of this bug: 1467007
You need to log in before you can comment on or make changes to this bug.