Closed Bug 1448086 Opened 2 years ago Closed 2 years ago

permafailing webdriver and webreftests testing backgrounds with Webrender when Gecko 61 merges to Beta on 2018-04-26


(Core :: Graphics: WebRender, defect)

Not set



Tracking Status
firefox-esr52 --- unaffected
firefox-esr60 --- fixed
firefox59 --- unaffected
firefox60 --- unaffected
firefox61 + fixed


(Reporter: aryx, Assigned: kats)




(1 file)

I'll take this, we shouldn't even be running QR tests on beta. I probably forgot something in the taskcluster configs somewhere.
Assignee: nobody → bugmail
Component: Graphics: Layers → Graphics: WebRender
Flags: needinfo?(rhunt)
They aren't running on beta, just in the central-as-beta simulation (it's difficult to get the same tasks across trees). Until now they were passing.
Until bug 1425589 landed yesterday they shouldn't have been running on any QR test platform anywhere. If you have a link to an older simulation we can check.
Ok, so there's like three problems here.

One is that I legitimately forgot to disable the tests on beta. When I simulate the taskgraph using a parameters.yml from the beta tree, I see the web-platform-tests showing up there. That's the first problem.

The second is that on beta I *also* see QR reftests, jsreftests, and mochitests. Which means I forgot to disable it for those suites as well - you can see them running uselessly on beta at [1]. They're running "with webrender enabled" but webrender isn't even built on beta, so they're just applying the webrender test expectations on a regular beta build. They *should* be failing, but out of sheer luck they're not. The mochitests are passing on beta are because they pass on central too (there's no mochitests which are annotated as fails-if webrender, although there are some that are skip-if webrender). And the reftest harness checks for webrender at runtime, so the expectations that it uses are the same as for the non-webrender.

Anyway, I have a patch to fix those two problems, will post it in a sec.

The third problem is what you said in comment 2 - the central-as-beta simulation doesn't run the same jobs as actually get run on beta. My patch to disable the tests on beta doesn't disable them in your simulation, so you'll just have to ignore the failures in your simulations, because the tests won't actually run on real beta.

Blocks: 1425589
Comment on attachment 8961501 [details]
Bug 1448086 - Ensure QR tests are disabled on beta and project branches.
Attachment #8961501 - Flags: review?(aryx.bugmail) → review+
Pushed by
Ensure QR tests are disabled on beta and project branches. r=aryx
Closed: 2 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla61
You need to log in before you can comment on or make changes to this bug.