Closed Bug 1399461 Opened 2 years ago Closed 2 years ago
.html == css-break/box-decoration-break-with-outset-box-shadow-1 .html | image comparison, max difference: 45, number of differing pixels: 439
Filed by: csabou [at] mozilla.com https://treeherder.mozilla.org/logviewer.html#?job_id=130633850&repo=mozilla-central https://queue.taskcluster.net/v1/task/U7I-31e-RQyjVatbYLY0yg/runs/0/artifacts/public/logs/live_backing.log https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://queue.taskcluster.net/v1/task/U7I-31e-RQyjVatbYLY0yg/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1
This started with this merge to m-c from inbound: https://hg.mozilla.org/mozilla-central/pushloghtml?changeset=9873269746359dbcccbd5b44f0e82c0225289945 Sadly, these jobs only run on autoland, m-c and try, so I can't see when it started on inbound. This is pretty close to permafailing. (I see maybe 4 out of 27 green rs4 jobs on autoland today since the merge.)
Summary: Intermittent css-break/box-decoration-break-with-outset-box-shadow-1.html == css-break/box-decoration-break-with-outset-box-shadow-1.html | image comparison, max difference: 45, number of differing pixels: 439 → Nearly-permafailing css-break/box-decoration-break-with-outset-box-shadow-1.html == css-break/box-decoration-break-with-outset-box-shadow-1.html | image comparison, max difference: 45, number of differing pixels: 439
Why do these jobs not run on mozilla-inbound?
(And how does one trigger them to run on mozilla-inbound?)
Matt, your changes seem the most likely cause, I think...
(In reply to David Baron :dbaron: ⌚️UTC-7 (catching up on backlog from vacation) from comment #4) > Why do these jobs not run on mozilla-inbound? IIRC, When stylo was set up, it was tier-2 and capacity issues had people lean toward running them on as few trees as possible. Since the only things that would theoretically break stylo would be the landings from the servo repo which only happen on autoland, it was decided that we didn't need to run them on inbound. I think it's probably time to reconsider that, since non-servo merges can clearly break stylo tests. (In reply to David Baron :dbaron: ⌚️UTC-7 (catching up on backlog from vacation) from comment #5) > (And how does one trigger them to run on mozilla-inbound?) I believe you can still manually add them to a push on inbound by signing in to Treeherder, filtering down to "rs4" in the textbox in Treeherder's header, and then clicking the "Add new jobs" button in the per-push menu for the desired pushes. After a few seconds, you should see a few new rs4 jobs appear for the push. Click the linux64 version of the job, then click the "Trigger new jobs" button in the per-push header and confirm the prompt that appears. After a few minutes, you should see the actual job start running on the push. You could also try pushing various revisions from inbound in the regression range to try, only requesting these jobs to run, then check those results in a few hours. Linux64 should finish decently quickly.
Filed bug 1399742 for trying to get these jobs scheduled on inbound.
I've triggered the rs4 job on inbound using the steps I mentioned and requested a 50-push backfill, which hopefully will be enough to get to where the failure started. I might be up long enough to monitor things and backfill further if needed.
Regression range down to https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&fromchange=68fc77ea1ef10f5ae69dc67899566391be636ee3&noautoclassify&filter-searchStr=64%20deb%20rs4&tochange=9e7cdda04cbfc5d4dae19e071498d3351923196e This still includes Matt's patches.
Not sure you'll get much more specific than this without doing some clever try pushes, because one of the patches from  busted all builds up to the backout at the top of the regression range. If you want to try figuring this out on try, you'd need to back out  to get usable builds. 1. https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=b507473b9b75dfcf08e4529483583a169f598ff6&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=running&filter-resultStatus=pending&filter-resultStatus=runnable 2. https://hg.mozilla.org/integration/mozilla-inbound/rev/371a03aa9b2469730b3fa300509eaeeb2ba5e255
Got bored, pushed things to try... So the tip of the current regression range is with a backout of just bug 1397056. Test fails. https://treeherder.mozilla.org/#/jobs?repo=try&revision=514131daf5fd08a4722bca5b86c62c2e0f741d28 will be the tip of the regression range, plus a backout of bug 1397060. Jury's still out on that one. https://treeherder.mozilla.org/#/jobs?repo=try&revision=34c56848b03f54f9b9587582ae38e1932593973c will be the prior try push, but bug 1397058 is also backed out. No results yet. https://treeherder.mozilla.org/#/jobs?repo=try&revision=e9a20bf2db60dc4e74d695e3a9f331e3618053a0 will be the prior try push, but bug 1397057 is also backed out. Nothing yet here. https://treeherder.mozilla.org/#/jobs?repo=try&revision=ab393c505a66529ae2c5ee5e818ee05ea619e510 will be all of Matt's commits from that push backed out. No results yet. Hope that helps, I'm gonna go sleep now.
Thank you for the Try pushes. Fixed by the backouts of bug 1397058 and bug 1397060.
You need to log in before you can comment on or make changes to this bug.