1484053 - Frequent to perma-failing browser-chrome failures on some platforms when Gecko 63 merges to Beta on 2018-08-27

Status: RESOLVED WORKSFORME

(Firefox :: General, defect, P1)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

mochitest browser-chrome failures have become more failures, some failures even have all 5 retriggers failing.

August 14th's beta simulation was without issues (one bc failure per platform is a known issue): https://treeherder.mozilla.org/#/jobs?repo=try&revision=7f2fbccfc0a525116d76c93a563bbdee252635e3&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=runnable&filter-searchStr=browser-chrome

August 15th's had frequent bc failures: https://treeherder.mozilla.org/#/jobs?repo=try&revision=ae5df0fec8ffc3539b5c97c2aef00ab852f57e1b&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=runnable&filter-searchStr=browser-chrome&selectedJob=194176741

Bisection:

good: 78d5bc33afd0 https://treeherder.mozilla.org/#/jobs?repo=try&revision=a08022e3994715e77e65d85f60b63c324a802516

bad: e72eac515afc https://treeherder.mozilla.org/#/jobs?repo=try&revision=8a2d897fb4d64cd5cf5e80d1895bf3ea28624f6b

Pushlog: https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=78d5bc33afd0&tochange=e72eac515afc

Kris, please take a look if this is from one of your changes in that pushlog.

Comment 1

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Does this still need to be fixed? The first merge of Nightly63 -> Beta63 was completed today.

Comment 2

[Ryan VanderMeulen [:RyanVM]]

Yes, this is still an important bug to fix. I emailed Kris directly today about it.

Comment 3

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Hi Mike, can you take a look why docshell/test/browser/browser_search_notification.js frequently fails on beta for some platforms, e.g. OS X Nightly?

Log example: https://treeherder.mozilla.org/logviewer.html#?job_id=196648559&repo=mozilla-beta

Thank you.

Comment 4

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

FYI: The test got disabled on beta (bug 1443137), so it has to be re-enabled before testing it e.g. on Try.

Comment 5

[Justin Dolske [:Dolske]]

Not a search-specific issue AIUI; kmag seems to have the blame from the regression range, so investigation should start there.

Assigning since this is on the tracking list.

Comment 6

[Kris Maglione [:kmag]]

If my changes are responsible, it's almost certainly only because they significantly sped a lot of things up and made existing races more apparent. They definitely had that effect elsewhere.

Comment 7

[David Durst [:ddurst]]

Because the changes in bug 1480244 (via the pushlog referenced in #c0, https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=78d5bc33afd0&tochange=e72eac515afc) are not simple or narrow, and have been verified as speeding things up, I feel like it would be safe/sane to choose a small test to verify that theory. But I would also prefer that verification to be done by the test owner.

Since #c4 indicates that docshell/test/browser/browser_search_notification.js is no longer a candidate for that, Sebastian, can you suggest another test we could use as a candidate?

[I realize this is unusual, but I don't yet have a good way to alert lots of groups when foundational changes might affect their tests. Ideas welcome, just not in this bug please.]

Comment 8

[Justin Dolske [:Dolske]]

(Now assigning to ddurst for a resolution :)

Comment 9

[David Durst [:ddurst]]

I still think the approach in #c7 is sensible. Looking for guidance from :aryx.

Comment 10

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

browser_search_notification.js got fixed in bug 1443137, other tests might have been disabled by project Stockwell (disable frequent failures if there is no action on them) or also fixed. The trees for beta and release don't show any of those frequent failures which were on beta when this bug got filed.