1443364 - Intermittent browser/components/customizableui/test/browser_901207_searchbar_in_panel.js | Uncaught exception - Panel did not show within 20 seconds.

Status: RESOLVED FIXED

(Firefox :: Toolbars and Customization, defect)

Description

[Treeherder Bug Filer]

Filed by: nbeleuzu [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=166072126&repo=autoland

https://queue.taskcluster.net/v1/task/HF7DDkSgTAi47krO3XEU-w/runs/0/artifacts/public/logs/live_backing.log

Comment 1

[Intermittent Failures Robot]

19 failures in 116 pushes (0.164 failures/push) were associated with this bug yesterday.    

Repository breakdown:
* autoland: 8
* mozilla-inbound: 4
* mozilla-central: 3
* mozilla-beta: 3
* try: 1

Platform breakdown:
* linux64-stylo-disabled: 6
* linux64: 4
* osx-10-10: 3
* linux32-stylo-disabled: 3
* linux32: 2
* linux32-devedition: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1443364&startday=2018-03-06&endday=2018-03-06&tree=all

Comment 2

[:Gijs (he/him)]

Off-hand this failure just looks like "we're too slow".

17 failures over 24 hours still means you'd need something on the order of 30-50 retriggers per push to narrow this down, and I likely won't be able to reproduce locally. I don't have cycles to chase this by manually running hundreds of retriggers to try to find the regressor in the pile of autoland checkins. Do we have tools for this at this point?

Basically, I'd really like this to be as simple as feeding an orange factor bug reference (which will have the first occurrence, failure message, most-frequently-failing platform, and frequency information to extrapolate a safe number of retriggers from) and a backward timeframe (default: 1 day) into a webpage, and have it come back to me a few hours later with an answer.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

actually making a |./mach try intermittent <bugid>| would be cool.  It could determine the failure rate and platforms/configs and push to try with --rebuild X (although we are limited to a max of 20 for --rebuild as it seems to get abused too often).  The failures are primarily debug with a few opt failures mixed in there.  I have filed bug 1443773 to track this idea.

For the concept of retriggering recent failures on the trees, that does take a lot of work and we don't have anything specific in place for a process or tools to do this.  Our plan is to have something in place for the code sheriffs to see these high frequency failures that started in the last 3 days and retrigger and find the root cause.  The best bet here is to build a view on orangefactor or other dashboard where we show recent failures that occur at least 15 times in and are less than 3 days old- that would give us a place to focus efforts.

We have tried a few different times to solve this for performance alerts and it has failed in so many ways each time due to all the edge cases.

I have done some retriggers to see if this happened when the test moved from chunk 2 -> chunk 8:
https://treeherder.mozilla.org/#/jobs?repo=autoland&filter-searchStr=linux%20debug%20browser-chrome-e10s%20%20x64%20stylo&tochange=83df6cf24cf8fa59eb338ce0bd7bedac289c5d33&fromchange=916d91f23f895a441ffe375ff245d4c62a4bb41a

Comment 4

[Geoff Brown [:gbrown]]

I think intelligent-backfilling&retriggering is a great idea...but it does have a lot of moving parts. I have some fear that such a project might end up creating a feature that works like magic one day of the year but otherwise has us chasing a million bugs.

I have aspirations of combining test-verify with backfilling to provide a similar feature: In treeherder, request regression tracking for a named test and changeset; then run test-verify on that test and work backward in time until the test runs reliably. However, this is just an idea -- I haven't even thought about coding this yet, there's no bug on file or anything.

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

looks to be related to bug 1442651:
https://hg.mozilla.org/integration/autoland/rev/562966f195b0f071cf1d973f7b4fd51ec8594343

::dao, can you take a look at this?

Comment 6

[Geoff Brown [:gbrown]]

btw, this can be reproduced with test-verify on try: https://treeherder.mozilla.org/#/jobs?repo=try&revision=5c085cfa2535f0f9d2ed4016501653ff8bf0e3e9

Comment 7

[Dão Gottwald [:dao]]

Interestingly, browser_901207_searchbar_in_panel.js doesn't seem to interact with tabbrowser in any meaningful way, and I don't see any errors in the log.

(In reply to :Gijs (under the weather; responses will be slow) from comment #2)
> Off-hand this failure just looks like "we're too slow".

20 seconds to show the overflow panel...?

I just ran the test locally and can actually reproduce the failure. The toolbar overflows as it should but the panel doesn't open. I can open it manually and focus the search bar, and then the test continues and passes.

Comment 8

[Dão Gottwald [:dao]]

(In reply to Dão Gottwald [::dao] from comment #7)
> I just ran the test locally and can actually reproduce the failure. The
> toolbar overflows as it should but the panel doesn't open. I can open it
> manually and focus the search bar, and then the test continues and passes.

Apparently it's sendWebSearchKeyCommand that fails. I can hit accel+k manually and that lets the test pass as well.

Comment 9

[:Gijs (he/him)]

(In reply to Dão Gottwald [::dao] from comment #7)
> Interestingly, browser_901207_searchbar_in_panel.js doesn't seem to interact
> with tabbrowser in any meaningful way, and I don't see any errors in the log.
> 
> (In reply to :Gijs (under the weather; responses will be slow) from comment
> #2)
> > Off-hand this failure just looks like "we're too slow".
> 
> 20 seconds to show the overflow panel...?

Well, some of the screenshots in failing tests do show the overflow panel, with the search bar in it, and it's mostly failing on debug and asan runs, so yes that was my starting assumption. The regression window makes it a bit moot, of course...

Comment 10

[Dão Gottwald [:dao]]

I'm going to attach a patch that fixes the test over here, but I really don't understand why. the only rational explanation I can come up with right now for how bug 1442651 broke this is that document.activeElement is the <browser> when sendWebSearchKeyCommand gets called first, whereas before bug 1442651 it probably was <tabbrowser> because the browser elements were anonymous. But the focused element should be the browser element regardless. I don't know why this would affect EventUtils.synthesizeKey, let alone why this would be intermittent. Like I said manually hitting accel+k worked fine, so I don't think we have a real regression here.

Comment 11

[Dão Gottwald [:dao]]

Attachment: Bug 1443364 - Fix browser_901207_searchbar_in_panel.js intermittent failure.

Review commit: https://reviewboard.mozilla.org/r/225850/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/225850/

Comment 12

[:Gijs (he/him)]

 (In reply to Geoff Brown [:gbrown] from comment #4)
> However, this is just an idea -- I haven't
> even thought about coding this yet, there's no bug on file or anything.

I filed https://bugzilla.mozilla.org/show_bug.cgi?id=1443867 . :-)

Comment 13

[:Gijs (he/him)]

Comment on attachment 8956896 [details]
Bug 1443364 - Fix browser_901207_searchbar_in_panel.js intermittent failure.

https://reviewboard.mozilla.org/r/225850/#review231734

rs=me, though I wonder if we should file a followup about the default focus change having happened and just check in with Enn about why that would be / if there are going to be more serious repercussions than random tests failing (which of course is unfortunate, but doesn't ultimately hurt users directly).

Thanks for tracking this down and fixing it so quickly, Dão, Joel and Geoff!

Comment 14

[Dão Gottwald [:dao]]

(In reply to :Gijs (under the weather; responses will be slow) from comment #13)
> Comment on attachment 8956896 [details]
> Bug 1443364 - Fix browser_901207_searchbar_in_panel.js intermittent failure.
> 
> https://reviewboard.mozilla.org/r/225850/#review231734
> 
> rs=me, though I wonder if we should file a followup about the default focus
> change having happened and just check in with Enn about why that would be /
> if there are going to be more serious repercussions than random tests
> failing (which of course is unfortunate, but doesn't ultimately hurt users
> directly).
> 
> Thanks for tracking this down and fixing it so quickly, Dão, Joel and Geoff!

I'll file a followup once we know that this actually fixes the intermittent failure rather than just my local one. I tried to run test-verify on try but failed to figure out how to do it, so I'll just land this...

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/39f31c0be9f4
Fix browser_901207_searchbar_in_panel.js intermittent failure. r=Gijs

Comment 16

[Stefan Hindli [:stefan_hindli]]

https://hg.mozilla.org/mozilla-central/rev/39f31c0be9f4

Comment 17

[Intermittent Failures Robot]

16 failures in 160 pushes (0.1 failures/push) were associated with this bug yesterday.    

Repository breakdown:
* autoland: 9
* mozilla-inbound: 6
* mozilla-central: 1

Platform breakdown:
* linux64-stylo-disabled: 7
* linux32: 5
* linux32-stylo-disabled: 2
* windows7-32: 1
* windows10-64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1443364&startday=2018-03-07&endday=2018-03-07&tree=all

Comment 19

[Intermittent Failures Robot]

36 failures in 747 pushes (0.048 failures/push) were associated with this bug in the last 7 days. 

This is the #34 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **  

Repository breakdown:
* autoland: 18
* mozilla-inbound: 10
* mozilla-central: 4
* mozilla-beta: 3
* try: 1

Platform breakdown:
* linux64-stylo-disabled: 15
* linux32: 7
* linux64: 4
* linux32-stylo-disabled: 4
* osx-10-10: 3
* windows7-32: 1
* windows10-64: 1
* linux32-devedition: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1443364&startday=2018-03-05&endday=2018-03-11&tree=all