1325530 - Intermittent browser/base/content/test/tabcrashed/browser_shown.js | This test exceeded the timeout threshold. It should be rewritten or split up. If that's not possible, use requestLongerTimeout(N), but only as a last resort. -

Status: RESOLVED FIXED

(Firefox :: General, defect)

Description

[Treeherder Bug Filer]

Filed by: philringnalda [at] gmail.com

https://treeherder.mozilla.org/logviewer.html#?job_id=8342249&repo=autoland

https://archive.mozilla.org/pub/firefox/tinderbox-builds/autoland-linux-debug/1482428453/autoland_ubuntu32_vm-debug_test-mochitest-e10s-browser-chrome-7-bm142-tests1-linux32-build27.txt.gz

Comment 1

[Intermittent Failures Robot]

2 failures in 814 pushes (0.002 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-central: 1
* autoland: 1

Platform breakdown:
* windows7-32-vm: 1
* linux64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1325530&startday=2017-06-12&endday=2017-06-18&tree=all

Comment 2

[Intermittent Failures Robot]

1 failures in 720 pushes (0.001 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 1

Platform breakdown:
* osx-cross: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1325530&startday=2017-07-10&endday=2017-07-16&tree=all

Comment 3

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Well, that's upsetting. The spike of these failures that's going to be mentioned tonight started on the merge of this backout to autoland: https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=de22d4444e76e5b8cdb80e772776d283210eaaf2&filter-searchStr=6af427563d6f0bdccde10a845b45a4a0fc23a359

https://treeherder.mozilla.org/#/jobs?repo=autoland&fromchange=3f27c55fa17fd02ec659235dc9ddbc03155e4f24&noautoclassify&filter-searchStr=6af427563d6f0bdccde10a845b45a4a0fc23a359&selectedJob=116133106&group_state=expanded

Comment 4

[Intermittent Failures Robot]

120 failures in 190 pushes (0.632 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 99
* mozilla-inbound: 21

Platform breakdown:
* linux32: 92
* linux64: 28

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1325530&startday=2017-07-20&endday=2017-07-20&tree=all

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

is this perma fail, can we back this out :KWierso?

Comment 6

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Not permafailing. I'd say it's about 50% failure rate. 

And I was wrong in comment 3. The chunk the test runs in keeps shifting, so it only looked like that backout merge caused the spike. I can try further tracking down where this started.

Comment 7

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Actually, I would guess it'd be https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=653720e594d17796bf940e69933c8a712a4db38c&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=running&filter-resultStatus=pending&filter-resultStatus=runnable&filter-searchStr=linux32+debug+s(bc that caused this.

Comment 8

[Intermittent Failures Robot]

64 failures in 144 pushes (0.444 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 49
* mozilla-inbound: 14
* mozilla-central: 1

Platform breakdown:
* linux32: 33
* linux64: 31

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1325530&startday=2017-07-21&endday=2017-07-21&tree=all

Comment 9

[Intermittent Failures Robot]

207 failures in 822 pushes (0.252 failures/push) were associated with this bug in the last 7 days. 

This is the #5 most frequent failure this week. 

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **  

Repository breakdown:
* autoland: 159
* mozilla-inbound: 47
* mozilla-central: 1

Platform breakdown:
* linux32: 143
* linux64: 64

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1325530&startday=2017-07-17&endday=2017-07-23&tree=all

Comment 10

[Intermittent Failures Robot]

78 failures in 152 pushes (0.513 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 39
* mozilla-inbound: 29
* try: 8
* mozilla-central: 2

Platform breakdown:
* linux32: 50
* linux64: 28

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1325530&startday=2017-07-24&endday=2017-07-24&tree=all

Comment 11

[Geoff Brown [:gbrown]]

This was a long-running test before mconley's change:

Test durations for browser/base/content/test/tabcrashed/browser_shown.js on mozilla-central,mozilla-inbound,autoland between 2017-07-11 and 2017-07-18
linux32/debug-e10s:            103.66 s (82.71 s - 123.08 s over 193 runs)
linux32/opt-e10s:               24.69 s (20.94 s - 47.71 s over 231 runs)
linux64/debug-e10s:             97.67 s (79.46 s - 138.65 s over 285 runs)

With those durations, it seems odd that frequent failures were not reported earlier.

Regardless, we have the standard options: Find a way to optimize or simplify the test, split it up into 2 or more tests, or requestLongerTimeout(). 

Hoping mconley has time to fix this soon...

Comment 12

[Geoff Brown [:gbrown]]

Attachment: requestLongerTimeout

The simplest "solution"!

Also, I noticed https://dxr.mozilla.org/mozilla-central/rev/7d2e89fb92331d7e4296391213c1e63db628e046/browser/base/content/test/tabcrashed/browser_showForm.js#5 and other tabcrashed tests already requestLongerTimeout().

Comment 13

[Intermittent Failures Robot]

41 failures in 181 pushes (0.227 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 35
* mozilla-central: 3
* try: 1
* oak: 1
* mozilla-inbound: 1

Platform breakdown:
* linux32: 33
* linux64: 8

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1325530&startday=2017-07-25&endday=2017-07-25&tree=all

Comment 14

[Mike Conley (:mconley) (:⚙️)]

Comment on attachment 8890021 [details] [diff] [review]
requestLongerTimeout

Review of attachment 8890021 [details] [diff] [review]:
-----------------------------------------------------------------

Thanks!

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/d0a02adc8689
requestLongerTimeout for browser_shown.js; r=mconley

Comment 16

[Intermittent Failures Robot]

60 failures in 218 pushes (0.275 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 33
* mozilla-inbound: 21
* oak: 4
* try: 1
* mozilla-central: 1

Platform breakdown:
* linux32: 52
* linux64: 7
* windows7-32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1325530&startday=2017-07-26&endday=2017-07-26&tree=all

Comment 17

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/d0a02adc8689

Comment 18

[Intermittent Failures Robot]

192 failures in 1008 pushes (0.19 failures/push) were associated with this bug in the last 7 days. 

This is the #10 most frequent failure this week. 

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **  

Repository breakdown:
* autoland: 110
* mozilla-inbound: 50
* mozilla-central: 13
* try: 12
* oak: 7

Platform breakdown:
* linux32: 144
* linux64: 47
* windows7-32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1325530&startday=2017-07-24&endday=2017-07-30&tree=all