Last Comment Bug 1292460 - [32-bit Linux, maybe focus related] Permaorange or intermittent on Linux32 debug unaccelerated layout/reftests/bugs/613433-3.html,613433-2.html,613433-1.html | load failed: timed out waiting for reftest-wait to be removed
: [32-bit Linux, maybe focus related] Permaorange or intermittent on Linux32 de...
Status: RESOLVED FIXED
: intermittent-failure, regression
Product: Core
Classification: Components
Component: Layout (show other bugs)
: Trunk
: Unspecified Unspecified
P1 normal (vote)
: mozilla54
Assigned To: Jet Villegas (:jet)
:
: Jet Villegas (:jet)
Mentors:
: 1289014 (view as bug list)
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2016-08-05 01:13 PDT by Treeherder Bug Filer
Modified: 2017-01-30 11:55 PST (History)
8 users (show)
mats: in‑testsuite-
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---
wontfix
wontfix
fixed


Attachments
jet's patch to wait for focus before starting tests (4.37 KB, patch)
2017-01-23 14:10 PST, David Baron :dbaron: ⌚️UTC-8
dbaron: review+
mats: feedback+
Details | Diff | Splinter Review

Comment 1 User image OrangeFactor Robot 2016-08-07 18:06:51 PDT
42 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 19
* autoland: 12
* fx-team: 6
* mozilla-central: 5

Platform breakdown:
* linux32: 42

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1292460&startday=2016-08-01&endday=2016-08-07&tree=all
Comment 2 User image OrangeFactor Robot 2016-08-08 18:00:05 PDT
49 automation job failures were associated with this bug yesterday.

Repository breakdown:
* autoland: 31
* mozilla-inbound: 12
* fx-team: 5
* mozilla-central: 1

Platform breakdown:
* linux32: 49

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1292460&startday=2016-08-08&endday=2016-08-08&tree=all
Comment 3 User image OrangeFactor Robot 2016-08-09 18:00:05 PDT
43 automation job failures were associated with this bug yesterday.

Repository breakdown:
* autoland: 24
* mozilla-inbound: 12
* fx-team: 7

Platform breakdown:
* linux32: 43

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1292460&startday=2016-08-09&endday=2016-08-09&tree=all
Comment 4 User image OrangeFactor Robot 2016-08-10 18:00:05 PDT
45 automation job failures were associated with this bug yesterday.

Repository breakdown:
* autoland: 20
* mozilla-inbound: 15
* fx-team: 6
* try: 2
* mozilla-central: 2

Platform breakdown:
* linux32: 45

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1292460&startday=2016-08-10&endday=2016-08-10&tree=all
Comment 5 User image Phil Ringnalda (:philor) 2016-08-10 19:46:07 PDT
Some real goodness going on here: something, perhaps the landing of bug 1000957, moved the permaorange from -3 to -2.
Comment 6 User image OrangeFactor Robot 2016-08-11 18:00:04 PDT
51 automation job failures were associated with this bug yesterday.

Repository breakdown:
* autoland: 27
* mozilla-inbound: 14
* mozilla-central: 5
* fx-team: 4
* try: 1

Platform breakdown:
* linux32: 51

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1292460&startday=2016-08-11&endday=2016-08-11&tree=all
Comment 7 User image OrangeFactor Robot 2016-08-12 18:00:04 PDT
34 automation job failures were associated with this bug yesterday.

Repository breakdown:
* autoland: 15
* mozilla-inbound: 13
* fx-team: 5
* mozilla-central: 1

Platform breakdown:
* linux32: 34

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1292460&startday=2016-08-12&endday=2016-08-12&tree=all
Comment 8 User image OrangeFactor Robot 2016-08-14 18:06:24 PDT
238 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* autoland: 124
* mozilla-inbound: 67
* fx-team: 29
* mozilla-central: 12
* try: 6

Platform breakdown:
* linux32: 238

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1292460&startday=2016-08-08&endday=2016-08-14&tree=all
Comment 9 User image OrangeFactor Robot 2016-08-15 18:00:03 PDT
34 automation job failures were associated with this bug yesterday.

Repository breakdown:
* autoland: 19
* mozilla-inbound: 9
* try: 2
* mozilla-central: 2
* fx-team: 2

Platform breakdown:
* linux32: 34

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1292460&startday=2016-08-15&endday=2016-08-15&tree=all
Comment 10 User image David Baron :dbaron: ⌚️UTC-8 2016-08-15 18:19:45 PDT
Maybe worth seeing if only the test (order) changes in https://hg.mozilla.org/integration/mozilla-inbound/rev/b1dbce81bf3b6124577fb46414f811ec5f45f4e0 were enough to trigger this?
Comment 11 User image OrangeFactor Robot 2016-08-16 18:00:05 PDT
60 automation job failures were associated with this bug yesterday.

Repository breakdown:
* autoland: 30
* mozilla-inbound: 20
* fx-team: 6
* try: 4

Platform breakdown:
* linux32: 60

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1292460&startday=2016-08-16&endday=2016-08-16&tree=all
Comment 12 User image OrangeFactor Robot 2016-08-17 18:00:04 PDT
45 automation job failures were associated with this bug yesterday.

Repository breakdown:
* autoland: 17
* mozilla-inbound: 15
* fx-team: 6
* try: 4
* mozilla-central: 3

Platform breakdown:
* linux32: 45

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1292460&startday=2016-08-17&endday=2016-08-17&tree=all
Comment 13 User image OrangeFactor Robot 2016-08-18 18:00:04 PDT
23 automation job failures were associated with this bug yesterday.

Repository breakdown:
* autoland: 12
* fx-team: 5
* try: 3
* mozilla-central: 3

Platform breakdown:
* linux32: 23

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1292460&startday=2016-08-18&endday=2016-08-18&tree=all
Comment 14 User image OrangeFactor Robot 2016-08-21 18:06:50 PDT
196 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* autoland: 93
* mozilla-inbound: 46
* fx-team: 27
* try: 17
* mozilla-central: 13

Platform breakdown:
* linux32: 196

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1292460&startday=2016-08-15&endday=2016-08-21&tree=all
Comment 15 User image Phil Ringnalda (:philor) 2016-08-22 14:49:22 PDT
Sigh.

This bug is permaorange when either -2 or -3 is the first test to run in Ru3, not following -1.

When -1 runs in Ru3, bug 1289014 is intermittent.

The thing that started this was not Markus, or a layout-touching push in July, it was when we went from running Linux32 debug reftest-no-accel in 2 chunks to running it in 6 chunks, making it possible for the 613433 tests to run in a browser which hadn't previously run whatever previous test they depend on running after.
Comment 16 User image OrangeFactor Robot 2016-08-31 18:00:05 PDT
46 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 19
* autoland: 15
* mozilla-central: 5
* fx-team: 4
* try: 3

Platform breakdown:
* linux32: 46

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1292460&startday=2016-08-31&endday=2016-08-31&tree=all
Comment 17 User image Ting-Yu Lin [:TYLin] (UTC+8) 2016-09-01 00:45:32 PDT
I turn on the focus logging on try to what happens on Ru3.
https://treeherder.mozilla.org/logviewer.html#?job_id=26688681&repo=try#L2016-L2064

Since "613433-2.html" is the first test being run on this chunk, somehow the test file is loaded *before* the window "chrome://reftest/content/reftest.xul" is loaded, which makes the focus changing to the contenteditable failed. On the subsequent successful tests, the xul should be loaded before the test files.

To make the focus switching happens after xul is loaded by setTimeout 1000ms, the focus could be switched successfully, but this might not be a robust fix though.
https://treeherder.mozilla.org/#/jobs?repo=try&revision=d7b0ed18b771
Comment 18 User image David Baron :dbaron: ⌚️UTC-8 2016-09-01 00:52:33 PDT
Perhaps the reftest harness should be waiting longer before it starts running tests, rather than initiating everything in OnRefTestLoad?
Comment 19 User image OrangeFactor Robot 2016-09-01 18:00:04 PDT
49 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 20
* autoland: 16
* fx-team: 7
* mozilla-central: 6

Platform breakdown:
* linux32: 49

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1292460&startday=2016-09-01&endday=2016-09-01&tree=all
Comment 20 User image Markus Stange [:mstange] (away until Feb 22) 2016-09-02 14:06:02 PDT
I agree with comment 18. We need to have a focus listener somewhere. We probably don't need the full-blown SimpleTest.waitForFocus solution, though.
Comment 21 User image OrangeFactor Robot 2016-09-02 18:00:05 PDT
28 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 15
* mozilla-central: 4
* autoland: 4
* fx-team: 3
* try: 2

Platform breakdown:
* linux32: 28

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1292460&startday=2016-09-02&endday=2016-09-02&tree=all
Comment 22 User image OrangeFactor Robot 2016-09-03 18:00:03 PDT
19 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 9
* autoland: 6
* mozilla-central: 2
* fx-team: 2

Platform breakdown:
* linux32: 19

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1292460&startday=2016-09-03&endday=2016-09-03&tree=all
Comment 23 User image OrangeFactor Robot 2016-09-04 18:07:32 PDT
184 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 83
* autoland: 50
* mozilla-central: 24
* fx-team: 20
* try: 7

Platform breakdown:
* linux32: 184

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1292460&startday=2016-08-29&endday=2016-09-04&tree=all
Comment 24 User image Mats Palmgren (:mats) 2016-09-04 20:37:05 PDT
A two second delay before starting the first test seems to fix it:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=4fe52514712b
Comment 25 User image Mats Palmgren (:mats) 2016-09-04 20:40:05 PDT
Perhaps we should we take that ^ as a wallpaper for now?
Comment 26 User image OrangeFactor Robot 2016-09-05 18:00:03 PDT
23 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 9
* autoland: 8
* try: 2
* mozilla-central: 2
* fx-team: 2

Platform breakdown:
* linux32: 23

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1292460&startday=2016-09-05&endday=2016-09-05&tree=all
Comment 27 User image OrangeFactor Robot 2016-09-06 18:00:05 PDT
31 automation job failures were associated with this bug yesterday.

Repository breakdown:
* autoland: 11
* try: 6
* mozilla-inbound: 5
* fx-team: 5
* mozilla-central: 4

Platform breakdown:
* linux32: 31

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1292460&startday=2016-09-06&endday=2016-09-06&tree=all
Comment 28 User image OrangeFactor Robot 2016-09-11 18:05:25 PDT
58 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* autoland: 20
* mozilla-inbound: 14
* try: 11
* fx-team: 7
* mozilla-central: 6

Platform breakdown:
* linux32: 58

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1292460&startday=2016-09-05&endday=2016-09-11&tree=all
Comment 29 User image David Baron :dbaron: ⌚️UTC-8 2016-09-12 16:00:14 PDT
Shouldn't it be simple enough to just poll for focus, and avoid having to add a timeout that might not be quite reliable?
Comment 30 User image Phil Ringnalda (:philor) 2016-09-18 19:02:09 PDT
*** Bug 1289014 has been marked as a duplicate of this bug. ***
Comment 31 User image Phil Ringnalda (:philor) 2016-09-18 19:12:36 PDT
Our number one single-test failure, so I'll let you all decide what sort of hack or perfect fix you want to give it, with an accompanying patch to start running the tests on Linux32 again.
Comment 32 User image Pulsebot 2016-09-18 19:13:14 PDT
Pushed by philringnalda@gmail.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/785f1dbb4900
Disable 613433-1.html,613433-2.html,613433-3.html on Linux32 for needing focus which they don't get when they are the first test to run in a chunk
Comment 33 User image Carsten Book [:Tomcat] 2016-09-19 03:16:53 PDT
https://hg.mozilla.org/mozilla-central/rev/785f1dbb4900
Comment 34 User image OrangeFactor Robot 2016-09-25 18:04:21 PDT
6 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* autoland: 3
* mozilla-central: 2
* fx-team: 1

Platform breakdown:
* linux32: 6

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1292460&startday=2016-09-19&endday=2016-09-25&tree=all
Comment 35 User image Ryan VanderMeulen [:RyanVM] 2016-12-29 09:00:38 PST
Seems that this bug was making progress and then stalled out after the tests got disabled? Is anybody owning the harness fixing and re-enabling of these tests?
Comment 36 User image Jet Villegas (:jet) 2017-01-23 14:01:00 PST
(In reply to David Baron :dbaron: ⌚️UTC-8 from comment #29)
> Shouldn't it be simple enough to just poll for focus, and avoid having to
> add a timeout that might not be quite reliable?

I got a green Try run by adding a focus() handler:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=5425671e37bd00e8dfa2053e717dac512a081815

Mats: can you help land this one, if it looks good to you? Thx!
Comment 37 User image David Baron :dbaron: ⌚️UTC-8 2017-01-23 14:10:13 PST
Created attachment 8829647 [details] [diff] [review]
jet's patch to wait for focus before starting tests

from https://hg.mozilla.org/try/raw-rev/5425671e37bd00e8dfa2053e717dac512a081815
Comment 38 User image Mats Palmgren (:mats) 2017-01-23 16:04:42 PST
Comment on attachment 8829647 [details] [diff] [review]
jet's patch to wait for focus before starting tests

Looks good to me, fwiw.  One potential issue might be that 'gBrowser'
already has focus so our listener won't be called.  Probably worth
checking that by doing a Try run on all platforms.
Comment 39 User image Jet Villegas (:jet) 2017-01-23 17:41:44 PST
(In reply to Mats Palmgren (:mats) from comment #38)
> Probably worth checking that by doing a Try run on all platforms.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=035862cbbe8c78f27f6390705b9a79906baee41c
Comment 40 User image David Baron :dbaron: ⌚️UTC-8 2017-01-24 15:23:24 PST
Comment on attachment 8829647 [details] [diff] [review]
jet's patch to wait for focus before starting tests

OK, I'd suggest as a commit message:

Bug 1292460 - Focus the reftest browser before starting tests, except when filtering out needs-focus tests.
Comment 41 User image Jet Villegas (:jet) 2017-01-26 02:06:23 PST
Mats will take this one over the finish line. Thanks, All!
Comment 42 User image Pulsebot 2017-01-26 15:24:33 PST
Pushed by mpalmgren@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/d0d4bfd4c073
Focus the reftest browser before starting tests, except when filtering out needs-focus tests.  r=dbaron
Comment 43 User image Carsten Book [:Tomcat] 2017-01-27 01:56:30 PST
https://hg.mozilla.org/mozilla-central/rev/d0d4bfd4c073
Comment 44 User image Ryan VanderMeulen [:RyanVM] 2017-01-27 13:19:16 PST
https://hg.mozilla.org/releases/mozilla-aurora/rev/8575918c999c
Comment 45 User image Ryan VanderMeulen [:RyanVM] 2017-01-27 13:39:57 PST
https://hg.mozilla.org/releases/mozilla-beta/rev/160c08a8699e
Comment 46 User image Ryan VanderMeulen [:RyanVM] 2017-01-30 10:40:38 PST
For reason, this appears to have stuck on trunk just fine, but both Aurora and Beta started getting frequent startup hangs (presumably unable to get focus in the right place) after I uplifted it there.
https://treeherder.mozilla.org/logviewer.html#?job_id=73014965&repo=mozilla-beta

Anyway, I've backed it out from Beta and will do so from Aurora as well.
https://hg.mozilla.org/releases/mozilla-beta/rev/cfe1b0427178
Comment 47 User image Phil Ringnalda (:philor) 2017-01-30 11:31:06 PST
Curiously enough, on the trunk instead of lots of Linux reftest/crashtest startup hangs showing the "not the default browser" dialog, we're getting just a smattering of Win8 reftest startup hangs showing the Start screen.
Comment 48 User image Mats Palmgren (:mats) 2017-01-30 11:54:02 PST
Hmm, that seems odd given that the testing profile appears to disable that check:
https://dxr.mozilla.org/mozilla-central/rev/71224049c0b52ab190564d3ea0eab089a159a4cf/testing/profiles/prefs_general.js#24
Maybe there's an actual bug there - either that check isn't waiting for prefs to be read,
or the prefs are not read properly in some cases, or the pref was renamed or something.
Comment 49 User image Ryan VanderMeulen [:RyanVM] 2017-01-30 11:55:47 PST
Given how soon after the merge day this landed, I'm a bit worried that 54 will start hitting these failures too when it goes to Aurora.

Note You need to log in before you can comment on or make changes to this bug.