[32-bit Linux, maybe focus related] Permaorange or intermittent on Linux32 debug unaccelerated layout/reftests/bugs/613433-3.html,613433-2.html,613433-1.html | load failed: timed out waiting for reftest-wait to be removed

RESOLVED FIXED in Firefox 54

Status

()

Core
Layout
P1
normal
RESOLVED FIXED
10 months ago
4 months ago

People

(Reporter: Treeherder Bug Filer, Assigned: jet)

Tracking

({intermittent-failure, regression})

Trunk
mozilla54
intermittent-failure, regression
Points:
---
Bug Flags:
in-testsuite -

Firefox Tracking Flags

(firefox52 wontfix, firefox53 wontfix, firefox54 fixed)

Details

Attachments

(1 attachment)

(Reporter)

Description

10 months ago
treeherder
Filed by: tomcat [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=33374135&repo=mozilla-inbound

http://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-inbound-linux-debug/1470369322/mozilla-inbound_ubuntu32_vm-debug_test-reftest-no-accel-3-bm141-tests1-linux32-build70.txt.gz

https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=http://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-inbound-linux-debug/1470369322/mozilla-inbound_ubuntu32_vm-debug_test-reftest-no-accel-3-bm141-tests1-linux32-build70.txt.gz&only_show_unexpected=1

Comment 1

10 months ago
42 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 19
* autoland: 12
* fx-team: 6
* mozilla-central: 5

Platform breakdown:
* linux32: 42

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1292460&startday=2016-08-01&endday=2016-08-07&tree=all

Updated

10 months ago
Blocks: 1012752
Keywords: regression

Updated

10 months ago
Summary: Intermittent layout/reftests/bugs/613433-3.html | load failed: timed out waiting for reftest-wait to be removed → Permaorange on Linux32 debug unaccelerated layout/reftests/bugs/613433-3.html | load failed: timed out waiting for reftest-wait to be removed

Comment 2

10 months ago
49 automation job failures were associated with this bug yesterday.

Repository breakdown:
* autoland: 31
* mozilla-inbound: 12
* fx-team: 5
* mozilla-central: 1

Platform breakdown:
* linux32: 49

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1292460&startday=2016-08-08&endday=2016-08-08&tree=all

Comment 3

10 months ago
43 automation job failures were associated with this bug yesterday.

Repository breakdown:
* autoland: 24
* mozilla-inbound: 12
* fx-team: 7

Platform breakdown:
* linux32: 43

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1292460&startday=2016-08-09&endday=2016-08-09&tree=all

Comment 4

10 months ago
45 automation job failures were associated with this bug yesterday.

Repository breakdown:
* autoland: 20
* mozilla-inbound: 15
* fx-team: 6
* try: 2
* mozilla-central: 2

Platform breakdown:
* linux32: 45

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1292460&startday=2016-08-10&endday=2016-08-10&tree=all
Some real goodness going on here: something, perhaps the landing of bug 1000957, moved the permaorange from -3 to -2.
Summary: Permaorange on Linux32 debug unaccelerated layout/reftests/bugs/613433-3.html | load failed: timed out waiting for reftest-wait to be removed → Permaorange on Linux32 debug unaccelerated layout/reftests/bugs/613433-3.html,613433-2.html | load failed: timed out waiting for reftest-wait to be removed

Comment 6

10 months ago
51 automation job failures were associated with this bug yesterday.

Repository breakdown:
* autoland: 27
* mozilla-inbound: 14
* mozilla-central: 5
* fx-team: 4
* try: 1

Platform breakdown:
* linux32: 51

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1292460&startday=2016-08-11&endday=2016-08-11&tree=all

Comment 7

10 months ago
34 automation job failures were associated with this bug yesterday.

Repository breakdown:
* autoland: 15
* mozilla-inbound: 13
* fx-team: 5
* mozilla-central: 1

Platform breakdown:
* linux32: 34

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1292460&startday=2016-08-12&endday=2016-08-12&tree=all

Comment 8

9 months ago
238 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* autoland: 124
* mozilla-inbound: 67
* fx-team: 29
* mozilla-central: 12
* try: 6

Platform breakdown:
* linux32: 238

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1292460&startday=2016-08-08&endday=2016-08-14&tree=all

Comment 9

9 months ago
34 automation job failures were associated with this bug yesterday.

Repository breakdown:
* autoland: 19
* mozilla-inbound: 9
* try: 2
* mozilla-central: 2
* fx-team: 2

Platform breakdown:
* linux32: 34

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1292460&startday=2016-08-15&endday=2016-08-15&tree=all
Maybe worth seeing if only the test (order) changes in https://hg.mozilla.org/integration/mozilla-inbound/rev/b1dbce81bf3b6124577fb46414f811ec5f45f4e0 were enough to trigger this?
Flags: needinfo?(mstange)

Comment 11

9 months ago
60 automation job failures were associated with this bug yesterday.

Repository breakdown:
* autoland: 30
* mozilla-inbound: 20
* fx-team: 6
* try: 4

Platform breakdown:
* linux32: 60

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1292460&startday=2016-08-16&endday=2016-08-16&tree=all

Comment 12

9 months ago
45 automation job failures were associated with this bug yesterday.

Repository breakdown:
* autoland: 17
* mozilla-inbound: 15
* fx-team: 6
* try: 4
* mozilla-central: 3

Platform breakdown:
* linux32: 45

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1292460&startday=2016-08-17&endday=2016-08-17&tree=all

Comment 13

9 months ago
23 automation job failures were associated with this bug yesterday.

Repository breakdown:
* autoland: 12
* fx-team: 5
* try: 3
* mozilla-central: 3

Platform breakdown:
* linux32: 23

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1292460&startday=2016-08-18&endday=2016-08-18&tree=all

Comment 14

9 months ago
196 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* autoland: 93
* mozilla-inbound: 46
* fx-team: 27
* try: 17
* mozilla-central: 13

Platform breakdown:
* linux32: 196

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1292460&startday=2016-08-15&endday=2016-08-21&tree=all
Sigh.

This bug is permaorange when either -2 or -3 is the first test to run in Ru3, not following -1.

When -1 runs in Ru3, bug 1289014 is intermittent.

The thing that started this was not Markus, or a layout-touching push in July, it was when we went from running Linux32 debug reftest-no-accel in 2 chunks to running it in 6 chunks, making it possible for the 613433 tests to run in a browser which hadn't previously run whatever previous test they depend on running after.
Priority: -- → P1
See Also: → bug 1289014
Summary: Permaorange on Linux32 debug unaccelerated layout/reftests/bugs/613433-3.html,613433-2.html | load failed: timed out waiting for reftest-wait to be removed → [32-bit Linux, maybe focus related] Permaorange on Linux32 debug unaccelerated layout/reftests/bugs/613433-3.html,613433-2.html | load failed: timed out waiting for reftest-wait to be removed

Comment 16

9 months ago
46 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 19
* autoland: 15
* mozilla-central: 5
* fx-team: 4
* try: 3

Platform breakdown:
* linux32: 46

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1292460&startday=2016-08-31&endday=2016-08-31&tree=all
I turn on the focus logging on try to what happens on Ru3.
https://treeherder.mozilla.org/logviewer.html#?job_id=26688681&repo=try#L2016-L2064

Since "613433-2.html" is the first test being run on this chunk, somehow the test file is loaded *before* the window "chrome://reftest/content/reftest.xul" is loaded, which makes the focus changing to the contenteditable failed. On the subsequent successful tests, the xul should be loaded before the test files.

To make the focus switching happens after xul is loaded by setTimeout 1000ms, the focus could be switched successfully, but this might not be a robust fix though.
https://treeherder.mozilla.org/#/jobs?repo=try&revision=d7b0ed18b771
Perhaps the reftest harness should be waiting longer before it starts running tests, rather than initiating everything in OnRefTestLoad?

Comment 19

9 months ago
49 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 20
* autoland: 16
* fx-team: 7
* mozilla-central: 6

Platform breakdown:
* linux32: 49

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1292460&startday=2016-09-01&endday=2016-09-01&tree=all
I agree with comment 18. We need to have a focus listener somewhere. We probably don't need the full-blown SimpleTest.waitForFocus solution, though.
Flags: needinfo?(mstange)

Comment 21

9 months ago
28 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 15
* mozilla-central: 4
* autoland: 4
* fx-team: 3
* try: 2

Platform breakdown:
* linux32: 28

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1292460&startday=2016-09-02&endday=2016-09-02&tree=all

Comment 22

9 months ago
19 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 9
* autoland: 6
* mozilla-central: 2
* fx-team: 2

Platform breakdown:
* linux32: 19

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1292460&startday=2016-09-03&endday=2016-09-03&tree=all

Comment 23

9 months ago
184 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 83
* autoland: 50
* mozilla-central: 24
* fx-team: 20
* try: 7

Platform breakdown:
* linux32: 184

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1292460&startday=2016-08-29&endday=2016-09-04&tree=all
A two second delay before starting the first test seems to fix it:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=4fe52514712b
Perhaps we should we take that ^ as a wallpaper for now?
Flags: needinfo?(dbaron)

Comment 26

9 months ago
23 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 9
* autoland: 8
* try: 2
* mozilla-central: 2
* fx-team: 2

Platform breakdown:
* linux32: 23

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1292460&startday=2016-09-05&endday=2016-09-05&tree=all
No longer blocks: 1012752

Comment 27

9 months ago
31 automation job failures were associated with this bug yesterday.

Repository breakdown:
* autoland: 11
* try: 6
* mozilla-inbound: 5
* fx-team: 5
* mozilla-central: 4

Platform breakdown:
* linux32: 31

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1292460&startday=2016-09-06&endday=2016-09-06&tree=all

Comment 28

9 months ago
58 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* autoland: 20
* mozilla-inbound: 14
* try: 11
* fx-team: 7
* mozilla-central: 6

Platform breakdown:
* linux32: 58

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1292460&startday=2016-09-05&endday=2016-09-11&tree=all
Shouldn't it be simple enough to just poll for focus, and avoid having to add a timeout that might not be quite reliable?
Flags: needinfo?(dbaron)
Summary: [32-bit Linux, maybe focus related] Permaorange on Linux32 debug unaccelerated layout/reftests/bugs/613433-3.html,613433-2.html | load failed: timed out waiting for reftest-wait to be removed → [32-bit Linux, maybe focus related] Permaorange or intermittent on Linux32 debug unaccelerated layout/reftests/bugs/613433-3.html,613433-2.html,613433-1.html | load failed: timed out waiting for reftest-wait to be removed
Duplicate of this bug: 1289014
Our number one single-test failure, so I'll let you all decide what sort of hack or perfect fix you want to give it, with an accompanying patch to start running the tests on Linux32 again.
Keywords: leave-open
Whiteboard: [test disabled]

Comment 32

8 months ago
Pushed by philringnalda@gmail.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/785f1dbb4900
Disable 613433-1.html,613433-2.html,613433-3.html on Linux32 for needing focus which they don't get when they are the first test to run in a chunk

Comment 33

8 months ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/785f1dbb4900
Keywords: leave-open

Comment 34

8 months ago
6 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* autoland: 3
* mozilla-central: 2
* fx-team: 1

Platform breakdown:
* linux32: 6

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1292460&startday=2016-09-19&endday=2016-09-25&tree=all
Seems that this bug was making progress and then stalled out after the tests got disabled? Is anybody owning the harness fixing and re-enabling of these tests?
Flags: needinfo?(bugs)
Version: unspecified → Trunk
(Assignee)

Comment 36

4 months ago
(In reply to David Baron :dbaron: ⌚️UTC-8 from comment #29)
> Shouldn't it be simple enough to just poll for focus, and avoid having to
> add a timeout that might not be quite reliable?

I got a green Try run by adding a focus() handler:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=5425671e37bd00e8dfa2053e717dac512a081815

Mats: can you help land this one, if it looks good to you? Thx!
Flags: needinfo?(bugs) → needinfo?(mats)
Created attachment 8829647 [details] [diff] [review]
jet's patch to wait for focus before starting tests

from https://hg.mozilla.org/try/raw-rev/5425671e37bd00e8dfa2053e717dac512a081815
Comment on attachment 8829647 [details] [diff] [review]
jet's patch to wait for focus before starting tests

Looks good to me, fwiw.  One potential issue might be that 'gBrowser'
already has focus so our listener won't be called.  Probably worth
checking that by doing a Try run on all platforms.
Flags: needinfo?(mats)
Attachment #8829647 - Flags: review?(dbaron)
Attachment #8829647 - Flags: feedback+
(Assignee)

Comment 39

4 months ago
(In reply to Mats Palmgren (:mats) from comment #38)
> Probably worth checking that by doing a Try run on all platforms.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=035862cbbe8c78f27f6390705b9a79906baee41c
Comment on attachment 8829647 [details] [diff] [review]
jet's patch to wait for focus before starting tests

OK, I'd suggest as a commit message:

Bug 1292460 - Focus the reftest browser before starting tests, except when filtering out needs-focus tests.
Attachment #8829647 - Flags: review?(dbaron) → review+
(Assignee)

Comment 41

4 months ago
Mats will take this one over the finish line. Thanks, All!
Assignee: nobody → mats

Comment 42

4 months ago
Pushed by mpalmgren@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/d0d4bfd4c073
Focus the reftest browser before starting tests, except when filtering out needs-focus tests.  r=dbaron

Updated

4 months ago
Assignee: mats → bugs
Flags: in-testsuite-

Comment 43

4 months ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/d0d4bfd4c073
Status: NEW → RESOLVED
Last Resolved: 4 months ago
status-firefox54: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla54
status-firefox52: --- → affected
status-firefox53: --- → affected
Whiteboard: [test disabled]

Comment 44

4 months ago
bugherderuplift
https://hg.mozilla.org/releases/mozilla-aurora/rev/8575918c999c
status-firefox53: affected → fixed

Comment 45

4 months ago
bugherderuplift
https://hg.mozilla.org/releases/mozilla-beta/rev/160c08a8699e
status-firefox52: affected → fixed
For reason, this appears to have stuck on trunk just fine, but both Aurora and Beta started getting frequent startup hangs (presumably unable to get focus in the right place) after I uplifted it there.
https://treeherder.mozilla.org/logviewer.html#?job_id=73014965&repo=mozilla-beta

Anyway, I've backed it out from Beta and will do so from Aurora as well.
https://hg.mozilla.org/releases/mozilla-beta/rev/cfe1b0427178
status-firefox52: fixed → wontfix
status-firefox53: fixed → wontfix
Curiously enough, on the trunk instead of lots of Linux reftest/crashtest startup hangs showing the "not the default browser" dialog, we're getting just a smattering of Win8 reftest startup hangs showing the Start screen.
Hmm, that seems odd given that the testing profile appears to disable that check:
https://dxr.mozilla.org/mozilla-central/rev/71224049c0b52ab190564d3ea0eab089a159a4cf/testing/profiles/prefs_general.js#24
Maybe there's an actual bug there - either that check isn't waiting for prefs to be read,
or the prefs are not read properly in some cases, or the pref was renamed or something.
Given how soon after the merge day this landed, I'm a bit worried that 54 will start hitting these failures too when it goes to Aurora.
You need to log in before you can comment on or make changes to this bug.