1369253 - Intermittent dom/base/test/test_intersectionobserver_no_root.html | undefined assertion name - got "0,0,0,0,0,0,1280,915", expected true

Status: RESOLVED WONTFIX

(Core :: Layout, defect)

Description

[Treeherder Bug Filer]

Filed by: rvandermeulen [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=103559053&repo=mozilla-inbound

https://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-inbound-win32/1496273912/mozilla-inbound_win7_vm_test-mochitest-1-bm138-tests1-windows-build126.txt.gz

Comment 1

[Ryan VanderMeulen [:RyanVM]]

Tobias, can you please take a look?

Comment 2

[Tobias Schneider [:tobytailor]]

On it.

Comment 3

[Intermittent Failures Robot]

44 failures in 864 pushes (0.051 failures/push) were associated with this bug in the last 7 days. 

This is the #34 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. ** 

Repository breakdown:
* autoland: 21
* mozilla-inbound: 19
* mozilla-central: 4

Platform breakdown:
* windows7-32-vm: 40
* windows8-64: 2
* osx-10-10: 1
* linux32: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1369253&startday=2017-06-05&endday=2017-06-11&tree=all

Comment 4

[Intermittent Failures Robot]

16 failures in 150 pushes (0.107 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* mozilla-inbound: 6
* mozilla-beta: 4
* autoland: 4
* mozilla-central: 2

Platform breakdown:
* windows7-32-vm: 8
* windows8-64: 5
* windows7-32-vm-devedition: 1
* osx-10-10-devedition: 1
* osx-10-10: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1369253&startday=2017-06-13&endday=2017-06-13&tree=all

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

we have 73 failures in the last week, 44 the week prior, this is primarily on win/opt non-e10s tests.

:tobytailer, I see a couple weeks ago you started looking into this- do you need more information?  With the increased failures I would like to see a resolution this week (fix, reduction, disabling)

Comment 6

[Tobias Schneider [:tobytailor]]

I'm gonna focus on this to have a solution this week.

Comment 7

[Intermittent Failures Robot]

71 failures in 814 pushes (0.087 failures/push) were associated with this bug in the last 7 days. 

This is the #26 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. ** 

Repository breakdown:
* mozilla-inbound: 25
* autoland: 24
* mozilla-beta: 10
* mozilla-central: 8
* try: 4

Platform breakdown:
* windows7-32-vm: 48
* windows8-64: 12
* osx-10-10: 7
* windows8-64-devedition: 1
* windows7-32-vm-devedition: 1
* osx-10-10-devedition: 1
* linux64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1369253&startday=2017-06-12&endday=2017-06-18&tree=all

Comment 8

[Tobias Schneider [:tobytailor]]

Attachment: Gather more information when test fails

I wasn't able to reproduce the failures locally but I think I'm having a good idea why it fails. I would love to land a slightly changed test first and leave it in the three for a short while to gather a bit more information the next time it fails. I will provide a final fix once that additional information confirms my idea.

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/f5ad2f825cc5
Gather more information when test fails. r=jmaher

Comment 10

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/f5ad2f825cc5

Comment 11

[Tobias Schneider [:tobytailor]]

Attachment: Make sure window is fully loaded before testing intersection

Under some circumstances we test for intersection when the window was not fully loaded. Make sure we wait for onload event.

Comment 12

[Tobias Schneider [:tobytailor]]

Try run: https://treeherder.mozilla.org/#/jobs?repo=try&revision=8ca803bbd941d9152860bbdcd8a5f3adc3df0bf0&selectedJob=109622396

Comment 13

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8880862 [details] [diff] [review]
Make sure window is fully loaded before testing intersection

Review of attachment 8880862 [details] [diff] [review]:
-----------------------------------------------------------------

this looks good, thanks for fixing this!

Comment 14

[Tobias Schneider [:tobytailor]]

I tested this a bit more and it turned out the intermittent failures still occur. This is a pretty fragile test, it always was. This was our first test we had for testing exact root bounds used for intersection calculation. The reason we run this test in a separate window is to avoid it running in an iframe, as it might be the case when running via the mochitest harness. But this is also the reason why this test is so unstable, especially on slower machines. Meanwhile we have way better test coverage for this landed via https://bugzilla.mozilla.org/show_bug.cgi?id=1358666. Therefore I would like to disable this test as it will probably keep causing problems. Joel, any thoughts on this?

Comment 15

[Joel Maher ( :jmaher ) (UTC -8)]

given the fact that there is a newer test that covers the same things, this sounds like a duplicate test and I would have no worries about disabling it.

Comment 16

[Tobias Schneider [:tobytailor]]

Attachment: Disable test

As discussed, disabling this test for being redundant and highly unstable. We have create coverage for this test case landed via Bug 1358666.

Comment 17

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8881160 [details] [diff] [review]
Disable test

Review of attachment 8881160 [details] [diff] [review]:
-----------------------------------------------------------------

looks good!

Comment 18

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/9286413c8215
Disable redundant dom/base/test/test_intersectionobserver_no_root.html on all platforms. r=jmaher

Comment 19

[Intermittent Failures Robot]

62 failures in 892 pushes (0.07 failures/push) were associated with this bug in the last 7 days. 

This is the #31 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. ** 

Repository breakdown:
* autoland: 31
* mozilla-inbound: 19
* mozilla-central: 7
* try: 3
* mozilla-beta: 2

Platform breakdown:
* windows7-32-vm: 48
* osx-10-10: 5
* windows8-64: 4
* linux32: 2
* windows7-32-vm-devedition: 1
* linux64-jsdcov: 1
* linux64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1369253&startday=2017-06-19&endday=2017-06-25&tree=all

Comment 20

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/9286413c8215

Comment 21

[Intermittent Failures Robot]

6 failures in 718 pushes (0.008 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-beta: 4
* autoland: 2

Platform breakdown:
* windows7-32-vm: 2
* osx-10-10: 2
* windows7-32-vm-devedition: 1
* macosx64-devedition: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1369253&startday=2017-06-26&endday=2017-07-02&tree=all

Comment 22

[Intermittent Failures Robot]

9 failures in 656 pushes (0.014 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-beta: 9

Platform breakdown:
* windows7-32-vm-devedition: 5
* windows7-32-vm: 2
* osx-10-10: 1
* macosx64-devedition: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1369253&startday=2017-07-03&endday=2017-07-09&tree=all

Comment 23

[Tobias Schneider [:tobytailor]]

We disabled this test in Bug 1369253.

Comment 24

[Ryan VanderMeulen [:RyanVM]]

It sounds like we have no intention of fixing this test, so why was it disabled instead of being removed outright?

https://hg.mozilla.org/releases/mozilla-beta/rev/2c41c817d86a

Comment 25

[Tobias Schneider [:tobytailor]]

I will create a separate bug to remove this test entirely.

Comment 26

[Tobias Schneider [:tobytailor]]

See Bug 1381293.

Comment 27

[Intermittent Failures Robot]

2 failures in 720 pushes (0.003 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-beta: 2

Platform breakdown:
* windows7-32-vm-devedition: 1
* windows7-32-vm: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1369253&startday=2017-07-10&endday=2017-07-16&tree=all

Comment 28

[Firefox Bug Husbandry Bot]

Removing leave-open keyword from resolved bugs, per :sylvestre.