Permafailing android 315920-12c.html,315920-13a.html,315920-8a.html,315920-8b.html,315920-7c.html | image comparison, max difference: 229, number of differing pixels: 23,45

RESOLVED FIXED in Firefox 54

Status

()

RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: intermittent-bug-filer, Unassigned)

Tracking

(Depends on: 1 bug, {intermittent-failure})

unspecified
mozilla54
intermittent-failure
Points:
---

Firefox Tracking Flags

(firefox52 unaffected, firefox-esr52 unaffected, firefox53 unaffected, firefox54 fixed)

Details

(Whiteboard: [stockwell disabled])

Attachments

(1 attachment)

The merge from autoland to m-c (which had this test still passing): 
https://hg.mozilla.org/mozilla-central/pushloghtml?changeset=27dade5e0c8350189eeb6495d70a9fb25ce137a9

The merge from inbound to m-c (which had the test start permafailing):
https://hg.mozilla.org/mozilla-central/pushloghtml?changeset=c02dd6a7e9c193b488271eb53e3ea039042c9ed6
Looks like it expects the first checkbox to be checked, but it isn't.
61 failures in 146 pushes (0.418 failures/push) were associated with this bug yesterday.  
Repository breakdown:
* mozilla-inbound: 26
* autoland: 24
* mozilla-central: 9
* graphics: 2

Platform breakdown:
* android-4-3-armv7-api15: 61

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1342240&startday=2017-02-23&endday=2017-02-23&tree=all
:bz -- You worked on this test, long ago. Any idea what's gone wrong?
Flags: needinfo?(bzbarsky)
Well, the obvious fail is that the checkmark in the checkbox is not drawing.

The actual checked state is correct, since the text is green.  It's just the drawing of the checkmark that's broken.  And it's not broken in general (paints in the reference)....

So presumably some sort of invalidation bug.  But nothing in the pushlog jumps out at me.

This was green on both parents of the merge but broken on the merge?  :(
Flags: needinfo?(bzbarsky)
I mean nothing in the "inbound merge to m-c" pushlog.  On the autoland side, maybe bug 1026804 messes up something in the test harness?  May be worth doing a try with that turned back off.  Past that no sure.
(In reply to Boris Zbarsky [:bz] (still a bit busy) from comment #6)
> maybe bug 1026804 messes up something in the test harness?  May be worth
> doing a try with that turned back off.  Past that no sure.

I'll check on that - thanks.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=b528fced62ea37341048d1dabcae1a858948f585
Another possibility: In all of the failed cases, 315920-12c runs as the first test in the chunk; in all of the recent tests that passed before the merge, 315920-12c ran as the second test in the chunk, after 315920-12b.

OK:

REFTEST SUITE-START | Running 14321 tests
REFTEST INFO | Running chunk 15 out of 48 chunks.  tests 4585-4883/299
REFTEST TEST-START | http://10.0.2.2:8854/tests/layout/reftests/bugs/315920-12b.html == http://10.0.2.2:8854/tests/layout/reftests/bugs/315920-12-ref.html

Fail:

REFTEST SUITE-START | Running 14323 tests
REFTEST INFO | Running chunk 15 out of 48 chunks.  tests 4586-4884/299
REFTEST TEST-START | http://10.0.2.2:8854/tests/layout/reftests/bugs/315920-12c.html == http://10.0.2.2:8854/tests/layout/reftests/bugs/315920-12-ref.html
(In reply to Geoff Brown [:gbrown] from comment #7)
> (In reply to Boris Zbarsky [:bz] (still a bit busy) from comment #6)
> > maybe bug 1026804 messes up something in the test harness?  May be worth
> > doing a try with that turned back off.  Past that no sure.
> 
> I'll check on that - thanks.
> 
> https://treeherder.mozilla.org/#/
> jobs?repo=try&revision=b528fced62ea37341048d1dabcae1a858948f585

No, it's not bug 1026804.
136 failures in 182 pushes (0.747 failures/push) were associated with this bug yesterday.  
Repository breakdown:
* autoland: 78
* mozilla-inbound: 49
* mozilla-central: 5
* graphics: 4

Platform breakdown:
* android-4-3-armv7-api15: 136

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1342240&startday=2017-02-24&endday=2017-02-24&tree=all
On Android, the checkmark is an SVG image.  Perhaps we're firing onload despite the image not having loaded (or maybe it loaded but hasn't gone through the whole drawing pipeline yet)?

https://searchfox.org/mozilla-central/rev/c9ccb8ec1f5acda78e6344ffb87aa3a409031e3d/layout/style/res/forms.css#643
Good news, adding a couple of reftests turned the crank, and it's no longer permaorange on Android opt.

Bad news, if, as that did, you put 315920-13a.html first in a chunk, it'll fail to get the checkmark loaded in time, and now Android debug's permaorange.
Summary: Permafailing android 315920-12c.html == 315920-12-ref.html | image comparison, max difference: 229, number of differing pixels: 23 → Permafailing android 315920-12c.html,315920-13a.html | image comparison, max difference: 229, number of differing pixels: 23
I got the same result on try: I removed some unrelated reftests to shift the tests so that 315920-12c no longer runs first, and all passed.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=209941037623eedefc0e2f281428ce22f78351db
Using MozReftestInvalidate should fix this.  See bug 1334944.
31 failures in 47 pushes (0.66 failures/push) were associated with this bug yesterday.  
Repository breakdown:
* mozilla-inbound: 14
* autoland: 10
* mozilla-central: 5
* try: 2

Platform breakdown:
* android-4-3-armv7-api15: 31

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1342240&startday=2017-02-25&endday=2017-02-25&tree=all
Yeah, running first in a chunk before anything in that session has had a chance to load the image could have this effect.  After that the image would be cached and loads would complete sync.... That at least explains why this could happen on a merge.
259 failures in 812 pushes (0.319 failures/push) were associated with this bug in the last 7 days. 

This is the #5 most frequent failure this week. 

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:
* autoland: 131
* mozilla-inbound: 97
* mozilla-central: 22
* graphics: 6
* try: 3

Platform breakdown:
* android-4-3-armv7-api15: 259

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1342240&startday=2017-02-20&endday=2017-02-26&tree=all
I saw this failure in 315920-8a.html on my try run:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=9661df4cb88d5e5266bedb934afc61be59e303d4&selectedJob=80356303
Because I added some reftests?

Updated

2 years ago
Summary: Permafailing android 315920-12c.html,315920-13a.html | image comparison, max difference: 229, number of differing pixels: 23 → Permafailing android 315920-12c.html,315920-13a.html,315920-8a.html | image comparison, max difference: 229, number of differing pixels: 23,45
93 failures in 125 pushes (0.744 failures/push) were associated with this bug yesterday.  
Repository breakdown:
* autoland: 59
* mozilla-inbound: 26
* graphics: 6
* mozilla-central: 2

Platform breakdown:
* android-4-3-armv7-api15: 92
* linux64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1342240&startday=2017-02-27&endday=2017-02-27&tree=all
(In reply to Mats Palmgren (:mats) from comment #14)
> Using MozReftestInvalidate should fix this.  See bug 1334944.

It still seems wrong to me that this is required.  Shouldn't the reftest harness be waiting for images to load (perhaps via shouldWaitForPendingPaints() in reftest-content.js calling to nsPresContext::IsDOMPaintEventPending)?
Whiteboard: [stockwell needswork]
60 failures in 87 pushes (0.69 failures/push) were associated with this bug yesterday.  
Repository breakdown:
* autoland: 28
* mozilla-inbound: 23
* mozilla-central: 3
* try: 2
* oak: 2
* graphics: 2

Platform breakdown:
* android-4-3-armv7-api15: 60

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1342240&startday=2017-02-28&endday=2017-02-28&tree=all
122 failures in 181 pushes (0.674 failures/push) were associated with this bug yesterday.  
Repository breakdown:
* mozilla-inbound: 61
* autoland: 46
* try: 7
* graphics: 4
* mozilla-central: 3
* oak: 1

Platform breakdown:
* android-4-3-armv7-api15: 122

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1342240&startday=2017-03-01&endday=2017-03-01&tree=all
Summary: Permafailing android 315920-12c.html,315920-13a.html,315920-8a.html | image comparison, max difference: 229, number of differing pixels: 23,45 → Permafailing android 315920-12c.html,315920-13a.html,315920-8a.html,315920-8b.html | image comparison, max difference: 229, number of differing pixels: 23,45
Duplicate of this bug: 1343808
Duplicate of this bug: 1343809
Duplicate of this bug: 1343811
Summary: Permafailing android 315920-12c.html,315920-13a.html,315920-8a.html,315920-8b.html | image comparison, max difference: 229, number of differing pixels: 23,45 → Permafailing android 315920-12c.html,315920-13a.html,315920-8a.html,315920-8b.html,315920-7c.html | image comparison, max difference: 229, number of differing pixels: 23,45
I assume the fix for this is in bug 1334944, I will push on that, with 552 failures in the last week (60%+ of the time), this is a bit more than a nuisance.  If we cannot get bug 1334944 assigned and in progress by end of day tomorrow, I will disable all 315920 bugs until we have time.
106 failures in 149 pushes (0.711 failures/push) were associated with this bug yesterday.  
Repository breakdown:
* mozilla-inbound: 55
* autoland: 35
* graphics: 6
* mozilla-central: 5
* try: 4
* oak: 1

Platform breakdown:
* android-4-3-armv7-api15: 106

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1342240&startday=2017-03-02&endday=2017-03-02&tree=all
Created attachment 8843438 [details] [diff] [review]
add fuzzy-if to frequently failing tests.

this is sort of 'disabling' the tests until bug 1334944 can get completion.
Attachment #8843438 - Flags: review?(gbrown)
Comment on attachment 8843438 [details] [diff] [review]
add fuzzy-if to frequently failing tests.

Review of attachment 8843438 [details] [diff] [review]:
-----------------------------------------------------------------

Maybe add a comment pointing back to this bug.
Attachment #8843438 - Flags: review?(gbrown) → review+

Comment 30

2 years ago
Pushed by jmaher@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/1416b28a89c5
Permafailing android 315920*, add fuzzy-if. r=gbrown
Whiteboard: [stockwell needswork] → [stockwell disabled]
101 failures in 157 pushes (0.643 failures/push) were associated with this bug yesterday.  
Repository breakdown:
* autoland: 64
* mozilla-inbound: 30
* try: 3
* mozilla-central: 3
* graphics: 1

Platform breakdown:
* android-4-3-armv7-api15: 101

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1342240&startday=2017-03-03&endday=2017-03-03&tree=all
19 failures in 45 pushes (0.422 failures/push) were associated with this bug yesterday.  
Repository breakdown:
* autoland: 16
* mozilla-central: 3

Platform breakdown:
* android-4-3-armv7-api15: 19

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1342240&startday=2017-03-04&endday=2017-03-04&tree=all
546 failures in 783 pushes (0.697 failures/push) were associated with this bug in the last 7 days. 

This is the #1 most frequent failure this week. 

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:
* autoland: 267
* mozilla-inbound: 210
* mozilla-central: 23
* graphics: 19
* try: 18
* oak: 9

Platform breakdown:
* android-4-3-armv7-api15: 545
* linux64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1342240&startday=2017-02-27&endday=2017-03-05&tree=all

Comment 34

2 years ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/1416b28a89c5
Status: NEW → RESOLVED
Last Resolved: 2 years ago
status-firefox54: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla54
21 failures in 113 pushes (0.186 failures/push) were associated with this bug yesterday.  
Repository breakdown:
* autoland: 17
* oak: 2
* mozilla-central: 1
* graphics: 1

Platform breakdown:
* android-4-3-armv7-api15: 21

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1342240&startday=2017-03-06&endday=2017-03-06&tree=all
status-firefox52: --- → unaffected
status-firefox53: --- → unaffected
status-firefox-esr52: --- → unaffected
21 failures in 790 pushes (0.027 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 17
* oak: 2
* mozilla-central: 1
* graphics: 1

Platform breakdown:
* android-4-3-armv7-api15: 21

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1342240&startday=2017-03-06&endday=2017-03-12&tree=all
This is hitting -9 on Aurora today. I'll fuzzy-if it away like we did on trunk in comment 34 I guess.
https://treeherder.mozilla.org/logviewer.html#?job_id=87311362&repo=mozilla-aurora
Fuzz harder on Aurora:
https://hg.mozilla.org/releases/mozilla-aurora/rev/a55f31cc0fe1

I suppose I could land this on trunk too, but I guess I'll hold out hope that we'll fix this properly somewhere else before we end up having to.
You need to log in before you can comment on or make changes to this bug.