1411490 - [Win7] Intermittent editor/reftests/xul/number-3.xul == editor/reftests/xul/number-ref.xul | image comparison, max difference: 255, number of differing pixels: 56

Status: RESOLVED FIXED

(Core :: Layout, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: philringnalda [at] gmail.com

https://treeherder.mozilla.org/logviewer.html#?job_id=139365134&repo=autoland

https://queue.taskcluster.net/v1/task/NWXfadFHRTSwZE6dx3Banw/runs/0/artifacts/public/logs/live_backing.log

https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://queue.taskcluster.net/v1/task/NWXfadFHRTSwZE6dx3Banw/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1

Comment 1

[Intermittent Failures Robot]

20 failures in 912 pushes (0.022 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 12
* mozilla-inbound: 5
* mozilla-central: 2
* mozilla-beta: 1

Platform breakdown:
* windows7-32: 19
* windows7-32-devedition: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1411490&startday=2017-10-23&endday=2017-10-29&tree=all

Comment 2

[Intermittent Failures Robot]

6 failures in 857 pushes (0.007 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 3
* mozilla-inbound: 2
* mozilla-central: 1

Platform breakdown:
* windows7-32: 6

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1411490&startday=2017-10-30&endday=2017-11-05&tree=all

Comment 3

[Stefan Hindli [:stefan_hindli]]

There are 32 failures over the last 7 days.
Most of them occur on Windows 7 (pgo, debug, opt) and windows7-32-stylo-disabled (opt)

Here is a recent log example: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-central&job_id=144044657&lineNumber=2742

:jet can you please take a look?

Thank you!

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

here is the reftest analyzer:
https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://queue.taskcluster.net/v1/task/DoXHEW1OS2Snh_Qqz5Nj6w/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1

looking at it in detail, I see there is an input field with a 0 in it- the 0 is shifted in position slightly

Comment 5

[Intermittent Failures Robot]

35 failures in 849 pushes (0.041 failures/push) were associated with this bug in the last 7 days.   

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **  

Repository breakdown:
* autoland: 15
* mozilla-inbound: 12
* mozilla-central: 3
* mozilla-beta: 3
* mozilla-release: 2

Platform breakdown:
* windows7-32: 34
* windows7-32-stylo-disabled: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1411490&startday=2017-11-06&endday=2017-11-12&tree=all

Comment 6

[Jet Villegas (inactive)]

(In reply to Stefan Hindli [:stefan_hindli] from comment #3)
> There are 32 failures over the last 7 days.
> Most of them occur on Windows 7 (pgo, debug, opt) and
> windows7-32-stylo-disabled (opt)
> 
> Here is a recent log example:
> https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-
> central&job_id=144044657&lineNumber=2742
> 
> :jet can you please take a look?
> 
> Thank you!

This test should be a 100% fail on Windows:
https://searchfox.org/mozilla-central/source/editor/reftests/xul/reftest.list#21

Masayuki: should these turn into fails-if(winWidget)?
fails-if(windowsDefaultTheme&&/^Windows\x20NT\x20(5\.[12]|6\.[012]|10\.0)/.test(http.oscpu))

Comment 7

[Intermittent Failures Robot]

18 failures in 762 pushes (0.024 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 13
* mozilla-inbound: 4
* mozilla-beta: 1

Platform breakdown:
* windows7-32: 18

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1411490&startday=2017-11-13&endday=2017-11-19&tree=all

Comment 8

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

Attachment: Diff between result and reference

Sorry for the delay to reply.

The "0" is positioned differently. Could be sub-pixel positioning issue on Windows?

Comment 9

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

Attachment: Diff between result and reference

Oops, the previous image was tiff.

Comment 10

[Jonathan Kew [:jfkthame]]

The tests here are annotated with fails-if(windowsDefaultTheme...), and have been ever since they first landed in bug  but it appears from checking the reftest log that the windowsDefaultTheme flag is not always being set as expected.

When I look at the log from a passing run, I see that it has \"windowsDefaultTheme\":true, whereas the failing log has \"windowsDefaultTheme\":false. Do we really expect some of our reftest jobs to occasionally run under different theme conditions than others? Or is the problem that the windowsDefaultTheme flag is a bit unreliable?

I don't really know anything about how this works in the reftest harness, but bug 1050229 suggests Markus may be somewhat familiar with it.... passing needinfo? along to him.

Comment 11

[Intermittent Failures Robot]

41 failures in 744 pushes (0.055 failures/push) were associated with this bug in the last 7 days. 

This is the #22 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **  

Repository breakdown:
* mozilla-inbound: 19
* autoland: 16
* mozilla-central: 4
* mozilla-beta: 2

Platform breakdown:
* windows7-32: 40
* windows7-32-nightly: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1411490&startday=2017-11-20&endday=2017-11-26&tree=all

Comment 12

[Narcis Beleuzu [:NarcisB]]

In the last 7 days there are 31 failures. They occur on Windows 7. 
A recent log example: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-inbound&job_id=147429434&lineNumber=2727

:jet, can you please take a look?

Comment 13

[Jet Villegas (inactive)]

(In reply to Narcis Beleuzu [:NarcisB] from comment #12)
> In the last 7 days there are 31 failures. They occur on Windows 7. 
> A recent log example:
> https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-
> inbound&job_id=147429434&lineNumber=2727
> 
> :jet, can you please take a look?

Let's disable these for Win32 and move on.

Comment 14

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: disable the tests on windows

Comment 15

[Geoff Brown [:gbrown]]

Comment on attachment 8932875 [details] [diff] [review]
disable the tests on windows

Review of attachment 8932875 [details] [diff] [review]:
-----------------------------------------------------------------

I don't think this will be effective: I checked several recent failure logs, and all had windowsDefaultTheme=false.

I think we need to follow-up on comment 10 to better understand windowsDefaultTheme, or disable on all win32.

Comment 16

[Joel Maher ( :jmaher ) (UTC -8)]

ok, this is a great observations Geoff and I see that all failures are windowsDefaultTheme:False and passing instances are True.  We set this value here:
https://searchfox.org/mozilla-central/source/layout/tools/reftest/manifest.jsm#576

    sandbox.windowsDefaultTheme = g.containingWindow.matchMedia("(-moz-windows-default-theme)").matches;


while we don't run stylo-disabled, historically there has been only one instance of this failure on stylo-disabled which did have the theme- either way there is either a bug in:
1) the environment we are running in
2) how we query for moz-windows-default-theme


I see bug 1343049 and when these failures were greatly reduced a fix was landed (Nov 23rd).  This indicates that it is related to #1 and that #1 isn't fixed yet.

Comment 17

[Intermittent Failures Robot]

11 failures in 792 pushes (0.014 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 6
* autoland: 4
* mozilla-beta: 1

Platform breakdown:
* windows7-32: 11

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1411490&startday=2017-11-27&endday=2017-12-03&tree=all

Comment 18

[Tiberius Oros[:tiberius_oros]]

There have been 37 failures in the last week

This is occurring only on the Windows 7 platform.
As for the build type, I see 20 occurrences on opt, 16 on pgo and 1 on debug.

Here is a link from a recent log file: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-inbound&job_id=150953315&lineNumber=2761

And a link to the reftest-analyzer: https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://queue.taskcluster.net/v1/task/ST3H4OToSZu4EsO53LtxUA/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1

Please also take a look at Comment 16

Comment 19

[Intermittent Failures Robot]

39 failures in 889 pushes (0.044 failures/push) were associated with this bug in the last 7 days. 

This is the #22 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **  

Repository breakdown:
* autoland: 18
* mozilla-inbound: 14
* mozilla-central: 5
* try: 1
* mozilla-beta: 1

Platform breakdown:
* windows7-32: 39

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1411490&startday=2017-12-04&endday=2017-12-10&tree=all

Comment 20

[Jet Villegas (inactive)]

(In reply to Tiberius Oros[:tiberius_oros] from comment #18)
> Please also take a look at Comment 16

Comment 16 says that this bug requires that bug 1343049 gets fixed.

Comment 21

[Intermittent Failures Robot]

9 failures in 423 pushes (0.021 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 4
* autoland: 3
* try: 2

Platform breakdown:
* windows7-32: 9

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1411490&startday=2017-12-11&endday=2017-12-17&tree=all

Comment 22

[Intermittent Failures Robot]

15 failures in 170 pushes (0.088 failures/push) were associated with this bug yesterday.    

Repository breakdown:
* autoland: 9
* mozilla-inbound: 6

Platform breakdown:
* windows7-32: 15

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1411490&startday=2017-12-21&endday=2017-12-21&tree=all

Comment 23

[Dorel Luca [:dluca]]

Hi :Jet : Any update on this bug?

Comment 24

[Intermittent Failures Robot]

36 failures in 590 pushes (0.061 failures/push) were associated with this bug in the last 7 days. 

This is the #29 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **  

Repository breakdown:
* autoland: 22
* mozilla-inbound: 12
* mozilla-central: 2

Platform breakdown:
* windows7-32: 36

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1411490&startday=2017-12-18&endday=2017-12-24&tree=all

Comment 25

[Intermittent Failures Robot]

4 failures in 147 pushes (0.027 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 3
* mozilla-central: 1

Platform breakdown:
* windows7-32: 4

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1411490&startday=2017-12-25&endday=2017-12-31&tree=all

Comment 26

[Intermittent Failures Robot]

22 failures in 462 pushes (0.048 failures/push) were associated with this bug in the last 7 days. 

This is the #44 most frequent failure this week.   

Repository breakdown:
* autoland: 8
* mozilla-inbound: 5
* mozilla-central: 5
* try: 2
* mozilla-beta: 2

Platform breakdown:
* windows7-32: 20
* windows7-32-nightly: 1
* windows7-32-devedition: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1411490&startday=2018-01-01&endday=2018-01-07&tree=all

Comment 27

[Tiberius Oros[:tiberius_oros]]

Summary:
There have been 37 failures in the last week.
All of the failures are on Windows 7 and on pgo and opt build types.

:jet, please take a look at this after Bug 1343049 is fixed. Thank you.

Comment 28

[Jet Villegas (inactive)]

(In reply to Tiberius Oros[:tiberius_oros] from comment #27)
> :jet, please take a look at this after Bug 1343049 is fixed. Thank you.

I'm fairly certain that you will come find me if this bug still happens after bug 1343049 is fixed :)

Comment 29

[Intermittent Failures Robot]

45 failures in 788 pushes (0.057 failures/push) were associated with this bug in the last 7 days. 

This is the #31 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **  

Repository breakdown:
* autoland: 23
* mozilla-inbound: 15
* try: 4
* mozilla-central: 2
* mozilla-release: 1

Platform breakdown:
* windows7-32: 44
* windows7-32-nightly: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1411490&startday=2018-01-08&endday=2018-01-14&tree=all

Comment 30

[Eliza Balazs [:ebalazs_]]

We have 34 failures in the last 7 days.
They occur mostly on Windows7 (opt and pgo) and the rest o them on windows10-64(pgo), windows7-32-nightly (opt).

Reftest analyzer: https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://queue.taskcluster.net/v1/task/IF7-gHm8QZugEXd917b00w/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1

Recent log: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-central&job_id=155909359&lineNumber=8369

Waiting for bug 1343049 to be fixed.

Comment 31

[Intermittent Failures Robot]

29 failures in 657 pushes (0.044 failures/push) were associated with this bug in the last 7 days. 

This is the #34 most frequent failure this week.   

Repository breakdown:
* autoland: 15
* mozilla-inbound: 11
* mozilla-central: 2
* mozilla-release: 1

Platform breakdown:
* windows7-32: 27
* windows7-32-nightly: 1
* windows10-64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1411490&startday=2018-01-15&endday=2018-01-21&tree=all

Comment 32

[Eliza Balazs [:ebalazs_]]

In the last 7 days there are 34 failures.
They occur on Windows  and the affected builds type are pgo, opt and debug.

Reftest analyzer: https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://queue.taskcluster.net/v1/task/WZvPIwc7STScjKtFAoU0ZA/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1

Recent log: https://treeherder.mozilla.org/logviewer.html#?repo=autoland&job_id=158773406&lineNumber=2666

Still waiting for bug 1343049 to be fixed.

Comment 33

[Intermittent Failures Robot]

32 failures in 701 pushes (0.046 failures/push) were associated with this bug in the last 7 days. 

This is the #36 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **  

Repository breakdown:
* autoland: 21
* mozilla-inbound: 10
* mozilla-beta: 1

Platform breakdown:
* windows7-32: 31
* windows7-32-nightly: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1411490&startday=2018-01-22&endday=2018-01-28&tree=all

Comment 34

[Intermittent Failures Robot]

17 failures in 735 pushes (0.023 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 5
* mozilla-central: 5
* autoland: 5
* try: 1
* mozilla-beta: 1

Platform breakdown:
* windows7-32: 17

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1411490&startday=2018-01-29&endday=2018-02-04&tree=all

Comment 35

[Intermittent Failures Robot]

5 failures in 702 pushes (0.007 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 5

Platform breakdown:
* linux64: 3
* linux64-qr: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1411490&startday=2018-02-05&endday=2018-02-11&tree=all

Comment 36

[Jonathan Kew [:jfkthame]]

This was a Win7 incorrect-theme issue; but the failures noted in comment 35 are all from Linux. So they are actually something quite different, and should have been filed as a separate bug.

Comment 37

[Joel Maher ( :jmaher ) (UTC -8)]

this was either fixed by run-by-manifests or changes made to the image.