1374134 - Intermittent w3c-css/received/css-multicol-1/multicol-nested-column-rule-001.xht == w3c-css/received/css-multicol-1/multicol-nested-column-rule-001-ref.xht | image comparison, max difference: 255, number of differing pixels: 11455

Status: RESOLVED FIXED

(Core :: Layout, defect)

Description

[Treeherder Bug Filer]

Filed by: philringnalda [at] gmail.com

https://treeherder.mozilla.org/logviewer.html#?job_id=108035002&repo=mozilla-central

https://queue.taskcluster.net/v1/task/IKrJyd4CQJOlCKtXeC9HnQ/runs/0/artifacts/public/logs/live_backing.log

https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://queue.taskcluster.net/v1/task/IKrJyd4CQJOlCKtXeC9HnQ/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1

Comment 1

[Intermittent Failures Robot]

5 failures in 892 pushes (0.006 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* try: 2
* mozilla-inbound: 2
* mozilla-central: 1

Platform breakdown:
* windows7-32: 5

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1374134&startday=2017-06-19&endday=2017-06-25&tree=all

Comment 2

[Intermittent Failures Robot]

3 failures in 718 pushes (0.004 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 3

Platform breakdown:
* windows7-32: 3

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1374134&startday=2017-06-26&endday=2017-07-02&tree=all

Comment 3

[Cosmin Sabou [:CosminS]]

https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-searchStr=61e41981252c8d11b9a22a35bba0114ad4738c97&selectedJob=138708590

Comment 4

[Intermittent Failures Robot]

18 failures in 912 pushes (0.02 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 8
* mozilla-inbound: 5
* mozilla-central: 3
* try: 1
* mozilla-release: 1

Platform breakdown:
* windows7-32: 15
* windows7-32-stylo-disabled: 3

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1374134&startday=2017-10-23&endday=2017-10-29&tree=all

Comment 5

[Intermittent Failures Robot]

2 failures in 857 pushes (0.002 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-central: 1
* autoland: 1

Platform breakdown:
* windows7-32-stylo-disabled: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1374134&startday=2017-10-30&endday=2017-11-05&tree=all

Comment 6

[Eliza Balazs [:ebalazs_]]

There are 41 failures in the last 7 days.
They occur mostly on Windows-7-32 (enabled, disabled, nightly) and the affected build types are opt, debug, pgo.

Here is a recent log example: https://treeherder.mozilla.org/logviewer.html#?repo=autoland&job_id=144088154&lineNumber=2755

And a relevant part of the log where the test failed:

12:21:10     INFO -  REFTEST TEST-START | file:///Z:/task_1510488755/build/tests/reftest/tests/layout/reftests/w3c-css/received/css-multicol-1/multicol-nested-column-rule-001.xht == file:///Z:/task_1510488755/build/tests/reftest/tests/layout/reftests/w3c-css/received/css-multicol-1/multicol-nested-column-rule-001-ref.xht
2752
12:21:10     INFO -  REFTEST TEST-LOAD | file:///Z:/task_1510488755/build/tests/reftest/tests/layout/reftests/w3c-css/received/css-multicol-1/multicol-nested-column-rule-001.xht | 262 / 487 (53%)
2753
12:21:10     INFO -  REFTEST TEST-LOAD | file:///Z:/task_1510488755/build/tests/reftest/tests/layout/reftests/w3c-css/received/css-multicol-1/multicol-nested-column-rule-001-ref.xht | 262 / 487 (53%)
2754
12:21:10     INFO -  REFTEST INFO | REFTEST fuzzy test (0, 0) <= (255, 11455) <= (225, 13600)
2755
12:21:10    ERROR -  REFTEST TEST-UNEXPECTED-FAIL | file:///Z:/task_1510488755/build/tests/reftest/tests/layout/reftests/w3c-css/received/css-multicol-1/multicol-nested-column-rule-001.xht == file:///Z:/task_1510488755/build/tests/reftest/tests/layout/reftests/w3c-css/received/css-multicol-1/multicol-nested-column-rule-001-ref.xht | image comparison, max difference: 255, number of differing pixels: 11455

:jet, can you please take a look?

Comment 7

[Intermittent Failures Robot]

49 failures in 849 pushes (0.058 failures/push) were associated with this bug in the last 7 days. 

This is the #37 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **  

Repository breakdown:
* autoland: 20
* mozilla-inbound: 19
* mozilla-central: 6
* mozilla-beta: 2
* try: 1
* mozilla-release: 1

Platform breakdown:
* windows7-32: 37
* windows7-32-stylo-disabled: 9
* windows7-32-nightly: 3

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1374134&startday=2017-11-06&endday=2017-11-12&tree=all

Comment 8

[Jet Villegas (inactive)]

Neerja: these tests allow for a massive amount of fuzz. Do they actually test anything?

Comment 9

[Neerja Pancholi[:neerja]]

(In reply to Jet Villegas (:jet) from comment #8)
> Neerja: these tests allow for a massive amount of fuzz. Do they actually
> test anything?

I am going to hold off on answering this since the fix for Bug 616436 might mean we don't need this fuzzing anymore. 
I'm putting this bug in my queue to remind myself to get back to it after Bug 616436 is reviewed/resolved.

Comment 10

[Intermittent Failures Robot]

29 failures in 762 pushes (0.038 failures/push) were associated with this bug in the last 7 days. 

This is the #43 most frequent failure this week.   

Repository breakdown:
* autoland: 15
* mozilla-inbound: 9
* mozilla-central: 3
* mozilla-beta: 2

Platform breakdown:
* windows7-32: 23
* windows7-32-stylo-disabled: 6

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1374134&startday=2017-11-13&endday=2017-11-19&tree=all

Comment 11

[Tiberius Oros[:tiberius_oros]]

There have been 36 total failures in the last week.
 
Occurrences per platform:
 - 35 Windows 7 
 -  1 windows7-32-stylo-disabled
 
Occurrences per build type:
 - 15 opt
 - 13 pgo
 -  8 debug
  
Here is a relevant recent log: 
https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-inbound&job_id=147097810&lineNumber=2742
  
and a snippet from it:
  
12:21:03     INFO -  REFTEST TEST-START | file:///Z:/task_1511438848/build/tests/reftest/tests/layout/reftests/w3c-css/received/css-multicol-1/multicol-nested-column-rule-001.xht == file:///Z:/task_1511438848/build/tests/reftest/tests/layout/reftests/w3c-css/received/css-multicol-1/multicol-nested-column-rule-001-ref.xht
2739
12:21:03     INFO -  REFTEST TEST-LOAD | file:///Z:/task_1511438848/build/tests/reftest/tests/layout/reftests/w3c-css/received/css-multicol-1/multicol-nested-column-rule-001.xht | 259 / 487 (53%)
2740
12:21:03     INFO -  REFTEST TEST-LOAD | file:///Z:/task_1511438848/build/tests/reftest/tests/layout/reftests/w3c-css/received/css-multicol-1/multicol-nested-column-rule-001-ref.xht | 259 / 487 (53%)
2741
12:21:03     INFO -  REFTEST INFO | REFTEST fuzzy test (0, 0) <= (255, 11455) <= (225, 13600)
2742
12:21:03    ERROR -  REFTEST TEST-UNEXPECTED-FAIL | file:///Z:/task_1511438848/build/tests/reftest/tests/layout/reftests/w3c-css/received/css-multicol-1/multicol-nested-column-rule-001.xht == file:///Z:/task_1511438848/build/tests/reftest/tests/layout/reftests/w3c-css/received/css-multicol-1/multicol-nested-column-rule-001-ref.xht | image comparison, max difference: 255, number of differing pixels: 11455

:neerja, could you please take a look? Thank you.

Comment 12

[Intermittent Failures Robot]

38 failures in 744 pushes (0.051 failures/push) were associated with this bug in the last 7 days. 

This is the #28 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **  

Repository breakdown:
* autoland: 20
* mozilla-inbound: 11
* try: 3
* mozilla-central: 2
* mozilla-beta: 2

Platform breakdown:
* windows7-32: 36
* windows7-32-stylo-disabled: 1
* windows7-32-nightly: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1374134&startday=2017-11-20&endday=2017-11-26&tree=all

Comment 13

[Neerja Pancholi[:neerja]]

(In reply to Tiberius Oros[:tiberius_oros] from comment #11)
> :neerja, could you please take a look? Thank you.

Hi Tiberius,
I took a few minutes to look at this test and off hand I am not sure why the horizontal scroll bar appears intermittently. Although, per comment 9, I am working on related functionality for Bug 616436 and I think it would be wise to hold off on this until then. Hopefully, I can give you a more helpful answer next week.
Thanks!

Comment 14

[Intermittent Failures Robot]

2 failures in 792 pushes (0.003 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 1
* autoland: 1

Platform breakdown:
* windows7-32: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1374134&startday=2017-11-27&endday=2017-12-03&tree=all

Comment 15

[Intermittent Failures Robot]

28 failures in 889 pushes (0.031 failures/push) were associated with this bug in the last 7 days. 

This is the #41 most frequent failure this week.   

Repository breakdown:
* mozilla-inbound: 11
* autoland: 10
* mozilla-beta: 5
* try: 1
* mozilla-central: 1

Platform breakdown:
* windows7-32: 25
* windows7-32-devedition: 2
* windows7-32-nightly: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1374134&startday=2017-12-04&endday=2017-12-10&tree=all

Comment 16

[Intermittent Failures Robot]

16 failures in 423 pushes (0.038 failures/push) were associated with this bug in the last 7 days. 

This is the #50 most frequent failure this week.   

Repository breakdown:
* autoland: 6
* mozilla-inbound: 5
* try: 3
* mozilla-central: 2

Platform breakdown:
* windows7-32: 16

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1374134&startday=2017-12-11&endday=2017-12-17&tree=all

Comment 17

[Eliza Balazs [:ebalazs_]]

There are 32 failures in the last 7 days.
They occur on Windows 7 and windows7-32-nightly. The affected builds type are debug, opt and pgo.
A recent log example: https://treeherder.mozilla.org/logviewer.html#?repo=autoland&job_id=153087526&lineNumber=2921
04:21:35     INFO -  REFTEST TEST-START | file:///Z:/task_1513916195/build/tests/reftest/tests/layout/reftests/w3c-css/received/css-multicol-1/multicol-nested-column-rule-001.xht == file:///Z:/task_1513916195/build/tests/reftest/tests/layout/reftests/w3c-css/received/css-multicol-1/multicol-nested-column-rule-001-ref.xht
2918
04:21:35     INFO -  REFTEST TEST-LOAD | file:///Z:/task_1513916195/build/tests/reftest/tests/layout/reftests/w3c-css/received/css-multicol-1/multicol-nested-column-rule-001.xht | 287 / 489 (58%)
2919
04:21:35     INFO -  REFTEST TEST-LOAD | file:///Z:/task_1513916195/build/tests/reftest/tests/layout/reftests/w3c-css/received/css-multicol-1/multicol-nested-column-rule-001-ref.xht | 287 / 489 (58%)
2920
04:21:35     INFO -  REFTEST INFO | REFTEST fuzzy test (0, 0) <= (255, 11455) <= (225, 13600)
2921
04:21:35    ERROR -  REFTEST TEST-UNEXPECTED-FAIL | file:///Z:/task_1513916195/build/tests/reftest/tests/layout/reftests/w3c-css/received/css-multicol-1/multicol-nested-column-rule-001.xht == file:///Z:/task_1513916195/build/tests/reftest/tests/layout/reftests/w3c-css/received/css-multicol-1/multicol-nested-column-rule-001-ref.xht | image comparison, max difference: 255, number of differing pixels: 11455
2922
04:21:35     INFO -  REFTEST   IMAGE 1 (TEST): data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAyAAAAPoCAYAAAAmy5qxAAAXMElEQVR4nO3dW2pcyRZF0dN0NS17oqZkfckYuWTrsT1D2zkGLPRlcxFEhCcH6l53AACAyHX6fwAAAPA4BAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAMdc1+wA/hb3FcxxBIBjPOjAFu4rmOMIAMd40IEt3FcwxxEAjvGgA1u4r2COIwAc40EHtnBfwRxHADjGgw5s4b6COY4AcIwHHdjCfQVzHAHgGA86sIX7CuY4AsAxHnRgC/cVzHEEgGM86MAW7iuY4wgAx3jQgS3cVzDHEQC
:jet, can you please take at look at this?

Comment 18

[Intermittent Failures Robot]

44 failures in 590 pushes (0.075 failures/push) were associated with this bug in the last 7 days. 

This is the #19 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **  

Repository breakdown:
* autoland: 16
* mozilla-inbound: 11
* try: 7
* mozilla-central: 5
* mozilla-beta: 5

Platform breakdown:
* windows7-32: 38
* windows7-32-nightly: 5
* windows7-32-devedition: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1374134&startday=2017-12-18&endday=2017-12-24&tree=all

Comment 19

[Intermittent Failures Robot]

5 failures in 147 pushes (0.034 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 2
* autoland: 2
* mozilla-central: 1

Platform breakdown:
* windows7-32: 5

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1374134&startday=2017-12-25&endday=2017-12-31&tree=all

Comment 20

[Intermittent Failures Robot]

24 failures in 462 pushes (0.052 failures/push) were associated with this bug in the last 7 days. 

This is the #40 most frequent failure this week.   

Repository breakdown:
* autoland: 13
* mozilla-inbound: 6
* mozilla-central: 3
* try: 1
* mozilla-beta: 1

Platform breakdown:
* windows7-32: 23
* windows7-32-nightly: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1374134&startday=2018-01-01&endday=2018-01-07&tree=all

Comment 21

[Cristina Coroiu [:ccoroiu]]

There are 33 failures in the past 7 days.
For the failure pattern see Comment 17
A recent log example: https://treeherder.mozilla.org/logviewer.html#?repo=autoland&job_id=155425068&lineNumber=2935 
:jet, can you please take a look into this?

Comment 22

[Jet Villegas (inactive)]

(In reply to Cristina Coroiu [:ccoroiu] from comment #21)
> There are 33 failures in the past 7 days.
> For the failure pattern see Comment 17
> A recent log example:
> https://treeherder.mozilla.org/logviewer.
> html#?repo=autoland&job_id=155425068&lineNumber=2935 
> :jet, can you please take a look into this?

See comment 9.

Comment 23

[Neerja Pancholi[:neerja]]

Thanks to dholbert for pointing out that the test was wrong!
I have fixed the test and created a pull request on github. Since this is a CSSWG test, the fix must be made upstream in the Web-platform-tests repo.
Pull request -
https://github.com/w3c/web-platform-tests/pull/9021

Comment 24

[Intermittent Failures Robot]

27 failures in 788 pushes (0.034 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 16
* mozilla-inbound: 5
* mozilla-central: 4
* try: 1
* mozilla-beta: 1

Platform breakdown:
* windows7-32: 27

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1374134&startday=2018-01-08&endday=2018-01-14&tree=all

Comment 25

[Neerja Pancholi[:neerja]]

(In reply to Neerja Pancholi[:neerja] from comment #23)
> Thanks to dholbert for pointing out that the test was wrong!
> I have fixed the test and created a pull request on github. Since this is a
> CSSWG test, the fix must be made upstream in the Web-platform-tests repo.
> Pull request -
> https://github.com/w3c/web-platform-tests/pull/9021

This pull request was approved and so the fixed test is now on w3c/web-platform-tests. The right way to update layout tests from this repo is to use import-tests.py script that bulk imports updates. So, I have created a helper Bug 1430939 to fix a bug in this script and then to update the latest version of all existing tests. Once the patches for Bug 1430939 are checked in, this bug will be fixed. (Added a bug dependency accordingly.)

Comment 26

[Intermittent Failures Robot]

19 failures in 657 pushes (0.029 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 9
* mozilla-central: 5
* autoland: 4
* mozilla-release: 1

Platform breakdown:
* windows7-32: 19

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1374134&startday=2018-01-15&endday=2018-01-21&tree=all

Comment 27

[Neerja Pancholi[:neerja]]

Its been a week since the fix for Bug 1430939 landed and since there are no new reports of this failure happening since then, I am closing this bug as fixed.