1727207 - High freq Linux ASAN testing/marionette/harness/marionette_harness/tests/unit/test_reftest.py TestReftest.test_cache_multiple_sizes | AssertionError: 'PASS' != 'FAIL'

Status: RESOLVED FIXED

(Testing :: Marionette Client and Harness, defect, P1)

Description

[Treeherder Bug Filer]

Filed by: ncsoregi [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=349314752&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Y2iNppNtSGOGawwBL0AncA/runs/0/artifacts/public/logs/live_backing.log

---

[task 2021-08-23T22:44:12.180Z] 22:44:12     INFO -  1629758652180	Marionette	DEBUG	25 <- [1,23,null,{"value":null}]
[task 2021-08-23T22:44:12.182Z] 22:44:12    ERROR -  TEST-UNEXPECTED-FAIL | testing/marionette/harness/marionette_harness/tests/unit/test_reftest.py TestReftest.test_cache_multiple_sizes | AssertionError: 'PASS' != 'FAIL'
[task 2021-08-23T22:44:12.182Z] 22:44:12     INFO -  - PASS
[task 2021-08-23T22:44:12.182Z] 22:44:12     INFO -  + FAIL
[task 2021-08-23T22:44:12.182Z] 22:44:12     INFO -  Traceback (most recent call last):
[task 2021-08-23T22:44:12.182Z] 22:44:12     INFO -    File "C:\Users\task_1629748997\build\venv\lib\site-packages\marionette_harness\marionette_test\testcases.py", line 202, in run
[task 2021-08-23T22:44:12.182Z] 22:44:12     INFO -      testMethod()
[task 2021-08-23T22:44:12.183Z] 22:44:12     INFO -    File "C:\Users\task_1629748997\build\tests\marionette\tests\testing\marionette\harness\marionette_harness\tests\unit\test_reftest.py", line 91, in test_cache_multiple_sizes
[task 2021-08-23T22:44:12.183Z] 22:44:12     INFO -      self.assertEqual(u"PASS", rv[u"value"][u"status"])
[task 2021-08-23T22:44:12.183Z] 22:44:12     INFO -  TEST-INFO took 555ms

Comment 1

[Intermittent Failures Robot]

1 failures in 2580 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• windows10-64-asan-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1727207&startday=2021-08-23&endday=2021-08-29&tree=all

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit auto_nag documentation.

Comment 3

[Treeherder Bug Filer]

New failure instance: https://treeherder.mozilla.org/logviewer?job_id=368374629&repo=autoland

Comment 4

[Intermittent Failures Robot]

1 failures in 3281 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• linux1804-64-asan-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1727207&startday=2022-02-14&endday=2022-02-20&tree=all

Comment 5

[Intermittent Failures Robot]

3 failures in 2992 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 3

Platform and build breakdown:

• linux1804-64-asan-qr: 2
  • opt: 2
• linux1804-64-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1727207&startday=2022-02-21&endday=2022-02-27&tree=all

Comment 6

[Intermittent Failures Robot]

2 failures in 3063 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2

Platform and build breakdown:

• linux1804-64-asan-qr: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1727207&startday=2022-02-28&endday=2022-03-06&tree=all

Comment 7

[Intermittent Failures Robot]

1 failures in 2837 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 1

Platform and build breakdown:

• linux1804-64-asan-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1727207&startday=2022-03-07&endday=2022-03-13&tree=all

Comment 8

[Intermittent Failures Robot]

1 failures in 3304 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 1

Platform and build breakdown:

• linux1804-64-asan-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1727207&startday=2022-03-14&endday=2022-03-20&tree=all

Comment 9

[BugBot [:suhaib / :marco/ :calixte]]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit auto_nag documentation.

Comment 10

[Treeherder Bug Filer]

New failure instance: https://treeherder.mozilla.org/logviewer?job_id=374281897&repo=mozilla-central

Comment 11

[Intermittent Failures Robot]

3 failures in 2948 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1
• autoland: 2

Platform and build breakdown:

• linux1804-64-asan-qr: 2
  • opt: 2
• windows10-64-2004-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1727207&startday=2022-04-11&endday=2022-04-17&tree=all

Comment 12

[Intermittent Failures Robot]

3 failures in 3159 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1
• autoland: 2

Platform and build breakdown:

• linux1804-64-asan-qr: 3
  • opt: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1727207&startday=2022-04-18&endday=2022-04-24&tree=all

Comment 13

[Intermittent Failures Robot]

16 failures in 3289 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 15
• mozilla-central: 1

Platform and build breakdown:

• linux1804-64-asan-qr: 16
  • opt: 16

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1727207&startday=2022-04-25&endday=2022-05-01&tree=all

Comment 14

[Intermittent Failures Robot]

25 failures in 3540 pushes (0.007 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 1
• mozilla-beta: 4
• mozilla-central: 9
• autoland: 11

Platform and build breakdown:

• linux1804-64-asan-qr: 25
  • opt: 25

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1727207&startday=2022-05-02&endday=2022-05-08&tree=all

Comment 15

[Intermittent Failures Robot]

21 failures in 3317 pushes (0.006 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 6
• try: 3
• mozilla-beta: 5
• autoland: 7

Platform and build breakdown:

• linux1804-64-asan-qr: 21
  • opt: 21

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1727207&startday=2022-05-09&endday=2022-05-15&tree=all

Comment 16

[Intermittent Failures Robot]

28 failures in 2694 pushes (0.01 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 16
• mozilla-beta: 3
• mozilla-central: 7
• try: 2

Platform and build breakdown:

• linux1804-64-asan-qr: 28
  • opt: 28

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1727207&startday=2022-05-16&endday=2022-05-22&tree=all

Comment 17

[Andreea Pavel [:apavel]]

There are 41 total failures in the last 7 days on

• linux1804-64-asan-qr opt
  Recent failure log: https://treeherder.mozilla.org/logviewer?job_id=379234393&repo=autoland&lineNumber=47312

[task 2022-05-25T10:56:49.437Z] 10:56:49     INFO - TEST-START | testing/marionette/harness/marionette_harness/tests/unit/test_reftest.py TestReftest.test_cache_multiple_sizes
[task 2022-05-25T10:56:49.438Z] 10:56:49     INFO -  1653476209435	Marionette	DEBUG	Closed connection 23
[task 2022-05-25T10:56:49.441Z] 10:56:49     INFO -  1653476209436	Marionette	DEBUG	Accepted connection 24 from 127.0.0.1:49954
[task 2022-05-25T10:56:49.444Z] 10:56:49     INFO -  1653476209438	Marionette	DEBUG	24 -> [0,1,"WebDriver:NewSession",{"strictFileInteractability":true}]
[task 2022-05-25T10:56:49.447Z] 10:56:49     INFO -  1653476209439	Marionette	DEBUG	Waiting for initial application window
[task 2022-05-25T10:56:49.450Z] 10:56:49     INFO -  1653476209442	RemoteAgent	TRACE	[64] Document already finished loading: data:text/html;charset=utf-8,%3Cselect%20multiple%3E%20%3Coption%3Efirst%20%3C/select%3E
[task 2022-05-25T10:56:49.453Z] 10:56:49     INFO -  1653476209446	Marionette	DEBUG	24 <- [1,1,null,{"sessionId":"5dff837a-dfb9-452e-8159-a9604b5185b7","capabilities":{"browserName":"firefox","browserVersion":"102.0 ... wnTimeout":300000,"moz:useNonSpecCompliantPointerOrigin":false,"moz:webdriverClick":true,"moz:windowless":false,"proxy":{}}}]
[task 2022-05-25T10:56:49.456Z] 10:56:49     INFO -  1653476209449	Marionette	DEBUG	24 -> [0,2,"WebDriver:SetTimeouts",{"script":30000}]
[task 2022-05-25T10:56:49.458Z] 10:56:49     INFO -  1653476209450	Marionette	DEBUG	24 <- [1,2,null,{"value":null}]
[task 2022-05-25T10:56:49.461Z] 10:56:49     INFO -  1653476209451	Marionette	DEBUG	24 -> [0,3,"WebDriver:SetTimeouts",{"pageLoad":300000}]
[task 2022-05-25T10:56:49.463Z] 10:56:49     INFO -  1653476209452	Marionette	DEBUG	24 <- [1,3,null,{"value":null}]
[task 2022-05-25T10:56:49.466Z] 10:56:49     INFO -  1653476209454	Marionette	DEBUG	24 -> [0,4,"WebDriver:SetTimeouts",{"implicit":0}]
[task 2022-05-25T10:56:49.468Z] 10:56:49     INFO -  1653476209454	Marionette	DEBUG	24 <- [1,4,null,{"value":null}]
[task 2022-05-25T10:56:49.471Z] 10:56:49     INFO -  1653476209457	Marionette	DEBUG	24 -> [0,5,"Marionette:GetContext",{}]
[task 2022-05-25T10:56:49.474Z] 10:56:49     INFO -  1653476209458	Marionette	DEBUG	24 <- [1,5,null,{"value":"content"}]
[task 2022-05-25T10:56:49.476Z] 10:56:49     INFO -  1653476209460	Marionette	DEBUG	24 -> [0,6,"Marionette:SetContext",{"value":"content"}]
[task 2022-05-25T10:56:49.478Z] 10:56:49     INFO -  1653476209460	Marionette	DEBUG	24 <- [1,6,null,{"value":null}]
[task 2022-05-25T10:56:49.480Z] 10:56:49     INFO -  1653476209462	Marionette	DEBUG	24 -> [0,7,"WebDriver:GetWindowHandle",{}]
[task 2022-05-25T10:56:49.482Z] 10:56:49     INFO -  1653476209463	Marionette	DEBUG	24 <- [1,7,null,{"value":"1e1d39d5-be0e-4be6-b2f6-eae236583067"}]
[task 2022-05-25T10:56:49.484Z] 10:56:49     INFO -  1653476209465	Marionette	DEBUG	24 -> [0,8,"Marionette:SetContext",{"value":"content"}]
[task 2022-05-25T10:56:49.486Z] 10:56:49     INFO -  1653476209466	Marionette	DEBUG	24 <- [1,8,null,{"value":null}]
[task 2022-05-25T10:56:49.489Z] 10:56:49     INFO -  1653476209469	Marionette	DEBUG	24 -> [0,9,"Marionette:GetContext",{}]
[task 2022-05-25T10:56:49.491Z] 10:56:49     INFO -  1653476209470	Marionette	DEBUG	24 <- [1,9,null,{"value":"content"}]
[task 2022-05-25T10:56:49.493Z] 10:56:49     INFO -  1653476209471	Marionette	DEBUG	24 -> [0,10,"Marionette:SetContext",{"value":"chrome"}]
[task 2022-05-25T10:56:49.496Z] 10:56:49     INFO -  1653476209472	Marionette	DEBUG	24 <- [1,10,null,{"value":null}]
[task 2022-05-25T10:56:49.498Z] 10:56:49     INFO -  1653476209475	Marionette	DEBUG	24 -> [0,11,"WebDriver:ExecuteScript",{"script":"const { Preferences } = ChromeUtils.import(\n                  \"resource://gre/mo ... ox":true,"sandbox":"default","line":849,"filename":"../../venv/lib/python3.6/site-packages/marionette_driver/marionette.py"}]
[task 2022-05-25T10:56:49.500Z] 10:56:49     INFO -  1653476209482	Marionette	TRACE	[7] MarionetteCommands actor created for window id 2
[task 2022-05-25T10:56:49.503Z] 10:56:49     INFO -  1653476209498	Marionette	DEBUG	24 <- [1,11,null,{"value":null}]
[task 2022-05-25T10:56:49.505Z] 10:56:49     INFO -  1653476209501	Marionette	DEBUG	24 -> [0,12,"Marionette:SetContext",{"value":"content"}]
[task 2022-05-25T10:56:49.508Z] 10:56:49     INFO -  1653476209503	Marionette	DEBUG	24 <- [1,12,null,{"value":null}]
[task 2022-05-25T10:56:49.510Z] 10:56:49     INFO -  1653476209505	Marionette	DEBUG	24 -> [0,13,"Marionette:GetContext",{}]
[task 2022-05-25T10:56:49.513Z] 10:56:49     INFO -  1653476209506	Marionette	DEBUG	24 <- [1,13,null,{"value":"content"}]
[task 2022-05-25T10:56:49.515Z] 10:56:49     INFO -  1653476209508	Marionette	DEBUG	24 -> [0,14,"Marionette:SetContext",{"value":"chrome"}]
[task 2022-05-25T10:56:49.517Z] 10:56:49     INFO -  1653476209509	Marionette	DEBUG	24 <- [1,14,null,{"value":null}]
[task 2022-05-25T10:56:49.520Z] 10:56:49     INFO -  1653476209512	Marionette	DEBUG	24 -> [0,15,"WebDriver:ExecuteScript",{"script":"const { Preferences } = ChromeUtils.import(\n                  \"resource://gre/mo ... ox":true,"sandbox":"default","line":849,"filename":"../../venv/lib/python3.6/site-packages/marionette_driver/marionette.py"}]
[task 2022-05-25T10:56:49.546Z] 10:56:49     INFO -  1653476209546	Marionette	DEBUG	24 <- [1,15,null,{"value":null}]
[task 2022-05-25T10:56:49.554Z] 10:56:49     INFO -  1653476209553	Marionette	DEBUG	24 -> [0,16,"Marionette:SetContext",{"value":"content"}]
[task 2022-05-25T10:56:49.558Z] 10:56:49     INFO -  1653476209558	Marionette	DEBUG	24 <- [1,16,null,{"value":null}]
[task 2022-05-25T10:56:49.565Z] 10:56:49     INFO -  1653476209565	Marionette	DEBUG	24 -> [0,17,"reftest:setup",{"screenshot":"unexpected"}]
[task 2022-05-25T10:56:49.583Z] 10:56:49     INFO -  1653476209582	Marionette	DEBUG	24 <- [1,17,null,{"value":null}]
[task 2022-05-25T10:56:49.586Z] 10:56:49     INFO -  1653476209585	Marionette	DEBUG	24 -> [0,18,"reftest:run",{"test":"http://127.0.0.1:38182/reftest/teal-700x700.html","references":[["http://127.0.0.1:38182/reftest/mostly-teal-700x700.html",[],"=="]],"expected":"PASS","timeout":10000,"width":600,"height":600}]
[task 2022-05-25T10:56:49.587Z] 10:56:49     INFO -  1653476209587	Marionette	DEBUG	ensuring we have a window 600x600
[task 2022-05-25T10:56:49.588Z] 10:56:49     INFO -  1653476209587	Marionette	DEBUG	Using separate window
[task 2022-05-25T10:56:49.595Z] 10:56:49     INFO -  1653476209595	Marionette	TRACE	Received observer notification domwindowopened
[task 2022-05-25T10:56:49.692Z] 10:56:49     INFO -  1653476209691	RemoteAgent	TRACE	Received DOM event activate for [object Window]
[task 2022-05-25T10:56:49.694Z] 10:56:49     INFO -  1653476209692	RemoteAgent	TRACE	Received DOM event focus for [object HTMLDocument]
[task 2022-05-25T10:56:49.700Z] 10:56:49     INFO -  JavaScript error: chrome://remote/content/shared/TabManager.jsm, line 311: TypeError: can't access property "linkedBrowser", tab is undefined
[task 2022-05-25T10:56:49.701Z] 10:56:49     INFO -  1653476209701	Marionette	DEBUG	loaded initial URL: about:blank

[task 2022-05-25T10:56:51.706Z] 10:56:51     INFO -  1653476211706	Marionette	DEBUG	24 -> [0,25,"WebDriver:GetPageSource",{}]
[task 2022-05-25T10:56:51.716Z] 10:56:51     INFO -  1653476211715	Marionette	TRACE	[67] MarionetteCommands actor created for window id 10737418242
[task 2022-05-25T10:56:51.722Z] 10:56:51     INFO -  1653476211721	Marionette	DEBUG	24 <- [1,25,null,{"value":"<html><head><style>\n  * {\n      margin: 0;\n      padding: 0;\n  }\n\n  html {\n      background: teal ... ight: 100px;\n      width: 100px;\n      background: transparent;\n  }\n</style>\n</head><body><div></div>\n</body></html>"}]
[task 2022-05-25T10:56:51.726Z] 10:56:51     INFO -  1653476211725	Marionette	DEBUG	24 -> [0,26,"Marionette:SetContext",{"value":"content"}]
[task 2022-05-25T10:56:51.727Z] 10:56:51     INFO -  1653476211726	Marionette	DEBUG	24 <- [1,26,null,{"value":null}]
[task 2022-05-25T10:56:51.744Z] 10:56:51     INFO - TEST-UNEXPECTED-FAIL | testing/marionette/harness/marionette_harness/tests/unit/test_reftest.py TestReftest.test_cache_multiple_sizes | AssertionError: 'PASS' != 'FAIL'
[task 2022-05-25T10:56:51.744Z] 10:56:51     INFO - - PASS
[task 2022-05-25T10:56:51.744Z] 10:56:51     INFO - + FAIL
[task 2022-05-25T10:56:51.744Z] 10:56:51     INFO - Traceback (most recent call last):
[task 2022-05-25T10:56:51.744Z] 10:56:51     INFO -   File "/builds/worker/workspace/build/venv/lib/python3.6/site-packages/marionette_harness/marionette_test/testcases.py", line 202, in run
[task 2022-05-25T10:56:51.744Z] 10:56:51     INFO -     testMethod()
[task 2022-05-25T10:56:51.744Z] 10:56:51     INFO -   File "/builds/worker/workspace/build/tests/marionette/tests/testing/marionette/harness/marionette_harness/tests/unit/test_reftest.py", line 91, in test_cache_multiple_sizes
[task 2022-05-25T10:56:51.744Z] 10:56:51     INFO -     self.assertEqual(u"PASS", rv[u"value"][u"status"])
[task 2022-05-25T10:56:51.744Z] 10:56:51     INFO - TEST-INFO took 2293ms

Glenn, could you please take a look or assign this to someone?
Thank you!

Comment 18

[Glenn Watson [:gw]]

Hi Andreea, I'm not really sure what this failure means or even who would be good to look at it.

I think it comes from:

[task 2022-05-25T10:41:32.537Z] 10:41:32     INFO -  Assertion failure
[task 2022-05-25T10:41:32.538Z] 10:41:32     INFO -  assert@resource:///modules/AsyncTabSwitcher.jsm:500:36
[task 2022-05-25T10:41:32.539Z] 10:41:32     INFO -  onLayersCleared@resource:///modules/AsyncTabSwitcher.jsm:811:12
[task 2022-05-25T10:41:32.539Z] 10:41:32     INFO -  handleEvent@resource:///modules/AsyncTabSwitcher.jsm:1111:16

Emilio, any ideas what this means and/or who would be a good candidate to look at it?

Comment 19

[Emilio Cobos Álvarez (:emilio)]

I don't think the tab switcher assert is relevant. Henrik are there screenshots that could help debug the retest failure or something like that? Or is this a marionette bug?

Comment 20

[Intermittent Failures Robot]

54 failures in 2624 pushes (0.021 failures/push) were associated with this bug in the last 7 days.

This is the #12 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 31
• mozilla-central: 18
• mozilla-beta: 1
• try: 3
• mozilla-release: 1

Platform and build breakdown:

• linux1804-64-asan-qr: 53
  • opt: 53
• linux1804-64-ccov-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1727207&startday=2022-05-23&endday=2022-05-29&tree=all

Comment 21

[Henrik Skupin [:whimboo][⌚️UTC+2]]

So it looks like that only Linux ASAN builds are affected here and we saw an increase of test failures on autoland starting in the week of March 25th to May 2nd. But I cannot see that we changed something for Marionette for the 101 release. Maybe a regression check might be good here.

(In reply to Emilio Cobos Álvarez (:emilio) from comment #19)

I don't think the tab switcher assert is relevant. Henrik are there screenshots that could help debug the retest failure or something like that? Or is this a marionette bug?

Screenshots are taken as you can see here:
https://treeherder.mozilla.org/logviewer?job_id=379569697&repo=mozilla-central&lineNumber=75544-75546

But sadly we also truncate the line so the data URL is not usable. Someone could run a try build with the preference remote.log.truncate set to false. Otherwise you could also print the URL once received by Marionette here:

https://searchfox.org/mozilla-central/source/testing/marionette/harness/marionette_harness/runner/base.py#709-710

Maybe we could do this by default in such cases so help with debugging.

Otherwise I would defer to James who is more familiar with the appropriate code.

Comment 22

[Julian Descottes [:jdescottes]]

(In reply to Henrik Skupin (:whimboo) [⌚️UTC+1] from comment #21)

So it looks like that only Linux ASAN builds are affected here and we saw an increase of test failures on autoland starting in the week of March 25th to May 2nd. But I cannot see that we changed something for Marionette for the 101 release. Maybe a regression check might be good here.

(In reply to Emilio Cobos Álvarez (:emilio) from comment #19)

I don't think the tab switcher assert is relevant. Henrik are there screenshots that could help debug the retest failure or something like that? Or is this a marionette bug?

Screenshots are taken as you can see here:
https://treeherder.mozilla.org/logviewer?job_id=379569697&repo=mozilla-central&lineNumber=75544-75546

But sadly we also truncate the line so the data URL is not usable. Someone could run a try build with the preference remote.log.truncate set to false. Otherwise you could also print the URL once received by Marionette here:

I just did a run with truncate to false: https://treeherder.mozilla.org/jobs?repo=try&selectedTaskRun=DL_ADyb_Rka-aLSPOkfnkA.0&revision=f40bb43bbfedc494454c9cecf091e1683ed0c11e

I can attach the images for a failure, but basically the only difference is about the way the floating scrollbars are rendered. In one version their color is rgb(150, 163, 162), while in the other version it is rgb(163, 166, 165).

I can imagine that the scrollbars are progressively fading in/out, so the visible color used to draw them might be different depending on the timing of the test?

Comment 23

[Julian Descottes [:jdescottes]]

Attachment: image1.png

Comment 24

[Julian Descottes [:jdescottes]]

Attachment: image2.png

Comment 25

[Emilio Cobos Álvarez (:emilio)]

Well there's the question of why are we using overlay scrollbars at all too (I thought we disabled them for reftests generally), but anyways yeah, at least that's not too concerning :)

Comment 26

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Well, this is from a Marionette unit test and those have different preferences set compared to web-platform reftests which only make use of Marionette. Which preference is this specifically? Can it be changed at runtime? We could specifically set it for this particular test. Or what would be the differences / implications if we do in for all Marionette?

Comment 27

[Andreea Pavel [:apavel]]

Attachment: Bug 1727207 - updated disabling condition to include linux opt r?#intermittent-reviewers

Comment 28

[Intermittent Failures Robot]

27 failures in 687 pushes (0.039 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 23
• mozilla-central: 3
• mozilla-beta: 1

Platform and build breakdown:

• linux1804-64-asan-qr: 27
  • opt: 27

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1727207&startday=2022-06-01&endday=2022-06-01&tree=all

Comment 29

[Julian Descottes [:jdescottes]]

(In reply to Emilio Cobos Álvarez (:emilio) from comment #25)

Well there's the question of why are we using overlay scrollbars at all too (I thought we disabled them for reftests generally), but anyways yeah, at least that's not too concerning :)

Henrik mentioned this is a Marionette unit test, so we might miss a preference to disable the scrollbars. Do you remember which preference we could set here to avoid having scrollbars?

Comment 30

[Emilio Cobos Álvarez (:emilio)]

On Linux, it's this: https://searchfox.org/mozilla-central/rev/c8e15e17bc6fd28f558c395c948a6251b38774ff/testing/profiles/common/user.js#89

On macOS and Windows they should be disabled at the system level.

Comment 31

[Julian Descottes [:jdescottes]]

Thanks! I can see it's still set to true when running Marionette unit tests.
Here's a try push forcing it to false: https://treeherder.mozilla.org/jobs?repo=try&revision=5fab95d955e0c75895561887891dc1e981b6b649

Edit: wrong try push, I had a typo in the preference name. New push: https://treeherder.mozilla.org/#/jobs?repo=try&revision=d636124352d94e43712c9ab598e5f7ff05c20dca

Comment 32

[Julian Descottes [:jdescottes]]

As commented on https://phabricator.services.mozilla.com/D148090, the current skip-if is probably not working as expected: os == "linux" && asan && fission. fission cannot be found in the mozinfo for this test, so it makes the condition fail.

Comment 33

[Julian Descottes [:jdescottes]]

Attachment: Bug 1727207 - [marionette] Disable widget.gtk.overlay-scrollbars.enabled for test_reftest.py

Disabling scrollbars fixes the intermittent, also removing the skip-if since it always evaluated to false anyway.

Comment 34

[Norisz Fay [:noriszfay]]

According to the backfills/retriggers it seems to be failing quite a lot - backfill range

Comment 35

[Intermittent Failures Robot]

101 failures in 558 pushes (0.181 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 98
• mozilla-central: 2
• try: 1

Platform and build breakdown:

• linux1804-64-asan-qr: 101
  • opt: 101

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1727207&startday=2022-06-02&endday=2022-06-02&tree=all

Comment 36

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d3911aa2553b
[marionette] Disable widget.gtk.overlay-scrollbars.enabled for test_reftest.py r=webdriver-reviewers,whimboo

Comment 37

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Cosmin, is there a reason why we would have to leave the bug open? The landed patch fixes the test for real.

Comment 38

[Cosmin Sabou [:CosminS]]

If it fixes then yes I'll delete the tags. Given the commit I thought it was just disabled, my bad. Thanks for pointing that out.

Comment 39

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/d3911aa2553b

Comment 40

[Intermittent Failures Robot]

150 failures in 2917 pushes (0.051 failures/push) were associated with this bug in the last 7 days.

This is the #5 most frequent failure this week.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 133
• mozilla-beta: 4
• mozilla-central: 10
• try: 2
• mozilla-release: 1

Platform and build breakdown:

• linux1804-64-asan-qr: 148
  • opt: 148
• linux1804-64-ccov-qr: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1727207&startday=2022-05-30&endday=2022-06-05&tree=all

Comment 41

[Julian Descottes [:jdescottes]]

Removing the whiteboard flag, no new failure since the fix landed.

Comment 42

[Intermittent Failures Robot]

1 failures in 3336 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• mozilla-beta: 1

Platform and build breakdown:

• linux1804-64-asan-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1727207&startday=2022-06-06&endday=2022-06-12&tree=all

Comment 43

[Intermittent Failures Robot]

1 failures in 3075 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr102: 1

Platform and build breakdown:

• linux1804-64-asan-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1727207&startday=2022-06-20&endday=2022-06-26&tree=all

Comment 44

[Intermittent Failures Robot]

1 failures in 3312 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr102: 1

Platform and build breakdown:

• linux1804-64-asan-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1727207&startday=2022-06-27&endday=2022-07-03&tree=all

Comment 46

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Release is affected as well (see the just duplicated bug) but not sure if we are going to fix that if the volume of failures is low.

Comment 47

[Intermittent Failures Robot]

2 failures in 3072 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-release: 2

Platform and build breakdown:

• linux1804-64-asan-qr: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1727207&startday=2022-07-04&endday=2022-07-10&tree=all

Comment 48

[Donal Meehan [:dmeehan]]

https://hg.mozilla.org/releases/mozilla-esr102/rev/13466ea509d7

Comment 49

[Intermittent Failures Robot]

1 failures in 3888 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• windows10-32-2004-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1727207&startday=2022-11-21&endday=2022-11-27&tree=all

Comment 50

[BMO Automation]

Moving bug to Testing::Marionette Client and Harness component per bug 1815831.