1696161 - Frequent testing/marionette/harness/marionette_harness/tests/unit/test_reftest.py TestReftest.test_url_comparison | AssertionError: u'PASS' != u'FAIL'

Status: RESOLVED FIXED

(Testing :: Marionette Client and Harness, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: ncsoregi [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=331885577&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/BB3qL6BwTbutQaGxCzNfrQ/runs/0/artifacts/public/logs/live_backing.log

---

[task 2021-03-03T12:03:12.838Z] 12:03:12     INFO -  Failed to gather test failure debug: Browsing context has been discarded
[task 2021-03-03T12:03:12.838Z] 12:03:12     INFO -  stacktrace:
[task 2021-03-03T12:03:12.838Z] 12:03:12     INFO -  	WebDriverError@chrome://marionette/content/error.js:181:5
[task 2021-03-03T12:03:12.839Z] 12:03:12     INFO -  	NoSuchWindowError@chrome://marionette/content/error.js:415:5
[task 2021-03-03T12:03:12.839Z] 12:03:12     INFO -  	assert.that/<@chrome://marionette/content/assert.js:460:13
[task 2021-03-03T12:03:12.839Z] 12:03:12     INFO -  	assert.open@chrome://marionette/content/assert.js:168:4
[task 2021-03-03T12:03:12.839Z] 12:03:12     INFO -  	GeckoDriver.prototype.takeScreenshot@chrome://marionette/content/driver.js:2428:10
[task 2021-03-03T12:03:12.839Z] 12:03:12     INFO -  	despatch@chrome://marionette/content/server.js:295:40
[task 2021-03-03T12:03:12.839Z] 12:03:12     INFO -  	execute@chrome://marionette/content/server.js:265:16
[task 2021-03-03T12:03:12.839Z] 12:03:12     INFO -  	onPacket/<@chrome://marionette/content/server.js:238:20
[task 2021-03-03T12:03:12.839Z] 12:03:12     INFO -  	onPacket@chrome://marionette/content/server.js:239:9
[task 2021-03-03T12:03:12.840Z] 12:03:12     INFO -  	_onJSONObjectReady/<@chrome://marionette/content/transport.js:504:20
[task 2021-03-03T12:03:12.840Z] 12:03:12     INFO -  1614772992829	Marionette	DEBUG	8 <- [1,14,null,{"value":null}]
[task 2021-03-03T12:03:12.841Z] 12:03:12     INFO -  1614772992833	Marionette	DEBUG	8 -> [0,15,"WebDriver:TakeScreenshot",{"full":true,"hash":false,"id":null,"scroll":true}]
[task 2021-03-03T12:03:12.841Z] 12:03:12     INFO -  1614772992834	Marionette	DEBUG	8 <- [1,15,{"error":"no such window","message":"Browsing context has been discarded","stacktrace":"WebDriverError@chrome://marione ... t@chrome://marionette/content/server.js:239:9\n_onJSONObjectReady/<@chrome://marionette/content/transport.js:504:20\n"},null]
[task 2021-03-03T12:03:12.841Z] 12:03:12    ERROR -  TEST-UNEXPECTED-FAIL | testing/marionette/harness/marionette_harness/tests/unit/test_reftest.py TestReftest.test_url_comparison | AssertionError: u'PASS' != u'FAIL'
[task 2021-03-03T12:03:12.841Z] 12:03:12     INFO -  - PASS
[task 2021-03-03T12:03:12.842Z] 12:03:12     INFO -  + FAIL```

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+2]]

That seems to have been caused by bug 1687022. Emilio, can you please have a look?

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Maybe you can run mozregression with widget.non-native-theme.enabled set to true?

Also as it looks like it's only Windows 7 that is affected here.

Comment 3

[Emilio Cobos Álvarez [:emilio]]

Is the stack trace above relevant? I have zero idea of how that could be caused by the non-native theme. What is the relevant page that is getting snapshotted?

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+2]]

The test pages you can see here:

https://searchfox.org/mozilla-central/rev/63fcc3f1a2cc73488d8986f4cf91fce2cd4b7564/testing/marionette/harness/marionette_harness/tests/unit/test_reftest.py#56-71

Sadly the screenshot data gets truncated for the tests in that file. Otherwise it could be found here:

https://treeherder.mozilla.org/logviewer?job_id=331885577&repo=autoland&lineNumber=57348

As such it would be good to trigger a try build with the preference marionette.log.truncate set to false. You can add that in the setUp and tearDown methods similar to what gets already done with dom.send_after_paint_to_content.

With that info we should get both the image and ref image listed in the log. Then we could check what's actually different.

Comment 5

[Emilio Cobos Álvarez [:emilio]]

So that page does have a couple <input> / <button> elements etc. Am I reading correctly that the number of differing pixels is in the log:

[task 2021-03-03T12:03:12.785Z] 12:03:12     INFO -  1614772992776	Marionette	DEBUG	Page 1 maxDifferences: 1 pixelsDifferent: 1

If so it seems like it is somehow slightly fuzzy (probably because the non-native theme has rounded corners, which the old theme didn't have). So, questions:

• How are we rendering this page? (That is, do we use WebRender? SWGL? Or the old layers system?)
• Is it acceptable to just tweak the test to allow 1px of fuzz in win7?
• Alternatively, is it acceptable to just use input[type=text], input[type=button] { appearance: none } in the test.html file?

Thanks.

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Emilio Cobos Álvarez (:emilio) from comment #5)

• How are we rendering this page? (That is, do we use WebRender? SWGL? Or the old layers system?)

Given the job description it should be no webrender, but for the rest I don't know. It's just the basic Windows 7 test job.

• Is it acceptable to just tweak the test to allow 1px of fuzz in win7?

No, we should leave the max difference set to 1.

• Alternatively, is it acceptable to just use input[type=text], input[type=button] { appearance: none } in the test.html file?

Yes, feel free to relax the appearance. This test is a unit test only so we shouldn't care about that. But please also add the pref that I mentioned before. It would be good to have for future failures. Thanks.

Comment 7

[Emilio Cobos Álvarez [:emilio]]

Attachment: Bug 1696161 - Use appearance: none for buttons and text inputs in a marionette test, to prevent win7 fuzziness. r=whimboo

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/668ca019611c
Use appearance: none for buttons and text inputs in a marionette test, to prevent win7 fuzziness. r=whimboo,marionette-reviewers

Comment 9

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/668ca019611c

Comment 10

[Intermittent Failures Robot]

19 failures in 693 pushes (0.027 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 18
• mozilla-central: 1

Platform and build breakdown:

• windows7-32: 18
  • debug: 15
  • opt: 3
• windows7-32-shippable: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1696161&startday=2021-03-03&endday=2021-03-03&tree=all

Comment 11

[Intermittent Failures Robot]

19 failures in 3683 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 18
• mozilla-central: 1

Platform and build breakdown:

• windows7-32: 18
  • debug: 15
  • opt: 3
• windows7-32-shippable: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1696161&startday=2021-03-01&endday=2021-03-07&tree=all

Comment 12

[BMO Automation]

Moving bug to Testing::Marionette Client and Harness component per bug 1815831.