Closed Bug 1696161 Opened 8 months ago Closed 8 months ago

Frequent testing/marionette/harness/marionette_harness/tests/unit/test_reftest.py TestReftest.test_url_comparison | AssertionError: u'PASS' != u'FAIL'

Categories

(Testing :: Marionette, defect, P5)

x86
Windows 7
defect

Tracking

(firefox-esr78 unaffected, firefox86 unaffected, firefox87 unaffected, firefox88 fixed)

RESOLVED FIXED
88 Branch
Tracking Status
firefox-esr78 --- unaffected
firefox86 --- unaffected
firefox87 --- unaffected
firefox88 --- fixed

People

(Reporter: intermittent-bug-filer, Assigned: emilio)

References

(Regression)

Details

(Keywords: assertion, intermittent-failure, regression)

Attachments

(1 file)

Filed by: ncsoregi [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=331885577&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/BB3qL6BwTbutQaGxCzNfrQ/runs/0/artifacts/public/logs/live_backing.log


[task 2021-03-03T12:03:12.838Z] 12:03:12     INFO -  Failed to gather test failure debug: Browsing context has been discarded
[task 2021-03-03T12:03:12.838Z] 12:03:12     INFO -  stacktrace:
[task 2021-03-03T12:03:12.838Z] 12:03:12     INFO -  	WebDriverError@chrome://marionette/content/error.js:181:5
[task 2021-03-03T12:03:12.839Z] 12:03:12     INFO -  	NoSuchWindowError@chrome://marionette/content/error.js:415:5
[task 2021-03-03T12:03:12.839Z] 12:03:12     INFO -  	assert.that/<@chrome://marionette/content/assert.js:460:13
[task 2021-03-03T12:03:12.839Z] 12:03:12     INFO -  	assert.open@chrome://marionette/content/assert.js:168:4
[task 2021-03-03T12:03:12.839Z] 12:03:12     INFO -  	GeckoDriver.prototype.takeScreenshot@chrome://marionette/content/driver.js:2428:10
[task 2021-03-03T12:03:12.839Z] 12:03:12     INFO -  	despatch@chrome://marionette/content/server.js:295:40
[task 2021-03-03T12:03:12.839Z] 12:03:12     INFO -  	execute@chrome://marionette/content/server.js:265:16
[task 2021-03-03T12:03:12.839Z] 12:03:12     INFO -  	onPacket/<@chrome://marionette/content/server.js:238:20
[task 2021-03-03T12:03:12.839Z] 12:03:12     INFO -  	onPacket@chrome://marionette/content/server.js:239:9
[task 2021-03-03T12:03:12.840Z] 12:03:12     INFO -  	_onJSONObjectReady/<@chrome://marionette/content/transport.js:504:20
[task 2021-03-03T12:03:12.840Z] 12:03:12     INFO -  1614772992829	Marionette	DEBUG	8 <- [1,14,null,{"value":null}]
[task 2021-03-03T12:03:12.841Z] 12:03:12     INFO -  1614772992833	Marionette	DEBUG	8 -> [0,15,"WebDriver:TakeScreenshot",{"full":true,"hash":false,"id":null,"scroll":true}]
[task 2021-03-03T12:03:12.841Z] 12:03:12     INFO -  1614772992834	Marionette	DEBUG	8 <- [1,15,{"error":"no such window","message":"Browsing context has been discarded","stacktrace":"WebDriverError@chrome://marione ... t@chrome://marionette/content/server.js:239:9\n_onJSONObjectReady/<@chrome://marionette/content/transport.js:504:20\n"},null]
[task 2021-03-03T12:03:12.841Z] 12:03:12    ERROR -  TEST-UNEXPECTED-FAIL | testing/marionette/harness/marionette_harness/tests/unit/test_reftest.py TestReftest.test_url_comparison | AssertionError: u'PASS' != u'FAIL'
[task 2021-03-03T12:03:12.841Z] 12:03:12     INFO -  - PASS
[task 2021-03-03T12:03:12.842Z] 12:03:12     INFO -  + FAIL```
Summary: Intermittent testing/marionette/harness/marionette_harness/tests/unit/test_reftest.py TestReftest.test_url_comparison | AssertionError: u'PASS' != u'FAIL' → Frequent testing/marionette/harness/marionette_harness/tests/unit/test_reftest.py TestReftest.test_url_comparison | AssertionError: u'PASS' != u'FAIL'

That seems to have been caused by bug 1687022. Emilio, can you please have a look?

Flags: needinfo?(emilio)
Keywords: regression
Regressed by: win-nnt

Maybe you can run mozregression with widget.non-native-theme.enabled set to true?

Also as it looks like it's only Windows 7 that is affected here.

OS: Unspecified → Windows 7
Hardware: Unspecified → x86

Is the stack trace above relevant? I have zero idea of how that could be caused by the non-native theme. What is the relevant page that is getting snapshotted?

Flags: needinfo?(hskupin)

The test pages you can see here:

https://searchfox.org/mozilla-central/rev/63fcc3f1a2cc73488d8986f4cf91fce2cd4b7564/testing/marionette/harness/marionette_harness/tests/unit/test_reftest.py#56-71

Sadly the screenshot data gets truncated for the tests in that file. Otherwise it could be found here:

https://treeherder.mozilla.org/logviewer?job_id=331885577&repo=autoland&lineNumber=57348

As such it would be good to trigger a try build with the preference marionette.log.truncate set to false. You can add that in the setUp and tearDown methods similar to what gets already done with dom.send_after_paint_to_content.

With that info we should get both the image and ref image listed in the log. Then we could check what's actually different.

Flags: needinfo?(hskupin)

So that page does have a couple <input> / <button> elements etc. Am I reading correctly that the number of differing pixels is in the log:

[task 2021-03-03T12:03:12.785Z] 12:03:12     INFO -  1614772992776	Marionette	DEBUG	Page 1 maxDifferences: 1 pixelsDifferent: 1

If so it seems like it is somehow slightly fuzzy (probably because the non-native theme has rounded corners, which the old theme didn't have). So, questions:

  • How are we rendering this page? (That is, do we use WebRender? SWGL? Or the old layers system?)
  • Is it acceptable to just tweak the test to allow 1px of fuzz in win7?
  • Alternatively, is it acceptable to just use input[type=text], input[type=button] { appearance: none } in the test.html file?

Thanks.

Flags: needinfo?(hskupin)

(In reply to Emilio Cobos Álvarez (:emilio) from comment #5)

  • How are we rendering this page? (That is, do we use WebRender? SWGL? Or the old layers system?)

Given the job description it should be no webrender, but for the rest I don't know. It's just the basic Windows 7 test job.

  • Is it acceptable to just tweak the test to allow 1px of fuzz in win7?

No, we should leave the max difference set to 1.

  • Alternatively, is it acceptable to just use input[type=text], input[type=button] { appearance: none } in the test.html file?

Yes, feel free to relax the appearance. This test is a unit test only so we shouldn't care about that. But please also add the pref that I mentioned before. It would be good to have for future failures. Thanks.

Flags: needinfo?(hskupin)
Assignee: nobody → emilio
Status: NEW → ASSIGNED
Flags: needinfo?(emilio)
Pushed by ealvarez@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/668ca019611c
Use appearance: none for buttons and text inputs in a marionette test, to prevent win7 fuzziness. r=whimboo,marionette-reviewers
Status: ASSIGNED → RESOLVED
Closed: 8 months ago
Resolution: --- → FIXED
Target Milestone: --- → 88 Branch
See Also: → 1696767
You need to log in before you can comment on or make changes to this bug.