1510695 - Marionette reftest runner fails to notice page load when URL doesn't round-trip

Status: RESOLVED FIXED

(Remote Protocol :: Marionette, defect)

Comment 1

[Sam Sneddon [:gsnedders]]

Attachment: Bug 1510695 - Fix URL comparisons when loading reftests

Depends on D13281

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/8a7b62c82459
Fix URL comparisons in reftestWait r=jgraham

Comment 4

[Arthur Iakab [arthur_iakab]]

Backed out 2 changesets (bug 1510695, bug 1510693) for causing frequent marionette failures on unit/test_reftest.py

Backout revision https://hg.mozilla.org/integration/autoland/rev/e71d2b9ad78f452058d7dea868bbaa02c1327137

Failed push https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=os%2Cx%2C10.10%2Copt%2Ctest-macosx64%2Fopt-marionette-headless-e10s%2C%28mnh%29&selectedJob=216659327&revision=8a7b62c8245908d7057dd7bed99b69e698527473

Failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=216663565&repo=autoland

:gsnedders can you please take a look?

Comment 5

[Sam Sneddon [:gsnedders]]

Relatively likely to only get around to trying to reland these next year, now.

:jgraham That it gets a timeout suggests something deeper is wrong with the script (and probably the cause of some of the disabled tests in WPT itself), no?

Comment 6

[James Graham [:jgraham]]

Well I would guess that if there is an existing bug it's about the handling of the case where we have the same url for the test and ref, which isn't a thing that really happens very much.

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/aa26fbe413c4
Fix URL comparisons in reftestWait r=jgraham

Comment 8

[Pulsebot]

Backout by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3994da3447c3
Backed out 3 changesets (bug 1510695, bug 1510696, bug 1510693) for causing marionette reftest failures.

Comment 9

[Cosmin Sabou [:CosminS]]

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=success%2Ctestfailed%2Cbusted%2Cexception&tochange=3994da3447c34fbf333b51776fd91afeb7fa1dbf&fromchange=daf05866f9ddcd5a362b4d436477f3e01abd4234&searchStr=mn&selectedJob=221132110

Failure log:
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=221150086&repo=autoland&lineNumber=60695
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=221137313&repo=autoland&lineNumber=60619

Backout link: https://hg.mozilla.org/integration/autoland/rev/3994da3447c34fbf333b51776fd91afeb7fa1dbf

10:48:35 WARNING - Failed to gather test failure debug: Browsing context has been discarded
10:48:35 WARNING - stacktrace:
10:48:35 WARNING - WebDriverError@chrome://marionette/content/error.js:179:5
10:48:35 WARNING - NoSuchWindowError@chrome://marionette/content/error.js:411:5
10:48:35 WARNING - assert.that/<@chrome://marionette/content/assert.js:418:13
10:48:35 WARNING - assert.open@chrome://marionette/content/assert.js:174:10
10:48:35 WARNING - GeckoDriver.prototype.takeScreenshot@chrome://marionette/content/driver.js:2927:13
10:48:35 WARNING - despatch@chrome://marionette/content/server.js:290:20
10:48:35 WARNING - asyncexecute@chrome://marionette/content/server.js:263:11
10:48:35 WARNING - asynconPacket/<@chrome://marionette/content/server.js:236:15
10:48:35 WARNING - async*onPacket@chrome://marionette/content/server.js:235:8
10:48:35 WARNING - _onJSONObjectReady/<@chrome://marionette/content/transport.js:493:9
10:48:35 WARNING -
10:48:35 INFO - TEST-UNEXPECTED-FAIL | testing/marionette/harness/marionette_harness/tests/unit/test_reftest.py TestReftest.test_cache_multiple_sizes | AssertionError: u'FAIL' != u'TIMEOUT'
10:48:35 INFO - - FAIL
10:48:35 INFO - + TIMEOUT
10:48:35 INFO - Traceback (most recent call last):
10:48:35 INFO - File "/Users/cltbld/tasks/task_1547145020/build/venv/lib/python2.7/site-packages/marionette_harness/marionette_test/testcases.py", line 159, in run
10:48:35 INFO - testMethod()
10:48:35 INFO - File "/Users/cltbld/tasks/task_1547145020/build/tests/marionette/tests/testing/marionette/harness/marionette_harness/tests/unit/test_reftest.py", line 83, in test_cache_multiple_sizes
10:48:35 INFO - self.assertEqual(u"FAIL", rv[u"value"][u"status"])
10:48:35 INFO - TEST-INFO took 2522ms
10:48:35 INFO - 1547146115184 Marionette DEBUG 7 -> [0,18,"reftest:teardown",{}]
10:48:35 INFO - 1547146115187 Marionette DEBUG 7 <- [1,18,null,{"value":null}]
10:48:35 INFO - 1547146115192 Marionette DEBUG 7 -> [0,19,"WebDriver:SwitchToWindow",{"focus":true,"name":"2147483649"}]
10:48:35 INFO - 1547146115195 Marionette DEBUG 7 <- [1,19,null,{"value":null}]
10:48:35 INFO - 1547146115199 Marionette DEBUG 7 -> [0,20,"Marionette:SetContext",{"value":"content"}]
10:48:35 INFO - 1547146115201 Marionette DEBUG 7 <- [1,20,null,{"value":null}]
10:48:35 INFO - 1547146115205 Marionette DEBUG 7 -> [0,21,"Marionette:GetContext",{}]
10:48:35 INFO - 1547146115207 Marionette DEBUG 7 <- [1,21,null,{"value":"content"}]
10:48:35 INFO - 1547146115210 Marionette DEBUG 7 -> [0,22,"Marionette:SetContext",{"value":"chrome"}]
10:48:35 INFO - 1547146115212 Marionette DEBUG 7 <- [1,22,null,{"value":null}]
10:48:35 INFO - 1547146115216 Marionette DEBUG 7 -> [0,23,"WebDriver:ExecuteScript",{"script":"Components.utils.import("resource://gre/modules/Preferences.jsm");\n ... ontent"],"filename":"../../venv/lib/python2.7/site-packages/marionette_driver/marionette.py","sandbox":"default","line":870}]
10:48:35 INFO - [Child 2202, Main Thread] WARNING: '!gThread', file /builds/worker/workspace/build/src/xpcom/threads/nsTimerImpl.cpp, line 299
10:48:35 INFO - [Child 2202, Main Thread] WARNING: '!gThread', file /builds/worker/workspace/build/src/xpcom/threads/nsTimerImpl.cpp, line 299
10:48:35 INFO - 1547146115227 Marionette DEBUG 7 <- [1,23,null,{"value":null}]
10:48:35 INFO - 1547146115240 Marionette DEBUG 7 -> [0,24,"Marionette:SetContext",{"value":"content"}]
10:48:35 INFO - 1547146115244 Marionette DEBUG 7 <- [1,24,null,{"value":null}]
10:48:35 INFO - 1547146115261 Marionette DEBUG 7 -> [0,25,"WebDriver:DeleteSession",{}]
10:48:35 INFO - 1547146115268 Marionette DEBUG 7 <- [1,25,null,{"value":null}]

Comment 10

[Sam Sneddon [:gsnedders]]

That unexpected fail is part of bug 1510696; let's handle it there.

However, bug 1519195 was filed for "Intermittent testing/marionette/harness/marionette_harness/tests/unit/test_reftest.py TestReftest.test_url_comparison | AssertionError: u'PASS' != u'TIMEOUT'", which is part of this (but fixed in m-c by rolling this out).

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/0655bfda512a
Fix URL comparisons when loading reftests r=jgraham

Comment 12

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/mozilla-central/rev/0655bfda512a

Comment 13

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/15054 for changes under testing/web-platform/tests

Comment 14

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Upstream PR was closed without merging