1505142 - Permafail TvW /webdriver/tests/element_click/scroll_into_view.py | test_partially_visible_does_not_scroll[offset0] - In-test skip decorators are disallowed, please use WPT metadata to ignore tests.

Status: RESOLVED FIXED

(Remote Protocol :: Marionette, defect, P1)

Description

[Treeherder Bug Filer]

Filed by: ccoroiu [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=210149432&repo=mozilla-central

https://queue.taskcluster.net/v1/task/DG-IIMg3SOmLjRmqCVtQ4w/runs/0/artifacts/public/logs/live_backing.log

[task 2018-11-06T18:55:45.307Z] 18:55:45     INFO - TEST-PASS | /webdriver/tests/element_click/scroll_into_view.py | test_scroll_into_view 
[task 2018-11-06T18:55:45.308Z] 18:55:45     INFO - TEST-UNEXPECTED-ERROR | /webdriver/tests/element_click/scroll_into_view.py | test_partially_visible_does_not_scroll[offset0] - In-test skip decorators are disallowed, please use WPT metadata to ignore tests.
[task 2018-11-06T18:55:45.308Z] 18:55:45     INFO - TEST-OK | /webdriver/tests/element_click/scroll_into_view.py | took 420ms
[task 2018-11-06T18:55:45.373Z] 18:55:45     INFO - Closing logging queue
[task 2018-11-06T18:55:45.374Z] 18:55:45     INFO - queue closed
[task 2018-11-06T18:55:45.374Z] 18:55:45     INFO - Starting runner
[task 2018-11-06T18:55:45.405Z] 18:55:45     INFO - PID 4144 | 1541530545401	webdriver::httpapi	DEBUG	Creating routes
[task 2018-11-06T18:55:45.408Z] 18:55:45     INFO - PID 4144 | 1541530545405	geckodriver	DEBUG	Listening on 127.0.0.1:4444
[task 2018-11-06T18:55:45.908Z] 18:55:45     INFO - WebDriver HTTP server listening at http://127.0.0.1:4444/

Comment 1

[Intermittent Failures Robot]

23 failures in 617 pushes (0.037 failures/push) were associated with this bug yesterday.

Repository breakdown:
* mozilla-central: 12
* mozilla-inbound: 11

Platform breakdown:
* windows10-64: 6
* osx-10-10: 4
* windows7-32: 6
* linux64: 7

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1505142&startday=2018-11-06&endday=2018-11-06&tree=all

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+2]]

This test was added by Andreas on bug 1493115, and it permafails for webdriver specific tests in TvW. A quick skim at the patch didn't show me anything obvious. We don't use a skip in the test file itself. So not sure where this comes from.

Andreas, please have a look to prevent a backout of the patch. Thanks.

Comment 3

[Andreas Tolfsen ❲:ato❳]

I think the problem here is that reversed(range(1, 10)) returns an
iterator.  Let’s make it a literal list to see if that helps.

Comment 4

[Andreas Tolfsen ❲:ato❳]

Ah, so this can reproduced with @pytest.mark.parametrize("offset",
[]), which means the iterator returned by reversed() finally returns
None, which WPT interprets as "In-test skip decorators are disallowed,
please use WPT metadata to ignore tests.".

Comment 5

[Andreas Tolfsen ❲:ato❳]

Attachment: bug 1505142: webdriver: replace iterator with literal list;

The iterator returned by reversed() can apparently return None in
some configurations.  WPT treats this as an in-test skip and errors
the test.  To avoid this configuration-specific problem we use a
literal list.

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Andreas Tolfsen ❲:ato❳ from comment #4)
> Ah, so this can reproduced with @pytest.mark.parametrize("offset",
> []), which means the iterator returned by reversed() finally returns
> None, which WPT interprets as "In-test skip decorators are disallowed,
> please use WPT metadata to ignore tests.".

Interesting. Isn't that a bug in wptrunner? Also why doens't this show up in our normal wdspec jobs?

Comment 7

[Andreas Tolfsen ❲:ato❳]

I don’t know.

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Joel, can you please have a look at the previous comments? We wonder why TvW is failing here while Wd jobs are just fine. Thanks!

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/31930943e22e
webdriver: replace iterator with literal list; r=whimboo

Comment 10

[Joel Maher ( :jmaher ) (UTC -8)]

I don't know why TvW would fail, is there a custom way Wd jobs are run that maybe we are not able to do with TvW?  TvW essentially runs the test in a single test mode- maybe the repeated test runs or chaos mode would cause it to fail.

Comment 11

[Arthur Iakab [arthur_iakab]]

https://hg.mozilla.org/mozilla-central/rev/31930943e22e

Comment 12

[Intermittent Failures Robot]

23 failures in 3007 pushes (0.008 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-central: 12
* mozilla-inbound: 11

Platform breakdown:
* osx-10-10: 4
* linux64: 7
* windows7-32: 6
* windows10-64: 6

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1505142&startday=2018-11-05&endday=2018-11-11&tree=all

Comment 13

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/14056 for changes under testing/web-platform/tests

Comment 14

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Upstream PR merged

Comment 15

[BMO Automation]

Moving bug to Remote Protocol::Marionette component per bug 1815831.