Intermittent TEST-UNEXPECTED-TIMEOUT | /webdriver/actions/key.py | expected OK

RESOLVED FIXED in Firefox 55

Status

Testing
web-platform-tests
RESOLVED FIXED
7 months ago
4 months ago

People

(Reporter: Treeherder Bug Filer, Assigned: jgraham)

Tracking

({intermittent-failure})

Version 3
mozilla56
intermittent-failure
Points:
---

Firefox Tracking Flags

(firefox55 fixed, firefox56 fixed)

Details

(Whiteboard: [stockwell fixed:other])

MozReview Requests

Submitter Diff Changes Open Issues Last Updated
Loading...
Error loading review requests:

Attachments

(1 attachment)

(Reporter)

Description

7 months ago
treeherder
Filed by: philringnalda [at] gmail.com

https://treeherder.mozilla.org/logviewer.html#?job_id=89764655&repo=autoland

https://queue.taskcluster.net/v1/task/GsDioRxQQmyda5fcPKX2Tw/runs/0/artifacts/public/logs/live_backing.log
Duplicate of this bug: 1354751

Comment 2

4 months ago
18 failures in 153 pushes (0.118 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 15
* mozilla-central: 2
* mozilla-inbound: 1

Platform breakdown:
* linux64: 18

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1354750&startday=2017-06-09&endday=2017-06-09&tree=all

Comment 3

4 months ago
32 failures in 864 pushes (0.037 failures/push) were associated with this bug in the last 7 days. 

This is the #49 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. ** 

Repository breakdown:
* autoland: 22
* mozilla-central: 5
* mozilla-inbound: 4
* try: 1

Platform breakdown:
* linux64: 32

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1354750&startday=2017-06-05&endday=2017-06-11&tree=all

Comment 4

4 months ago
24 failures in 142 pushes (0.169 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* mozilla-inbound: 10
* autoland: 9
* mozilla-central: 4
* mozilla-beta: 1

Platform breakdown:
* linux64: 23
* linux64-nightly: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1354750&startday=2017-06-12&endday=2017-06-12&tree=all

Comment 5

4 months ago
18 failures in 150 pushes (0.12 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* mozilla-inbound: 5
* autoland: 5
* mozilla-central: 4
* mozilla-beta: 3
* pine: 1

Platform breakdown:
* linux64: 15
* linux64-nightly: 2
* linux64-devedition: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1354750&startday=2017-06-13&endday=2017-06-13&tree=all

Comment 6

4 months ago
18 failures in 168 pushes (0.107 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 9
* mozilla-inbound: 5
* try: 2
* mozilla-beta: 2

Platform breakdown:
* linux64: 16
* linux64-nightly: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1354750&startday=2017-06-14&endday=2017-06-14&tree=all
:jgraham- can you look at this given that this is failing often every day.
Flags: needinfo?(james)
Whiteboard: [stockwell needswork]

Comment 8

4 months ago
20 failures in 131 pushes (0.153 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 7
* mozilla-inbound: 5
* mozilla-central: 5
* try: 2
* mozilla-beta: 1

Platform breakdown:
* linux64: 19
* linux64-devedition: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1354750&startday=2017-06-15&endday=2017-06-15&tree=all

Comment 9

4 months ago
18 failures in 147 pushes (0.122 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* mozilla-inbound: 9
* autoland: 5
* mozilla-beta: 3
* try: 1

Platform breakdown:
* linux64: 16
* linux64-nightly: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1354750&startday=2017-06-16&endday=2017-06-16&tree=all

Comment 10

4 months ago
112 failures in 814 pushes (0.138 failures/push) were associated with this bug in the last 7 days. 

This is the #10 most frequent failure this week. 

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **  

Repository breakdown:
* autoland: 40
* mozilla-inbound: 37
* mozilla-central: 15
* mozilla-beta: 11
* try: 7
* pine: 2

Platform breakdown:
* linux64: 102
* linux64-nightly: 8
* linux64-devedition: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1354750&startday=2017-06-12&endday=2017-06-18&tree=all
(Assignee)

Comment 11

4 months ago
It looks like maybe when this fails the test file isn't loading at all? Otherwise I would expect some intermediate output. Also increasing the timeout doesn't help. If that's true disabling it *might* just make the following test intermittent in the same way.

Comment 12

4 months ago
21 failures in 151 pushes (0.139 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* mozilla-inbound: 7
* mozilla-beta: 7
* autoland: 5
* try: 1
* mozilla-central: 1

Platform breakdown:
* linux64: 15
* linux64-nightly: 4
* linux64-devedition: 1
* linux64-ccov: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1354750&startday=2017-06-19&endday=2017-06-19&tree=all
(Assignee)

Comment 13

4 months ago
Something bad happens in pytest startup so we never start running tests. I'm pretty sure disabling won't have any impact here except to change the test that's unstable. I'm doing an excessie number of try runs to figure out what's up.
Flags: needinfo?(james)
(Assignee)

Updated

4 months ago
Flags: needinfo?(james)
(Assignee)

Comment 14

4 months ago
So so there's good news and bad news. The good news is I think I know what the problem is. The bad news is what the problem is. 

So I think that what's happening here is a nasty interaction between multiprocessing and threading. On Linux multiprocessing calls fork() which provides a child process with a copy (well copy-on-write) of the parent process memory. That usually works fine. However in the case that you have a mutex in the parent process that happens to be locked at the time of the fork, and you attempt to acquire that same mutex in the child process it will now always appear locked because even if it were released in the parent that change is never reflected in the child's copy. In our case we see an intermittent deadlock when trying to set up the structured logger from pytest, despite the fact that there's only a single thread in that process. Keying the lock itself on pid makes the problem go away, but isn't a very general solution because there could be other bits of code that will fail in the same way at some later time.

I'm not sure what the best solution here is other than "don't use threading and multiprocessing together" (needless to say I didn't know about this issue when the code was written). Possibly landing some hack in the short term is a good idea.

Comment 15

4 months ago
23 failures in 173 pushes (0.133 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* mozilla-inbound: 10
* autoland: 6
* mozilla-central: 4
* try: 2
* mozilla-beta: 1

Platform breakdown:
* linux64: 22
* linux64-ccov: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1354750&startday=2017-06-20&endday=2017-06-20&tree=all
I am glad there is some news about this- do you think we can get a hack in this week, or should we disable this test (or wdspec) job until we can get a hack landed?
(Assignee)

Comment 17

4 months ago
Disabling this test won't help at all. Disabling the entire suite would be bad for other reasons. You would have to make the job Tier < 1. But I have an idea for a better solution that should be low effort (basically there's a backported version of the Python 3 multiprocessing module that allows you to use multiple processes in a way that doesn't hillariously violate posix semantics), so I'll try that.
Comment hidden (mozreview-request)

Comment 19

4 months ago
16 failures in 170 pushes (0.094 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 8
* mozilla-inbound: 7
* try: 1

Platform breakdown:
* linux64: 16

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1354750&startday=2017-06-21&endday=2017-06-21&tree=all

Comment 20

4 months ago
mozreview-review
Comment on attachment 8880151 [details]
Bug 1354750 - Disable loading mozlog plugin with pytest for wpt,

https://reviewboard.mozilla.org/r/151526/#review156614
Attachment #8880151 - Flags: review?(ato) → review+
Comment hidden (mozreview-request)

Comment 22

4 months ago
Pushed by james@hoppipolla.co.uk:
https://hg.mozilla.org/integration/autoland/rev/a7e6d0b5fbdd
Disable loading mozlog plugin with pytest for wpt, r=ato
(Assignee)

Updated

4 months ago
Flags: needinfo?(james)

Comment 23

4 months ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/a7e6d0b5fbdd
Status: NEW → RESOLVED
Last Resolved: 4 months ago
status-firefox56: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla56
should we uplift this to beta?
Whiteboard: [stockwell needswork] → [stockwell fixed:other]
Assignee: nobody → james

Comment 25

4 months ago
bugherderuplift
https://hg.mozilla.org/releases/mozilla-beta/rev/b4bd21b3d861
status-firefox55: --- → fixed

Comment 26

4 months ago
72 failures in 892 pushes (0.081 failures/push) were associated with this bug in the last 7 days. 

This is the #24 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. ** 

Repository breakdown:
* mozilla-inbound: 27
* autoland: 19
* mozilla-beta: 12
* try: 8
* mozilla-central: 6

Platform breakdown:
* linux64: 62
* linux64-nightly: 4
* linux64-devedition: 3
* linux64-ccov: 2
* windows10-64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1354750&startday=2017-06-19&endday=2017-06-25&tree=all

Comment 27

4 months ago
2 failures in 718 pushes (0.003 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-release: 1
* autoland: 1

Platform breakdown:
* linux64-nightly: 1
* linux64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1354750&startday=2017-06-26&endday=2017-07-02&tree=all
You need to log in before you can comment on or make changes to this bug.