1404647 - Intermittent PID 1487 | thread '<unnamed>' panicked at 'failed to create an OS RNG: Error { repr: Os { code: 5, message: "Input/output error" } }', /checkout/src/libcore/result.rs:860

Status: RESOLVED FIXED

(Remote Protocol :: Marionette, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: archaeopteryx [at] coole-files.de

https://treeherder.mozilla.org/logviewer.html#?job_id=134206438&repo=autoland

https://queue.taskcluster.net/v1/task/EBWFwPXZQoumGUmipGMTMw/runs/0/artifacts/public/logs/live_backing.log

[task 2017-09-30T16:37:54.574Z] 16:37:54     INFO - TEST-START | /webdriver/tests/minimize_window.py
[task 2017-09-30T16:37:54.598Z] 16:37:54     INFO - STDOUT: ============================= test session starts ==============================
[task 2017-09-30T16:37:54.599Z] 16:37:54     INFO - STDOUT: platform linux2 -- Python 2.7.12, pytest-2.9.1, py-1.4.31, pluggy-0.3.1 -- /builds/worker/workspace/build/venv/bin/python
[task 2017-09-30T16:37:54.599Z] 16:37:54     INFO - STDOUT: rootdir: /builds/worker/workspace/build/tests/web-platform/tests/webdriver/tests, inifile: 
[task 2017-09-30T16:37:54.600Z] 16:37:54     INFO - STDOUT: collecting ... 
[task 2017-09-30T16:37:54.622Z] 16:37:54     INFO - STDOUT: collected 10 items
[task 2017-09-30T16:37:54.624Z] 16:37:54     INFO - STDOUT: tests/web-platform/tests/webdriver/tests/minimize_window.py::test_no_browsing_context 
[task 2017-09-30T16:37:54.660Z] 16:37:54     INFO - PID 1487 | 1506789474655	Marionette	DEBUG	Register listener.js for window 2147483653
[task 2017-09-30T16:37:54.717Z] 16:37:54     INFO - PID 1487 | 1506789474706	Marionette	DEBUG	Register listener.js for window 2147483656
[task 2017-09-30T16:37:54.766Z] 16:37:54     INFO - STDOUT: PASSED
[task 2017-09-30T16:37:54.923Z] 16:37:54     INFO - STDOUT: tests/web-platform/tests/webdriver/tests/minimize_window.py::test_handle_prompt_dismiss_and_notify 
[task 2017-09-30T16:37:54.925Z] 16:37:54     INFO - STDOUT: PASSED
[task 2017-09-30T16:37:54.925Z] 16:37:54     INFO - STDOUT: tests/web-platform/tests/webdriver/tests/minimize_window.py::test_handle_prompt_accept_and_notify 
[task 2017-09-30T16:37:54.927Z] 16:37:54     INFO - STDOUT: PASSED
[task 2017-09-30T16:37:54.928Z] 16:37:54     INFO - STDOUT: tests/web-platform/tests/webdriver/tests/minimize_window.py::test_handle_prompt_ignore 
[task 2017-09-30T16:37:54.929Z] 16:37:54     INFO - STDOUT: PASSED
[task 2017-09-30T16:37:54.930Z] 16:37:54     INFO - STDOUT: tests/web-platform/tests/webdriver/tests/minimize_window.py::test_handle_prompt_accept 
[task 2017-09-30T16:37:54.931Z] 16:37:54     INFO - PID 1487 | 1506789474924	Marionette	INFO	New connections will no longer be accepted
[task 2017-09-30T16:37:55.097Z] 16:37:55     INFO - PID 1487 | [Child 1967, Chrome_ChildThread] WARNING: pipe error (3): Connection reset by peer: file /builds/worker/workspace/build/src/ipc/chromium/src/chrome/common/ipc_channel_posix.cc, line 353
[task 2017-09-30T16:37:55.098Z] 16:37:55     INFO - PID 1487 | Sandbox: Unexpected EOF, op 0 flags 02000000 path /dev/urandom
[task 2017-09-30T16:37:55.098Z] 16:37:55    ERROR - PID 1487 | thread '<unnamed>' panicked at 'failed to create an OS RNG: Error { repr: Os { code: 5, message: "Input/output error" } }', /checkout/src/libcore/result.rs:860

Comment 1

[Intermittent Failures Robot]

13 failures in 885 pushes (0.015 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 7
* mozilla-inbound: 3
* mozilla-central: 3

Platform breakdown:
* linux64: 13

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1404647&startday=2017-09-25&endday=2017-10-01&tree=all

Comment 3

[James Graham [:jgraham]]

This seems to have started rather recently, can we investigate if there was a rustup, or a change in the machine image that might be associated with this.

Comment 4

[Intermittent Failures Robot]

30 failures in 144 pushes (0.208 failures/push) were associated with this bug yesterday.    

Repository breakdown:
* autoland: 16
* mozilla-inbound: 11
* mozilla-central: 3

Platform breakdown:
* linux64: 30

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1404647&startday=2017-10-02&endday=2017-10-02&tree=all

Comment 5

[Jed Davis [:jld] ⟨⏰|UTC-7⟩ ⟦he/him⟧]

[task 2017-09-30T16:37:55.097Z] 16:37:55     INFO - PID 1487 | [Child 1967, Chrome_ChildThread] WARNING: pipe error (3): Connection reset by peer: file /builds/worker/workspace/build/src/ipc/chromium/src/chrome/common/ipc_channel_posix.cc, line 353
[task 2017-09-30T16:37:55.098Z] 16:37:55     INFO - PID 1487 | Sandbox: Unexpected EOF, op 0 flags 02000000 path /dev/urandom
[task 2017-09-30T16:37:55.098Z] 16:37:55    ERROR - PID 1487 | thread '<unnamed>' panicked at 'failed to create an OS RNG: Error { repr: Os { code: 5, message: "Input/output error" } }', /checkout/src/libcore/result.rs:860


The content process is sandboxed; when it tries to open /dev/urandom, this is handled by messaging a broker in the parent process (the content process can no longer open files by itself).  However, this seems to be during shutdown and the parent process has already exited (see the IPC error on the first line), so that fails and the child gets EIO.  

The fundamental problem here seems to be that the content process is still running around and trying to do (apparently?) Stylo things after the parent process has exited.  So this bug probably belongs in Core :: DOM: Content Processes, but the patches in bug 1385971 might take care of this particular instance of the problem and make it a moot point (for now).

Comment 6

[Intermittent Failures Robot]

31 failures in 173 pushes (0.179 failures/push) were associated with this bug yesterday.    

Repository breakdown:
* mozilla-inbound: 20
* autoland: 9
* mozilla-central: 2

Platform breakdown:
* linux64: 30
* linux64-nightly: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1404647&startday=2017-10-03&endday=2017-10-03&tree=all

Comment 7

[Joel Maher ( :jmaher ) (UTC -8)]

:ato, this is failing on pgo (nightly is pgo) only, which looks to me as almost perma fail- can we look into a fix for this sooner rather than later?

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Joel please see the previous comments. This should be not a problem anymore. At least OF doens't show any failures for today.

Comment 9

[Manish Goregaokar [:manishearth]]

Stylo should no longer be relying on an RNG, so this is probably no longer a problem.

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Manish, which bug exactly changed that?

Comment 11

[Manish Goregaokar [:manishearth]]

bug 1385971

Comment 12

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Fixed on central, and the uplift to beta is outstanding.

Comment 13

[Intermittent Failures Robot]

78 failures in 824 pushes (0.095 failures/push) were associated with this bug in the last 7 days. 

This is the #26 most frequent failure this week. 

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **   

Repository breakdown:
* mozilla-inbound: 37
* autoland: 29
* mozilla-central: 12

Platform breakdown:
* linux64: 76
* linux64-nightly: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1404647&startday=2017-10-02&endday=2017-10-08&tree=all

Comment 14

[BMO Automation]

Moving bug to Remote Protocol::Marionette component per bug 1815831.