1417189 - Intermittent IOError: [Errno 109] The pipe has been ended

Status: RESOLVED FIXED

(Testing :: General, defect, P1)

Description

[Treeherder Bug Filer]

Filed by: archaeopteryx [at] coole-files.de

https://treeherder.mozilla.org/logviewer.html#?job_id=144680687&repo=mozilla-inbound

https://queue.taskcluster.net/v1/task/YfYCHZ2DS62End-lmrF-OA/runs/0/artifacts/public/logs/live_backing.log

https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://queue.taskcluster.net/v1/task/YfYCHZ2DS62End-lmrF-OA/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1

18:27:53     INFO - Running pre test command run mouse & screen adjustment script with 'c:\mozilla-build\python\python.exe Z:\task_1510683682\mozharness\external_tools\mouse_and_screen_resolution.py --configuration-file Z:\task_1510683682\mozharness\external_tools\machine-configuration.json'
18:27:53     INFO - Running command: ('c:\\mozilla-build\\python\\python.exe', 'Z:\\task_1510683682\\mozharness\\external_tools\\mouse_and_screen_resolution.py', '--configuration-file', 'Z:\\task_1510683682\\mozharness\\external_tools\\machine-configuration.json') in Z:\task_1510683682\build
18:27:53     INFO - Copy/paste: c:\mozilla-build\python\python.exe Z:\task_1510683682\mozharness\external_tools\mouse_and_screen_resolution.py --configuration-file Z:\task_1510683682\mozharness\external_tools\machine-configuration.json
18:27:56     INFO -  Screen resolution (current): (1600, 1200)
18:27:56     INFO -  Changing the screen resolution...
18:27:56     INFO -  Screen resolution (new): (1280, 1024)
18:27:56     INFO -  Mouse position (current): (640, 512)
18:27:56     INFO -  Mouse position (new): (1010, 10)
18:27:56     INFO - Return code: 0
18:27:56     INFO - mkdir: Z:\task_1510683682\build\blobber_upload_dir
18:29:18     INFO - Unable to complete system-info.log: <type 'exceptions.WindowsError'>
18:29:18     INFO - #### Running reftest suites
[taskcluster 2017-11-14T18:29:20.735Z]   Exit Code: 3221225477

Comment 1

[Geoff Brown [:gbrown]]

Actually, I'm not sure that "Unable to complete system-info.log" is related to the task failure. I expect system-info.log generation to fail sometimes and generate that message, but I would not expect it to cause trouble.

Comment 2

[Intermittent Failures Robot]

48 failures in 166 pushes (0.289 failures/push) were associated with this bug yesterday.    

Repository breakdown:
* mozilla-inbound: 34
* autoland: 9
* mozilla-central: 3
* try: 2

Platform breakdown:
* windows7-32: 41
* windows7-32-stylo-disabled: 6
* windows7-32-nightly: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1417189&startday=2017-11-15&endday=2017-11-15&tree=all

Comment 3

[Geoff Brown [:gbrown]]

Attachment: try to avoid IOError generating system-info.log

I don't understand what's going wrong here. The IOError is coming from resourcemanager, which also uses psutil...maybe there is some funny interaction between the two callers of psutil??

I notice that this only happens on Windows and, from the system-info perspective, we fail when querying the list of users. So, as a speculative fix, let's not worry about getting the users on Windows. (I'm cleaning up a little too, so there are fewer nested exception handlers in use.)

Comment 4

[Geoff Brown [:gbrown]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=97e03b009f344df7db7f1406eb64f10c2f037016

Comment 5

[Intermittent Failures Robot]

19 failures in 116 pushes (0.164 failures/push) were associated with this bug yesterday.    

Repository breakdown:
* autoland: 10
* mozilla-inbound: 7
* try: 1
* mozilla-central: 1

Platform breakdown:
* windows7-32: 16
* windows7-32-stylo-disabled: 3

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1417189&startday=2017-11-16&endday=2017-11-16&tree=all

Comment 6

[Joel Maher ( :jmaher ) (UTC -8) (PTO back normal Nov 17)]

Comment on attachment 8929214 [details] [diff] [review]
try to avoid IOError generating system-info.log

Review of attachment 8929214 [details] [diff] [review]:
-----------------------------------------------------------------

this seems slightly better- we can assess where we are at after this is landed and we get more data.

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/62b42bf135c1
Try to avoid intermittent IOError following system-log generation; r=jmaher

Comment 8

[Geoff Brown [:gbrown]]

Leaving open to better track status if the intermittent persists.

Comment 9

[Andrei Ciure[:aciure]]

https://hg.mozilla.org/mozilla-central/rev/62b42bf135c1

Comment 10

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/7632efd3f660

Comment 11

[Intermittent Failures Robot]

91 failures in 762 pushes (0.119 failures/push) were associated with this bug in the last 7 days. 

This is the #4 most frequent failure this week. 

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **   

Repository breakdown:
* mozilla-inbound: 46
* autoland: 32
* mozilla-central: 10
* try: 3

Platform breakdown:
* windows7-32: 76
* windows7-32-stylo-disabled: 14
* windows7-32-nightly: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1417189&startday=2017-11-13&endday=2017-11-19&tree=all

Comment 12

[Geoff Brown [:gbrown]]

No new failures since my speculative fix landed.

Comment 13

[Intermittent Failures Robot]

3 failures in 423 pushes (0.007 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 2
* mozilla-inbound: 1

Platform breakdown:
* windows10-64: 3

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1417189&startday=2017-12-11&endday=2017-12-17&tree=all

Comment 14

[Intermittent Failures Robot]

4 failures in 590 pushes (0.007 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 4

Platform breakdown:
* windows10-64: 4

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1417189&startday=2017-12-18&endday=2017-12-24&tree=all