1191877 - Intermittent OSX build Automation Error: mozprocess timed out after 2400 seconds running ['/tools/buildbot/bin/python', 'mach', '--log-no-times', 'build', '-v']

Reporter

Description

•

10 years ago

+++ This bug was initially created as a clone of Bug #1165763 +++ +++ This bug was initially created as a clone of Bug #1154377 +++ +++ This bug was initially created as a clone of Bug #1145507 +++

Comment hidden (Legacy TBPL/Treeherder Robot)

Chris AtLee [:catlee]

Comment 8

•

10 years ago

this looks like a buildsymbols problem

Flags: needinfo?(ted)

Comment hidden (Legacy TBPL/Treeherder Robot)

Gregory Szorc [:gps]

Comment 23

•

10 years ago

From comment #22: 11:09:46 INFO - 44623: Worker processing files: ('dist/universal/test-stage/cppunittest/TestJSONWriter.dSYM', 'dist/universal/test-stage/cppunittest/TestJSONWriter') 11:09:46 INFO - 44622: Worker processing files: ('dist/universal/test-stage/cppunittest/TestLineBreak.dSYM', 'dist/universal/test-stage/cppunittest/TestLineBreak') 11:09:47 INFO - 44623: Worker processing files: ('dist/universal/test-stage/cppunittest/TestLineBreak.dSYM', 'dist/universal/test-stage/cppunittest/TestLineBreak') 11:09:47 INFO - 44620: Worker processing files: ('dist/universal/test-stage/cppunittest/TestMacroArgs.dSYM', 'dist/universal/test-stage/cppunittest/TestMacroArgs') 11:09:47 INFO - Exception in thread Thread-2: 11:09:47 INFO - Traceback (most recent call last): 11:09:47 INFO - File "/tools/python/lib/python2.7/threading.py", line 551, in __bootstrap_inner 11:09:47 INFO - self.run() 11:09:47 INFO - File "/tools/python/lib/python2.7/threading.py", line 504, in run 11:09:47 INFO - self.__target(*self.__args, **self.__kwargs) 11:09:47 INFO - File "/tools/python/lib/python2.7/multiprocessing/pool.py", line 319, in _handle_tasks 11:09:47 INFO - put(task) 11:09:47 INFO - RuntimeError: dictionary changed size during iteration 11:21:43 INFO - 44613: Submitting jobs for files: ('dist/universal/firefox/Nightly.app/Contents/MacOS/XUL.dSYM', 'dist/universal/firefox/Nightly.app/Contents/MacOS/XUL') 12:01:43 INFO - Automation Error: mozprocess timed out after 2400 seconds running ['/tools/buildbot/bin/python', 'mach', '--log-no-times', 'build', '-v'] 12:01:43 ERROR - timed out after 2400 seconds of no output 12:01:43 ERROR - Return code: -9 12:01:43 WARNING - setting return code to 2 12:01:43 FATAL - 'mach build' did not run successfully. Please check log for errors. This is a bug in symbolstore.py.

Gregory Szorc [:gps]

Comment 24

•

10 years ago

So, there is a bug where multiple threads or something else modifies a dictionary during iteraction. This causes an exception in a thread. And, chances are the code for waiting on threads is also buggy in that it doesn't detect aborted threads.

Gregory Szorc [:gps]

Comment 25

•

10 years ago

This smells like https://bugs.python.org/issue23051. The bug was fixed in Python 2.7.10.

Gregory Szorc [:gps]

•

10 years ago

Pretty sure bug 1164816 fixed this.

Depends on: 1164816

Comment hidden (Legacy TBPL/Treeherder Robot)

log: https://treeherder.mozilla.org/logviewer.html#?repo=b2g-inbound&job_id=2754533 repository: b2g-inbound start_time: 2015-09-14T05:30:34 who: philringnalda[at]gmail[dot]com machine: bld-lion-r5-076 buildname: b2g_b2g-inbound_macosx64_gecko build revision: 842451239eb7 Failed to log stats. Exception = [Errno 1] _ssl.c:504: error:1407742E:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert protocol version CalledProcessError: Command '['hg', 'path', 'default']' returned non-zero exit status 1 CalledProcessError: Command '['hg', 'unbundle', '--traceback', 'https://ftp-ssl.mozilla.org/pub/mozilla.org/firefox/bundles/gu.hg']' returned non-zero exit status 255 CalledProcessError: Command '['hg', 'path', 'default']' returned non-zero exit status 1 CalledProcessError: Command '['hg', 'unbundle', '--traceback', 'https://ftp-ssl.mozilla.org/pub/mozilla.org/firefox/bundles/ht.hg']' returned non-zero exit status 255 CalledProcessError: Command '['hg', 'path', 'default']' returned non-zero exit status 1 CalledProcessError: Command '['hg', 'unbundle', '--traceback', 'https://ftp-ssl.mozilla.org/pub/mozilla.org/firefox/bundles/ha.hg']' returned non-zero exit status 255 CalledProcessError: Command '['hg', 'path', 'default']' returned non-zero exit status 1 CalledProcessError: Command '['hg', 'unbundle', '--traceback', 'https://ftp-ssl.mozilla.org/pub/mozilla.org/firefox/bundles/he.hg']' returned non-zero exit status 255 CalledProcessError: Command '['hg', 'path', 'default']' returned non-zero exit status 1 CalledProcessError: Command '['hg', 'unbundle', '--traceback', 'https://ftp-ssl.mozilla.org/pub/mozilla.org/firefox/bundles/ee.hg']' returned non-zero exit status 255 CalledProcessError: Command '['hg', 'path', 'default']' returned non-zero exit status 1 CalledProcessError: Command '['hg', 'unbundle', '--traceback', 'https://ftp-ssl.mozilla.org/pub/mozilla.org/firefox/bundles/et.hg']' returned non-zero exit status 255 CalledProcessError: Command '['hg', 'path', 'default']' returned non-zero exit status 1 CalledProcessError: Command '['hg', 'unbundle', '--traceback', 'https://ftp-ssl.mozilla.org/pub/mozilla.org/firefox/bundles/lij.hg']' returned non-zero exit status 255 CalledProcessError: Command '['hg', 'path', 'default']' returned non-zero exit status 1 CalledProcessError: Command '['hg', 'unbundle', '--traceback', 'https://ftp-ssl.mozilla.org/pub/mozilla.org/firefox/bundles/ig.hg']' returned non-zero exit status 255 CalledProcessError: Command '['hg', 'path', 'default']' returned non-zero exit status 1 CalledProcessError: Command '['hg', 'unbundle', '--traceback', 'https://ftp-ssl.mozilla.org/pub/mozilla.org/firefox/bundles/be.hg']' returned non-zero exit status 255 CalledProcessError: Command '['hg', 'path', 'default']' returned non-zero exit status 1 CalledProcessError: Command '['hg', 'unbundle', '--traceback', 'https://ftp-ssl.mozilla.org/pub/mozilla.org/firefox/bundles/ast.hg']' returned non-zero exit status 255 CalledProcessError: Command '['hg', 'path', 'default']' returned non-zero exit status 1 CalledProcessError: Command '['hg', 'unbundle', '--traceback', 'https://ftp-ssl.mozilla.org/pub/mozilla.org/firefox/bundles/wo.hg']' returned non-zero exit status 255 CalledProcessError: Command '['hg', 'path', 'default']' returned non-zero exit status 1 CalledProcessError: Command '['hg', 'unbundle', '--traceback', 'https://ftp-ssl.mozilla.org/pub/mozilla.org/firefox/bundles/fi.hg']' returned non-zero exit status 255 CalledProcessError: Command '['hg', 'path', 'default']' returned non-zero exit status 1 CalledProcessError: Command '['hg', 'unbundle', '--traceback', 'https://ftp-ssl.mozilla.org/pub/mozilla.org/firefox/bundles/hy-AM.hg']' returned non-zero exit status 255 IOError: [Errno 22] Invalid argument IOError: [Errno 22] Invalid argument IOError: [Errno 22] Invalid argument IOError: [Errno 22] Invalid argument OSError: [Errno 22] Invalid argument: 'dist/bin/jsapi-tests.dSYM/Contents/Info.plist' IOError: [Errno 22] Invalid argument Automation Error: mozprocess timed out after 2400 seconds running ['/tools/buildbot/bin/python', 'mach', '--log-no-times', 'build', '-v'] IOError: [Errno 22] Invalid argument IOError: [Errno 6] Device not configured timed out after 2400 seconds of no output IOError: [Errno 22] Invalid argument IOError: [Errno 6] Device not configured IOError: [Errno 6] Device not configured IOError: [Errno 22] Invalid argument Return code: -9 IOError: [Errno 6] Device not configured IOError: [Errno 6] Device not configured IOError: [Errno 6] Device not configured IOError: [Errno 22] Invalid argument IOError: [Errno 6] Device not configured IOError: [Errno 6] Device not configured IOError: [Errno 6] Device not configured 'mach build' did not run successfully. Please check log for errors. IOError: [Errno 22] Invalid argument IOError: [Errno 6] Device not configured IOError: [Errno 6] Device not configured IOError: [Errno 22] Invalid argument IOError: [Errno 22] Invalid argument IOError: [Errno 22] Invalid argument Running post_fatal callback... IOError: [Errno 22] Invalid argument IOError: [Errno 6] Device not configured IOError: [Errno 6] Device not configured IOError: [Errno 22] Invalid argument IOError: [Errno 22] Invalid argument IOError: [Errno 22] Invalid argument Exiting -1 IOError: [Errno 6] Device not configured IOError: [Errno 6] Device not configured IOError: [Errno 6] Device not configured IOError: [Errno 22] Invalid argument IOError: [Errno 22] Invalid argument IOError: [Errno 22] Invalid argument IOError: [Errno 22] Invalid argument IOError: [Errno 6] Device not configured IOError: [Errno 22] Invalid argument IOError: [Errno 6] Device not configured IOError: [Errno 22] Invalid argument IOError: [Errno 6] Device not configured # TBPL FAILURE # IOError: [Errno 6] Device not configured IOError: [Errno 6] Device not configured IOError: [Errno 6] Device not configured IOError: [Errno 22] Invalid argument IOError: [Errno 6] Device not configured IOError: [Errno 6] Device not configured IOError: [Errno 22] Invalid argument IOError: [Errno 6] Device not configured IOError: [Errno 6] Device not configured IOError: [Errno 6] Device not configured # TBPL FAILURE # IOError: [Errno 6] Device not configured IOError: [Errno 6] Device not configured IOError: [Errno 6] Device not configured IOError: [Errno 22] Invalid argument remoteFailed: [Failure instance: Traceback from remote host -- Traceback (most recent call last): exceptions.OSError: [Errno 6] Device not configured: '/builds/slave/b2g-in-m64_g-00000000000000000/properties' OSError: [Errno 6] Device not configured

Comment hidden (Legacy TBPL/Treeherder Robot)

Comment hidden (Intermittent Failures Robot)

Nobody; OK to take it and work on it

Assignee

Updated

•

7 years ago

Component: General Automation → General

Mihai Tabara [:mtabara]⌚️GMT

Comment 48

•

7 years ago

I think this bug outlived its usefulness. Closing this, feel free to reopen if I'm wrong.

Status: NEW → RESOLVED

Closed: 7 years ago

Resolution: --- → WONTFIX