Closed Bug 920153 Opened 12 years ago Closed 11 years ago

Cloning of hg.mozilla.org/build/tools and hg.mozilla.org/integration/gaia-central often times out, as does downloading/unzipping test zips

Categories

(Release Engineering :: General, defect)

Product:
Release Engineering
Component:
General
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: emorley, Assigned: mozilla)

References

Details

(Keywords: intermittent-failure, sheriffing-P1)

Attachments

(8 files, 1 obsolete file)

mercurial_timeout_test.py
1.05 KB, text/plain
Details
buildbotcustom default_timeout_1800
864 bytes, patch
catlee
: review+
mozilla
: checked-in+
Details | Diff | Splinter Review
timeout_fixes
4.54 KB, patch
jgriffin
: review+
mozilla
: checked-in+
Details | Diff | Splinter Review
download_timeout_test.py
1.00 KB, patch
catlee
: review+
Details | Diff | Splinter Review
download_timeout
8.33 KB, patch
catlee
: review+
mozilla
: checked-in-
Details | Diff | Splinter Review
add a vcs_output_timeout to |hg pull|
1.27 KB, patch
jgriffin
: review+
mozilla
: checked-in+
Details | Diff | Splinter Review
output timeout for gaia git clone
730 bytes, patch
jgriffin
: review+
mozilla
: checked-in+
Details | Diff | Splinter Review
unzip_timeout
1.20 KB, patch
jgriffin
: review+
mozilla
: checked-in+
Details | Diff | Splinter Review
eg: https://tbpl.mozilla.org/php/getParsedLog.php?id=28094665&tree=Mozilla-Inbound { 08:00:05 INFO - Running command: ['tar', 'zxf', '/builds/slave/test/build/emulator.tar.gz'] in /builds/slave/test/build/emulator 08:00:05 INFO - Copy/paste: tar zxf /builds/slave/test/build/emulator.tar.gz 08:00:10 INFO - Return code: 0 08:00:10 INFO - retry: Calling <bound method B2GEmulatorTest._get_revision of <__main__.B2GEmulatorTest object at 0x1fa3f50>> with args: (<mozharness.base.vcs.mercurial.MercurialVCS object at 0x201b450>, '/builds/slave/test/build/tools'), kwargs: {}, attempt #1 08:00:10 INFO - Setting /builds/slave/test/build/tools to http://hg.mozilla.org/build/tools. 08:00:10 INFO - Cloning http://hg.mozilla.org/build/tools to /builds/slave/test/build/tools. 08:00:10 INFO - Running command: ['hg', '--config', 'ui.merge=internal:merge', 'clone', 'http://hg.mozilla.org/build/tools', '/builds/slave/test/build/tools'] 08:00:10 INFO - Copy/paste: hg --config ui.merge=internal:merge clone http://hg.mozilla.org/build/tools /builds/slave/test/build/tools command timed out: 1200 seconds without output, attempting to kill process killed by signal 9 program finished with exit code -1 elapsedTime=1246.914894 ========= Finished '/tools/buildbot/bin/python scripts/scripts/b2g_emulator_unittest.py ...' failed (results: 2, elapsed: 20 mins, 55 secs) (at 2013-09-19 08:20:19.665482) ========= } Expected: * Few retries of the hg clone * "Automation Error: Unable to clone build tools repo" (and no "command timed out: 1200 seconds without output, attempting to kill"). I thought bug 840305 had fixed this..?
Keywords: intermittent-failure
Summary: Cloning of build tools does not retry and/or output a TBPL compatible failure message → Cloning of build tools does not retry and/or output a TBPL compatible failure message ("command timed out: 1200 seconds without output, attempting to kill")
Blocks: 840305
There's also this (non-retrying) failure mode: https://tbpl.mozilla.org/php/getParsedLog.php?id=28035195&tree=Mozilla-Inbound { 05:26:02 INFO - ##### 05:26:02 INFO - ##### Running pull step. 05:26:02 INFO - ##### 05:26:02 INFO - Running pre-action listener: _resource_record_pre_action 05:26:02 INFO - Running main action method: pull 05:26:02 INFO - Changing directory to /builds/slave/talos-slave/test/build. 05:26:02 INFO - retry: Calling <bound method DesktopUnittest._get_revision of <__main__.DesktopUnittest object at 0x1010dc7d0>> with args: (<mozharness.base.vcs.mercurial.MercurialVCS object at 0x100409fd0>, 'tools'), kwargs: {}, attempt #1 05:26:02 INFO - Setting /builds/slave/talos-slave/test/build/tools to http://hg.mozilla.org/build/tools. 05:26:02 INFO - Cloning http://hg.mozilla.org/build/tools to /builds/slave/talos-slave/test/build/tools. 05:26:02 INFO - Getting output from command: ['/builds/slave/talos-slave/test/build/venv/bin/python', '-c', 'from distutils.sysconfig import get_python_lib; print(get_python_lib())'] 05:26:02 INFO - Copy/paste: /builds/slave/talos-slave/test/build/venv/bin/python -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())" 05:26:02 INFO - Copy/paste: /builds/slave/talos-slave/test/build/venv/bin/python -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())" 05:26:02 INFO - Reading from file tmpfile_stdout 05:26:02 INFO - Output received: 05:26:02 INFO - /builds/slave/talos-slave/test/build/venv/lib/python2.7/site-packages 05:26:02 INFO - retry: Calling <built-in function remove> with args: ('tmpfile_stderr',), kwargs: {}, attempt #1 05:26:02 INFO - retry: Calling <built-in function remove> with args: ('tmpfile_stdout',), kwargs: {}, attempt #1 05:26:02 INFO - Running command: ['hg', '--config', 'ui.merge=internal:merge', 'clone', 'http://hg.mozilla.org/build/tools', '/builds/slave/talos-slave/test/build/tools'] 05:26:02 INFO - Copy/paste: hg --config ui.merge=internal:merge clone http://hg.mozilla.org/build/tools /builds/slave/talos-slave/test/build/tools 05:26:02 INFO - Calling ['hg', '--config', 'ui.merge=internal:merge', 'clone', 'http://hg.mozilla.org/build/tools', '/builds/slave/talos-slave/test/build/tools'] with output_timeout 1000 05:42:42 INFO - mozprocess timed out 05:42:42 ERROR - timed out after 1000 seconds of no output 05:42:42 ERROR - Return code: 9 }
Keywords: sheriffing-P1
Chris, is there someone that could take a look at this for us? :-)
Flags: needinfo?(catlee)
Should we add --verbose and --debug to the clone command to diagnose what is going on?
I was going to file on the gaia tests cloning https://hg.mozilla.org//integration/gaia-central, but I guess we long since decided to just throw them in here.
Summary: Cloning of build tools does not retry and/or output a TBPL compatible failure message ("command timed out: 1200 seconds without output, attempting to kill") → Cloning of build tools and gaia repo does not retry and/or output a TBPL compatible failure message ("command timed out: 1200 seconds without output, attempting to kill")
Also doesn't use the hg share, the hg mirror, or a bundle (which admittedly doesn't exist). This is not at all wonderful given gaia-central has a .hg more than a GB in size, which approaches repos like mozilla-central.
But at least it gives us a handy indicator for the slightly reduced phase of bug 957502, where we don't get the failures downloading from ftp.m.o but we do get this going from one every eight hours to eight an hour.
Blocks: 960072
Breaking out the "output a TBPL compatible failure message & attempt to clone more than once before giving up" parts to other bugs, one for build tools cloning: https://tbpl.mozilla.org/php/getParsedLog.php?id=32237843&tree=Mozilla-Inbound { 12:50:20 INFO - retry: Calling <bound method B2GDesktopTest._get_revision of <__main__.B2GDesktopTest object at 0x23f45d0>> with args: (<mozharness.base.vcs.mercurial.MercurialVCS object at 0x2403910>, '/builds/slave/test/build/tools'), kwargs: {}, attempt #1 12:50:20 INFO - Setting /builds/slave/test/build/tools to http://hg.mozilla.org/build/tools. 12:50:20 INFO - Cloning http://hg.mozilla.org/build/tools to /builds/slave/test/build/tools. 12:50:20 INFO - Running command: ['hg', '--config', 'ui.merge=internal:merge', 'clone', 'http://hg.mozilla.org/build/tools', '/builds/slave/test/build/tools'] 12:50:20 INFO - Copy/paste: hg --config ui.merge=internal:merge clone http://hg.mozilla.org/build/tools /builds/slave/test/build/tools command timed out: 1200 seconds without output, attempting to kill } And the other for gaia-central cloning: https://tbpl.mozilla.org/php/getParsedLog.php?id=33150237&tree=B2g-Inbound { 22:01:35 INFO - ##### 22:01:35 INFO - ##### Running pull step. 22:01:35 INFO - ##### 22:01:35 INFO - Running pre-action listener: _resource_record_pre_action 22:01:35 INFO - Running main action method: pull 22:01:35 INFO - retry: Calling <bound method GaiaIntegrationTest.load_json_from_url of <__main__.GaiaIntegrationTest object at 0x995bfcc>> with args: ('https://hg.mozilla.org/integration/b2g-inbound/raw-file/328bad2599f2/b2g/config/gaia.json',), kwargs: {}, attempt #1 22:01:41 INFO - Changing directory to /builds/slave/test. 22:01:41 INFO - retry: Calling <bound method GaiaIntegrationTest._get_revision of <__main__.GaiaIntegrationTest object at 0x995bfcc>> with args: (<mozharness.base.vcs.mercurial.MercurialVCS object at 0x99352ec>, '/builds/slave/test/gaia'), kwargs: {}, attempt #1 22:01:41 INFO - Setting /builds/slave/test/gaia to https://hg.mozilla.org//integration/gaia-central revision 9e00ea980c1de438590396b5d990b3a567d2edc6. 22:01:41 INFO - Cloning https://hg.mozilla.org//integration/gaia-central to /builds/slave/test/gaia. 22:01:41 INFO - Running command: ['hg', '--config', 'ui.merge=internal:merge', 'clone', 'https://hg.mozilla.org//integration/gaia-central', '/builds/slave/test/gaia'] 22:01:41 INFO - Copy/paste: hg --config ui.merge=internal:merge clone https://hg.mozilla.org//integration/gaia-central /builds/slave/test/gaia command timed out: 1200 seconds without output, attempting to kill }
Summary: Cloning of build tools and gaia repo does not retry and/or output a TBPL compatible failure message ("command timed out: 1200 seconds without output, attempting to kill") → Cloning of hg.mozilla.org/build/tools and hg.mozilla.org/integration/gaia-central often times out with "command timed out: 1200 seconds without output, attempting to kill"
Depends on: 961048
Hah, as if 1800 seconds would be enough! https://tbpl.mozilla.org/php/getParsedLog.php?id=33335100&tree=Mozilla-Inbound
Summary: Cloning of hg.mozilla.org/build/tools and hg.mozilla.org/integration/gaia-central often times out with "command timed out: 1200 seconds without output, attempting to kill" → Cloning of hg.mozilla.org/build/tools and hg.mozilla.org/integration/gaia-central often times out with "command timed out: 1200 seconds without output, attempting to kill", "command timed out: 1800 seconds without output, attempting to kill"
One interesting possibility is that the awful gaia-central parts of this (not only no retry, but no bundle, no share) are actually the cause of the every-night multiple-hour closures as we clog the VPN. It's not at all impossible that we start out by clogging it with gaia-central clones, and then build up to clogging ftp.m.o downloads.
Depends on: 964411
Depends on: 965519
Assignee: nobody → aki
Flags: needinfo?(catlee)
This is a quick test script to make sure that the pull() method retries, by cloning mozilla-central with a 5 second idle timeout. Verified: 00:18:49 INFO - Running command: ['hg', '--config', 'ui.merge=internal:merge', 'clone', 'https://hg.mozilla.org/mozilla-central', '/src/tools-timeout/build/mozilla-central'] [22/365] 00:18:49 INFO - Copy/paste: hg --config ui.merge=internal:merge clone https://hg.mozilla.org/mozilla-central /src/tools-timeout/build/mozilla-central 00:18:49 INFO - Calling ['hg', '--config', 'ui.merge=internal:merge', 'clone', 'https://hg.mozilla.org/mozilla-central', '/src/tools-timeout/build/mozilla-central'] with output_timeout 5 00:18:54 INFO - mozprocess timed out 00:18:54 ERROR - timed out after 5 seconds of no output 00:18:54 ERROR - Return code: -9 00:18:54 INFO - rmtree: mozilla-central 00:18:54 INFO - retry: Calling <function rmtree at 0x10c0fc8c0> with args: ('mozilla-central',), kwargs: {}, attempt #1 00:18:54 INFO - retry: Failed, sleeping 60 seconds before retrying Adding vcs_output_timeout to various scripts first.
(In reply to TBPL Robot from comment #692) > Tomcat > https://tbpl.mozilla.org/php/getParsedLog.php?id=34212654&tree=Mozilla- > Inbound > Ubuntu VM 12.04 mozilla-inbound opt test mochitest-1 on 2014-02-06 06:25:43 > revision: 42d91a4ff0b0 > slave: tst-linux32-spot-091 > > command timed out: 1200 seconds without output, attempting to kill Ok, this, and a number of other timeouts, is a download timeout, not a clone timeout. Still needs fixing, but it's a different fix.
(In reply to TBPL Robot from comment #689) > philor > https://tbpl.mozilla.org/php/getParsedLog.php?id=34192660&tree=Mozilla- > Inbound > b2g_ubuntu64_vm mozilla-inbound opt test gaia-unit on 2014-02-05 22:40:10 > revision: 9185c4a586df > slave: tst-linux64-ec2-118 > > command timed out: 1200 seconds without output, attempting to kill This is a timeout while running the tests.
Bumps the mozharness test default idle timeout to 1800 from 1200. This will give more time to clone gaia. (When this lands we may need to change this bug's summary for sheriffs)
Attachment #8372785 - Flags: review?(catlee)
(In reply to TBPL Robot from comment #648) > Tomcat > https://tbpl.mozilla.org/php/getParsedLog.php?id=33865343&tree=Mozilla- > Inbound > Windows 7 32-bit mozilla-inbound debug test reftest on 2014-01-30 23:45:31 > revision: 2a7a697f2d88 > slave: t-w732-ix-071 > > command timed out: 1200 seconds without output, attempting to kill This is actually an *unzip* that timed out *boggle*
Attached patch timeout_fixes — — Splinter Review
I'm not even going to pretend I went through all those logs. These seem to cover the most recent few dozen.
Attachment #8372787 - Flags: review?(jgriffin)
Thank you for your work on all of these bugs Aki! <3 :-)
Attachment #8372785 - Flags: review?(catlee) → review+
Attachment #8372787 - Flags: review?(jgriffin) → review+
I have an idea for the download timeout: 1) split _download_file() into a helper script. Then we can call it via run_command() with an output_timeout. bonus points: try to import requests (adding the venv if hasattr('query_virtualenv_path')), and use that instead if it's available. I think #1 is faster to get to without the bonus points, though.
Alternately we can go straight to run_command(["wget", ...], output_timeout=1760) which gets us output_timeout and certificate checking. The downside is if certain machines fail on the certificate check, which is true of the osx and windows boxen iirc.
Comment on attachment 8372787 [details] [diff] [review] timeout_fixes http://hg.mozilla.org/build/mozharness/rev/0e4ebc409c16 Kicking off some tests on Cypress.
Attachment #8372787 - Flags: checked-in+
in production
Attached file download_timeout_test.py (obsolete) —
Test script to test download_file(), with a 2 second output timeout.
This moves _download_file() into external_tools/download_file.py (removing various mozharness-isms along the way), so we can call it with run_command() with an output_timeout. I added a PYTHONPATH in case we ever have requests support in download_file.py, but that's just a stub atm. The default output_timeout is 300 seconds atm. I'm not sure if that's a good default... a more conservative one might be 900, 1160, or 1760. If it doesn't end up burning things, though, it might save time waiting for broken downloads to time out. (We can always override this timeout per-download_file()-call, but that won't stop initial bustage.) I'm more confident that this will work than not, but not 100% confident this will roll out without bustage. This deserves at least a full Cypress test run at the least, and sheriff notification.
Attachment #8375294 - Flags: review?(catlee)
(In reply to TBPL Robot from comment #719) > KWierso > https://tbpl.mozilla.org/php/getParsedLog.php?id=34642416&tree=Fx-Team This is actually a mock timeout =\ Not covered by any of the patches in this bug. See comment 718 if the download patch gets review and we want to roll that out before I come back from PTO.
Assignee: aki → nobody
Attachment #8375294 - Flags: review?(catlee) → review+
Assignee: nobody → aki
Attachment #8375290 - Attachment is obsolete: true
Attached patch download_timeout — — Splinter Review
Oops, I meant to attach this patch for review =\
Attachment #8390762 - Flags: review?(catlee)
(In reply to TBPL Robot from comment #725) > Tomcat > https://tbpl.mozilla.org/php/getParsedLog.php?id=35987715&tree=Mozilla- > Central > Windows 7 32-bit mozilla-central debug test reftest on 2014-03-12 01:28:25 > revision: 44ae8462d6ab > slave: t-w732-ix-034 > > command timed out: 1800 seconds without output, attempting to kill This is an unzip timeout. Removing -q would make the logs more spammy but could help avoid the timeout. (In reply to TBPL Robot from comment #724) > KWierso > https://tbpl.mozilla.org/php/getParsedLog.php?id=35628077&tree=Jetpack > jetpack-fx-team-win7-ix-opt on 2014-03-04 16:52:43 > revision: 0d8d87176b7a > slave: t-w732-ix-069 > > command timed out: 1200 seconds without output, attempting to kill > remoteFailed: [Failure instance: Traceback from remote host -- Traceback > (most recent call last): Non-mozharness in-test-harness timeout. (In reply to TBPL Robot from comment #723) > RyanVM > https://tbpl.mozilla.org/php/getParsedLog.php?id=35556110&tree=Mozilla-B2g28- > v1.3 > b2g_mozilla-b2g28_v1_3_linux32_gecko build on 2014-03-03 14:30:52 > revision: 9bbbbdfacfd8 > slave: bld-linux64-spot-329 > > command timed out: 1200 seconds without output, attempting to kill > remoteFailed: [Failure instance: Traceback from remote host -- Traceback > (most recent call last): Non-mozharness mock timeout. (In reply to TBPL Robot from comment #722) > Tomcat > https://tbpl.mozilla.org/php/getParsedLog.php?id=35515115&tree=Fx-Team > WINNT 6.2 fx-team opt test mochitest-5 on 2014-03-02 22:06:25 > revision: c543d5d38e29 > slave: t-w864-ix-092 > > command timed out: 1200 seconds without output, attempting to kill > remoteFailed: [Failure instance: Traceback from remote host -- Traceback > (most recent call last): Non-mozharness clobber scripts timeout =\ (In reply to TBPL Robot from comment #721) > Tomcat > https://tbpl.mozilla.org/php/getParsedLog.php?id=35504213&tree=B2g-Inbound > b2g_b2g-inbound_macosx64_gecko build on 2014-03-02 12:21:32 > revision: a4b9da63545e > slave: bld-lion-r5-094 > > command timed out: 1200 seconds without output, attempting to kill This is in an 'hg pull'. I only added a timeout for an 'hg clone'... I can add it for hg pull.
Attachment #8390814 - Flags: review?(jgriffin)
Attachment #8390814 - Flags: review?(jgriffin) → review+
Comment on attachment 8390814 [details] [diff] [review] add a vcs_output_timeout to |hg pull| https://hg.mozilla.org/build/mozharness/rev/ded0eefe5c3e
Attachment #8390814 - Flags: checked-in+
Attachment #8390762 - Flags: review?(catlee) → review+
Comment on attachment 8390762 [details] [diff] [review] download_timeout script_obj bustage fix: http://hg.mozilla.org/build/mozharness/rev/e48924cead7e Still busted on various tests on Cypress. This depends on the virtualenv existing and we download test zips before creating the virtualenv. I think we can get around this by checking for the existence of the virtualenv and falling back to sys.executable. Backed out both download_file patches for now. :(
Attachment #8390762 - Flags: checked-in+ → checked-in-
(In reply to Aki Sasaki [:aki] from comment #731) > Comment on attachment 8390762 [details] [diff] [review] > download_timeout > > script_obj bustage fix: > http://hg.mozilla.org/build/mozharness/rev/e48924cead7e > Still busted on various tests on Cypress. This depends on the virtualenv > existing and we download test zips before creating the virtualenv. I think > we can get around this by checking for the existence of the virtualenv and > falling back to sys.executable. > > Backed out both download_file patches for now. :( 13:05:42 INFO - Downloading https://ftp-ssl.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/cypress-win32-debug/1395154525/firefox-31.0a1.en-US.win32.tests.zip to C:\slave\test\build\firefox-31.0a1.en-US.win32.tests.zip 13:05:42 INFO - Getting output from command: ['C:\\slave\\test\\build\\venv\\Scripts\\python', '-c', 'from distutils.sysconfig import get_python_lib; print(get_python_lib())'] 13:05:42 INFO - Copy/paste: C:\slave\test\build\venv\Scripts\python -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())" 13:05:42 INFO - Running post-action listener: _resource_record_post_action 13:05:42 FATAL - Uncaught exception: Traceback (most recent call last): 13:05:42 FATAL - File "C:\slave\test\scripts\mozharness\base\script.py", line 1170, in run 13:05:42 FATAL - self.run_action(action) 13:05:42 FATAL - File "C:\slave\test\scripts\mozharness\base\script.py", line 1112, in run_action 13:05:42 FATAL - self._possibly_run_method(method_name, error_if_missing=True) 13:05:42 FATAL - File "C:\slave\test\scripts\mozharness\base\script.py", line 1053, in _possibly_run_method 13:05:42 FATAL - return getattr(self, method_name)() 13:05:42 FATAL - File "scripts/scripts/desktop_unittest.py", line 315, in download_and_extract 13:05:42 FATAL - super(DesktopUnittest, self).download_and_extract(target_unzip_dirs=target_unzip_dirs) 13:05:42 FATAL - File "C:\slave\test\scripts\mozharness\mozilla\testing\testbase.py", line 281, in download_and_extract 13:05:42 FATAL - self._download_test_zip() 13:05:42 FATAL - File "C:\slave\test\scripts\mozharness\mozilla\testing\testbase.py", line 193, in _download_test_zip 13:05:42 FATAL - error_level=FATAL) 13:05:42 FATAL - File "C:\slave\test\scripts\mozharness\base\script.py", line 201, in download_file 13:05:42 FATAL - env = {"PYTHONPATH": self.query_python_site_packages_path()} 13:05:42 FATAL - File "C:\slave\test\scripts\mozharness\base\python.py", line 122, in query_python_site_packages_path 13:05:42 FATAL - 'print(get_python_lib())']) 13:05:42 FATAL - File "C:\slave\test\scripts\mozharness\base\script.py", line 744, in get_output_from_command 13:05:42 FATAL - cwd=cwd, stderr=tmp_stderr, env=env) 13:05:42 FATAL - File "c:\mozilla-build\python27\lib\subprocess.py", line 679, in __init__ 13:05:42 FATAL - errread, errwrite) 13:05:42 FATAL - File "c:\mozilla-build\python27\lib\subprocess.py", line 896, in _execute_child 13:05:42 FATAL - startupinfo) 13:05:42 FATAL - WindowsError: [Error 2] The system cannot find the file specified 13:05:42 FATAL - Running post_fatal callback... 13:05:42 FATAL - Exiting -1
Live in production.
(In reply to TBPL Robot from comment #734) > jryans > https://tbpl.mozilla.org/php/getParsedLog.php?id=36536511&tree=Try > b2g_ubuntu64_vm try opt test gaia-integration on 2014-03-21 18:43:20 > revision: ab37f8a74948 > slave: tst-linux64-spot-422 > > command timed out: 1800 seconds without output, attempting to kill Someone trying to clone their user github repo inside a test run.
Attachment #8395741 - Flags: review?(jgriffin)
Comment on attachment 8395741 [details] [diff] [review] output timeout for gaia git clone Review of attachment 8395741 [details] [diff] [review]: ----------------------------------------------------------------- Great, thanks.
Attachment #8395741 - Flags: review?(jgriffin) → review+
Aki++ for covering all these corner cases :-)
(In reply to TBPL Robot from comment #743) > Tomcat > https://tbpl.mozilla.org/php/getParsedLog.php?id=36783978&tree=Fx-Team > WINNT 6.2 fx-team opt test mochitest-other on 2014-03-26 21:21:25 > revision: 47dcbe2b1fd0 > slave: t-w864-ix-020 > > command timed out: 1800 seconds without output, attempting to kill Another unzip. Maybe we need a timeout there.
(One more unzip, and two hgtool timeouts (one pull, one update).)
Attached patch unzip_timeout — — Splinter Review
Attachment #8401065 - Flags: review?(jgriffin)
Comment on attachment 8401065 [details] [diff] [review] unzip_timeout Review of attachment 8401065 [details] [diff] [review]: ----------------------------------------------------------------- I wonder if run_command shouldn't have a default timeout...
Attachment #8401065 - Flags: review?(jgriffin) → review+
(In reply to Jonathan Griffin (:jgriffin) from comment #747) > Comment on attachment 8401065 [details] [diff] [review] > unzip_timeout > > Review of attachment 8401065 [details] [diff] [review]: > ----------------------------------------------------------------- > > I wonder if run_command shouldn't have a default timeout... We could, since mozprocess is there for every mozharness job now. That could be disruptive when it first lands, if there are long-running processes that need longer-than-default timeouts, but would likely be beneficial in the long run.
Live in production.
The last comment is a mis-star; let's close this out for now and file new bugs for any remaining cases. Adjusting summary to avoid more TBPL false-positives. Thank you Aki for your work on this! :-)
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Summary: Cloning of hg.mozilla.org/build/tools and hg.mozilla.org/integration/gaia-central often times out with "command timed out: 1200 seconds without output, attempting to kill", "command timed out: 1800 seconds without output, attempting to kill" → Cloning of hg.mozilla.org/build/tools and hg.mozilla.org/integration/gaia-central often times out, as does downloading/unzipping test zips
I've filed bug 995167 to improve the handling of a timeout during unzip.
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: