Closed Bug 1027668 Opened 10 years ago Closed 9 years ago

Intermittent B2G build timeouts cloning gecko ("command timed out: 3600 seconds without output, attempting to kill" or "abort: connection ended unexpectedly")

Categories

(Release Engineering :: General, defect)

All
Gonk (Firefox OS)
defect
Not set
normal

Tracking

(Not tracked)

RESOLVED WORKSFORME

People

(Reporter: emorley, Unassigned)

References

(Blocks 1 open bug)

Details

(Keywords: intermittent-failure)

eg:
https://tbpl.mozilla.org/php/getParsedLog.php?id=42058274&full=1&branch=mozilla-inbound#error0

{
07:39:47     INFO - Running command: ['hgtool.py', '-r', u'3ff9374610b71410b65fc44b01200559953eaca4', '--bundle', u'https://ftp-ssl.mozilla.org/pub/mozilla.org/firefox/bundles/mozilla-inbound.hg', u'https://hg.mozilla.org/integration/mozilla-inbound', '/builds/slave/b2g_m-in_ham_dep-0000000000000/build/gecko']
07:39:47     INFO - Copy/paste: hgtool.py -r 3ff9374610b71410b65fc44b01200559953eaca4 --bundle https://ftp-ssl.mozilla.org/pub/mozilla.org/firefox/bundles/mozilla-inbound.hg https://hg.mozilla.org/integration/mozilla-inbound /builds/slave/b2g_m-in_ham_dep-0000000000000/build/gecko
07:39:47     INFO - Using env: {'HG_SHARE_BASE_DIR': '/builds/hg-shared',
07:39:47     INFO -  'PATH': '/usr/local/bin:/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cltbld/bin'}
07:39:47     INFO -  Reporting hg version in use
07:39:47     INFO -  command: START
07:39:47     INFO -  command: hg -q version
07:39:47     INFO -  command: cwd: .
07:39:47     INFO -  command: output:
07:39:47     INFO -  Mercurial Distributed SCM (version 2.5.4)
07:39:47     INFO -  command: END (0.19s elapsed)
07:39:47     INFO -  Checking if share extension works
07:39:47     INFO -  command: START
07:39:47     INFO -  command: hg help share
07:39:47     INFO -  command: cwd: /builds/slave/b2g_m-in_ham_dep-0000000000000
07:39:47     INFO -  command: END (0.26 elapsed)
07:39:47     INFO -  command: START
07:39:47     INFO -  command: hg path default
07:39:47     INFO -  command: cwd: /builds/hg-shared/integration/mozilla-inbound
07:39:48     INFO -  command: output:
07:39:48     INFO -  https://hg.mozilla.org/integration/mozilla-inbound
07:39:48     INFO -  command: END (0.21 elapsed)
07:39:48     INFO -  Updating shared repo
07:39:48     INFO -  Reporting hg version in use
07:39:48     INFO -  command: START
07:39:48     INFO -  command: hg -q version
07:39:48     INFO -  command: cwd: .
07:39:48     INFO -  command: output:
07:39:48     INFO -  Mercurial Distributed SCM (version 2.5.4)
07:39:48     INFO -  command: END (0.17s elapsed)
07:39:48     INFO -  command: START
07:39:48     INFO -  command: hg path default
07:39:48     INFO -  command: cwd: /builds/hg-shared/integration/mozilla-inbound
07:39:48     INFO -  command: output:
07:39:48     INFO -  https://hg.mozilla.org/integration/mozilla-inbound
07:39:48     INFO -  command: END (0.20 elapsed)
07:39:48     INFO -  command: START
07:39:48     INFO -  command: hg pull -r 3ff9374610b71410b65fc44b01200559953eaca4 https://hg.mozilla.org/integration/mozilla-inbound
07:39:48     INFO -  command: cwd: /builds/hg-shared/integration/mozilla-inbound
07:39:48     INFO -  command: output:
07:39:48     INFO -  warning: hg.mozilla.org certificate with fingerprint af:27:b9:34:47:4e:e5:98:01:f6:83:2b:51:c9:aa:d8:df:fb:1a:27 not verified (check hostfingerprints or web.cacerts config setting)
07:39:49     INFO -  pulling from https://hg.mozilla.org/integration/mozilla-inbound
07:39:49     INFO -  no changes found
07:39:49     INFO -  command: END (1.21s elapsed)
07:39:49     INFO -  Trying to share /builds/hg-shared/integration/mozilla-inbound to /builds/slave/b2g_m-in_ham_dep-0000000000000/build/gecko
07:39:49     INFO -  command: START
07:39:49     INFO -  command: hg share -U /builds/hg-shared/integration/mozilla-inbound /builds/slave/b2g_m-in_ham_dep-0000000000000/build/gecko
07:39:49     INFO -  command: cwd: /builds/slave/b2g_m-in_ham_dep-0000000000000
07:39:49     INFO -  command: output:
07:39:49     INFO -  command: END (0.22s elapsed)
07:39:49     INFO -  command: START
07:39:49     INFO -  command: hg update -C -r 3ff9374610b71410b65fc44b01200559953eaca4
07:39:49     INFO -  command: cwd: /builds/slave/b2g_m-in_ham_dep-0000000000000/build/gecko
07:39:49     INFO -  command: output:

command timed out: 3600 seconds without output, attempting to kill
process killed by signal 9
program finished with exit code -1
elapsedTime=3719.177584
========= Finished 'scripts/scripts/b2g_build.py --target ...' failed (results: 2, elapsed: 1 hrs, 2 mins, 4 secs) (at 2014-06-19 08:39:55.352483) =========
}

We should be timing out with a TBPL parsable error message.
Split out from bug 873928:

(In reply to TBPL Robot from comment #127)
> edmorley
> https://tbpl.mozilla.org/php/getParsedLog.php?id=41360976&tree=Fx-Team
> b2g_fx-team_emulator-jb-debug_dep on 2014-06-09 07:33:24
> revision: 6e5607b09d8f
> slave: bld-linux64-spot-450
> 
> command timed out: 3600 seconds without output, attempting to kill

(In reply to TBPL Robot from comment #128)
> edmorley
> https://tbpl.mozilla.org/php/getParsedLog.php?id=41361262&tree=Mozilla-
> Central
> b2g_mozilla-central_emulator-jb_dep on 2014-06-09 07:34:15
> revision: 8bbb61b80df4
> slave: bld-linux64-spot-492
> 
> command timed out: 3600 seconds without output, attempting to kill
https://tbpl.mozilla.org/php/getParsedLog.php?id=42063830&tree=Mozilla-Central
Summary: Intermittent B2G build timeouts cloning gecko ("command timed out: 3600 seconds without output, attempting to kill") → Intermittent B2G build timeouts cloning gecko ("command timed out: 3600 seconds without output, attempting to kill" or "abort: connection ended unexpectedly")
Inactive; closing (see bug 1180138).
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → WORKSFORME
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.