Intermittent B2G build timeouts cloning gecko ("command timed out: 3600 seconds without output, attempting to kill" or "abort: connection ended unexpectedly")

RESOLVED WORKSFORME

Status

Release Engineering
General Automation
RESOLVED WORKSFORME
4 years ago
3 years ago

People

(Reporter: emorley, Unassigned)

Tracking

(Blocks: 1 bug, {intermittent-failure})

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

4 years ago
eg:
https://tbpl.mozilla.org/php/getParsedLog.php?id=42058274&full=1&branch=mozilla-inbound#error0

{
07:39:47     INFO - Running command: ['hgtool.py', '-r', u'3ff9374610b71410b65fc44b01200559953eaca4', '--bundle', u'https://ftp-ssl.mozilla.org/pub/mozilla.org/firefox/bundles/mozilla-inbound.hg', u'https://hg.mozilla.org/integration/mozilla-inbound', '/builds/slave/b2g_m-in_ham_dep-0000000000000/build/gecko']
07:39:47     INFO - Copy/paste: hgtool.py -r 3ff9374610b71410b65fc44b01200559953eaca4 --bundle https://ftp-ssl.mozilla.org/pub/mozilla.org/firefox/bundles/mozilla-inbound.hg https://hg.mozilla.org/integration/mozilla-inbound /builds/slave/b2g_m-in_ham_dep-0000000000000/build/gecko
07:39:47     INFO - Using env: {'HG_SHARE_BASE_DIR': '/builds/hg-shared',
07:39:47     INFO -  'PATH': '/usr/local/bin:/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cltbld/bin'}
07:39:47     INFO -  Reporting hg version in use
07:39:47     INFO -  command: START
07:39:47     INFO -  command: hg -q version
07:39:47     INFO -  command: cwd: .
07:39:47     INFO -  command: output:
07:39:47     INFO -  Mercurial Distributed SCM (version 2.5.4)
07:39:47     INFO -  command: END (0.19s elapsed)
07:39:47     INFO -  Checking if share extension works
07:39:47     INFO -  command: START
07:39:47     INFO -  command: hg help share
07:39:47     INFO -  command: cwd: /builds/slave/b2g_m-in_ham_dep-0000000000000
07:39:47     INFO -  command: END (0.26 elapsed)
07:39:47     INFO -  command: START
07:39:47     INFO -  command: hg path default
07:39:47     INFO -  command: cwd: /builds/hg-shared/integration/mozilla-inbound
07:39:48     INFO -  command: output:
07:39:48     INFO -  https://hg.mozilla.org/integration/mozilla-inbound
07:39:48     INFO -  command: END (0.21 elapsed)
07:39:48     INFO -  Updating shared repo
07:39:48     INFO -  Reporting hg version in use
07:39:48     INFO -  command: START
07:39:48     INFO -  command: hg -q version
07:39:48     INFO -  command: cwd: .
07:39:48     INFO -  command: output:
07:39:48     INFO -  Mercurial Distributed SCM (version 2.5.4)
07:39:48     INFO -  command: END (0.17s elapsed)
07:39:48     INFO -  command: START
07:39:48     INFO -  command: hg path default
07:39:48     INFO -  command: cwd: /builds/hg-shared/integration/mozilla-inbound
07:39:48     INFO -  command: output:
07:39:48     INFO -  https://hg.mozilla.org/integration/mozilla-inbound
07:39:48     INFO -  command: END (0.20 elapsed)
07:39:48     INFO -  command: START
07:39:48     INFO -  command: hg pull -r 3ff9374610b71410b65fc44b01200559953eaca4 https://hg.mozilla.org/integration/mozilla-inbound
07:39:48     INFO -  command: cwd: /builds/hg-shared/integration/mozilla-inbound
07:39:48     INFO -  command: output:
07:39:48     INFO -  warning: hg.mozilla.org certificate with fingerprint af:27:b9:34:47:4e:e5:98:01:f6:83:2b:51:c9:aa:d8:df:fb:1a:27 not verified (check hostfingerprints or web.cacerts config setting)
07:39:49     INFO -  pulling from https://hg.mozilla.org/integration/mozilla-inbound
07:39:49     INFO -  no changes found
07:39:49     INFO -  command: END (1.21s elapsed)
07:39:49     INFO -  Trying to share /builds/hg-shared/integration/mozilla-inbound to /builds/slave/b2g_m-in_ham_dep-0000000000000/build/gecko
07:39:49     INFO -  command: START
07:39:49     INFO -  command: hg share -U /builds/hg-shared/integration/mozilla-inbound /builds/slave/b2g_m-in_ham_dep-0000000000000/build/gecko
07:39:49     INFO -  command: cwd: /builds/slave/b2g_m-in_ham_dep-0000000000000
07:39:49     INFO -  command: output:
07:39:49     INFO -  command: END (0.22s elapsed)
07:39:49     INFO -  command: START
07:39:49     INFO -  command: hg update -C -r 3ff9374610b71410b65fc44b01200559953eaca4
07:39:49     INFO -  command: cwd: /builds/slave/b2g_m-in_ham_dep-0000000000000/build/gecko
07:39:49     INFO -  command: output:

command timed out: 3600 seconds without output, attempting to kill
process killed by signal 9
program finished with exit code -1
elapsedTime=3719.177584
========= Finished 'scripts/scripts/b2g_build.py --target ...' failed (results: 2, elapsed: 1 hrs, 2 mins, 4 secs) (at 2014-06-19 08:39:55.352483) =========
}

We should be timing out with a TBPL parsable error message.
(Reporter)

Comment 1

4 years ago
Split out from bug 873928:

(In reply to TBPL Robot from comment #127)
> edmorley
> https://tbpl.mozilla.org/php/getParsedLog.php?id=41360976&tree=Fx-Team
> b2g_fx-team_emulator-jb-debug_dep on 2014-06-09 07:33:24
> revision: 6e5607b09d8f
> slave: bld-linux64-spot-450
> 
> command timed out: 3600 seconds without output, attempting to kill

(In reply to TBPL Robot from comment #128)
> edmorley
> https://tbpl.mozilla.org/php/getParsedLog.php?id=41361262&tree=Mozilla-
> Central
> b2g_mozilla-central_emulator-jb_dep on 2014-06-09 07:34:15
> revision: 8bbb61b80df4
> slave: bld-linux64-spot-492
> 
> command timed out: 3600 seconds without output, attempting to kill
(Reporter)

Updated

4 years ago
Keywords: intermittent-failure
(Reporter)

Comment 2

4 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=42063830&tree=Mozilla-Central
Summary: Intermittent B2G build timeouts cloning gecko ("command timed out: 3600 seconds without output, attempting to kill") → Intermittent B2G build timeouts cloning gecko ("command timed out: 3600 seconds without output, attempting to kill" or "abort: connection ended unexpectedly")
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
(Reporter)

Comment 60

3 years ago
Inactive; closing (see bug 1180138).
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.