1018779 - b2g nonunified builds frequently "command timed out: 3600 seconds without output, attempting to kill" untarring Gecko on bld-centos6-hp-* and bld-linux64-ix-*

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Phil Ringnalda (:philor)]

Because disk space is the usual bld-centos6-hp-* problem, I want to blame that, but having seen one do a green build of jb-opt with 22.13GB free, while another hung on the same job with 23.60GB free, maybe not.

https://tbpl.mozilla.org/php/getParsedLog.php?id=40827193&tree=Mozilla-Central
bld-centos6-hp-009

https://tbpl.mozilla.org/php/getParsedLog.php?id=40806741&tree=Mozilla-Central
bld-centos6-hp-009

https://tbpl.mozilla.org/php/getParsedLog.php?id=40807700&tree=Mozilla-Central
bld-centos6-hp-013

https://tbpl.mozilla.org/php/getParsedLog.php?id=40806926&tree=Mozilla-Central
bld-centos6-hp-015

07:51:44     INFO -  cd out/target/product/generic/system && tar xvfz /builds/slave/b2g_m-cen_emu-jb_nu-0000000000/build/out/target/product/generic/obj/DATA/gecko_intermediates/gecko
07:51:44     INFO -  b2g/
07:51:44     INFO -  b2g/ua-update.json
07:51:44     INFO -  b2g/dependentlibs.list
07:51:44     INFO -  b2g/run-mozilla.sh
07:51:44     INFO -  b2g/omni.ja
07:51:44     INFO -  b2g/libnssckbi.so
07:51:44     INFO -  b2g/dictionaries/
07:51:44     INFO -  b2g/dictionaries/en-US.aff
07:51:44     INFO -  b2g/dictionaries/en-US.dic
07:51:44     INFO -  b2g/libnss3.so
07:51:44     INFO -  b2g/libfreebl3.so
07:51:44     INFO -  b2g/crashreporter.ini
07:51:44     INFO -  b2g/Throbber-small.gif
07:51:44     INFO -  b2g/removed-files
07:51:44     INFO -  b2g/updater
07:51:44     INFO -  b2g/platform.ini
07:51:44     INFO -  b2g/b2g
07:51:44     INFO -  b2g/libxul.so
07:51:44     INFO -  b2g/application.ini
07:51:44     INFO -  b2g/libmozglue.so
07:51:44     INFO -  b2g/libsoftokn3.so
07:51:44     INFO -  b2g/libreplace_jemalloc.so
07:51:44     INFO -  b2g/plugin-container
07:51:44     INFO -  b2g/precomplete

command timed out: 3600 seconds without output, attempting to kill

Comment 1

[Ed Morley [:emorley]]

https://tbpl.mozilla.org/php/getParsedLog.php?id=40789682&tree=Fx-Team

Comment 2

[Legacy TBPL/Treeherder Robot]

KWierso
https://tbpl.mozilla.org/php/getParsedLog.php?id=40901847&tree=Fx-Team
b2g_fx-team_emulator-kk_nonunified on 2014-06-02 11:30:19
revision: 0e6f1d91ca26
slave: bld-linux64-ix-034

command timed out: 3600 seconds without output, attempting to kill

Comment 3

[Legacy TBPL/Treeherder Robot]

edmorley
https://tbpl.mozilla.org/php/getParsedLog.php?id=40957981&tree=B2g-Inbound
b2g_b2g-inbound_emulator-jb_nonunified on 2014-06-03 02:35:29
revision: 8b3454b9896d
slave: bld-linux64-ix-034

command timed out: 3600 seconds without output, attempting to kill

Comment 4

[Legacy TBPL/Treeherder Robot]

RyanVM
https://tbpl.mozilla.org/php/getParsedLog.php?id=40975701&tree=Mozilla-Inbound
b2g_mozilla-inbound_emulator-jb_nonunified on 2014-06-03 07:38:22
revision: d6dc9b7afda4
slave: bld-centos6-hp-013

command timed out: 3600 seconds without output, attempting to kill

Comment 5

[Legacy TBPL/Treeherder Robot]

RyanVM
https://tbpl.mozilla.org/php/getParsedLog.php?id=41006847&tree=Mozilla-B2g30-v1.4
b2g_mozilla-b2g30_v1_4_emulator-kk_periodic on 2014-06-03 18:30:42
revision: db3b521e5a4c
slave: bld-centos6-hp-013

command timed out: 3600 seconds without output, attempting to kill

Comment 6

[Legacy TBPL/Treeherder Robot]

RyanVM
https://tbpl.mozilla.org/php/getParsedLog.php?id=41045502&tree=Mozilla-B2g30-v1.4
b2g_mozilla-b2g30_v1_4_emulator-kk-debug_periodic on 2014-06-04 06:30:21
revision: d8e93180b601
slave: bld-linux64-ix-034

command timed out: 3600 seconds without output, attempting to kill

Comment 7

[Legacy TBPL/Treeherder Robot]

RyanVM
https://tbpl.mozilla.org/php/getParsedLog.php?id=41160607&tree=Mozilla-Inbound
b2g_mozilla-inbound_emulator-kk-debug_nonunified on 2014-06-05 13:30:11
revision: 039b9e7b3616
slave: bld-centos6-hp-013

command timed out: 3600 seconds without output, attempting to kill

Comment 8

[Legacy TBPL/Treeherder Robot]

philor
https://tbpl.mozilla.org/php/getParsedLog.php?id=41173887&tree=Mozilla-Inbound
b2g_mozilla-inbound_emulator-jb-debug_nonunified on 2014-06-05 16:41:12
revision: b9d6fb7448b3
slave: bld-linux64-ix-034

command timed out: 3600 seconds without output, attempting to kill

Comment 9

[Legacy TBPL/Treeherder Robot]

edmorley
https://tbpl.mozilla.org/php/getParsedLog.php?id=41216251&tree=Mozilla-Central
b2g_mozilla-central_emulator-jb_nonunified on 2014-06-06 04:33:31
revision: c8288d0c7a15
slave: bld-centos6-hp-015

command timed out: 3600 seconds without output, attempting to kill

Comment 10

[Legacy TBPL/Treeherder Robot]

RyanVM
https://tbpl.mozilla.org/php/getParsedLog.php?id=41300377&tree=Mozilla-Central
b2g_mozilla-central_emulator-kk_periodic on 2014-06-07 13:30:35
revision: a2f0e0619332
slave: bld-linux64-ix-034

command timed out: 3600 seconds without output, attempting to kill

Comment 11

[Legacy TBPL/Treeherder Robot]

RyanVM
https://tbpl.mozilla.org/php/getParsedLog.php?id=41321652&tree=Mozilla-Central
b2g_mozilla-central_emulator-kk_periodic on 2014-06-08 10:30:38
revision: 8bd92dc9ef59
slave: bld-linux64-ix-034

command timed out: 3600 seconds without output, attempting to kill

Comment 12

[Legacy TBPL/Treeherder Robot]

edmorley
https://tbpl.mozilla.org/php/getParsedLog.php?id=41361821&tree=Mozilla-Central
b2g_mozilla-central_emulator-jb-debug_nonunified on 2014-06-09 07:42:12
revision: 8bbb61b80df4
slave: bld-linux64-spot-462

command timed out: 3600 seconds without output, attempting to kill

Comment 13

[Legacy TBPL/Treeherder Robot]

edmorley
https://tbpl.mozilla.org/php/getParsedLog.php?id=41362374&tree=B2g-Inbound
b2g_b2g-inbound_emulator-jb-debug_nonunified on 2014-06-09 05:42:46
revision: 81f9bffef707
slave: bld-linux64-spot-347

abort: connection ended unexpectedly
Automation Error: hg not responding
CalledProcessError: Command '['hg', 'clone', '-U', 'https://hg.mozilla.org/integration/b2g-inbound', '/builds/hg-shared/integration/b2g-inbound']' returned non-zero exit status 255
abort: connection ended unexpectedly
Automation Error: hg not responding
CalledProcessError: Command '['hg', 'clone', 'https://hg.mozilla.org/integration/b2g-inbound', '/builds/slave/b2g_b2g-in_emu-jb-d_nu-0000000/build/gecko']' returned non-zero exit status 255
subprocess.CalledProcessError: Command '['hg', 'clone', 'https://hg.mozilla.org/integration/b2g-inbound', '/builds/slave/b2g_b2g-in_emu-jb-d_nu-0000000/build/gecko']' returned non-zero exit status 255
Return code: 1
command timed out: 3600 seconds without output, attempting to kill

Comment 14

[Legacy TBPL/Treeherder Robot]

edmorley
https://tbpl.mozilla.org/php/getParsedLog.php?id=41440882&tree=Mozilla-Inbound
b2g_mozilla-inbound_emulator_nonunified on 2014-06-10 04:57:41
revision: 85ddc0b5770e
slave: bld-linux64-spot-1096

command timed out: 3600 seconds without output, attempting to kill

Comment 15

[Legacy TBPL/Treeherder Robot]

edmorley
https://tbpl.mozilla.org/php/getParsedLog.php?id=41441619&tree=Mozilla-Inbound
b2g_mozilla-inbound_emulator_dep on 2014-06-10 05:29:51
revision: e6d504f62767
slave: bld-linux64-spot-315

command timed out: 3600 seconds without output, attempting to kill

Comment 16

[Legacy TBPL/Treeherder Robot]

edmorley
https://tbpl.mozilla.org/php/getParsedLog.php?id=41443416&tree=Mozilla-Inbound
b2g_mozilla-inbound_emulator-jb-debug_dep on 2014-06-10 06:25:38
revision: 46f584afbed5
slave: bld-linux64-spot-366

command timed out: 3600 seconds without output, attempting to kill

Comment 17

[Legacy TBPL/Treeherder Robot]

edmorley
https://tbpl.mozilla.org/php/getParsedLog.php?id=41445109&tree=Mozilla-Aurora
b2g_mozilla-aurora_hamachi_eng_dep on 2014-06-10 04:54:03
revision: 486712e65383
slave: bld-linux64-spot-419

abort: connection ended unexpectedly
Automation Error: hg not responding
CalledProcessError: Command '['hg', 'clone', '-U', 'https://hg.mozilla.org/releases/mozilla-aurora', '/builds/hg-shared/releases/mozilla-aurora']' returned non-zero exit status 255
abort: connection ended unexpectedly
Automation Error: hg not responding
CalledProcessError: Command '['hg', 'clone', 'https://hg.mozilla.org/releases/mozilla-aurora', '/builds/slave/b2g_m-aurora_ham_eng_dep-00000/build/gecko']' returned non-zero exit status 255
subprocess.CalledProcessError: Command '['hg', 'clone', 'https://hg.mozilla.org/releases/mozilla-aurora', '/builds/slave/b2g_m-aurora_ham_eng_dep-00000/build/gecko']' returned non-zero exit status 255
Return code: 1
abort: connection ended unexpectedly
Automation Error: hg not responding
CalledProcessError: Command '['hg', 'clone', '-U', 'https://hg.mozilla.org/releases/mozilla-aurora', '/builds/hg-shared/releases/mozilla-aurora']' returned non-zero exit status 255
command timed out: 3600 seconds without output, attempting to kill

Comment 18

[Legacy TBPL/Treeherder Robot]

KWierso
https://tbpl.mozilla.org/php/getParsedLog.php?id=41475018&tree=Mozilla-Inbound
b2g_mozilla-inbound_emulator-jb_dep on 2014-06-10 13:53:21
revision: c8ca0026a72b
slave: bld-linux64-spot-367

command timed out: 3600 seconds without output, attempting to kill

Comment 19

[Legacy TBPL/Treeherder Robot]

RyanVM
https://tbpl.mozilla.org/php/getParsedLog.php?id=41539449&tree=B2g-Inbound
b2g_b2g-inbound_emulator-kk-debug_nonunified on 2014-06-11 08:33:08
revision: c5f010ecfe9e
slave: bld-linux64-spot-433

abort: connection ended unexpectedly
Automation Error: hg not responding
CalledProcessError: Command '['hg', 'clone', '-U', 'https://hg.mozilla.org/integration/b2g-inbound', '/builds/hg-shared/integration/b2g-inbound']' returned non-zero exit status 255
abort: connection ended unexpectedly
Automation Error: hg not responding
CalledProcessError: Command '['hg', 'clone', 'https://hg.mozilla.org/integration/b2g-inbound', '/builds/slave/b2g_b2g-in_emu-kk-d_nu-0000000/build/gecko']' returned non-zero exit status 255
subprocess.CalledProcessError: Command '['hg', 'clone', 'https://hg.mozilla.org/integration/b2g-inbound', '/builds/slave/b2g_b2g-in_emu-kk-d_nu-0000000/build/gecko']' returned non-zero exit status 255
Return code: 1
abort: connection ended unexpectedly
Automation Error: hg not responding
CalledProcessError: Command '['hg', 'clone', '-U', 'https://hg.mozilla.org/integration/b2g-inbound', '/builds/hg-shared/integration/b2g-inbound']' returned non-zero exit status 255
command timed out: 3600 seconds without output, attempting to kill

Comment 20

[Legacy TBPL/Treeherder Robot]

RyanVM
https://tbpl.mozilla.org/php/getParsedLog.php?id=41611226&tree=Mozilla-Aurora
b2g_mozilla-aurora_emulator_dep on 2014-06-12 09:12:50
revision: 18e2630727d4
slave: bld-linux64-spot-1086

command timed out: 3600 seconds without output, attempting to kill

Comment 21

[Legacy TBPL/Treeherder Robot]

RyanVM
https://tbpl.mozilla.org/php/getParsedLog.php?id=41611943&tree=Mozilla-B2g30-v1.4
b2g_mozilla-b2g30_v1_4_emulator-jb_dep on 2014-06-12 09:22:35
revision: d001f9f42d0d
slave: bld-linux64-spot-063

command timed out: 3600 seconds without output, attempting to kill

Comment 22

[Legacy TBPL/Treeherder Robot]

KWierso
https://tbpl.mozilla.org/php/getParsedLog.php?id=41640969&tree=Mozilla-Central
b2g_mozilla-central_emulator_nightly on 2014-06-12 16:09:30
revision: aab3362f97e9
slave: bld-linux64-spot-458

command timed out: 3600 seconds without output, attempting to kill

Comment 23

[Legacy TBPL/Treeherder Robot]

KWierso
https://tbpl.mozilla.org/php/getParsedLog.php?id=41641113&tree=Mozilla-Central
b2g_mozilla-central_nexus-4_eng_nightly on 2014-06-12 16:09:35
revision: aab3362f97e9
slave: bld-linux64-spot-478

command timed out: 3600 seconds without output, attempting to kill

Comment 24

[Legacy TBPL/Treeherder Robot]

Tomcat
https://tbpl.mozilla.org/php/getParsedLog.php?id=41788971&tree=Mozilla-Inbound
b2g_mozilla-inbound_emulator-kk-debug_nonunified on 2014-06-16 04:41:08
revision: 97a467d8ec47
slave: bld-linux64-spot-429

Return code: 1
command timed out: 3600 seconds without output, attempting to kill

Comment 25

[Legacy TBPL/Treeherder Robot]

RyanVM
https://tbpl.mozilla.org/php/getParsedLog.php?id=41981460&tree=B2g-Inbound
b2g_b2g-inbound_emulator-debug_nonunified on 2014-06-18 08:44:31
revision: 7c41391a9bc7
slave: bld-linux64-spot-351

Return code: 1
command timed out: 3600 seconds without output, attempting to kill

Comment 26

[Legacy TBPL/Treeherder Robot]

edmorley
https://tbpl.mozilla.org/php/getParsedLog.php?id=42062038&tree=Fx-Team
b2g_fx-team_emulator-kk_nonunified on 2014-06-19 08:31:43
revision: 9e8c00218dcc
slave: bld-linux64-spot-477

command timed out: 3600 seconds without output, attempting to kill

Comment 27

[Legacy TBPL/Treeherder Robot]

Tomcat
https://tbpl.mozilla.org/php/getParsedLog.php?id=43320862&tree=B2g-Inbound
b2g_b2g-inbound_hamachi_eng_dep on 2014-07-07 23:58:44
revision: 87b7fee443c8
slave: bld-linux64-spot-383

Return code: 1
command timed out: 3600 seconds without output, attempting to kill

Comment 28

[Legacy TBPL/Treeherder Robot]

:nigelb
https://tbpl.mozilla.org/php/getParsedLog.php?id=43391255&tree=Fx-Team
b2g_fx-team_emulator-jb_nonunified on 2014-07-08 17:40:08
revision: 6598d8f28ded
slave: bld-linux64-spot-460

command timed out: 3600 seconds without output, attempting to kill

Comment 29

[Legacy TBPL/Treeherder Robot]

KWierso
https://tbpl.mozilla.org/php/getParsedLog.php?id=46291452&tree=Mozilla-Inbound
b2g_mozilla-inbound_win32_gecko-debug build on 2014-08-19 12:30:32
revision: 3c8adc8ca61b
slave: b-2008-ix-0104

command timed out: 3600 seconds without output, attempting to kill

Comment 30

[Legacy TBPL/Treeherder Robot]

KWierso
https://tbpl.mozilla.org/php/getParsedLog.php?id=46577246&tree=Mozilla-Inbound
b2g_mozilla-inbound_win32_gecko build on 2014-08-22 10:56:57
revision: 73a965cfceba
slave: b-2008-ix-0007

command timed out: 3600 seconds without output, attempting to kill

Comment 31

[Legacy TBPL/Treeherder Robot]

KWierso
https://tbpl.mozilla.org/php/getParsedLog.php?id=46811970&tree=Mozilla-Inbound
b2g_mozilla-inbound_win32_gecko-debug build on 2014-08-26 17:02:51
revision: 0e73004b33f2
slave: b-2008-ix-0161

command timed out: 3600 seconds without output, attempting to kill

Comment 32

[Legacy TBPL/Treeherder Robot]

KWierso
https://tbpl.mozilla.org/php/getParsedLog.php?id=46884514&tree=Mozilla-Inbound
b2g_mozilla-inbound_win32_gecko build on 2014-08-27 13:01:01
revision: f1dcee212697
slave: b-2008-ix-0111

command timed out: 3600 seconds without output, attempting to kill

Comment 33

[Legacy TBPL/Treeherder Robot]

KWierso
https://tbpl.mozilla.org/php/getParsedLog.php?id=46884514&tree=Mozilla-Inbound
b2g_mozilla-inbound_win32_gecko build on 2014-08-27 13:01:01
revision: f1dcee212697
slave: b-2008-ix-0111

command timed out: 3600 seconds without output, attempting to kill

Comment 34

[Legacy TBPL/Treeherder Robot]

KWierso
https://tbpl.mozilla.org/php/getParsedLog.php?id=47073919&tree=B2g-Inbound
b2g_b2g-inbound_win32_gecko build on 2014-08-29 17:14:33
revision: ca310908e620
slave: b-2008-ix-0134

command timed out: 3600 seconds without output, attempting to kill

Comment 35

[Legacy TBPL/Treeherder Robot]

KWierso
https://tbpl.mozilla.org/php/getParsedLog.php?id=47252483&tree=Mozilla-Central
b2g_mozilla-central_win32_gecko_localizer nightly on 2014-09-02 12:14:42
revision: 372ce1f36116
slave: b-2008-ix-0153

command timed out: 3600 seconds without output, attempting to kill
C:/Users/cltbld/AppData/Local/Temp/make2736-2.sh: line 1: compare-locales: command not found
pymake\..\..\mozmake.exe: *** [merge-es-ES] Error 127
Return code: 2
C:/Users/cltbld/AppData/Local/Temp/make496-2.sh: line 1: compare-locales: command not found
pymake\..\..\mozmake.exe: *** [merge-pl] Error 127
Return code: 2
C:/Users/cltbld/AppData/Local/Temp/make284-2.sh: line 1: compare-locales: command not found
pymake\..\..\mozmake.exe: *** [merge-pt-BR] Error 127
Return code: 2
pymake\..\..\mozmake.exe: *** [package-tests] Error 1
Return code: 2
2 not in success codes: [0]
Halting on failure while running C:\mozilla-build\python27\python.exe c:\builds\moz2_slave\m-cen-w32_g_lz-ntly-0000000000\./build/build/pymake/make.py package-tests AB_CD=multi
Running post_fatal callback...
Exiting 2

Comment 36

[Legacy TBPL/Treeherder Robot]

submit_timestamp: 2014-11-28T02:04:49
log: https://treeherder.mozilla.org/ui/logviewer.html#?repo=mozilla-inbound&job_id=4279285
repository: mozilla-inbound
who: tomcat[at]mozilla[dot]com
machine: b-2008-ix-0082
buildname: b2g_mozilla-inbound_win32_gecko build
revision: 840a80f401a6

command timed out: 3600 seconds without output, attempting to kill

Comment 37

[Rail Aliiev [:rail]]

Sounds like it was fixed!