Frequent "command timed out: 3600 seconds without output, attempting to kill" after build/tools/mktarball.sh ... step in b2g device builds

RESOLVED FIXED

Status

Release Engineering
General
RESOLVED FIXED
5 years ago
19 days ago

People

(Reporter: philor, Unassigned)

Tracking

({intermittent-failure})

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

5 years ago
e.g.

https://tbpl.mozilla.org/php/getParsedLog.php?id=20792981&tree=Mozilla-Inbound
b2g_mozilla-inbound_panda_dep on 2013-03-18 12:59:14 PDT for push d764382ed4cf
slave: bld-linux64-ix-008

3:44:37     INFO -  Target system fs tarball: out/target/product/panda/system.tar.bz2
13:44:37     INFO -  build/tools/mktarball.sh out/host/linux-x86/bin/fs_get_stats out/target/product/panda system out/target/product/panda/system.tar out/target/product/panda/system.tar.bz2

command timed out: 3600 seconds without output, attempting to kill
process killed by signal 9
program finished with exit code -1
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
I took a look at this. A failing log looks like this:

07:57:20     INFO -  Created filesystem with 707/32768 inodes and 33712/131072 blocks
07:57:20     INFO -  Install system fs image: out/target/product/unagi/system.img
07:57:21     INFO -  out/target/product/unagi/system.img+ total size is 131546892

command timed out: 3600 seconds without output, attempting to kill

If it doesn't stall it's:
10:09:59     INFO -  Created filesystem with 707/32768 inodes and 33761/131072 blocks
10:09:59     INFO -  Install system fs image: out/target/product/unagi/system.img
10:09:59     INFO -  out/target/product/unagi/system.img+ total size is 131747596
10:09:59     INFO -  real	47m27.244s
10:09:59     INFO -  user	147m20.280s
10:09:59     INFO -  sys	21m23.225s
10:09:59     INFO -  Run |./flash.sh| to flash all partitions of your device
10:10:00     INFO - Return code: 0

The time output comes from https://github.com/mozilla-b2g/B2G/blob/master/build.sh#L61:
  time nice -n19 make $MAKE_FLAGS $@
ie make isn't finishing

Catching a machine (bld-centos5-hp-007) in the act yields this process list:
 2213 ?        Sl     0:01 /tools/buildbot-0.8.4-pre-moz2/bin/python2.7 /tools/buildbot/bin/twistd --no_save --logfile /builds/slave/twistd.log --python /builds/slave/buildbot.tac
 2336 ?        S      0:02  \_ python scripts/scripts/b2g_build.py --target unagi --config b2g/releng.py --gaia-languages-file locales/languages_dev.json --gecko-languages-file gecko/b2g/locales/all-locales
 2840 ?        S      0:00      \_ /usr/bin/python -tt /usr/sbin/mock_mozilla -r mozilla-centos6-i386 -q --cwd /builds/slave/b2g_m-in_unagi_dep-00000000000/build --unpriv --shell /usr/bin/env VARIANT=user GAIA_OPTIMIZE=1 "LESSOPEN=|/usr/bin/lesspipe
 2867 ?        S      0:00          \_ /bin/bash ./build.sh
 3253 ?        SN     0:05              \_ make -j6

strace says 3253 is trying to read from fd 11, which is a closed pipe.
catlee was doing emulator builds today and saw a similar issue, and traced it an 'adb fork-server server' process. Killing that unwedged make. I haven't confirmed that in the panda or unagi builds yet.
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Thanks for tracking that down mwu. FTR, inbound rev 805254513390 is the first build to use that.
22 builds since the landing and no failures.
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
(Assignee)

Updated

5 years ago
Product: mozilla.org → Release Engineering
(Assignee)

Updated

19 days ago
Component: General Automation → General
Product: Release Engineering → Release Engineering
You need to log in before you can comment on or make changes to this bug.