1037130 - Errors running tooltool.py

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Ryan VanderMeulen [:RyanVM]]

All trees closed.

https://tbpl.mozilla.org/php/getParsedLog.php?id=43549005&full=1&branch=mozilla-inbound

Fetching...
retry: Calling <function run_with_timeout at 0x1004fc8c0> with args: (['/tools/tooltool.py', '--url', 'http://runtime-binaries.pvt.build.mozilla.org/tooltool', '--overwrite', '-m', 'browser/config/tooltool-manifests/macosx64/releng.manifest', 'fetch', '-c', '/builds/tooltool_cache'], 300, None, None, False, True), kwargs: {}, attempt #1
retry: Failed, sleeping 30 seconds before retrying
retry: Calling <function run_with_timeout at 0x1004fc8c0> with args: (['/tools/tooltool.py', '--url', 'http://runtime-binaries.pvt.build.mozilla.org/tooltool', '--overwrite', '-m', 'browser/config/tooltool-manifests/macosx64/releng.manifest', 'fetch', '-c', '/builds/tooltool_cache'], 300, None, None, False, True), kwargs: {}, attempt #2
retry: Failed, sleeping 60 seconds before retrying
retry: Calling <function run_with_timeout at 0x1004fc8c0> with args: (['/tools/tooltool.py', '--url', 'http://runtime-binaries.pvt.build.mozilla.org/tooltool', '--overwrite', '-m', 'browser/config/tooltool-manifests/macosx64/releng.manifest', 'fetch', '-c', '/builds/tooltool_cache'], 300, None, None, False, True), kwargs: {}, attempt #3
retry: Failed, sleeping 120 seconds before retrying
retry: Calling <function run_with_timeout at 0x1004fc8c0> with args: (['/tools/tooltool.py', '--url', 'http://runtime-binaries.pvt.build.mozilla.org/tooltool', '--overwrite', '-m', 'browser/config/tooltool-manifests/macosx64/releng.manifest', 'fetch', '-c', '/builds/tooltool_cache'], 300, None, None, False, True), kwargs: {}, attempt #4
retry: Failed, sleeping 240 seconds before retrying
retry: Calling <function run_with_timeout at 0x1004fc8c0> with args: (['/tools/tooltool.py', '--url', 'http://runtime-binaries.pvt.build.mozilla.org/tooltool', '--overwrite', '-m', 'browser/config/tooltool-manifests/macosx64/releng.manifest', 'fetch', '-c', '/builds/tooltool_cache'], 300, None, None, False, True), kwargs: {}, attempt #5
retry: Failed, sleeping 300 seconds before retrying
retry: Calling <function run_with_timeout at 0x1004fc8c0> with args: (['/tools/tooltool.py', '--url', 'http://runtime-binaries.pvt.build.mozilla.org/tooltool', '--overwrite', '-m', 'browser/config/tooltool-manifests/macosx64/releng.manifest', 'fetch', '-c', '/builds/tooltool_cache'], 300, None, None, False, True), kwargs: {}, attempt #6
retry: Failed, sleeping 300 seconds before retrying
retry: Calling <function run_with_timeout at 0x1004fc8c0> with args: (['/tools/tooltool.py', '--url', 'http://runtime-binaries.pvt.build.mozilla.org/tooltool', '--overwrite', '-m', 'browser/config/tooltool-manifests/macosx64/releng.manifest', 'fetch', '-c', '/builds/tooltool_cache'], 300, None, None, False, True), kwargs: {}, attempt #7
retry: Failed, sleeping 300 seconds before retrying
retry: Calling <function run_with_timeout at 0x1004fc8c0> with args: (['/tools/tooltool.py', '--url', 'http://runtime-binaries.pvt.build.mozilla.org/tooltool', '--overwrite', '-m', 'browser/config/tooltool-manifests/macosx64/releng.manifest', 'fetch', '-c', '/builds/tooltool_cache'], 300, None, None, False, True), kwargs: {}, attempt #8
retry: Failed, sleeping 300 seconds before retrying
retry: Calling <function run_with_timeout at 0x1004fc8c0> with args: (['/tools/tooltool.py', '--url', 'http://runtime-binaries.pvt.build.mozilla.org/tooltool', '--overwrite', '-m', 'browser/config/tooltool-manifests/macosx64/releng.manifest', 'fetch', '-c', '/builds/tooltool_cache'], 300, None, None, False, True), kwargs: {}, attempt #9
retry: Failed, sleeping 300 seconds before retrying
retry: Calling <function run_with_timeout at 0x1004fc8c0> with args: (['/tools/tooltool.py', '--url', 'http://runtime-binaries.pvt.build.mozilla.org/tooltool', '--overwrite', '-m', 'browser/config/tooltool-manifests/macosx64/releng.manifest', 'fetch', '-c', '/builds/tooltool_cache'], 300, None, None, False, True), kwargs: {}, attempt #10
retry: Giving up on <function run_with_timeout at 0x1004fc8c0>
Unable to successfully run ['/tools/tooltool.py', '--url', 'http://runtime-binaries.pvt.build.mozilla.org/tooltool', '--overwrite', '-m', 'browser/config/tooltool-manifests/macosx64/releng.manifest', 'fetch', '-c', '/builds/tooltool_cache'] after 10 attempts
program finished with exit code 1
elapsedTime=1950.639170
========= Finished 'sh /builds/slave/m-in-osx64-d-00000000000000000/tools/scripts/tooltool/tooltool_wrapper.sh ...' failed (results: 2, elapsed: 32 mins, 30 secs) (at 2014-07-10 13:09:45.959230) =========

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

The changes in bug 1037040 accidentally left $python unset, which left

[[e-mail] ~]$ head -n1 /tools/tooltool.py
#!

which meant tooltool.py didn't do much.

http://hg.mozilla.org/build/puppet/rev/76a76fdd927a fixes this, and it's in place on the puppet masters now, but any machine on which the most recent puppet run had this error will still have problems, whether it's started a job already or not.

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

Bug 1037163 for the logging issues.

Ryan, I'll leave you to close this when you're happy with the results.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

All the build machines are fixed by puppet on reboot, or doing builds which haven't failed and will reboot to get the fix.

Comment 4

[Ryan VanderMeulen [:RyanVM]]

Trees reopened.