684768 - buildbot for Win64 always fails due to job timeout

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Makoto Kato [:m_kato]]

https://tbpl.mozilla.org/php/getParsedLog.php?id=6280903&full=1
https://tbpl.mozilla.org/php/getParsedLog.php?id=6283972&full=1
etc


Win64 buildbot is always orange due to the following job.  I think that this is buildbot issue.

========= Started sendchange failed (results: 2, elapsed: 2 mins, 6 secs) ==========
    master: talos-master.mozilla.org:9010
    branch: mozilla-central-win64-talos
    revision: d2c6783fa7153ef892236c16dd39c3da198b3064
    comments: 
    user: sendchange
    files: ['http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-win64/1315247919/firefox-9.0a1.en-US.win64-x86_64.zip']
    properties: [('buildid', '20110905113839'), ('builduid', u'a6db2516527f47c9a71653fcd24ca6a0')]
'python' 'e:/builds/moz2_slave/m-cen-w64/tools/buildfarm/utils/retry.py' '-s' '5' '-t' '1800' '-r' '5' '--stdout-regexp' 'change sent successfully' 'buildbot' 'sendchange' '--master' 'talos-master.mozilla.org:9010' '--username' 'sendchange' '--branch' 'mozilla-central-win64-talos' '--revision' 'd2c6783fa7153ef892236c16dd39c3da198b3064' '--property' 'buildid:20110905113839' '--property' u'builduid:a6db2516527f47c9a71653fcd24ca6a0' 'http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-win64/1315247919/firefox-9.0a1.en-US.win64-x86_64.zip'
 in dir e:\builds\moz2_slave\m-cen-w64\build (timeout 1200 secs)
 watching logfiles {}
 argv: ['python', 'e:/builds/moz2_slave/m-cen-w64/tools/buildfarm/utils/retry.py', '-s', '5', '-t', '1800', '-r', '5', '--stdout-regexp', 'change sent successfully', 'buildbot', 'sendchange', '--master', 'talos-master.mozilla.org:9010', '--username', 'sendchange', '--branch', 'mozilla-central-win64-talos', '--revision', 'd2c6783fa7153ef892236c16dd39c3da198b3064', '--property', 'buildid:20110905113839', '--property', u'builduid:a6db2516527f47c9a71653fcd24ca6a0', 'http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-win64/1315247919/firefox-9.0a1.en-US.win64-x86_64.zip']
 environment:
 :
 :
Giving up on <function run_with_timeout at 0x027979B0>
Unable to successfully run ['buildbot.bat', 'sendchange', '--master', 'talos-master.mozilla.org:9010', '--username', 'sendchange', '--branch', 'mozilla-central-win64-talos', '--revision', 'd2c6783fa7153ef892236c16dd39c3da198b3064', '--property', 'buildid:20110905113839', '--property', 'builduid:a6db2516527f47c9a71653fcd24ca6a0', 'http://stage.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-win64/1315247919/firefox-9.0a1.en-US.win64-x86_64.zip'] after 5 attempts
program finished with exit code 1

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

talos-master.mozilla.org is an odd-one-out host that we only use for Tiger Talos on the mozilla-1.9.2 branch. It's in a different network to the rest of the build machines, so those failures indicate a difference in the setup of the windows subnet. However, we should just make this go away by only doing sendchanges to talos-master.m.o for mac builds on 1.9.2.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [buildbot-configs] Only use talos-master on macosx platform

This prevents man+dog of trying to sendchange to talos-master for unrelated platforms, and we don't do a talos sendchange for the debug builds we still run on 10.5/macosx.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [buildbotcustom] Use platform level talos_masters var

We use the platform copy of talos_masters for non-release builds, so this just gets release back in sync with that.

With both these changes, the only builds to sendchange to talos-master (in a dump master) are:
  OS X 10.5.2 mozilla-1.9.2 build
  OS X 10.5.2 mozilla-1.9.2 nightly
  release-mozilla-1.9.2-macosx_build

Comment 4

[Armen [:armenzg]]

Comment on attachment 558370 [details] [diff] [review]
[buildbotcustom] Use platform level talos_masters var

Review of attachment 558370 [details] [diff] [review]:
-----------------------------------------------------------------

Well thought Nick. Thanks!

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 558369 [details] [diff] [review]
[buildbot-configs] Only use talos-master on macosx platform

http://hg.mozilla.org/build/buildbot-configs/rev/b6d26c7160e5

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 558370 [details] [diff] [review]
[buildbotcustom] Use platform level talos_masters var

http://hg.mozilla.org/build/buildbotcustom/rev/7ae4faa5538c

Comment 7

[Armen [:armenzg]]

Merged to production and reconfigured.

Comment 8

[Armen [:armenzg]]

This got fixed and the builds are going green:
https://tbpl.mozilla.org/?noignore=1&jobname=WINNT%206.1%20x86-64&usebuildbot=1&rev=787ebf02a8c1