1166420 - buildbot configs need to be adjusted to use new tooltool, too

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

We changed mozharness to address the relengapi-based tooltool, but android builds are still running tooltool_wrapper.sh directly from a buildstep.

========= Started 'sh /builds/slave/try-and-api-11-000000000000000/tools/scripts/tooltool/tooltool_wrapper.sh ...' failed (results: 2, elapsed: 34 mins, 27 secs) (at 2015-05-19 10:48:31.061844) =========
sh /builds/slave/try-and-api-11-000000000000000/tools/scripts/tooltool/tooltool_wrapper.sh mobile/android/config/tooltool-manifests/android/releng.manifest http://tooltool.pvt.build.mozilla.org/build setup.sh /tools/tooltool.py
in dir /builds/slave/try-and-api-11-000000000000000/build (timeout 1200 secs)
watching logfiles {}
argv: ['sh', '/builds/slave/try-and-api-11-000000000000000/tools/scripts/tooltool/tooltool_wrapper.sh', 'mobile/android/config/tooltool-manifests/android/releng.manifest', 'http://tooltool.pvt.build.mozilla.org/build', 'setup.sh', '/tools/tooltool.py']

That needs to be (a) fixed to use the right URL and (b) fixed to use --authentication-file.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: buildbot-configs.patch

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: buildbotcustom.patch

This should work on Linux, but not on Windows.  Do we still use this code on Windows?

Similar to https://github.com/mozilla/build-mozharness/commit/e2a945297ac65326d8dfc83b3ffe15b157be05ac

Comment 3

[Michael Shal [:mshal]]

Comment on attachment 8607708 [details] [diff] [review]
buildbotcustom.patch

Looks like mozilla-release still uses buildbot on Windows... I guess we need a way to support that for the time being.

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: buildbotcustom-2.patch

Comment 5

[Michael Shal [:mshal]]

Comment on attachment 8608278 [details] [diff] [review]
buildbotcustom-2.patch

LGTM

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8608278 [details] [diff] [review]
buildbotcustom-2.patch

remote:   https://hg.mozilla.org/build/buildbotcustom/rev/ac3cd748df85

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8607705 [details] [diff] [review]
buildbot-configs.patch

remote:   https://hg.mozilla.org/build/buildbot-configs/rev/02abe4e20bf9

Comment 8

[Rail Aliiev [:rail]]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/02abe4e20bf9

Comment 9

[[e-mail] (:bhearsum)]

I think you accidentally landed on production-0.8 of buildbotcustom instead of default. We caught an error during a release config because of it:
cd master && /builds/buildbot/build1/bin/buildbot checkconfig
Traceback (most recent call last):
  File "/builds/buildbot/build1/lib/python2.7/site-packages/buildbot-0.8.2_hg_808f68cc2960_production_0.8-py2.7.egg/buildbot/scripts/runner.py", line 1042, in doCheckConfig
    ConfigLoader(configFileName=configFileName)
  File "/builds/buildbot/build1/lib/python2.7/site-packages/buildbot-0.8.2_hg_808f68cc2960_production_0.8-py2.7.egg/buildbot/scripts/checkconfig.py", line 31, in __init__
    self.loadConfig(configFile, check_synchronously_only=True)
  File "/builds/buildbot/build1/lib/python2.7/site-packages/buildbot-0.8.2_hg_808f68cc2960_production_0.8-py2.7.egg/buildbot/master.py", line 652, in loadConfig
    exec f in localDict
  File "/builds/buildbot/build1/master/master.cfg", line 78, in <module>
    getattr(passwords, 'secrets', None))
  File "/builds/buildbot/build1/lib/python2.7/site-packages/buildbotcustom/misc.py", line 2232, in generateBranchObjects
    tooltool_url_list=config.get('tooltool_url_list', []),
  File "/builds/buildbot/build1/lib/python2.7/site-packages/buildbotcustom/process/factory.py", line 5059, in __init__
    self.addTooltoolStep()
  File "/builds/buildbot/build1/lib/python2.7/site-packages/buildbotcustom/process/factory.py", line 393, in addTooltoolStep
    if self.platform.startswith('win'):
AttributeError: 'NoneType' object has no attribute 'startswith'


I backed out just the buildbotcustom portion, but I suspect it needs landing on default...

Comment 10

[Dustin J. Mitchell [:dustin] (he/him)]

Indeed.  That'll teach me to believe that 'hg rebase' "does the right thing" :(

Grafted to the correct branch.

remote:   https://hg.mozilla.org/build/buildbotcustom/rev/2d4ff2155676

Comment 11

[Justin Wood (:Callek)]

(In reply to Dustin J. Mitchell [:dustin] from comment #10)
> Indeed.  That'll teach me to believe that 'hg rebase' "does the right thing"
> :(
> 
> Grafted to the correct branch.
> 
> remote:   https://hg.mozilla.org/build/buildbotcustom/rev/2d4ff2155676

And I backed this out again since its still failing (see travis results for e.g. buildbot-configs repo)

https://hg.mozilla.org/build/buildbotcustom/rev/fd6476cc8f48

I also had to backout the configs patch (that switched tooltool servers) due to no NDK/SDK's being gettable without said token from the previous patch (broke on aurora):

https://hg.mozilla.org/build/buildbot-configs/rev/665c6d3d0b6f
https://hg.mozilla.org/build/buildbot-configs/rev/ddbb5bd78a83

c.f. http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-aurora-android-x86/1432246097/mozilla-aurora-android-x86-bm91-build1-build4.txt.gz


Order of Operations for this bug now:
* Fixup both patch (in any way required)
* Land custom's patch, make sure it deploys to prod (since it should be useable without configs patch)
* Land the config patch and then make sure it goes to prod and doesn't break aurora android builds.

Comment 12

[Dustin J. Mitchell [:dustin] (he/him)]

OK, I'll need some help with this patch, then, or just to wait until it's no longer relevant.

Until this lands or until we are building Android with mozharness (bug 1155349), Android builds can only use the old tooltool, which means that any updates to tooltool dependencies will need to be added by hand.

From comment 3, releases on Windows may also be affected, but I think the tooltool stuff required there is less dynamic.

Comment 13

[Jordan Lund (:jlund)]

we ran into an issue where we uplifted gecko's m-a->m-b which caused android manifests to assume new tooltool api/url: 1193052

Comment 14

[Chris AtLee [:catlee]]

Attachment: pass in relengapi authentication file to tooltool

looks like the valgrind builder was the only culprit here. I passed in platform to the factory and it now it passes both buildbotcustom and buildbot-config tests.

Comment 15

[Chris AtLee [:catlee]]

Comment on attachment 8607705 [details] [diff] [review]
buildbot-configs.patch

this got backed out

Comment 16

[Justin Wood (:Callek)]

Comment on attachment 8723278 [details] [diff] [review]
pass in relengapi authentication file to tooltool

Review of attachment 8723278 [details] [diff] [review]:
-----------------------------------------------------------------

if this passes checkconfig r+

Comment 17

[Chris AtLee [:catlee]]

https://hg.mozilla.org/build/buildbotcustom/rev/e3b7e854569bb8f1ea68288b4af165887fd42a1a
Bug 1166420 - pass in relengapi authentication file to tooltool r=Callek

Comment 18

[Chris AtLee [:catlee]]

https://hg.mozilla.org/build/buildbot-configs/rev/b335375d516b3cea969fa35bc5ce0ae9f30ffec4
Bug 1166420 - Switch to new tooltool endpoint r=mshal

Comment 19

[aleth [:aleth]]

(In reply to Chris AtLee [:catlee] from comment #17)
> https://hg.mozilla.org/build/buildbotcustom/rev/
> e3b7e854569bb8f1ea68288b4af165887fd42a1a
> Bug 1166420 - pass in relengapi authentication file to tooltool r=Callek

This broke c-c Linux builds, as the file doesn't seem to be found:
https://treeherder.mozilla.org/logviewer.html#?job_id=34384&repo=comm-central#L2113

tooltool_url_list seems to be correct.

Comment 20

[Chris AtLee [:catlee]]

Attachment: MozReview Request: Bug 1166420 - Add relengapi.tok to TB bb configs. r=dustin

Review commit: https://reviewboard.mozilla.org/r/38861/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/38861/

Comment 21

[Rail Aliiev [:rail]]

Comment on attachment 8728203 [details]
MozReview Request: Bug 1166420 - Add relengapi.tok to TB bb configs. r=dustin

https://reviewboard.mozilla.org/r/38861/#review35561

easy!

Comment 22

[Chris AtLee [:catlee]]

https://hg.mozilla.org/build/buildbot-configs/rev/d7776ec49bb4b3f5fd01ad195be24ab98b48b0c6
Bug 1166420 - Add relengapi.tok to TB bb configs. r=rail

Comment 23

[Release Engineering SlaveAPI Service]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/d7776ec49bb4

Comment 24

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8728203 [details]
MozReview Request: Bug 1166420 - Add relengapi.tok to TB bb configs. r=dustin

https://reviewboard.mozilla.org/r/38861/#review35579

I'll defer to rail's expertise