1155238 - convert all uses of tooltool in automation to use the new relengapi tooltool

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

The will require changing URLs and using a relengapi token with permission to download public and internal files from tooltool.  The easy bit should be Mozharness.  Slightly harder are buildbot steps and shell scripts running tooltool.  Harder still will be URLs referring directly to files on tooltool.  Those should be converted to use tooltool with a cache, rather than fetching from S3 on every run.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

I have confirmed that the new tooltool.py can download from the old URLs, so the first step will be to deploy the new script everywhere.

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: bug1155238.patch

This is just copied from the tooltool repo.

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

1155256

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: bug1155238-mozharness.patch

This is a basic mozharness patch to update just the URLs.  I'll need to figure out how to test this in try, once the new tooltool.py is in place.

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: bug1155238-mozilla-central.patch

I'm not sure whether this would have to ride the trains -- is this used in automation, or just by devs?

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

The explicitly-linked files have sha512's

372c89f9dccaf5ee3b9d35fd1cfeb089e1e5db3ff1c04e35aa3adc8800bc61a2ae10e321f37ae7bab20b56e60941f91bb003bcb22035902a73
0748e900821820f1a42e2f1f3fa4d9002ef257c351b9e6b78e7de0ddd0202eace351f440372fbb1ae0b7e69e8361b036f6bd3362df99e67fc585082a311fc0df
4d8d7a37d90c34a2a2fda3066a8fe85c189b183d05389cb957fc6fed31f10a6924e50c1b84488ff61c015293803f58a3aed5d4819374d04c8e0ee2b9e3997278
dc9503b21c87b5a469118746f99e4f41d73888972ce735fa10a80f6d218086da0e3da525d9a4cd8e4ea497ec199fef720e4a525873d77a1af304ac505e076462

and all are public.

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: bug1155238-direct-links.patch

This takes the cheap way out and downloads these files directly from S3 on every run, without any caching.  That's slow and expensive, though.  Jonathan, can you help me figure out how to test the scripts that use these variables, so I can make the necessary changes?

mozharness/mozilla/gaia.py:    def extract_xre(self, xre_url, xre_path=None, parent_dir=None):
mozharness/mozilla/gaia.py:        filename = self.download_file(xre_url, parent_dir=parent_dir)
mozharness/mozilla/gaia.py:                  xre_url=None, build_config_path=None):
mozharness/mozilla/gaia.py:        if xre_url:
mozharness/mozilla/gaia.py:            self.extract_xre(xre_url, xre_path=xre_dir, parent_dir=gaia_dir)
mozharness/mozilla/testing/gaia_test.py:         "dest": "xre_url",
scripts/android_emulator_unittest.py:         "dest": "xre_url",
scripts/b2g_emulator_unittest.py:         "dest": "xre_url",
scripts/b2g_emulator_unittest.py:        self._download_unzip(self.config['xre_url'],
scripts/gaia_unit.py:                       xre_url=self.config.get('xre_url'),
scripts/luciddream_unittest.py:         "dest": "xre_url",
scripts/luciddream_unittest.py:                       xre_url=self.config.get('xre_url'),
scripts/marionette.py:         "dest": "xre_url",
scripts/marionette.py:                           xre_url=self.config.get('xre_url'),

scripts/b2g_emulator_unittest.py:         "dest": "busybox_url",
scripts/b2g_emulator_unittest.py:        if self.config.get('busybox_url'):
scripts/b2g_emulator_unittest.py:            self.download_file(self.config['busybox_url'],

Comment 9

[Michael Shal [:mshal]]

Comment on attachment 8593445 [details] [diff] [review]
bug1155238.patch

-#! <%= @python %>
+#!/usr/bin/env python

I thought the <%= @python %> thing was needed to get the python path correct on all platforms, but it's been a while since I've looked. Has that changed? If you're sure it's fine, I can switch to r+.

Comment 10

[Dustin J. Mitchell [:dustin] (he/him)]

Oh, thanks for spotting that -- the hashbang line should remain unchanged.

Comment 11

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: bug1155238-p2.patch

Comment 12

[Michael Shal [:mshal]]

Comment on attachment 8593480 [details] [diff] [review]
bug1155238-mozharness.patch

If there's only a single tooltool URL now, I think they can probably be removed from configs/* and just use the defaults in scripts/*, but no need to redo the patch.

If you don't feel like setting up a mozharness user repo to test on try, feel free to ping me & I can help test it.

Comment 13

[Chris Cooper [:coop] (he/him)]

Comment on attachment 8593486 [details] [diff] [review]
bug1155238-mozilla-central.patch

Review of attachment 8593486 [details] [diff] [review]:
-----------------------------------------------------------------

::: js/src/devtools/automation/winbuildenv.sh
@@ +33,5 @@
>  # newer mozilla-build or tooltool.
>  if ! which mozmake 2>/dev/null; then
>      export PATH="$PATH:$SOURCE/.."
>      if ! which mozmake 2>/dev/null; then
> +	TT_SERVER=${TT_SERVER:-https://api.pub.build.mozilla.org/tooltool/}

Only concern is the trailing slash. Previous version didn't have it.

Comment 14

[Dustin J. Mitchell [:dustin] (he/him)]

The trailing slash is necessary now.

I'll try the defaults-in-scripts idea.  There's always been only a single tooltool URL, so I'm not sure why that wasn't done to begin with!

Comment 15

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8593511 [details] [diff] [review]
bug1155238-p2.patch

remote:   https://hg.mozilla.org/build/puppet/rev/0cdc5066f976
remote:   https://hg.mozilla.org/build/puppet/rev/34c98b742ec0

Note that this still has to be deployed on Windows before I can land any of the other patches.

Comment 16

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: bug1155238-mozharness-r2.patch

This removes all tooltool_servers config, but leaves the old-old-old tooltool_url config in place.  This is a little more than a search-and-replace, but hopefully testing it on a few builders will be enough to indicate it's correct.

Comment 17

[Michael Shal [:mshal]]

Comment on attachment 8593555 [details] [diff] [review]
bug1155238-mozharness-r2.patch

>-    "busybox_url": "http://tooltool.pvt.build.mozilla.org/build/sha512/0748e900821820f1a42e2f1f3fa4d9002ef257c351b9e6b78e7de0ddd0202eace351f440372fbb1ae0b7e69e8361b036f6bd3362df99e67fc585082a311fc0df",
>-    "xre_url": "http://tooltool.pvt.build.mozilla.org/build/sha512/dc9503b21c87b5a469118746f99e4f41d73888972ce735fa10a80f6d218086da0e3da525d9a4cd8e4ea497ec199fef720e4a525873d77a1af304ac505e076462",
>-    "tooltool_servers": ["https://api.pub.build.mozilla.org/tooltool/"],
>+    "busybox_url": "https://api.pub.build.mozilla.org/tooltool/sha512/0748e900821820f1a42e2f1f3fa4d9002ef257c351b9e6b78e7de0ddd0202eace351f440372fbb1ae0b7e69e8361b036f6bd3362df99e67fc585082a311fc0df",
>+    "xre_url": "https://api.pub.build.mozilla.org/tooltool/sha512/dc9503b21c87b5a469118746f99e4f41d73888972ce735fa10a80f6d218086da0e3da525d9a4cd8e4ea497ec199fef720e4a525873d77a1af304ac505e076462",

It looks like you combined this with the direct-links patch? Not sure if that was intended.

In any case, this looks good to me, though we definitely need to test a bunch of builds.

Comment 18

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: bug1155238-puppet-token.patch

Comment 19

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: bug1155238-mozharness-r3.patch

Corrected patch, carrying forward review from attachment 8593555 [details] [diff] [review].

Comment 20

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: bug1155238-auth-file.patch

Comment 21

[Jordan Lund (:jlund)]

Comment on attachment 8594039 [details] [diff] [review]
bug1155238-auth-file.patch

Review of attachment 8594039 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozharness/mozilla/tooltool.py
@@ +21,5 @@
>  ]
>  
> +# set the default authentication file based on platform; this corresponds to where
> +# puppet puts the token
> +if sys.platform == 'win32':

can we define this within the TooltoolMixin class so we can avail of things like _is_windows():
http://mxr.mozilla.org/build/source/mozharness/mozharness/base/script.py#122

Comment 22

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: bug1155238-auth-file-r2.patch

Comment 23

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8593999 [details] [diff] [review]
bug1155238-puppet-token.patch

remote:   https://hg.mozilla.org/build/puppet/rev/d613d86bc91d
remote:   https://hg.mozilla.org/build/puppet/rev/6d5f6e258698

Comment 24

[Dustin J. Mitchell [:dustin] (he/him)]

All MH patches are being tested in https://treeherder.mozilla.org/#/jobs?repo=try&revision=dab022900494

If that goes well, I'll merge the m-c patch (attachment 8593486 [details] [diff] [review]) and do a full try run.

Comment 25

[Jonathan Griffin (:jgriffin)]

(In reply to Dustin J. Mitchell [:dustin] from comment #7)
> Created attachment 8593502 [details] [diff] [review]
> bug1155238-direct-links.patch
> 
> This takes the cheap way out and downloads these files directly from S3 on
> every run, without any caching.  That's slow and expensive, though. 
> Jonathan, can you help me figure out how to test the scripts that use these
> variables, so I can make the necessary changes?
> 

The easiest way would be to land these changes on mozharness default and then update the mozharness pointer on cedar, and see what happens.

Comment 26

[Jonathan Griffin (:jgriffin)]

Or, try works too.  :)

Comment 27

[Dustin J. Mitchell [:dustin] (he/him)]

I added attachment 8593486 [details] [diff] [review] and pushed a full run to try:
  https://treeherder.mozilla.org/#/jobs?repo=try&revision=c5a3d30c4152

Comment 28

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: bug1155238-puppetagain-all.patch

My try run failed a bunch of test jobs because the token was only installed on builders.  Oops :)

Comment 29

[Dustin J. Mitchell [:dustin] (he/him)]

Just waiting on deployment on Windows before I can test this again.

Comment 30

[Jordan Lund (:jlund)]

Comment on attachment 8594770 [details] [diff] [review]
bug1155238-auth-file-r2.patch

Review of attachment 8594770 [details] [diff] [review]:
-----------------------------------------------------------------

I've been at a work-week the last few days and thus slow to clear bug review queue. Apologies for the delay!

Comment 31

[Dustin J. Mitchell [:dustin] (he/him)]

I'm re-triggering a fwe jobs on my last try push to see if things are working now.  If so, I'll push a fresh commit to try, since this one may be a bit rotten.

Comment 32

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: bug1155238-config.patch

Comment 33

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8598841 [details] [diff] [review]
bug1155238-config.patch

wrong patch

Comment 34

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: bug1155238-puppetagain-config.patch

Comment 35

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8598851 [details] [diff] [review]
bug1155238-puppetagain-config.patch

remote:   https://hg.mozilla.org/build/puppet/rev/212a8079cac5
remote:   https://hg.mozilla.org/build/puppet/rev/d6e2251b93c1

and again we wait for AMIs

Comment 36

[Dustin J. Mitchell [:dustin] (he/him)]

New, rebased try push: https://treeherder.mozilla.org/#/jobs?repo=try&revision=7329bf847055

Comment 37

[Dustin J. Mitchell [:dustin] (he/him)]

That's looking green, and I'm seeing tooltool hits at
  https://papertrailapp.com/groups/1141234/events?q=api+%2Ftooltool%2F
so I'm going to push a full try job.

Comment 38

[Dustin J. Mitchell [:dustin] (he/him)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=3315413ccd0a

Comment 39

[Dustin J. Mitchell [:dustin] (he/him)]

Hm, tooltool_wrapper.sh appears to not be passing any authentication info:

11:09:51     INFO - ['sh', '/builds/slave/try-lx-00000000000000000000000/build/tools/scripts/tooltool/tooltool_wrapper.sh', '/builds/slave/try-lx-00000000000000000000000/build/src/browser/config/tooltool-manifests/linux32/releng.manifest', 'https://api.pub.build.mozilla.org/tooltool/', 'setup.sh', '/builds/tooltool.py']
11:09:51     INFO - Running command: ['sh', '/builds/slave/try-lx-00000000000000000000000/build/tools/scripts/tooltool/tooltool_wrapper.sh', '/builds/slave/try-lx-00000000000000000000000/build/src/browser/config/tooltool-manifests/linux32/releng.manifest', 'https://api.pub.build.mozilla.org/tooltool/', 'setup.sh', '/builds/tooltool.py'] in /builds/slave/try-lx-00000000000000000000000/build/src
11:09:51     INFO - Copy/paste: sh /builds/slave/try-lx-00000000000000000000000/build/tools/scripts/tooltool/tooltool_wrapper.sh /builds/slave/try-lx-00000000000000000000000/build/src/browser/config/tooltool-manifests/linux32/releng.manifest https://api.pub.build.mozilla.org/tooltool/ setup.sh /builds/tooltool.py
11:09:51     INFO -  /builds/slave/try-lx-00000000000000000000000/build/src/browser/config/tooltool-manifests/linux32/releng.manifest content
11:09:51     INFO -  =======================================================
11:09:51     INFO -  [
11:09:51     INFO -  {
11:09:51     INFO -  "size": 82,
11:09:51     INFO -  "digest": "70a6126249e40aa1da32248bf6bfe45e0d8c87334579ec0cf69403e61b635e27c766d9bf08d530978286552f158ee24c74b0168a57cc6b734dcfed4fc5e09cff",
11:09:51     INFO -  "algorithm": "sha512",
11:09:51     INFO -  "filename": "setup.sh"
11:09:51     INFO -  },
11:09:51     INFO -  {
11:09:51     INFO -  "size": 80458572,
11:09:51     INFO -  "digest": "e5101f9dee1e462f6cbd3897ea57eede41d23981825c7b20d91d23ab461875d54d3dfc24999aa58a31e8b01f49fb3140e05ffe5af2957ef1d1afb89fd0dfe1ad",
11:09:51     INFO -  "algorithm": "sha512",
11:09:51     INFO -  "filename": "gcc.tar.xz"
11:09:51     INFO -  },
11:09:51     INFO -  {
11:09:51     INFO -  "size": 167175,
11:09:51     INFO -  "digest": "0b71a936edf5bd70cf274aaa5d7abc8f77fe8e7b5593a208f805cc9436fac646b9c4f0b43c2b10de63ff3da671497d35536077ecbc72dba7f8159a38b580f831",
11:09:51     INFO -  "algorithm": "sha512",
11:09:51     INFO -  "filename": "sccache.tar.bz2"
11:09:51     INFO -  }
11:09:51     INFO -  ]
11:09:51     INFO -  =======================================================
11:09:51     INFO -  Fetching...
11:09:51     INFO -  retry: Calling run_with_timeout with args: (['/builds/tooltool.py', '--url', 'https://api.pub.build.mozilla.org/tooltool/', '--overwrite', '-m', '/builds/slave/try-lx-00000000000000000000000/build/src/browser/config/tooltool-manifests/linux32/releng.manifest', 'fetch', '-c', '/builds/tooltool_cache'], 300, None, None, False, True), kwargs: {}, attempt #1
11:09:51     INFO -  Executing: ['/builds/tooltool.py', '--url', 'https://api.pub.build.mozilla.org/tooltool/', '--overwrite', '-m', '/builds/slave/try-lx-00000000000000000000000/build/src/browser/config/tooltool-manifests/linux32/releng.manifest', 'fetch', '-c', '/builds/tooltool_cache']
11:09:51     INFO -  INFO - File setup.sh not present in local cache folder /builds/tooltool_cache

which is working for most builds because they're all public files.

Comment 40

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: MozReview Request: bz://1155238/dustin

/r/7891 - Bug 1155238: replace direct links to old filenames with direct links to new filenames
/r/7893 - Bug 1155238: use the RelengAPI tooltool URL everywhere
/r/7895 - Bug 1155238: pass --authentication-file to tooltool
/r/7897 - Bug 1155238: pass --authentication-file to the old tooltool_wrapper.sh, too

Pull down these commits:

hg pull -r fbe718fe444062cc439d8ccd8736cbb77bdd6be3 https://reviewboard-hg.mozilla.org/build-mozharness

Comment 41

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8599464 [details]
MozReview Request: bz://1155238/dustin

I'm a little new to mozreview, but three of the four csets here have already been reviewed via bugzilla, so I'm only looking for r+ for the last one.

Comment 42

[Dustin J. Mitchell [:dustin] (he/him)]

https://reviewboard.mozilla.org/r/7891/#review6647

Carried forward from :jgriffin on bugzilla

Comment 43

[Dustin J. Mitchell [:dustin] (he/him)]

https://reviewboard.mozilla.org/r/7893/#review6649

r+ carried forward from :mshal on bugzilla

Comment 44

[Dustin J. Mitchell [:dustin] (he/him)]

https://reviewboard.mozilla.org/r/7895/#review6651

r+ carried forward from :jlund on bugzilla

Comment 45

[Dustin J. Mitchell [:dustin] (he/him)]

Looks like there are some XP and W7 hosts that don't have c:\builds\relengapi.tok installed -- getting some reds like https://treeherder.mozilla.org/logviewer.html#?job_id=7068715&repo=try

Mark will have a look-see.

Comment 46

[Dustin J. Mitchell [:dustin] (he/him)]

OK, that looks to have been per-machine issues.  I made one more try push with attachment 8599464 [details] included (using --authentication-file with tooltool_wrapper.sh), both to test that commit and to get a little more confidence that this isn't going to burn trees tomorrow morning.  Speaking of which, I'd like to land this tomorrow morning, mshal :)

https://treeherder.mozilla.org/#/jobs?repo=try&revision=c7b277b5d8b9

Comment 47

[Armen [:armenzg]]

With these changes, we have lost the ability of a developer to run any mozharness jobs which requires tooltool.
This is important to allow people run mozharness without having to request a loaner.

Bug 1088208 contains the information about this feature.

Comment 48

[Dustin J. Mitchell [:dustin] (he/him)]

Only with the latest tooltool.py.  I didn't realize things were automatically pulling the latest.  Using an older tooltool works fine, or applying the mozharness patches here (which I'm working on pushing to production today).

Comment 49

[Pulsebot]

https://hg.mozilla.org/integration/fx-team/rev/c982ce92229b

Comment 50

[Dustin J. Mitchell [:dustin] (he/him)]

https://hg.mozilla.org/build/mozharness/rev/06a2bc3e6b42

Like a dummy I forgot the r='s in the component patches.

Armen's asked me to make a new try push with `mozharness.json` pointing to the latest default.  I'll get that started shortly (I don't have a local clone of m-c so it will be a while).  Once that's done, I think we can bump the revision in m-c.

Comment 51

[Dustin J. Mitchell [:dustin] (he/him)]

Armen kindly pushed to try for me:
  https://treeherder.mozilla.org/#/jobs?repo=try&revision=67428d309732

Comment 52

[Michael Shal [:mshal]]

https://reviewboard.mozilla.org/r/7891/#review6695

Ship It!

Comment 53

[Pulsebot]

https://hg.mozilla.org/integration/fx-team/rev/90cf2e995537

Comment 54

[Dustin J. Mitchell [:dustin] (he/him)]

As the pulsebot posts are indicating, this has been completely landed, merged, bumped, and otherwise *'ed; I'll be watching for use of the old vhosts over the next few weeks, and eventually shut them off.

Comment 55

[Rail Aliiev [:rail]]

mozharness production tag moved to: https://hg.mozilla.org/build/mozharness/rev/ca580fcfa627

Comment 56

[Rail Aliiev [:rail]]

mozharness production tag moved to: https://hg.mozilla.org/build/mozharness/rev/ca580fcfa627

Comment 57

[Rail Aliiev [:rail]]

mozharness production tag moved to: https://hg.mozilla.org/build/mozharness/rev/ca580fcfa627

Comment 58

[Rail Aliiev [:rail]]

mozharness production tag moved to: https://hg.mozilla.org/build/mozharness/rev/ca580fcfa627

Comment 59

[Rail Aliiev [:rail]]

mozharness production tag moved to: https://hg.mozilla.org/build/mozharness/rev/ca580fcfa627

Comment 60

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/c982ce92229b
https://hg.mozilla.org/mozilla-central/rev/90cf2e995537

Comment 61

[Rail Aliiev [:rail]]

mozharness production tag moved to: https://hg.mozilla.org/build/mozharness/rev/4d260376921f

Comment 62

[Rail Aliiev [:rail]]

mozharness production tag moved to: https://hg.mozilla.org/build/mozharness/rev/4d260376921f

Comment 63

[Rail Aliiev [:rail]]

mozharness production tag moved to: https://hg.mozilla.org/build/mozharness/rev/4d260376921f

Comment 64

[Rail Aliiev [:rail]]

mozharness production tag moved to: https://hg.mozilla.org/build/mozharness/rev/4d260376921f

Comment 65

[Rail Aliiev [:rail]]

mozharness production tag moved to: https://hg.mozilla.org/build/mozharness/rev/4d260376921f

Comment 67

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: MozReview Request: Bug 1155238: pass --authentication-file to the old tooltool_wrapper.sh, too

Comment 68

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: MozReview Request: Bug 1155238: replace direct links to old filenames with direct links to new filenames

Comment 69

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: MozReview Request: Bug 1155238: use the RelengAPI tooltool URL everywhere

Comment 70

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: MozReview Request: Bug 1155238: pass --authentication-file to tooltool