1191930 - Migrate Thunderbird from tooltool.pvt to api.pub

Status: RESOLVED FIXED

(Release Engineering :: Release Automation, defect)

Description

[Philipp Kewisch [:Fallen][📱📆]]

(Edmund Wong (:ewong) from bug 1190124 comment #5)
> TB's releng run_tooltool step is using tooltool.pvt.build.mozilla.org and
> Firefox is using the new api.pub.build.mozilla.org.
> 
> IOW, I'm guessing there's another sync issue with the api repo having
> the new gtk3.tar.xz and  the tooltool.pvt (which is being phased out soon)
> doesn't.
> 
> The short term solution is to copy the gtk3.tar.xz over to the tooltool repo.
> Long term... fix TB's releng run_tooltool step to use the api.pub.. if it
> is ready.. (this part, I don't know) 


It was determined that moving to api.pub is the right step. It seems we just need to change the url.

Comment 1

[Philipp Kewisch [:Fallen][📱📆]]

Callek, is it correct that all we have to change is the following locations to the new URL?

https://mxr.mozilla.org/build/search?string=tooltool.pvt&find=buildbot-configs&tree=build

Comment 2

[Justin Wood (:Callek)]

(In reply to Philipp Kewisch [:Fallen] from comment #1)
> Callek, is it correct that all we have to change is the following locations
> to the new URL?
> 
> https://mxr.mozilla.org/build/search?string=tooltool.pvt&find=buildbot-
> configs&tree=build

yes, IFF nothing that is using tooltool.pvt is also trying to download private archives.

That said, the ideal is to get off tooltool.pvt anyway, and I don't know of anything in TB land that needs a private archive so we can probably go forward with this, and if anything breaks we know that is something we need to add a relengapi token to.

Comment 3

[Philipp Kewisch [:Fallen][📱📆]]

(In reply to Justin Wood (:Callek) from comment #2)
> (In reply to Philipp Kewisch [:Fallen] from comment #1)
> yes, IFF nothing that is using tooltool.pvt is also trying to download
> private archives.
How would I found out about this? I'm not aware of anything Thunderbird specific, except maybe that oauth file.

Comment 4

[Philipp Kewisch [:Fallen][📱📆]]

Attachment: Fix - v1

In addition to my question from comment 3, what is the right url? Looking at the tooltool source and api doc, it seems it would be https://api.pub.build.mozilla.org/tooltool/

I'm really just stabbing in the dark here as I don't have the environment set up to test this. Maybe someone could give me a full overview of what needs to be done?

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8645345 [details] [diff] [review]
Fix - v1

Review of attachment 8645345 [details] [diff] [review]:
-----------------------------------------------------------------

That looks about right, to the extent I understand buildbot-configs.

Comment 6

[Philipp Kewisch [:Fallen][📱📆]]

Comment on attachment 8645345 [details] [diff] [review]
Fix - v1

Great, thanks for the feedback. In that case upgrading to review.

Comment 7

[Justin Wood (:Callek)]

Comment on attachment 8645345 [details] [diff] [review]
Fix - v1

Review of attachment 8645345 [details] [diff] [review]:
-----------------------------------------------------------------

I'm pretty sure this is accurate, lets go for it.

Comment 8

[Philipp Kewisch [:Fallen][📱📆]]

https://hg.mozilla.org/build/buildbot-configs/rev/46beb09ecb87

Comment 9

[Release Engineering SlaveAPI Service]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/46beb09ecb87

Comment 10

[Jordan Lund (:jlund)]

hrm, did we address https://bugzilla.mozilla.org/show_bug.cgi?id=1166420#c12 ?

I thought we couldn't use new tooltool api on buildbot based builds yet?

Comment 11

[Jordan Lund (:jlund)]

Comment on attachment 8645345 [details] [diff] [review]
Fix - v1

Review of attachment 8645345 [details] [diff] [review]:
-----------------------------------------------------------------

backed out because of https://bugzilla.mozilla.org/show_bug.cgi?id=1198571 and comment 10

reconfig'd backout to all masters at 20:10 PT

Comment 12

[Dustin J. Mitchell [:dustin] (he/him)]

We can't use the new tooltool *authentication* on buildbot-based builds.  Which means that internal-only files like the NDK and SDK aren't accessible from the new tooltool URL.

Comment 13

[Philipp Kewisch [:Fallen][📱📆]]

Reopening this issue since we want to migrate for Thunderbird (and leave the others alone this time).

Comment 14

[Philipp Kewisch [:Fallen][📱📆]]

Attachment: Fix - v2

This patch adds the same keys to Thunderbird's GLOBAL_VARS, which IIUC overwrites the global vars defined in production_config/staging_config. If all goes well this patch would make the changes for Thunderbird only.

Comment 15

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8657540 [details] [diff] [review]
Fix - v2

I'm not a good person for a buildbot-configs review.  Nick, perhaps, as module owner?

Comment 16

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8657540 [details] [diff] [review]
Fix - v2

>diff --git a/mozilla/thunderbird_config.py b/mozilla/thunderbird_config.py
>+    'tooltool_url_list': ['https://api.pub.build.mozilla.org/tooltool/'],

This one isn't necessary, as it's in the production and staging configs. I leave it up to you if you land the other changes in this file.

I've double checked this only affects Thunderbird jobs.

Comment 17

[Philipp Kewisch [:Fallen][📱📆]]

(In reply to Nick Thomas [:nthomas] from comment #16)
> Comment on attachment 8657540 [details] [diff] [review]
> Fix - v2
> 
> >diff --git a/mozilla/thunderbird_config.py b/mozilla/thunderbird_config.py
> >+    'tooltool_url_list': ['https://api.pub.build.mozilla.org/tooltool/'],
> 
> This one isn't necessary, as it's in the production and staging configs. I
> leave it up to you if you land the other changes in this file.
Ah yes, I see now that thunderbird_localconfig is symlinked to either production or staging config. In that case, wouldn't it make more sense to leave out the changes in the {production,staging} config files and just add it to thunderbird_config.py? Or are there other scripts that only load the production or staging config?

Comment 18

[Nick Thomas [:nthomas] (UTC+12)]

We could do that I guess, my comment was just following the example Firefox is setting.

Comment 19

[Philipp Kewisch [:Fallen][📱📆]]

Ok, then lets do what Firefox is doing :)

https://hg.mozilla.org/build/buildbot-configs/rev/e4776b864fa4

Comment 20

[Philipp Kewisch [:Fallen][📱📆]]

Attachment: Fix - v3

(patch as checked in)

Comment 21

[Chris Cooper [:coop] (he/him)]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/e4776b864fa4