1497097 - Automate bouncer aliases for partner builds

Status: RESOLVED FIXED

(Release Engineering :: Release Automation, enhancement, P2)

Description

[Nick Thomas [:nthomas] (UTC+12)]

We've started to get into longer-lived partnerships where we also host the download page and builds, so there is configuration in bouncer which needs updating on each new release from mozilla-release. Bug 1495050 is the most recent example of this. Updating the locations in bouncer is a manual task for releaseduty and this bug is about automating that.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

I had a look at the bouncerscipt code [1] which interacts with Bouncer (download.m.o) and it would need some changes to support partner builds. 

In an ideal world
* we'd have parameters in repack.cfg for each partner build which wanted bouncer changes, if it's acceptable to have this outside the tree
* use those to create partner-equivalent jobs for 
  * release-bouncer-sub-firefox (promote), which creates a Firefox-{version}-{partner} product and associated locations
  * release-bouncer-aliases-firefox (ship), which updates the partner alias to point to the product above when we ship

NB: we're currently setting the locations to the candidates directory. Bonus points for beetmoving those into releases instead as part of push.

We'd want to do this on mozilla-release but not mozilla-beta, even though we create partner builds for the 2nd half of the beta cycle using the same configuration.

Any thoughts Mihai ?

[1] https://github.com/mozilla-releng/bouncerscript/

Comment 2

[Mihai Tabara [:mtabara]⌚️GMT]

(In reply to Nick Thomas [:nthomas] (UTC+13) from comment #1)
> I had a look at the bouncerscipt code [1] which interacts with Bouncer
> (download.m.o) and it would need some changes to support partner builds. 
> 
> In an ideal world
> * we'd have parameters in repack.cfg for each partner build which wanted
> bouncer changes, if it's acceptable to have this outside the tree
> * use those to create partner-equivalent jobs for 
>   * release-bouncer-sub-firefox (promote), which creates a
> Firefox-{version}-{partner} product and associated locations
>   * release-bouncer-aliases-firefox (ship), which updates the partner alias
> to point to the product above when we ship

Suppose we have those outside the tree. We'd need to use them at transforms generation time anyway in order to create the tasks we want in the release graphs, right? So I think I'm fine with that, since the bouncerworker tasks won't actually fail CoT as logic will happen in the transforms. Question is though - can we get our hands at transforms level on those repack.cfg for each partner build?

> NB: we're currently setting the locations to the candidates directory. Bonus
> points for beetmoving those into releases instead as part of push.

This should be easy from beetmover perspective as we'd only need to change the regexes that allow more stuff to be published. Since we have them in ~candidates, I assume having them in ~releases is equally fine from a privacy point of view.
 
> We'd want to do this on mozilla-release but not mozilla-beta, even though we
> create partner builds for the 2nd half of the beta cycle using the same
> configuration.

This is doable, AFAIK. I was going to suggest we already have this behavior implemented in the release-mark-as-started tasks (which is suppose to create that task for mozilla-beta releases only as we turned Ship-it v2 on solely for betas. However, by looking at the code[0], I don't necessarily see what I was hoping for as the code exists on mozilla-beta but not on central, nor on relase (rides the trains?). In any case, it's definitely doable with amending some of the graphs configurations. (run-on-projects or something) 

> Any thoughts Mihai ?

What's the main blocker thing from your perspective?


[0]: https://hg.mozilla.org/releases/mozilla-beta/file/default/taskcluster/ci/release-mark-as-started/kind.yml

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #2)
> Suppose we have those outside the tree. We'd need to use them at transforms
> generation time anyway in order to create the tasks we want in the release
> graphs, right?

Yes, we read all the partner configs from their github repos when the decision task runs, via https://hg.mozilla.org/mozilla-central/file/default/taskcluster/taskgraph/util/partners.py#l284, or pass in a config directly. Bouncer could be one or more extra parameters coming in that same mechanism. Now that you point it out it doesn't seem like a change at all.

> So I think I'm fine with that, since the bouncerworker tasks
> won't actually fail CoT as logic will happen in the transforms. Question is
> though - can we get our hands at transforms level on those repack.cfg for
> each partner build?

Yes, we have a few transforms calling get_partner_config_by_kind() already. I don't really mind if we have a partner bouncer per graph or per partner.
 
> This should be easy from beetmover perspective as we'd only need to change
> the regexes that allow more stuff to be published. Since we have them in
> ~candidates, I assume having them in ~releases is equally fine from a
> privacy point of view.

Great! Just need to bikeshed on a location then :) pub/firefox/partner-releases/ would require us to modify AWS permissions, so pub/firefox/releases/<version>/partners or pub/firefox/releases/partners/<version> ?
  
> > We'd want to do this on mozilla-release but not mozilla-beta, even though we
> > create partner builds for the 2nd half of the beta cycle using the same
> > configuration.
> 
> This is doable, AFAIK. I was going to suggest we already have this behavior
> implemented in the release-mark-as-started tasks (which is suppose to create
> that task for mozilla-beta releases only as we turned Ship-it v2 on solely
> for betas. 

Tom pointed me to the run-on-releases work he added in https://phabricator.services.mozilla.com/D5788 which could be useful here. It's in 64+.

> What's the main blocker thing from your perspective?

There's a moderate amount of work here so I wanted to make sure the approach was sound (sounds like it is), and it's worth doing (the future is hard to predict).

Comment 4

[Mihai Tabara [:mtabara]⌚️GMT]

(In reply to Nick Thomas [:nthomas] (UTC+13) from comment #3)
> Yes, we have a few transforms calling get_partner_config_by_kind() already.
> I don't really mind if we have a partner bouncer per graph or per partner.

Great!

> Great! Just need to bikeshed on a location then :)
> pub/firefox/partner-releases/ would require us to modify AWS permissions, so
> pub/firefox/releases/<version>/partners or
> pub/firefox/releases/partners/<version> ?

Frankly, both LGTM. We already have http://archive.mozilla.org/pub/firefox/releases/partners/ so we could stick to it?
  
> > > We'd want to do this on mozilla-release but not mozilla-beta, even though we
> > > create partner builds for the 2nd half of the beta cycle using the same
> > > configuration.
> > 
> > This is doable, AFAIK. I was going to suggest we already have this behavior
> > implemented in the release-mark-as-started tasks (which is suppose to create
> > that task for mozilla-beta releases only as we turned Ship-it v2 on solely
> > for betas. 
> 
> Tom pointed me to the run-on-releases work he added in
> https://phabricator.services.mozilla.com/D5788 which could be useful here.
> It's in 64+.

Oh, that's even better, TIL! 
 
> > What's the main blocker thing from your perspective?
> 
> There's a moderate amount of work here so I wanted to make sure the approach
> was sound (sounds like it is), and it's worth doing (the future is hard to
> predict).

Yep, it does sound good to me. Depending on the timeline, I might plugin to help too at some point in Q4.
If not, am happy to help with reviews.

Thanks for working on this!

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #4)
> Frankly, both LGTM. We already have
> http://archive.mozilla.org/pub/firefox/releases/partners/ so we could stick
> to it?

I'd forgotten about that, looks like a good spot.

> Thanks for working on this!

I see what you did there. :-)

Comment 6

[Mihai Tabara [:mtabara]⌚️GMT]

Found this in triaging.

In the context of election-bundle alias update that we seem to forget to update at times (thanks Nick for taking care of it when releaseduty missed this step ^_^ ), do we want to prioritize this a bit this quarter or are we stil blocked on not having the parameters in repack.cfg for each partner build?

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

I've re-asked if we're going to continue with the election bundle over in bug 1495050. Based on the DAU we're not getting a lot of traffic to the download page, but there may be other considerations.

It's relatively simple to add new configuration into the appropriate repack.cfg, and there's little risk of it causing issues elsewhere. Most of the work is in taskgraph generation and beetmover adjustments. I don't see any time for that in my near future.

Comment 8

[Mihai Tabara [:mtabara]⌚️GMT]

(In reply to Nick Thomas [:nthomas] (UTC+13) from comment #7)

I've re-asked if we're going to continue with the election bundle over in bug 1495050. Based on the DAU we're not getting a lot of traffic to the download page, but there may be other considerations.

It's relatively simple to add new configuration into the appropriate repack.cfg, and there's little risk of it causing issues elsewhere. Most of the work is in taskgraph generation and beetmover adjustments. I don't see any time for that in my near future.

Thanks for the input. I think the amount of in-tree and beetmoverscript changes should be fairly minimal.
Will put this bug in my backlog, maybe I'll have some time in February to glance at it.

Comment 9

[Jordan Lund (:jlund)]

bumped manually to 65.0.1

Comment 10

[Mihai Tabara [:mtabara]⌚️GMT]

bumped manually to 66.0

Comment 11

[Nick Thomas [:nthomas] (UTC+12)]

A lot of this is going to happen in bug 1538995 where I'm adding support for partner stub installers, with slightly different implementation than discussed here

• new bouncer submission job for partner products, release-partner-repack-bouncer-sub-firefox
• extended firefox-push-to-release to copy into firefox/releases/partners/
• extended release-bouncer-aliases-firefox to update partner aliases, which have a slightly different form - partner-firefox-<branch>-<partner>-<partner_distro>-<suffix>, where branch is beta/release/esr, suffix is latest or stub

It doesn't handle the full but not stub installer case, but generally we want the stub installer because it makes install-time decisions about the 32/64-bit architecture to use. There may be some followup work to adjust for the funnelcake case, either in the automation or in bedrock, but the preference is to deprecate funnelcake if we can.

Comment 12

[Nick Thomas [:nthomas] (UTC+12)]

I'm going to call this fixed.