1083328 - Update bouncer entries for firefox-nightly-latest and firefox-aurora-latest

Status: RESOLVED DUPLICATE of bug 1432656

(Release Engineering :: Release Automation: Bouncer, defect)

Description

[Rail Aliiev [:rail]]

https://wiki.mozilla.org/ReleaseEngineering/Merge_Duty/Steps#Kick_off_m-c_and_m-a_nightlies requires us to update the location for these products (used but stub installer).

We could add another step in our nightly automation to check the current path and update it if it doesn't match.

Comment 1

[[e-mail] (:bhearsum)]

Going to have a look at this soon.

Comment 2

[[e-mail] (:bhearsum)]

After rereading the docs I realized there's a few things here. First off, we need a script that can do location edits. Then we need to call it as part of the nightly build automation (probably through mozharness mach, since that's where nightly builds will be done very soon now), but only for Firefox mozilla-central nightlies. Ideally we'd only do this on one platform. Then we need to update the aurora throttling docs to do something similar when we unthrottle it: https://wiki.mozilla.org/ReleaseEngineering/How_To/Enable_or_Disable_Updates_on_Aurora#Enabling_Updates

Comment 3

[[e-mail] (:bhearsum)]

Attachment: refactor bouncer submitter script into a mixin

Looks like Mozharness contains our Bouncer submitter logic these days, but it's all in a single script at the moment. This patch refactors it to put the core api stuff into a mixin, so that I can re-use it in a new script. This is pretty much just copy+paste, and I tested it against bouncer stage.

Comment 4

[[e-mail] (:bhearsum)]

Attachment: add script to update bouncer locations

So, this script seems to work fine. However, I'm starting to think about how to integrate with the nightly builds - because we'll want to be doing this as part of the mozilla-central en-US builds. I'm wondering if I should completely move the logic to the mixin so that the mozharness mach script can use that, instead of shelling out to this script. Jordan, what do you think?

Comment 5

[Rail Aliiev [:rail]]

Comment on attachment 8522341 [details] [diff] [review]
add script to update bouncer locations

Review of attachment 8522341 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm

::: configs/bouncer/mozilla-aurora.py
@@ +13,5 @@
> +            "win": "/%(product)s/nightly/latest-mozilla-aurora/%(product)s-%(version)s.en-US.win32.installer.exe",
> +        },
> +        # TODO: why does this l10n product exist? can't %(product)s-aurora-latest just use :lang?
> +        "%(product)s-aurora-latest-l10n": {
> +            "linux64": "/%(product)s/nightly/latest-mozilla-aurora/%(product)s-%(version)s.:lang.linux-x86_64.tar.bz2",

I think this is supposed to be "latest-mozilla-aurora-l10n", this is why we need this entry. :/

Comment 6

[[e-mail] (:bhearsum)]

(In reply to Rail Aliiev [:rail] from comment #5)
> Comment on attachment 8522341 [details] [diff] [review]
> add script to update bouncer locations
> 
> Review of attachment 8522341 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> lgtm
> 
> ::: configs/bouncer/mozilla-aurora.py
> @@ +13,5 @@
> > +            "win": "/%(product)s/nightly/latest-mozilla-aurora/%(product)s-%(version)s.en-US.win32.installer.exe",
> > +        },
> > +        # TODO: why does this l10n product exist? can't %(product)s-aurora-latest just use :lang?
> > +        "%(product)s-aurora-latest-l10n": {
> > +            "linux64": "/%(product)s/nightly/latest-mozilla-aurora/%(product)s-%(version)s.:lang.linux-x86_64.tar.bz2",
> 
> I think this is supposed to be "latest-mozilla-aurora-l10n", this is why we
> need this entry. :/

Oops, looks like I misread the live ones - they're pointing at -l10n. Thanks for catching this.

Comment 7

[Jordan Lund (:jlund)]

Comment on attachment 8522341 [details] [diff] [review]
add script to update bouncer locations

Review of attachment 8522341 [details] [diff] [review]:
-----------------------------------------------------------------

I am not familiar with bouncer and how it works. However, if we are going to be adding steps to nightly builds (I'm assuming by en-us nightlies you mean normal desktop ff nightlies and not l10n?), we should do this in a way that FxDesktopBuild can reach it.

Though there are a lot of criticisms of mozharness and how it is bursting at the seams with mixins. What might be best is making a stand alone obj:
    self.bouncer = Bouncer(self.config, self.log_obj)  # where self is instance of FxDesktopBuild and Bouncer is a combo of BouncerSubmitter and BouncerUpdater
    self.bouncer.submit() # or whatever

see what massimo did for proxy for context: http://hg.mozilla.org/build/mozharness/rev/574bfaee9e63

Comment 8

[[e-mail] (:bhearsum)]

Diving back into this now...

(In reply to Jordan Lund (:jlund) from comment #7)
> Comment on attachment 8522341 [details] [diff] [review]
> add script to update bouncer locations
> 
> Review of attachment 8522341 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> I am not familiar with bouncer and how it works. However, if we are going to
> be adding steps to nightly builds (I'm assuming by en-us nightlies you mean
> normal desktop ff nightlies and not l10n?), we should do this in a way that
> FxDesktopBuild can reach it.

en-US for sure. We need to update l10n bouncer entries too, but it's not clear to me where the right place to do that is. We can't update them until _all_ l10n nightlies have completed. I'll have to think on that part some more...

> Though there are a lot of criticisms of mozharness and how it is bursting at
> the seams with mixins. What might be best is making a stand alone obj:
>     self.bouncer = Bouncer(self.config, self.log_obj)  # where self is
> instance of FxDesktopBuild and Bouncer is a combo of BouncerSubmitter and
> BouncerUpdater
>     self.bouncer.submit() # or whatever
> 
> see what massimo did for proxy for context:
> http://hg.mozilla.org/build/mozharness/rev/574bfaee9e63

This is very helpful, thank you!

Comment 9

[[e-mail] (:bhearsum)]

(In reply to Ben Hearsum [:bhearsum] from comment #8)
> Diving back into this now...
> 
> (In reply to Jordan Lund (:jlund) from comment #7)
> > Comment on attachment 8522341 [details] [diff] [review]
> > add script to update bouncer locations
> > 
> > Review of attachment 8522341 [details] [diff] [review]:
> > -----------------------------------------------------------------
> > 
> > I am not familiar with bouncer and how it works. However, if we are going to
> > be adding steps to nightly builds (I'm assuming by en-us nightlies you mean
> > normal desktop ff nightlies and not l10n?), we should do this in a way that
> > FxDesktopBuild can reach it.
> 
> en-US for sure. We need to update l10n bouncer entries too, but it's not
> clear to me where the right place to do that is. We can't update them until
> _all_ l10n nightlies have completed. I'll have to think on that part some
> more...

The l10n part of this will be much easier once bug 740142 is completed, which will moved repacks to a set of chunked builders. Once that exists, it should be possible to have a job fire _after_ all chunks complete, which is the appropriate time to change the bouncer entry...

Comment 10

[[e-mail] (:bhearsum)]

We may need need the -l10n entries after all. They were a temporary measure for dev edition. I don't see us linking to them anywhere. Note to self: investigate this more. bug 1096393 may be useful.

Comment 11

[[e-mail] (:bhearsum)]

This work has been deprioritized for now.

Comment 12

[Nick Thomas [:nthomas] (UTC+12)]

In the meantime we have more products - https://bugzilla.mozilla.org/show_bug.cgi?id=994886#c3

Comment 13

[[e-mail] (:bhearsum)]

Is this fixed now?

Comment 14

[Rail Aliiev [:rail]]

Hmmmm, I'm not sure. Let me look around. :)

Comment 15

[Rail Aliiev [:rail]]

I don't see anything in mozharness that would update the aliases... Sounds like we still have some room for improvements. :D

Comment 16

[Mihai Tabara [:mtabara]⌚️GMT]

We added automation support for this last August. Work was tracked in bug 1432656, closing this as dupe.