Closed Bug 1387622 Opened 4 years ago Closed 3 years ago

Add latest-ssl aliases for bedrock to use

Categories

(Release Engineering :: Release Automation: Other, enhancement, P1)

enhancement

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: mtabara, Assigned: nthomas)

References

Details

(Whiteboard: [releaseduty])

Attachments

(3 files)

So this is quite an interesting thing I discovered today while talking to :catlee about the 55.0 release.

Context:
===
1. Among the first tasks to run in the beta/release graph, there is the bouncer submission one. This task lies here[1] and uses this config file[2] (e.g. beta). For each of those products, per each platform it submits all those paths (paths to mirrors future location of the artifacts) to bouncer[3].

2. "release-localtest" updates are served directly from the candidates directory so this is easy.

3. "release-cdntest" updates are served via downloads page[4]. The downloads page gets its information from bouncer. If we haven't pushed to mirrors yet, all these updates, even though retrieved as valid urls from Balrog, would result in 404. That is because bouncer checks (well, technically Sentry) if files exist at the locations before redirecting to them. Since no push-to-mirrors happened, there's gonna be 404. So as soon as we complete this step (push-to-mirrors), updates to "release-cdntest" become available.

4. After we publish to "release" channel, there are a couple of builders which get unblocked, out of which are:

4.1 mark the release as shipped in Ship-it
Once we mark a release as shipped, there's a chain of tools that eventually get updated:  "ship-it -> product-details -> product-details-json -> bedrock -> www". This usually takes about 30mins or less.

4.2 update bouncer aliases
We've got a small script[5] that talks to bouncer aliases based on this[2] configs, and tells it to update the `firefox-latest` and `firefox-stub` to the current release. This means that once this builder is completed, all urls like this[6], regardless of the platform, will serve the latest version.

Then what is the problem then?
===
If we glance at the source of the downloads page[7], we notice that except the Win32 installer which gets served by querying the `firefox-stub`, none of the other platforms do so, but instead have versioned urls:

https://download.mozilla.org/?product=firefox-stub&os=win&lang=en-US
https://download.mozilla.org/?product=firefox-54.0.1-SSL&os=win64&lang=en-US
https://download.mozilla.org/?product=firefox-54.0.1-SSL&os=osx&lang=en-US
https://download.mozilla.org/?product=firefox-54.0.1-SSL&os=linux&lang=en-US
https://download.mozilla.org/?product=firefox-54.0.1-SSL&os=linux64&lang=en-US

Historically, the bouncer aliases were added specifically to support the `firefox-stub` so that we can offer the latest stub installer to win32 users. But question is, now that we have the context and logic to support this operation for all the other platforms, why are we still having this urls versioned?

Solution
If we move these urls to point to "firefox-latest", they will be available as soon as the bouncer aliases builder job runs in our graph. That is ~30 mins faster than waiting for "ship-it -> product-details -> product-details-json -> bedrock -> www" chain.

Resolution
* we need to investigate more to make sure we're not misunderstanding something here
* we need to rope in all the teams involved along the line
* see if we can do this and if this makes sense.

Observation:
* bedrock is just for releases, not for nightly. nightlies version from downloads page[7] talks directly to bouncer and is part of the reason why we manually update[8] these at each nightly => beta version bump. This is the reason why we update bouncer directly during mergeduty[8] each time we bump the version in nightly.

[1]: https://dxr.mozilla.org/mozilla-central/rev/52285ea5e54c73d3ed824544cef2ee3f195f05e6/testing/mozharness/scripts/bouncer_submitter.py
[2]: https://dxr.mozilla.org/mozilla-central/rev/52285ea5e54c73d3ed824544cef2ee3f195f05e6/testing/mozharness/configs/releases/bouncer_firefox_release.py
[3]: http://bounceradmin.mozilla.com/
[4]: http://download.mozilla.org/?product=firefox-55.0....
[5]: https://dxr.mozilla.org/mozilla-central/rev/52285ea5e54c73d3ed824544cef2ee3f195f05e6/testing/mozharness/scripts/release/postrelease_bouncer_aliases.py
[6]: https://download.mozilla.org/?product=firefox-54.0.1-SSL&os=OS_TO_FILL&lang=en-US
[7]: view-source:https://www.mozilla.org/en-US/firefox/new/?scene=2
[8]: https://github.com/mozilla/releasewarrior/blob/master/how-tos/mergeduty.md#bump-bouncer-versions
I tested today during the 55.0 and it does seem like https://download.mozilla.org/?product=firefox-latest&os=osx&lang=en-GB was updated as soon as the `bouncer-aliases` builder run successfully.

This means that if we can have a broader conversation with downloads.page website admins, we can change those urls to point to latest-firefox or alike so that we go live right away and we no longer need to wait for all these changes to propagate via ship-it => product details => bedrock => www, etc.
Dupe of bug 937865 ?
(In reply to Nick Thomas [:nthomas] from comment #3)
> Dupe of bug 937865 ?

Holy smokes, a six-digits bug to reference! :) 
Reading your comment https://bugzilla.mozilla.org/show_bug.cgi?id=937865#c13  I think we can keep this and rename it once we have some RelEng work to do.
See Also: → 937865
Sounds like we'll soon be reaching a consensus in bug 937865 comment 14 so we should pay a bit more attention do this. With a bit of luck, we may be shipping 56 faster in September!
Priority: -- → P1
Whiteboard: [releaseduty]
Assignee: mtabara → nobody
Priority: P1 → P3
bug 937865 comment 14 asks for
* firefox-beta-latest-ssl
* firefox-latest-ssl
* firefox-esr-latest-ssl

Should be easy to add.
Assignee: nobody → nthomas
Blocks: 937865
Priority: P3 → P1
See Also: 937865
Summary: investigate why not all platforms from https://www.mozilla.org/en-US/firefox/new/ query the `firefox-latest` in bouncer → Add latest-ssl aliases for bedrock to use
I've manually added those 3 products in comment #6 as well as firefox-sha1-ssl, and set them to point at the current -SSL release. And removed devedition-latest, devedition-latest-ssl, and devedition-stub, and firefox-latest-euballot.
TODO - make sure people know how www.mozilla.org will behave differently after this goes live
Thanks Nick! Can't wait to update bedrock to use this!
Comment on attachment 8903101 [details]
Bug 1387622 - add latest-ssl aliases for bedrock,

https://reviewboard.mozilla.org/r/174884/#review180984

This is looking good!
Excuse the n00b question, but please remind me again, why are we killing devedition bouncer entries?
Attachment #8903101 - Flags: review?(mtabara) → review+
The www folks wanted to use firefox-devedition-* ones, see bug 937865 comment #14.
Pushed by nthomas@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/b5cbf8e3f776
add latest-ssl aliases for bedrock, r=mtabara
No rush on this, please leave it until you get back.

If you haven't seen this before it's a nagios check that runs on bm81 to make sure the important bouncer products are working. The code lives at https://hg.mozilla.org/build/nagios-tools/.

I've added checks for firefox-latest-ssl and firefox-beta-latest-ssl, in anticipation of bedrock using them. Noticed we don't have anything for esr so corrected that with the new aliases. For nightly I'm just tidying up based on what we'll want after bug 1377842 (one day!). And a version bump because this is deployed as a python package via puppet.
Attachment #8904389 - Flags: review?(mtabara)
(In reply to Nick Thomas [:nthomas] from comment #13)
> The www folks wanted to use firefox-devedition-* ones, see bug 937865
> comment #14.

Ah, makes sense. Misread the bouncer names at first glance.
Comment on attachment 8904389 [details] [diff] [review]
[nagios-tools] Update bouncer checks

Lgtm!

I broke this script before ^_^ we've made our acquaintances when I last changed some bouncer entries and used " " instead of "%20".
Attachment #8904389 - Flags: review?(mtabara) → review+
Added nagios-tools 0.6 to the puppet server and deployed it with
https://hg.mozilla.org/build/puppet/rev/b8ddcebb1cde8a820f4b9ff5ed374ab0af7cdc03
https://hg.mozilla.org/build/puppet/rev/615a2af805b3a1d6c840e89eb1c140c1754ddad6

All done here.
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
Forgot to say that all future releases will update the aliases in an automated fashion. I'll do that manually for 56.0b9 when it ships in the next 12-24 hours.
I'm implementing this in bedrock now. I have a question:

What will firefox-esr-latest-ssl return when there are 2 current ESR releases. In product-details we get a version for ESR and one for ESR_NEXT during the overlapping time. Will firefox-esr-latest-ssl return the current ESR or the NEXT one during that time? Given an option I'd prefer for it to stick with the previous release until ESR_NEXT becomes the only active release so that I can always use the "latest" product for the current ESR and only bring out the version specific URLs for the "next" version when it's active.

I should be able to work with it either way, but I just need to know for implementation purposes.
Flags: needinfo?(nthomas)
Reopening this, because we stopped updating "firefox-sha1" by https://hg.mozilla.org/mozilla-central/rev/b5cbf8e3f776#l2.30, which is used by bouncer: https://github.com/rail/go-bouncer/blob/master/handlers_test.go#L183 to redirect XP users to latest ESR build.

I don't think we use firefox-sha1-ssl anywhere publicly available, so probably we can just revert that line.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(In reply to Rail Aliiev [:rail] ⌚️ET from comment #24)
> Reopening this, because we stopped updating "firefox-sha1" by
> https://hg.mozilla.org/mozilla-central/rev/b5cbf8e3f776#l2.30, which is used
> by bouncer:
> https://github.com/rail/go-bouncer/blob/master/handlers_test.go#L183 to
> redirect XP users to latest ESR build.
> 
> I don't think we use firefox-sha1-ssl anywhere publicly available, so
> probably we can just revert that line.

Manually updated firefox-sha to point to latest ESR for both production and staging instances.
Had no idea staging is in such a good shape, not sure who's updating it but it's almost a production mirror.
(In reply to Paul [:pmac] McLanahan ⏰ET needinfo? me from comment #23)
> What will firefox-esr-latest-ssl return when there are 2 current ESR
> releases. In product-details we get a version for ESR and one for ESR_NEXT
> during the overlapping time. Will firefox-esr-latest-ssl return the current
> ESR or the NEXT one during that time? Given an option I'd prefer for it to
> stick with the previous release until ESR_NEXT becomes the only active
> release so that I can always use the "latest" product for the current ESR
> and only bring out the version specific URLs for the "next" version when
> it's active.

The short answer is: it's not well defined in our process. When starting up ESR52 we had bug 1347754, which disabled the aliasing on ESR45 about the time of 52.0.1. I have a memory that we were offering 52.x.y on www.mozilla.org during the overlap with 45, but perhaps that's faulty based on your comments. We can pretty much do whatever we want here though, same amount of work either way.
Flags: needinfo?(nthomas)
The way it's handled on the site is that both versions are listed on the download page when FIREFOX_ESR_NEXT is defined in product-details[0]. You're right that there is a planned 6 week overlap after a new ESR release and they're both available at:

https://www.mozilla.org/en-US/firefox/organizations/all/

Which is the only page from which ESR is downloadable. So either way will work for me I think. What I believe will be easiest is for the firefox-esr-latest-ssl alias to stay on the old version during the transitional period, and we can use the value of ESR_NEXT in product-details to produce those download buttons. A really-really easy thing would be for there to exist a firefox-esr-next-latest-ssl alias when a new ESR is released so that we can use aliases for both, but that is really just me wishing out loud (no idea how difficult these things are to do).

Hope that's clear and helpful.

[0] https://product-details.mozilla.org/1.0/firefox_versions.json
Attachment #8915742 - Flags: review?(rail) → review+
Comment on attachment 8915742 [details] [diff] [review]
[gecko] Revert firefox-sha1 alias change

https://hg.mozilla.org/releases/mozilla-esr52/rev/31949068fe2c9cf147f00df0b8ccda57d74a337f

TODO: landed on central/beta/release when they reopen.
(In reply to Nick Thomas [:nthomas] (PTO 9-11 Oct) from comment #29)
> Comment on attachment 8915742 [details] [diff] [review]
> [gecko] Revert firefox-sha1 alias change
> 
> https://hg.mozilla.org/releases/mozilla-esr52/rev/
> 31949068fe2c9cf147f00df0b8ccda57d74a337f
> 
> TODO: landed on central/beta/release when they reopen.

Found in triaging: did we land this already or still need to cherry-pick that?
Pushed by nthomas@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/c49fa16162cb
revert firefox-sha1 alias change, r=rail a=release DONTBUILD
Removing leave-open keyword from resolved bugs, per :sylvestre.
Keywords: leave-open
See Also: → 1471228
You need to log in before you can comment on or make changes to this bug.