1444406 - AUS links to download.m.o should use HTTPS

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, enhancement, P2)

Description

[Julien Vehent]

AUS XML documents currently link to the HTTP endpoint of download.m.o [1]. Those links should point to HTTPS instead, which points to the same file on the download CDN.

Note that this change is blocked by bug 1444399 to make sure we are ready to serve HTTPS traffic to all our users on the CDN side beforehand.

HTTP downloads redirect to download.cdn.mozilla.net

> $ curl -si 'http://download.mozilla.org/?product=firefox-56.0-complete-bz2&os=win&lang=en-US'|grep Location
> Location: http://download.cdn.mozilla.net/pub/firefox/releases/56.0/update/win32/en-US/firefox-56.0.bz2.complete.mar

HTTPS downloads redirect to download-installer.cdn.mozilla.net

> $ curl -si 'https://download.mozilla.org/?product=firefox-56.0-complete-bz2&os=win&lang=en-US'|grep Location
> Location: https://download-installer.cdn.mozilla.net/pub/firefox/releases/56.0/update/win32/en-US/firefox-56.0.bz2.complete.mar

[1] https://aus5.mozilla.org/update/6/Firefox/53.0a1/20161225030206/WINNT_x86-msvc-x64/en-US/release/Windows_NT%2010.0.0.0%20(x64)(noBug1296630v1)/SSE3/default/default/update.xml

Comment 1

[[e-mail] (:bhearsum)]

This requires a change to the submission tools, not Balrog itself. Most likely https://github.com/mozilla/build-tools/blob/master/lib/python/balrog/submitter/cli.py, and possibly something in balrog scriptworker.

Comment 2

[Chris AtLee [:catlee]]

Hold off on implementing this until we can determine the impact of serving updates over HTTPS. We know that some users are unable to update from cdn.mozilla.net due to the HPKP pins we have in place.

Comment 3

[Eric Rescorla (:ekr)]

Several points

1. We should generally be doing HTTPS for everything, even if it's separately authenticated.
2. I don't quite understand why HPKP would be a problem here. Is the concern MITM devices? Because as a general matter, those work fine with HPKP.

Comment 4

[Jeremy Orem [:oremj]]

catlee, how can we push this forward?

Comment 5

[Chris AtLee [:catlee]]

we're waiting on the outcome of bug 1444399

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

bhearsum, catlee, and myself have been looking into how we can make this change. Comments 0 and 1 are still accurate,

• Balrog specifies http://download.m.o, which redirects to http://download.cdn.mozilla.net/
• Switching to https://download.m.o redirects to https://download-installer.cdn.mozilla.net/

The former redirect is from the setup of download.m.o (bouncer), where the update products don't have the 'Use SSL' flag set like some of the installer products do. This is controlled by the craft_ssl_only() function at [1]. The latter is from bug 1422844 ensuring bouncer doesn't downgrade from SSL.

So the minimal change is to have Balrog specify https - the code for this is at [2] and currently hardcodes http. To do a gradual rollout we'll need to make the protocol configurable, possibly by combining it with the domain we already set in the task payload at [3][4].

For completeness we should tidy up the 'Use SSL' on the bouncer products, and set it everywhere. I'd also like to simplify the products we set up in [5], eg get rid of the the -SSL products, remove overlap. This will require some co-ordination with bedrock and various checks that run.

[1] https://searchfox.org/mozilla-central/rev/567b68b8ff4b6d607ba34a6f1926873d21a7b4d7/taskcluster/taskgraph/transforms/bouncer_submission.py#266
[2] https://github.com/mozilla-releng/scriptworker-scripts/blob/34e0aa45bc5226e446dd2972c91539d686f1e5e4/balrogscript/src/balrogscript/submitter/cli.py#L66
[3] https://searchfox.org/mozilla-central/rev/567b68b8ff4b6d607ba34a6f1926873d21a7b4d7/taskcluster/ci/release-balrog-submit-toplevel/kind.yml#26
[4] https://searchfox.org/mozilla-central/rev/567b68b8ff4b6d607ba34a6f1926873d21a7b4d7/taskcluster/ci/release-secondary-balrog-submit-toplevel/kind.yml#26
[5] https://searchfox.org/mozilla-central/rev/567b68b8ff4b6d607ba34a6f1926873d21a7b4d7/testing/mozharness/configs/releases/bouncer_firefox_release.py

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

That's all for beta, release, and esr updates. For nightly updates, Balrog uses https://archive.mozilla.org, while the stub installer and various download boxes on www.m.o use https://download.m.o.

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: DevEdition 77.0b8 balrog diff

Per the plan in our last meeting, I've updated the Devedition-77.0b8-build1 release in Balrog to point at https://download.m.o, and the aurora-localtest to use https://archive.m.o for kicks.

Comment 10

[Julien Cristau [:jcristau]]

bhearsum updated the Devedition-77.0b9-build1 release to use https today.

Comment 11

[Nick Thomas [:nthomas] (UTC+12)]

DevEdition-78.0b1-build1 is set to use https too.

Comment 12

[Chris AtLee [:catlee]]

We haven't seen any problems with DevEdition. Let's make this change permanent for DevEdition and Beta.

Comment 13

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: GitHub Pull Request

We didn't manually update for 78.0b2, b3, or b4. This patch will switch to https for devedition and firefox beta.

Comment 14

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to Nick Thomas [:nthomas] (UTC+13) from comment #13)

Created attachment 9155487 [details] [review]
GitHub Pull Request

We didn't manually update for 78.0b2, b3, or b4. This patch will switch to https for devedition and firefox beta.

This is deployed in time for 78.0b6 today.

Comment 15

[Julien Cristau [:jcristau]]

Looks good:

$ curl -s https://aus-api.mozilla.org/api/v1/releases/Devedition-78.0b6-build1 | jq '.fileUrls."*"'
{
  "completes": {
    "*": "https://download.mozilla.org/?product=devedition-78.0b6-complete&os=%OS_BOUNCER%&lang=%LOCALE%"
  },
  "partials": {
    "Devedition-78.0b3-build1": "https://download.mozilla.org/?product=devedition-78.0b6-partial-78.0b3&os=%OS_BOUNCER%&lang=%LOCALE%",
    "Devedition-78.0b4-build1": "https://download.mozilla.org/?product=devedition-78.0b6-partial-78.0b4&os=%OS_BOUNCER%&lang=%LOCALE%",
    "Devedition-78.0b5-build1": "https://download.mozilla.org/?product=devedition-78.0b6-partial-78.0b5&os=%OS_BOUNCER%&lang=%LOCALE%"
  }
}
$ curl -s https://aus-api.mozilla.org/api/v1/releases/Firefox-78.0b6-build1 | jq '.fileUrls."*"'
{
  "completes": {
    "*": "https://download.mozilla.org/?product=firefox-78.0b6-complete&os=%OS_BOUNCER%&lang=%LOCALE%"
  },
  "partials": {
    "Firefox-78.0b3-build1": "https://download.mozilla.org/?product=firefox-78.0b6-partial-78.0b3&os=%OS_BOUNCER%&lang=%LOCALE%",
    "Firefox-78.0b4-build1": "https://download.mozilla.org/?product=firefox-78.0b6-partial-78.0b4&os=%OS_BOUNCER%&lang=%LOCALE%",
    "Firefox-78.0b5-build1": "https://download.mozilla.org/?product=firefox-78.0b6-partial-78.0b5&os=%OS_BOUNCER%&lang=%LOCALE%"
  }
}

Comment 16

[Rachel Tublitz [:rachel]]

As a timeline update for anyone following along: we plan to monitor the changes made against Beta, and assuming we run into no problems make this change for release for FX 79. ESR details come next and are still TBD. See this doc for more info.

Comment 18

[[e-mail] (:bhearsum)]

I'll be taking this work over to drive it home. Big thanks to Nick for all of his existing work here.

For various reasons, this stalled a bit. I've been getting myself up to speed and I believe the current state is that we've been shipping Beta & DevEdition via https for a couple of months, but haven't carried it forward to release & esr. I believe the next steps are:

• Test this out in an ad-hoc manner on release & esr, by changing the currently shipping release blobs (as Nick did for DevEdition originally).
• Update balrogscript to always use https for these channels before we go to build on 81.0.

Nick mentioned to me privately that we may want to have this only happen for Firefox, too. rjl, what do you want to do? It's no big deal keep this off for Thunderbird if you like.

Comment 19

[Rob Lemley [:rjl]]

If it's possible to use https, then we should.
I'll set up our beta-localtest channel to test, but I don't expect there to be any problems with this.

Comment 20

[[e-mail] (:bhearsum)]

We pointed the release & esr channels at https updates today. Rachel, I believe the next step is for you to verify that everything looks OK in the telemetry data.

Comment 21

[Rachel Tublitz [:rachel]]

Thanks Ben! So, initial telemetry (looking at the DAU based update orphan dash and the update ping read/success metrics) looks ok. I'll keep the ni? for myself here for another week or two to continue monitoring for issues before we close this out completely.

Comment 22

[Rachel Tublitz [:rachel]]

(In reply to [e-mail] (:bhearsum) from comment #20)

Rachel, I believe the next step is for you to verify that everything looks OK in the telemetry data.

Confirmed on my end; all looks normal.

Comment 23

[[e-mail] (:bhearsum)]

(In reply to Rachel Tublitz [:rachel] from comment #22)

(In reply to [e-mail] (:bhearsum) from comment #20)

Rachel, I believe the next step is for you to verify that everything looks OK in the telemetry data.

Confirmed on my end; all looks normal.

Great! We already landed the releng bits to enable this for new releases in time for 81.0, so there shouldn't be anything left to do here. I'll leave this bug open until 81.0 ships.

Comment 24

[[e-mail] (:bhearsum)]

We shipped 81.0 today, which has its updates configured for https. This is also the new default for all releases going forward.

Big thanks to Nick for all his work organizing and verifying the RelEng parts of this work.