1419189 - provide devedition 58.0b1 updates to 58.0b5 over SSL

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, enhancement)

Description

[Jordan Lund (:jlund)]

first for testing purposes, then for real once QE sign off.

Comment 1

[Jordan Lund (:jlund)]

context:

POA:
catlee> can we create SSL-only bouncer aliases for 58.0bX devedition to check if we can restore updates for users on 58.0b1?
...
nthomas> can probably update fileUrls for special cases, and use a different bouncer product for those entries
22:09:08 
<•catlee> do we even need to use bouncer I wonder?
22:09:15 if we're going to be updating balrog
22:10:06 
<•nthomas> hmm, interesting idea


testing so far:
15:05:14 bogdan_maris: could you check if 58.0b1 still fails to update when you get a chance?
<bogdan_maris> catlee: Sure thing
15:40:45 catlee: still fails if I test on aurora-cdntest
15:41:41 catlee: aurora channel as well

nick is mocking something up. Testing a patch similar to: https://diff.pastebin.mozilla.org/9073277

Comment 2

[Rail Aliiev [:rail]]

Should we use SSL by default? It's just toggling "ssl-only" to True everywhere in bouncer configs, https://dxr.mozilla.org/mozilla-central/rev/b056526be38e96b3e381b7e90cd8254ad1d96d9d/testing/mozharness/configs/releases/bouncer_firefox_release.py#9 for example.

Comment 3

[Chris AtLee [:catlee]]

(In reply to Rail Aliiev [:rail] ⌚️ET from comment #2)
> Should we use SSL by default? It's just toggling "ssl-only" to True
> everywhere in bouncer configs,
> https://dxr.mozilla.org/mozilla-central/rev/
> b056526be38e96b3e381b7e90cd8254ad1d96d9d/testing/mozharness/configs/releases/
> bouncer_firefox_release.py#9 for example.

I think we should...but not just yet :)

Comment 4

[Robert Strong (they/them - no direct email)]

AIUI cert pinning is enabled on SSL downloads and we intentionally chose not to use cert pinning on app update per bug 1063111 (please read the bug for the rationale).

catlee / jlund, will these SSL downloads be cert pinned?

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to Robert Strong [:rstrong] (use needinfo to contact me) from comment #4)
> AIUI cert pinning is enabled on SSL downloads ...

Sounds like you are referring to cdn.mozilla.net appearing in https://hg.mozilla.org/releases/mozilla-beta/file/default/security/manager/ssl/StaticHPKPins.h#l674, which presumably includes our SSL CDN download-installer.cdn.mozilla.net. 

Should we separate the one-off rescue of 58.0b1 users from the broader question of serving updates over SSL ?

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

I've added a Devedition-58.0b5-build1-SSL release to Balrog, which has a modified fileUrls block to download mar files from https://archive.mozilla.org. We can't use download-installer.cdn.mozilla.net without a change to the whitelist used by Balrog.

Also added rules 687 and 688 to use that release for requests coming from DevEd 58.0b1 (buildID 20171103003834). Was able to update 58.0b1 mac en-US on aurora-cdntest OK. 

Bogdan, could you please test aurora-cdntest to see if it fixes up the issues you've been seeing with 58.0b1.

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

Rerunning the automated update verify as well, although it's using wget for the downloads.

Comment 8

[Bogdan Maris, Desktop Test Engineering]

(In reply to Nick Thomas [:nthomas] (UTC+13) from comment #6)
> I've added a Devedition-58.0b5-build1-SSL release to Balrog, which has a
> modified fileUrls block to download mar files from
> https://archive.mozilla.org. We can't use download-installer.cdn.mozilla.net
> without a change to the whitelist used by Balrog.
> 
> Also added rules 687 and 688 to use that release for requests coming from
> DevEd 58.0b1 (buildID 20171103003834). Was able to update 58.0b1 mac en-US
> on aurora-cdntest OK. 
> 
> Bogdan, could you please test aurora-cdntest to see if it fixes up the
> issues you've been seeing with 58.0b1.

Retested across platforms (Windows 10 64bit, macOS 10.13 and Ubuntu 16.04 32bit) and DevEd 58.0b1 now updates to 58.0b5 without issues.

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

Great! I've added a Balrog scheduled change to do this on the aurora channel too, to go with the main change for 58.0b5.

Comment 10

[Nick Thomas [:nthomas] (UTC+12)]

Updates are live.

Comment 11

[Nick Thomas [:nthomas] (UTC+12)]

We're hitting some update verify failures for 58.0b6 because of the special rule.

Comment 12

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [tools] Fix update verify

The changes to the u.v. configs are to fix for 58.0b6 testing, while the patcher config is for subsequent releases.

As we discussed, we'll have a static 58.0b1 -> 58.0b5 SSL path for the 58 cycle, then likely remove that in 59 when users have had a chance to update. We're kinda flying blind since we don't know how many users are affected by the download issue QE saw.

Comment 13

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8931511 [details] [diff] [review]
[tools] Fix update verify

https://hg.mozilla.org/build/tools/rev/017ead33eb4260f2de190c36391e60f12d54e425
https://hg.mozilla.org/build/tools/rev/c43e705ae16463f39e9216de73749fee9b453775

Will rerun the failing u.v. jobs.

Comment 14

[Nick Thomas [:nthomas] (UTC+12)]

Green!