Closed
Bug 1391297
Opened 7 years ago
Closed 7 years ago
Add LZMA/SHA384 watershed when 56.0b3 ships
Categories
(Release Engineering :: Release Requests, enhancement, P1)
Release Engineering
Release Requests
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: mtabara, Assigned: mtabara)
References
Details
Attachments
(3 files, 1 obsolete file)
757.77 KB,
patch
|
mtabara
:
review+
mtabara
:
checked-in+
|
Details | Diff | Splinter Review |
6.14 KB,
patch
|
Details | Diff | Splinter Review | |
59 bytes,
text/x-review-board-request
|
catlee
:
review+
mtabara
:
checked-in+
|
Details |
For updates to >= 56.0b4, we need to add watershed in 56.0b3 so that all betas prior to this can have the support for LZMA/SHA384.
Assignee | ||
Updated•7 years ago
|
Priority: -- → P1
Comment 1•7 years ago
|
||
I looked at your lzma/sha384 balrog rule and it looked sane to me.
Comment 2•7 years ago
|
||
That new rule will override the partial rollout we have right now, is that the plan ? And we'll need a new rule for aurora/devedition too.
Assignee | ||
Comment 3•7 years ago
|
||
(In reply to Nick Thomas [:nthomas] from comment #2) > That new rule will override the partial rollout we have right now, is that > the plan ? And we'll need a new rule for aurora/devedition too. re: rolled out plan for beta - my understanding from channel meeting last night was that they intend to go 100% with beta re: rule in aurora/devedition - good point, I'll add that as well as soon as I test. So sounds like we need RelMan to increase beta population to 100% first and then schedule these rules.
Comment 4•7 years ago
|
||
Ah, OK. We can delete the partial rollout rule later then. Another thing we'll need to do is modify https://hg.mozilla.org/build/tools/file/default/release/patcher-configs/mozBeta-branch-patcher2.cfg so that we only do update verify from 56.0b3 to subsequent releases. Check the history on that file for how we did that last time. It can be done between b4 and b5.
Comment 5•7 years ago
|
||
The patcher config drives the update verify configs, so that change sets us up for 56.0b5 when the updates task runs. I'm fairly sure http://hg.mozilla.org/build/tools/file/default/release/patcher-config-bump.pl is going to do the right thing and repopulate <partials> and past-update lines. It might be helpful to manually set the partials to only b3 and higher in ship-it though. The release/updates/beta-firefox-* changes are for testing updates to 56.0b4. We'll need to move the FIREFOX_56_0b4_{BUILD2,RELEASE}_RUNTIME tags when we land this, but uv tasks should go green after that (assuming all is well in lzma land). On the plus side update verify will be lightning fast after this.
Attachment #8898728 -
Flags: review?(mtabara)
Comment 6•7 years ago
|
||
Adds the devedition configs too.
Attachment #8898728 -
Attachment is obsolete: true
Attachment #8898728 -
Flags: review?(mtabara)
Attachment #8898742 -
Flags: review?(mtabara)
Comment 7•7 years ago
|
||
If you really really wanted to test old -> 56.0b3, as well as 56.0b3 --> 56.0b4 this one time we could make this sort of manual change, on the 2nd line of all 10 update verify configs. It takes the to=, to_build_id=, to_display_version=, and to_app_version= from the first line and changes the b4 values to b3. I don't think we've ever done this for real but I think the code [1] supports it. [1] https://dxr.mozilla.org/build-central/source/tools/release/updates/verify.sh
Assignee | ||
Comment 8•7 years ago
|
||
Comment on attachment 8898742 [details] [diff] [review] [tools] Remove everything older than 56.0b3 (beta and deved) Review of attachment 8898742 [details] [diff] [review]: ----------------------------------------------------------------- Looks good to me. Wasn't sure why beta on linux goes only to 44.0 but I guess there was another watershed there for this particular platform and we had to trim the same way.
Attachment #8898742 -
Flags: review?(mtabara) → review+
Assignee | ||
Comment 9•7 years ago
|
||
Comment on attachment 8898742 [details] [diff] [review] [tools] Remove everything older than 56.0b3 (beta and deved) https://hg.mozilla.org/build/tools/rev/85763394a95655a1fdce6129bde09586c29a6f20
Attachment #8898742 -
Flags: checked-in+
Assignee | ||
Comment 10•7 years ago
|
||
Comment on attachment 8898742 [details] [diff] [review] [tools] Remove everything older than 56.0b3 (beta and deved) Retagged with the changes - https://hg.mozilla.org/build/tools/rev/82950f5cdae8
Assignee | ||
Comment 11•7 years ago
|
||
For the record, I've added the same watershed for aurora as well.
Comment hidden (mozreview-request) |
Comment 13•7 years ago
|
||
mozreview-review |
Comment on attachment 8899186 [details] Bug 1391297 - Update MAR to sha384 past 56.0b4 in update-packaging Makefile. a=release https://reviewboard.mozilla.org/r/170456/#review175608
Attachment #8899186 -
Flags: review?(catlee) → review+
Comment 14•7 years ago
|
||
Pushed by mtabara@mozilla.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/531b8905ad4d Update MAR to sha384 past 56.0b4 in update-packaging Makefile. r=catlee a=release DONTBUILD
Assignee | ||
Comment 15•7 years ago
|
||
Comment on attachment 8899186 [details] Bug 1391297 - Update MAR to sha384 past 56.0b4 in update-packaging Makefile. a=release https://hg.mozilla.org/releases/mozilla-beta/rev/05a4dc75c125a80cbaacd44c7b7937c77dc384a8 https://hg.mozilla.org/integration/mozilla-inbound/rev/531b8905ad4dc655105f59b1d5b9a3e336554a9f
Attachment #8899186 -
Flags: checked-in+
Assignee | ||
Updated•7 years ago
|
Keywords: leave-open
Comment 16•7 years ago
|
||
bugherder |
https://hg.mozilla.org/mozilla-central/rev/531b8905ad4d
Assignee | ||
Comment 17•7 years ago
|
||
So sounds like build5 solved the update verification issues. But ... Good news: all MacOSx and Windows update verification now work like a charm, no more signing issues! Bad news: All Linux based update verification fail + final verification is failing too (used to work). The reason for failing is mismatch between size is Balrog and the one downloaded from the CDN. For this particular size-reason, "beta final verification" job is somehow expected to fail as that is only checking if Balrog serves the right updates, doesn't actually test the update themselves as update verification jobs. On another hand, I've done some testing and found no problem so I presume this is just caching issues. Sample: Balrog url[1] returns * URL: [2] * size: 43691628 But [2] is a bouncer entry which 302s to cdn url[3]. In the failing logs, it says: Downloading 'http://download.mozilla.org/?product=firefox-56.0b4-complete&os=linux&lang=gu-IN&force=1' and placing in cache... ... HTTP/1.1 302 Found Location: http://download.cdn.mozilla.net/pub/firefox/releases/56.0b4/update/linux-i686/gu-IN/firefox-56.0b4.complete.mar Content-Length: 134 Connection: keep-alive Location: http://download.cdn.mozilla.net/pub/firefox/releases/56.0b4/update/linux-i686/gu-IN/firefox-56.0b4.complete.mar [following] --2017-08-20 11:31:36-- http://download.cdn.mozilla.net/pub/firefox/releases/56.0b4/update/linux-i686/gu-IN/firefox-56.0b4.complete.mar Resolving download.cdn.mozilla.net (download.cdn.mozilla.net)... 104.96.221.145, 104.96.221.66, 2600:1408:20::6860:dda9, ... Connecting to download.cdn.mozilla.net (download.cdn.mozilla.net)|104.96.221.145|:80... connected. HTTP request sent, awaiting response... ... Content-Length: 43689852 ... Length: 43689852 (42M) [application/octet-stream] Saving to: 'update/complete.mar' Obviously, 43689852 != 43691628. But, the fun part now starts. The actual build5, gu-IN locale, update mar for Linux CMAR is this[4] while its corresponding build4 counterpart is this[5]. [5]'s size is actually 43689852 while [4]'s size is 43691628 which means the CDN caches are still serving build4 for Linux. Not sure why this happens. But I'll rerun the jobs and see if I get any luck at a follow-up run. Otherwise, I'll escalate in bug 1391843 to CloudOps for further investigation. [1]: https://aus5.mozilla.org/update/3/Firefox/56.0/20170815141045/Linux_x86-gcc3/gu-IN/beta-cdntest/default/default/default/update.xml?force=1 [2]: http://download.mozilla.org/?product=firefox-56.0b4-complete&os=linux&lang=gu-IN&force=1 [3]: http://download.cdn.mozilla.net/pub/firefox/releases/56.0b4/update/linux-i686/gu-IN/firefox-56.0b4.complete.mar [4]: https://archive.mozilla.org/pub/firefox/candidates/56.0b4-candidates/build5/update/linux-i686/gu-IN/firefox-56.0b4.complete.mar [5]: https://archive.mozilla.org/pub/firefox/candidates/56.0b4-candidates/build4/update/linux-i686/gu-IN/firefox-56.0b4.complete.mar
Assignee | ||
Comment 18•7 years ago
|
||
Retriggered update verification went green so it was definitely a CDN caching issue. We successfully shipped 56.0b4-build5 yesterday and the watersheds are in place. We're also rolling out beta5 as well in a few minutes so I think we can close this bug for now.
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Comment 19•6 years ago
|
||
Removing leave-open keyword from resolved bugs, per :sylvestre.
Keywords: leave-open
You need to log in
before you can comment on or make changes to this bug.
Description
•