Closed Bug 1391297 Opened 2 years ago Closed 2 years ago

Add LZMA/SHA384 watershed when 56.0b3 ships

Categories

(Release Engineering :: Release Requests, enhancement, P1)

enhancement

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: mtabara, Assigned: mtabara)

References

Details

Attachments

(3 files, 1 obsolete file)

For updates to >= 56.0b4, we need to add watershed in 56.0b3 so that all betas prior to this can have the support for LZMA/SHA384.
Priority: -- → P1
I looked at your lzma/sha384 balrog rule and it looked sane to me.
That new rule will override the partial rollout we have right now, is that the plan ? And we'll need a new rule for aurora/devedition too.
(In reply to Nick Thomas [:nthomas] from comment #2)
> That new rule will override the partial rollout we have right now, is that
> the plan ? And we'll need a new rule for aurora/devedition too.

re: rolled out plan for beta - my understanding from channel meeting last night was that they intend to go 100% with beta
re: rule in aurora/devedition - good point, I'll add that as well as soon as I test.

So sounds like we need RelMan to increase beta population to 100% first and then schedule these rules.
Ah, OK. We can delete the partial rollout rule later then.

Another thing we'll need to do is modify https://hg.mozilla.org/build/tools/file/default/release/patcher-configs/mozBeta-branch-patcher2.cfg so that we only do update verify from 56.0b3 to subsequent releases. Check the history on that file for how we did that last time. It can be done between b4 and b5.
The patcher config drives the update verify configs, so that change sets us up for 56.0b5 when the updates task runs. I'm fairly sure http://hg.mozilla.org/build/tools/file/default/release/patcher-config-bump.pl is going to do the right thing and repopulate <partials> and past-update lines. It might be helpful to manually set the partials to only b3 and higher in ship-it though.

The release/updates/beta-firefox-* changes are for testing updates to 56.0b4. We'll need to move the FIREFOX_56_0b4_{BUILD2,RELEASE}_RUNTIME tags when we land this, but uv tasks should go green after that (assuming all is well in lzma land).

On the plus side update verify will be lightning fast after this.
Attachment #8898728 - Flags: review?(mtabara)
Adds the devedition configs too.
Attachment #8898728 - Attachment is obsolete: true
Attachment #8898728 - Flags: review?(mtabara)
Attachment #8898742 - Flags: review?(mtabara)
If you really really wanted to test old -> 56.0b3, as well as 56.0b3 --> 56.0b4 this one time we could make this sort of manual change, on the 2nd line of all 10 update verify configs. It takes the to=, to_build_id=, to_display_version=, and to_app_version= from the first line and changes the b4 values to b3. I don't think we've ever done this for real but I think the code [1] supports it.

[1] https://dxr.mozilla.org/build-central/source/tools/release/updates/verify.sh
Comment on attachment 8898742 [details] [diff] [review]
[tools] Remove everything older than 56.0b3 (beta and deved)

Review of attachment 8898742 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good to me. Wasn't sure why beta on linux goes only to 44.0 but I guess there was another watershed there for this particular platform and we had to trim the same way.
Attachment #8898742 - Flags: review?(mtabara) → review+
Comment on attachment 8898742 [details] [diff] [review]
[tools] Remove everything older than 56.0b3 (beta and deved)

https://hg.mozilla.org/build/tools/rev/85763394a95655a1fdce6129bde09586c29a6f20
Attachment #8898742 - Flags: checked-in+
Comment on attachment 8898742 [details] [diff] [review]
[tools] Remove everything older than 56.0b3 (beta and deved)

Retagged with the changes - https://hg.mozilla.org/build/tools/rev/82950f5cdae8
For the record, I've added the same watershed for aurora as well.
Blocks: 1391718
See Also: → 1391680
Comment on attachment 8899186 [details]
Bug 1391297 - Update MAR to sha384 past 56.0b4 in update-packaging Makefile.  a=release

https://reviewboard.mozilla.org/r/170456/#review175608
Attachment #8899186 - Flags: review?(catlee) → review+
Pushed by mtabara@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/531b8905ad4d
Update MAR to sha384 past 56.0b4 in update-packaging Makefile. r=catlee a=release DONTBUILD
Keywords: leave-open
Blocks: 1391843
So sounds like build5 solved the update verification issues. But ...

Good news: all MacOSx and Windows update verification now work like a charm, no more signing issues!
Bad news: All Linux based update verification fail + final verification is failing too (used to work).

The reason for failing is mismatch between size is Balrog and the one downloaded from the CDN.
For this particular size-reason, "beta final verification" job is somehow expected to fail as that is only checking if Balrog serves the right updates, doesn't actually test the update themselves as update verification jobs.

On another hand, I've done some testing and found no problem so I presume this is just caching issues.

Sample:
Balrog url[1] returns
* URL: [2]
* size: 43691628

But [2] is a bouncer entry which 302s to cdn url[3]. In the failing logs, it says:

Downloading 'http://download.mozilla.org/?product=firefox-56.0b4-complete&os=linux&lang=gu-IN&force=1' and placing in cache...
... 
  HTTP/1.1 302 Found
  Location: http://download.cdn.mozilla.net/pub/firefox/releases/56.0b4/update/linux-i686/gu-IN/firefox-56.0b4.complete.mar
  Content-Length: 134
  Connection: keep-alive
Location: http://download.cdn.mozilla.net/pub/firefox/releases/56.0b4/update/linux-i686/gu-IN/firefox-56.0b4.complete.mar [following]
--2017-08-20 11:31:36--  http://download.cdn.mozilla.net/pub/firefox/releases/56.0b4/update/linux-i686/gu-IN/firefox-56.0b4.complete.mar
Resolving download.cdn.mozilla.net (download.cdn.mozilla.net)... 104.96.221.145, 104.96.221.66, 2600:1408:20::6860:dda9, ...
Connecting to download.cdn.mozilla.net (download.cdn.mozilla.net)|104.96.221.145|:80... connected.
HTTP request sent, awaiting response... 
...
  Content-Length: 43689852
...
Length: 43689852 (42M) [application/octet-stream]
Saving to: 'update/complete.mar'

Obviously, 43689852 != 43691628.

But, the fun part now starts.
The actual build5, gu-IN locale, update mar for Linux CMAR is this[4] while its corresponding build4 counterpart is this[5].

[5]'s size is actually 43689852 while [4]'s size is 43691628 which means the CDN caches are still serving build4 for Linux.
Not sure why this happens. But I'll rerun the jobs and see if I get any luck at a follow-up run. Otherwise, I'll escalate in bug 1391843 to CloudOps for further investigation.

[1]: https://aus5.mozilla.org/update/3/Firefox/56.0/20170815141045/Linux_x86-gcc3/gu-IN/beta-cdntest/default/default/default/update.xml?force=1
[2]: http://download.mozilla.org/?product=firefox-56.0b4-complete&os=linux&lang=gu-IN&force=1
[3]: http://download.cdn.mozilla.net/pub/firefox/releases/56.0b4/update/linux-i686/gu-IN/firefox-56.0b4.complete.mar
[4]: https://archive.mozilla.org/pub/firefox/candidates/56.0b4-candidates/build5/update/linux-i686/gu-IN/firefox-56.0b4.complete.mar
[5]: https://archive.mozilla.org/pub/firefox/candidates/56.0b4-candidates/build4/update/linux-i686/gu-IN/firefox-56.0b4.complete.mar
Retriggered update verification went green so it was definitely a CDN caching issue.
We successfully shipped 56.0b4-build5 yesterday and the watersheds are in place.

We're also rolling out beta5 as well in a few minutes so I think we can close this bug for now.
Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Blocks: 1407304
Removing leave-open keyword from resolved bugs, per :sylvestre.
Keywords: leave-open
You need to log in before you can comment on or make changes to this bug.