Provide a partial mar file to update from Firefox 47.0.2

RESOLVED FIXED

Status

enhancement
P2
normal
RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: rstrong, Assigned: jlorenzo)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

+++ This bug was initially created as a clone of Bug #1319905 +++

We no longer have a partial mar for 47.0.2 as was implemented in bug 1319905. Currently 47.0.2 represents around 33% of the out of date population since it is an update watershed for both websense detection and CPU instruction set detection.

I also noticed another uptick in no updates found for out of date clients.
https://telemetry.mozilla.org/update-orphaning/

Clients that can update to the next release will only get advertised an update if they manually check.

Background update
https://aus5.mozilla.org/update/6/Firefox/47.0.2/20161031133903/WINNT_x86-msvc-x64/en-US/release/Windows_NT%2010.0.0.0%20(x64)(nowebsense)/SSE3/default/default/update.xml

Manual update
https://aus5.mozilla.org/update/6/Firefox/47.0.2/20161031133903/WINNT_x86-msvc-x64/en-US/release/Windows_NT%2010.0.0.0%20(x64)(nowebsense)/SSE3/default/default/update.xml?force=1
> I also noticed another uptick in no updates found for out of date clients.
> https://telemetry.mozilla.org/update-orphaning/
> 
> Clients that can update to the next release will only get advertised an
> update if they manually check.
> 
> Background update
> https://aus5.mozilla.org/update/6/Firefox/47.0.2/20161031133903/WINNT_x86-
> msvc-x64/en-US/release/Windows_NT%2010.0.0.0%20(x64)(nowebsense)/SSE3/
> default/default/update.xml
> 
> Manual update
> https://aus5.mozilla.org/update/6/Firefox/47.0.2/20161031133903/WINNT_x86-
> msvc-x64/en-US/release/Windows_NT%2010.0.0.0%20(x64)(nowebsense)/SSE3/
> default/default/update.xml?force=1

Updates to 52.0 are currently set at 10% update rate, so this isn't surprising.
We've just set the fallback update to 51.0.1build3, so 90% of users will be getting an update to 51.0.1, and 10% will get 52.0. We're discussing how to get these fallbacks set as part of automation.
we may want to hold off on this until we are at 100%. There is a possibility of a dot release coming.

rstrong, rather than taking the 47.0.2 partial request one at a time and forcing you to remember and request, maybe it would be better if releng automatically handled this as part of new version checklist?

In order to do that, could we set some expectations about how long we plan to continue creating partials for 47.0.2? 1, 2, N more major releases?

There are long term efforts to make this possible within automation: 'partials on demand' is the big one, but being able to specify more partials than the current 3 or 4 for each release is something we can't do yet but are working on it.
Flags: needinfo?(robert.strong.bugs)
I'm told this may not be so easy to add to our manual checklist for each major version bump. I don't believe I have enough context to request a change in protocol so we will leave things for now and continue requesting partials via bugs when you want them.
Flags: needinfo?(robert.strong.bugs)
Didn't we try this at one point and there were too many things for taskcluster to handle? We hit some sort of limit when we tried to have an extra partial on top of what we usually build.
I mean, we tried adding it to the normal build process and it failed.  But as Jordan says we can still do this outside of the routine of shipping a new version.
(In reply to Jordan Lund (:jlund) from comment #3)
> we may want to hold off on this until we are at 100%. There is a possibility
> of a dot release coming.
Understandable and I'm fine with this.

> rstrong, rather than taking the 47.0.2 partial request one at a time and
> forcing you to remember and request, maybe it would be better if releng
> automatically handled this as part of new version checklist?
> 
> In order to do that, could we set some expectations about how long we plan
> to continue creating partials for 47.0.2? 1, 2, N more major releases?
> 
> There are long term efforts to make this possible within automation:
> 'partials on demand' is the big one, but being able to specify more partials
> than the current 3 or 4 for each release is something we can't do yet but
> are working on it.
As far as expectations go it would be good to have a partial for at least the largest distribution of out of date clients which happens to be 47.0.2 at this time.
We do pick partials based on ADI but the datasource has a limitation. For a X.0 release we use the last release build, the last beta build (so we can provide the release build to beta as a release candidate), and then the next highest ADI. For a X.0.Y it's the last release build, and the two highest ADI. The data comes from crash-stats, and that doesn't return ADI for older versions like 47.0.2. The oldest right now is 50.0.1. I'll ask the Socorro folks if there is a way around that.
One thing that would be great is to always have a partial from any watershed release since clients have to go through that release to get to the latest.
Now that we have shipped a chemspill we can look at this.
Priority: -- → P2
(Assignee)

Comment 12

2 years ago
The patch above was created by blindly following the steps described in bug 1319905 comment 3. The 47.0.2 data was already there, so no need to manually tweak the patch.
Assignee: nobody → jlorenzo
(Assignee)

Comment 13

2 years ago
I should probably delete some lines in the patch, just like you suggested in bug 1319905 comment 17

Comment 14

2 years ago
mozreview-review
Comment on attachment 8855289 [details]
Bug 1347030 - Provide a partial mar file to update from Firefox 47.0.2 to 53.0

https://reviewboard.mozilla.org/r/127162/#review129964

::: release/updates/release-firefox-win32.cfg:18
(Diff revision 1)
>  release="49.0" product="Firefox" platform="WINNT_x86-msvc-x86" build_id="20160916101415" locales="de en-US ru" channel="release-localtest" patch_types="complete" from="/firefox/releases/49.0/win32/%locale%/Firefox Setup 49.0.exe" ftp_server_from="https://archive.mozilla.org/pub" ftp_server_to="https://archive.mozilla.org/pub"
>  release="49.0" product="Firefox" platform="WINNT_x86-msvc-x86" build_id="20160916101415" locales="ach af an ar as ast az bg bn-BD bn-IN br bs ca cak cs cy da dsb el en-GB en-ZA eo es-AR es-CL es-ES es-MX et eu fa ff fi fr fy-NL ga-IE gd gl gn gu-IN he hi-IN hr hsb hu hy-AM id is it ja kk km kn ko lij lt lv mai mk ml mr ms nb-NO nl nn-NO or pa-IN pl pt-BR pt-PT rm ro si sk sl son sq sr sv-SE ta te th tr uk uz vi xh zh-CN zh-TW" channel="release-localtest" patch_types="complete"
>  release="48.0.2" product="Firefox" platform="WINNT_x86-msvc-x86" build_id="20160823121617" locales="ach af an ar as ast az bg bn-BD bn-IN br bs ca cak cs cy da de dsb el en-GB en-US en-ZA eo es-AR es-CL es-ES es-MX et eu fa ff fi fr fy-NL ga-IE gd gl gn gu-IN he hi-IN hr hsb hu hy-AM id is it ja kk km kn ko lij lt lv mai mk ml mr ms nb-NO nl nn-NO or pa-IN pl pt-BR pt-PT rm ro ru si sk sl son sq sr sv-SE ta te th tr uk uz vi xh zh-CN zh-TW" channel="release-localtest" patch_types="complete"
>  release="48.0.1" product="Firefox" platform="WINNT_x86-msvc-x86" build_id="20160817112116" locales="ach af an ar as ast az bg bn-BD bn-IN br bs ca cak cs cy da de dsb el en-GB en-US en-ZA eo es-AR es-CL es-ES es-MX et eu fa ff fi fr fy-NL ga-IE gd gl gn gu-IN he hi-IN hr hsb hu hy-AM id is it ja kk km kn ko lij lt lv mai mk ml mr ms nb-NO nl nn-NO or pa-IN pl pt-BR pt-PT rm ro ru si sk sl son sq sr sv-SE ta te th tr uk uz vi xh zh-CN zh-TW" channel="release-localtest" patch_types="complete"
>  release="48.0" product="Firefox" platform="WINNT_x86-msvc-x86" build_id="20160726073904" locales="ach af an ar as ast az bg bn-BD bn-IN br bs ca cak cs cy da de dsb el en-GB en-US en-ZA eo es-AR es-CL es-ES es-MX et eu fa ff fi fr fy-NL ga-IE gd gl gn gu-IN he hi-IN hr hsb hu hy-AM id is it ja kk km kn ko lij lt lv mai mk ml mr ms nb-NO nl nn-NO or pa-IN pl pt-BR pt-PT rm ro ru si sk sl son sq sr sv-SE ta te th tr uk uz vi xh zh-CN zh-TW" channel="release-localtest" patch_types="complete"
> -release="47.0.2" product="Firefox" platform="WINNT_x86-msvc-x86" build_id="20161031133903" locales="de en-US ru" channel="release-localtest" patch_types="complete" from="/firefox/releases/47.0.2/win32/%locale%/Firefox Setup 47.0.2.exe" ftp_server_from="https://archive.mozilla.org/pub" ftp_server_to="https://archive.mozilla.org/pub"
> +release="47.0.2" product="Firefox" platform="WINNT_x86-msvc-x86" build_id="20161031133903" locales="ach af an ar as ast az bg bn-BD bn-IN br bs ca cak cs cy da de dsb el en-GB en-US en-ZA eo es-AR es-CL es-ES es-MX et eu fa ff fi fr fy-NL ga-IE gd gl gn gu-IN he hi-IN hr hsb hu hy-AM id is it ja kk km kn ko lij lt lv mai mk ml mr ms nb-NO nl nn-NO or pa-IN pl pt-BR pt-PT rm ro ru si sk sl son sq sr sv-SE ta te th tr uk uz vi xh zh-CN zh-TW" channel="release-localtest" patch_types="complete partial" from="/firefox/releases/47.0.2/win32/%locale%/Firefox Setup 47.0.2.exe" ftp_server_from="https://archive.mozilla.org/pub" ftp_server_to="https://archive.mozilla.org/pub"

Feel free to delete everything except 47.0.2
Attachment #8855289 - Flags: review?(rail) → review+
Comment hidden (mozreview-request)
(Assignee)

Comment 17

2 years ago
https://tools.taskcluster.net/task-group-inspector/#/t39qwTCNT-6_nNniiqTDDQ?_k=5i46se

This taskgraph was created with the following method:
1. Connect onto bm85 (with the build user)
2. Download https://bugzilla.mozilla.org/attachment.cgi?id=8818876
3. Modify runme.py to point to 52.0.2
4. Copy the creds to config.yml
5. Source the virtualenv of releaserunner
6. `python runme.py`
7. If the graph is printed out, uncomment the last line of runme.py and do step 6.
(Assignee)

Comment 18

2 years ago
Graph green so far. Just like bug 1319905 comment 21, I added:

> ["fileUrls"]["release-localtest"]["partials"].append("Firefox-47.0.2-build2": "http://archive.mozilla.org/pub/firefox/candidates/52.0.2-candidates/build1/update/%OS_FTP%/%LOCALE%/firefox-47.0.2-52.0.2.partial.mar")

It's a now a scheduled change that's gonna be enacted in 3 hours[1]. Does it look good to you, rail? 

[1] https://aus4-admin.mozilla.org/releases/scheduled_changes
Flags: needinfo?(rail)
(Assignee)

Comment 19

2 years ago
Link in comment 18 doesn't show the new data. You can find it there: https://aus4-admin.mozilla.org/api/scheduled_changes/releases/2/revisions
We chatted on IRC
Flags: needinfo?(rail)
(Assignee)

Comment 21

2 years ago
Eh, release-locatest now points to 53.0, which made some of the update tests fail[1]. I'll restart the partial update process with 53.0, once its out.

[1] https://tools.taskcluster.net/task-group-inspector/#/t39qwTCNT-6_nNniiqTDDQ
Comment hidden (mozreview-request)
(Assignee)

Comment 23

2 years ago
Comment on attachment 8855289 [details]
Bug 1347030 - Provide a partial mar file to update from Firefox 47.0.2 to 53.0

Resetting review. Patch is now from 47.0.2 to 53.0.
Attachment #8855289 - Flags: review+ → review?(rail)

Comment 24

2 years ago
mozreview-review
Comment on attachment 8855289 [details]
Bug 1347030 - Provide a partial mar file to update from Firefox 47.0.2 to 53.0

https://reviewboard.mozilla.org/r/127162/#review135744
Attachment #8855289 - Flags: review?(rail) → review+
(Assignee)

Comment 26

2 years ago
This has been live on release since yesterday 7pm UTC. Let me detail below what was needed:

1. Tasks in comment 25 went green but only after doing comment 18 again.
2. Manually add the bouncer entries, which means:
  a. Create a new product called "Firefox-53.0-Partial-47.0.2". Don't include the build numbers in the name. Build numbers are for *-cdntest channels, only. For instance: https://bounceradmin.mozilla.com/admin/mirror/product/6993/
  b. Create locations for this new product. In this particular case, we needed 2 locations, one for win32 and the other for win64. Ask for a review, because typos can happen. (Corrected) example: https://bounceradmin.mozilla.com/admin/mirror/location/?product__id__exact=6993
3. Resolve the human decision of the graph.
4. Wait for https://bounceradmin.mozilla.com/stats/locations/?p=6993 to show some products (replace the ID of the product by yours)
5. Manually test out the bouncer. Change this link http://download.mozilla.org/?product=firefox-53.0-partial-47.0.2&os=win&lang=fr with your product name and all the locations you created. That's case insensitive.
6. On balrog, perform this addition in the release blob,
> ["fileUrls"]["*"]["partials"].append("Firefox-47.0.2-build2": "http://download.mozilla.org/?product=firefox-53.0-partial-47.0.2&os=%OS_BOUNCER%&lang=%LOCALE%")
7. If you have a what's new page blob on balrog
  a. Download the regular blob (the one just updated in step 6), which now contains all the partials.
  b. Add the whatsnew bits to it, which is detailed in https://wiki.mozilla.org/Release:Release_Automation_on_Mercurial:Updates_through_Shipping#Set-up_whatsnew_page (Don't forget to change the name)
  c. Ask for another review.
8. You can now test out the balrog rule. In order to to so"
  a. Go to the logs of one update verify job.
  b. Copy one of URL, like:  https://aus4.mozilla.org/update/3/Firefox/47.0.2/20161031133903/WINNT_x86-msvc-x86/eo/release-localtest/default/default/default/update.xml?force=1
  c. Modify it to now point to release (and to aus5, which has a valid certificate): https://aus5.mozilla.org/update/3/Firefox/47.0.2/20161031133903/WINNT_x86-msvc-x86/eo/release/default/default/default/update.xml?force=1
  d. See the new partial entry in the XML:
> <patch type="partial" URL="http://download.mozilla.org/?product=firefox-53.0-partial-47.0.2&os=win&lang=eo&force=1" hashFunction="sha512" hashValue="1f8a24ec43002477b21ffdc06cdbb7c885ebf57eecc08e45c31089ceac1a9e57cf9ec2c4becc638ec346335e5d84e9384f2503d61a259232151cb6e173126cce" size="42336047"/>
  e. Copy/paste the URL, just to make sure it's a valid one.


Because this procedure is quite long, I'll document it in the wiki. Kudos to Rail, who explained the process to me, and spotted errors.
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
(Assignee)

Comment 27

2 years ago
Process being documented in https://github.com/mozilla/releasewarrior/pull/75
You need to log in before you can comment on or make changes to this bug.