575317 - Support partial updates for more than the last release

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, enhancement, P3)

Description

[LegNeato]

It would be nice to support bspatch update packages for more than just the latest build. It will reduce the amount of bandwidth for both users and mozilla, decrease the amount of time it takes users to download (and potentially apply) an update, etc.

Also, by looking at the download stats we could potentially get some useful metrics.

We wouldn't have to do all builds, perhaps the 3 or 4 most recent (and the major version). So, if we are releasing Firefox 3.6.9 we would make a a patch package for 3.6.4-8 and 3.6.

Comment 1

[Chris Cooper [:coop] (he/him)]

This has been proposed at various times in the past and has been routinely shot down due to the expanded testing matrix required by QA. Can we loop someone in from the QA team to address this?

I did some cursory testing by generating a partial mar from 3.6->3.6.6. The resulting partial mar was 2.1MB vs. 18MB for the complete mar, which admittedly is a bigger saving than I thought initially.

Do we have any data about how many users come from >1 dot version back, and hence how many people we would be helping?

Another factor is time. Each extra partial mar we generate can take up to 2 minutes to complete. We could make mar generation for the different OSes run in parallel (as we do for nightlies), but we're still talking about 70+ locales multiplied by # of previous dot releases.

Comment 2

[[e-mail] (:bhearsum)]

One way we've talked about implementing this is doing on-demand partials on AUS2. This would eliminate any concerns about extra time during the build process.

Comment 3

[Robert Strong (they/them - no direct email)]

LegNeato, bug 598649 is for Android using apk files and no other platforms.

Comment 4

[LegNeato]

Whoops, apologies.

Comment 6

[Carsten Otto]

Please consider the numbers given in bug 639802 and also take into consideration that for a chemspill release only very little users have the (buggy) previous version at the time of the update.

Comment 7

[LegNeato]

Yes, this should be made a priority based on bug 639802 and the fact that we are going to quicker/more releases in a shorter time

Comment 8

[carlos]

These are the percentages of requests for complete and partial mar's since January 2010 in the mozilla.c3sl mirror:

month  complete.mar  partial.mar
           (%)         (%)      
10-01     63.53       18.72     
10-02     59.75       27.79     
10-03     58.28       24.53     
10-04     63.03       21.31     
10-05     80.17       02.58     
10-06     53.31       31.62     
10-07     63.34       16.41     
10-08     81.21       00.52     
10-09     67.87       15.08     
10-10     65.32       18.91     
10-11     81.85       02.71     
10-12     60.79       26.16     
11-01     68.31       05.74     
11-02     69.67       03.89     
11-03     54.42       21.22

The stats are taken for each month. It's not clear that chemspills increase the proportion of complete.mar requests significantly. Paradoxically, this month has one of the smaller proportions in the time frame considered.

Comment 9

[Chris AtLee [:catlee]]

(In reply to comment #8)
> These are the percentages of requests for complete and partial mar's since
> January 2010 in the mozilla.c3sl mirror:
> 
> month  complete.mar  partial.mar
>            (%)         (%)      
> 10-01     63.53       18.72     
> 10-02     59.75       27.79     
> 10-03     58.28       24.53     
> 10-04     63.03       21.31     
> 10-05     80.17       02.58     
> 10-06     53.31       31.62     
> 10-07     63.34       16.41     
> 10-08     81.21       00.52     
> 10-09     67.87       15.08     
> 10-10     65.32       18.91     
> 10-11     81.85       02.71     
> 10-12     60.79       26.16     
> 11-01     68.31       05.74     
> 11-02     69.67       03.89     
> 11-03     54.42       21.22
> 
> The stats are taken for each month. It's not clear that chemspills increase the
> proportion of complete.mar requests significantly. Paradoxically, this month
> has one of the smaller proportions in the time frame considered.

Very interesting, thanks for the breakdown!

Comment 10

[Vic]

With Aurora Release channel this becomes more of an issue, see bug 662208.

Comment 11

[Vic]

From Bug 639802 

"Dear all,

this is a feature request regarding the handling of auto-updates in case of a chemspill release (e.g. 3.6.14 to 3.6.15).

I am an administrator of a primary Mozilla mirror, serving roughly 40 TByte a day in times of a Firefox update.

When a chemspill release is distributed, this is done in the usual way: a complete upgrade ".mar" file is provided, in addition to a partial update from the previous version. Because a chemspill release naturally occurs only a few days after the release of the previous version, the percentage of Firefox users that already upgraded to the pre-chemspill version is very low.

Based on my numbers (from mozilla.ftp.halifax.rwth-aachen.de, logs from yesterday) only about 18% of all requests are directed to the partial update. In other words, by providing an additional update file from 3.6.13 to 3.6.15, a lot of traffic (and related resources) could be saved (consider the size difference of the partial and the complete update file!).

Considering the effort that is needed to handle even a non-chemspill Firefox update, we as a Mozilla mirror would be very happy to see Mozilla take away some of our burden here.

Summary: Please provide partial update files skipping two versions in case of a chemspill release.

Best regards,
Carsten Otto"

Comment 12

[John Hopkins (:jhopkins)]

At a previous job, we generated additional deltas with almost no additional computation:

t0: compute n-1 delta -> patch1
    patch1 = [ file1.patch, file2 delete, file3 add ]
t1: compute n-1 delta -> patch2
    patch2 = [ file1.patch, file4.patch ]
    combine patch2 + patch1 -> patch2+1
    patch2+1 = [ file1.patch(t0), file1.patch(t1), file2 delete, file3 add ]

Repeat steps in t1 for additional levels of delta.  Compute time is extremely low, trading off for slightly non-optimal n-2,.. deltas (the "wasteage" accumulates the more revisions you go back) though they still save the user a lot of bandwidth.

Since you're re-using n-1 binary diffs that were tested, there may be some testing benefit as well.

Comment 13

[John Hopkins (:jhopkins)]

Typo fix:

< patch2+1 = [ file1.patch(t0), file1.patch(t1), file2 delete, file3 add ]
> patch2+1 = [ file1.patch(t0), file1.patch(t1), file2 delete, file3 add, file4.patch ]

Comment 16

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

found in triage.

This will be much easier to do once Balrog is live, hence blocking on bug#583244.

Comment 17

[[e-mail] (:bhearsum)]

A few of us talked in detail about this today and came up with a short list of things that need to done to work this work:
- Tuxedo submitter - needs to create products for each partial we plan to create
- Uptake monitoring - needs to monitor all partials
- Build/Repack - needs to know how to create partial MARs for more than one version
- Updates - needs to know how to create partial snippets for more than one reason
- Update verify - needs to test partial updates for more than one version

For everything except updates, this means either turning oldVersion into the release config into a list or perhaps a better approach is to be explicit and create 'partialVersions' or something.

For updates, we talked about taking this opportunity to start rewriting patcher in python. We could create a minimal thing that understands enough to create partial MARs, and run it after the initial patcher run, but before we upload/push snippets. We might be able to hack into the existing patcher, too, but I'm not sure it's worth the effort. We also need the patcher config bumper to put the list of partials we want in the patcher config.

For update verify, we need the bumper to have any versions with partials set to test them. We may want to consider using additional locales (other than the 3 we use for n-2 and older versions) as our set to test, not sure about that yet.

I'm going to provide a more thorough outline tomorrow.

Comment 18

[[e-mail] (:bhearsum)]

We figured out a way to do this without balrog. This bug isn't going to affect nightly or aurora, thus won't help with bug 598649.

Comment 19

[[e-mail] (:bhearsum)]

Blockers all fixed, we used the new feature in Firefox 15.0b6, and we'll be using it 15.0 too! Thanks to everyone who helped on this, especially Nick and Rail for their extensive reviews and patches.

Comment 20

[Daniel C]

(In reply to Ben Hearsum [:bhearsum] from comment #19)
> Blockers all fixed, we used the new feature in Firefox 15.0b6, and we'll be
> using it 15.0 too! Thanks to everyone who helped on this, especially Nick
> and Rail for their extensive reviews and patches.

What's the plan for how many, and which releases to provide partial updates from going forward?

Comment 21

[[e-mail] (:bhearsum)]

(In reply to Daniel Cater from comment #20)
> (In reply to Ben Hearsum [:bhearsum] from comment #19)
> > Blockers all fixed, we used the new feature in Firefox 15.0b6, and we'll be
> > using it 15.0 too! Thanks to everyone who helped on this, especially Nick
> > and Rail for their extensive reviews and patches.
> 
> What's the plan for how many, and which releases to provide partial updates
> from going forward?

For 15.0, we're going to provide partials for 14.0.1, 13.0.1, and 12.0 (which represents more than 75% of the userbase). Going forward, this is going to be a release-by-release call -- extra partials are far from free to generate, so we will not be doing them for everything, only where the cost/benefit makes sense.