1023624 - Figure out tools versioning for partial generation

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Nick Thomas [:nthomas] (UTC+12)]

In the next few days we (I mean lucky me!) need to address this question:

Can we snapshot the current set of scripts for partial generation, and add some binaries, for use in the partial update server ?

Related issues:
* current scripts to v2 and v3 manifests, how far back does v2 support go ? aka what is the oldest version we can create a partial for ? 
* is that far enough ? what about those watersheds we have ?
* are there other issues we need to be aware of (eg channel tagging on mar files)
* what do we do in the future when files change ? Will the plans for signed mars on non-windows make any difference

Also, we need to verify that make_incremental_updates.py produces the same output as make_incremental_update.sh, before we start in on modifying it.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

I've been creating some docs at
 https://wiki.mozilla.org/User:NThomas:Mar_Generation
 https://wiki.mozilla.org/User:NThomas:Watersheds
to capture the history of the update system. They may move to another location later, redirects will save us.

If we were to take the current scripts in tools/update-packaging/, plus binaries for mar and mbsdiff, then we would generate mar which would be supported as far back as Firefox/Thunderbird 5.0. 

For Firefox on the release channel, we wouldn't actually want to do any older than 9.0 --> latest due to the watershed for Win 2K and XP RM/SP1 desupport. We could do 5.0+ --> 12.0 partials if it had value (dubious after all this time!). It's similar for Firefox on beta, except the watershed is earlier at 29.0, so we could do 29.0 --> latest, or 9.0+ --> 29.0b9.

That seems like far enough to me, what do you think Chris ? We could always compile older code for mar and mbsdiff if we need to.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

As for tracking future changes, we're essentially interested in 
  modules/libmar/ other-licenses/bsdiff/ modules/libbz2/ tools/update-packaging/
for mar, mbsdiff (which includes libbz2), and the scripts respectively. 

In the first three there are many changes like
  http://hg.mozilla.org/mozilla-central/rev/9a71fbba0656
  http://hg.mozilla.org/mozilla-central/rev/20d7a9037429
which don't change the fundamentals and are build system work, which is likely to continue as more definitions move into moz.build. And then there are the real functional changes we care about. 

If we add an in-tree 'update tools version' then it's easy for a Senbonzakura client to specify what version of tools to use, and our job of managing things as they go up the trains is zero/minimal. But we have the problem of devs not knowing to bump the tools version if they land changes. Maybe a b2gbumper-like cronjob can handle that, and we bump when anything changes, regardless of how trivial it is just to avoid having to think about it. 

We could poll hg.m.o for changes in in-tree version and create a new tools dir on Senbonzakura. Scripts can come from hg, binary utils from a special compile, or from
 http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux64/latest/mar
 http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux64/latest/mbsdiff
once the on-change build is done (hash checks?). Maybe even make the new tools dir the default automatically.

Comment 3

[Chris AtLee [:catlee]]

Agreed - doing partials back to Firefox 9 seems like a stretch. We can still reach a good chunk of our non-latest userbase going back to Firefox 18 or so.

Comment 4

[Hal Wine [:hwine] use NI!]

(In reply to Nick Thomas [:nthomas] from comment #1)
> For Firefox on the release channel, we wouldn't actually want to do any
> older than 9.0 --> latest due to the watershed for Win 2K and XP RM/SP1
> desupport. We could do 5.0+ --> 12.0 partials if it had value (dubious after
> all this time!). It's similar for Firefox on beta, except the watershed is
> earlier at 29.0, so we could do 29.0 --> latest, or 9.0+ --> 29.0b9.

These are important points. How much of this do we want enforced by Senbonzakura, and how much is on the client to not ask for something out-of-policy?

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

I think we should capture that information somewhere, and the server has a certain appeal because there's only one of it and there may be several client scripts. The restrictions are specific to a number of details (app, versions, channel) which aren't currently being submitted though. They could be inferred from URLs or the mar files themselves I guess, but I'm fairly comfortable with the docs I wrote.

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

I've addressed everything in comment #0 by taking the current state and v0, except for

(In reply to Nick Thomas [:nthomas] from comment #0)
> * what do we do in the future when files change ? Will the plans for signed
> mars on non-windows make any difference

This can be dealt with in another bug which includes the update developers.
 
> Also, we need to verify that make_incremental_updates.py produces the same
> output as make_incremental_update.sh, before we start in on modifying it.

The vibe I'm getting is that we're not going to use this in the initial implementation.