813921 - Figure out why mac partials from 16 -> 17.0 are so large

Status: RESOLVED WONTFIX

(Release Engineering :: Release Automation, defect)

Description

[Nick Thomas [:nthomas] (UTC+12)]

NB: This is not a problem for windows or linux updates to 17.0.

For mac, partials between .0 releases, or to a b1, tend to be less than 50% of the size of a complete mar. eg
 http://ftp.mozilla.org/pub/mozilla.org/firefox/releases/16.0/update/mac/en-US/
  firefox-15.0.1-16.0.partial.mar         15M
  firefox-16.0.complete.mar               32M
and similar for 15.0b6 to 16.0b1.

But for 16.0.2 -> 17.0 they're ~85% of the size
 http://ftp.mozilla.org/pub/mozilla.org/firefox/releases/17.0/update/mac/en-US/
  firefox-16.0.2-17.0.partial.mar         28M
  firefox-17.0.complete.mar               33M
Similar for 16.0b6 -> 17.0b1. This will slow the rate that users update to 17.0.


For some reason some executables have add instructions in the update.manifest, rather than patch instructions. This is particularly a problem for Contents/MacOX/XUL, which is 66MB uncompressed, but libnssdbm3.dylib, libnss3.dylib, libnspr4.dylib, and libmozsqlite3.dylib are affected too.

I think we must be getting oversize patch files that go down the wrong branch of this if block:
 http://hg.mozilla.org/releases/mozilla-release/file/919435c6f654/tools/update-packaging/make_incremental_update.sh#l172

We switched to clang between 16 and 17, so perhaps that tickles mbsdiff somehow. I can generate patch files larger than the original with a 2010 vintage mbsdiff, so it's something about the binaries being diffed.

There's also a stray '</textarea>' in
 http://hg.mozilla.org/releases/mozilla-release/rev/bfd0d06e1a78#l3.13
from bug 776537 too, although that doesn't appears in the build log https://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/17.0-candidates/build2/logs/release-mozilla-release-macosx64_build-bm30-build1-build1.txt.gz

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

From a skim of how bsdiff works (http://www.daemonology.net/papers/bsdiff.pdf) it seems plausible that we'd get large diffs if binaries generated by clang have a lot of differences from what gcc created previously. 

For 17.0b6 -> 18.0b1 we have partials which are 66% of the complete size, so clang seems to churn a bit from release to release too.

I'm inclined to not persue this any further so please reopen if you disagree and have someone with serious compiler chops to dig into it.