243739 - Generate xdelta patches for incremental source tarball update between releases

Status: RESOLVED WONTFIX

(mozilla.org :: Miscellaneous, task)

Description

[Alexey Molchanov]

User-Agent:       Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.5) Gecko/20031031
Build Identifier: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.5) Gecko/20031031

I propose mozilla.org system administrators to generate xdelta files between
releases and upload them to ftp servers along with source taballs.

First, few words about xdelta itself (from xdelta man page):
=====
    Xdelta provides the ability to generate deltas between a pair of files and
later apply those deltas. It operates similar to the diff and patch commands,
but works on binary files and does not produce a human readable output.
    Xdelta  has three subcommands, delta, patch, and info. Delta accepts two
file versions and produces a delta, while patch accepts the original file
version and delta and produces the second version. The info command prints
useful information about a delta.
=====

xdelta files is small: for example xdelta generated between
mozilla-source-1.5.tar.bz2 and mozilla-source-1.6.tar.bz2 tarballs is 5.25 Mb in
size. So if I have already downloaded mozilla-source-1.5.tar.bz2 and now I want
to upgrade mozilla to 1.6 version I need to download only this small xdelta file
(name it mozilla-source-1.5-1.6.tar.xdelta) and using 'xdelta patch' command I
can recreate mozilla-source-1.6.tar.bz2 file on my computer. Using xdelta in
such a way will save me about 25 Mb of traffic (30 Mb (full source size) - 5 Mb
(xdelta size)).
Releasing xdelta files will give opportunity to users who like constantly
upgrade their mozilla version to save a few bandwidth (very important for
dial-up users), also this will help to save bandwidth to mozilla servers.

By the way, KDE project successfully uses xdelta in their release process for a
long time.

Reproducible: Always
Steps to Reproduce:

Comment 1

[Alexey Molchanov]

To generate xdelta between releases all you need to do is to run this command:
$ xdelta delta mozilla-source-1.5.tar.gz mozilla-source-1.6.tar.gz
mozilla-source-1.5-1.6.tar.xdelta
Generation of this xdelta on Duron 750/256Mb takes about 15 minutes but this
time will dramatically decrease if you have 512 Mb RAM (better 1 Gb).

P.S. Download xdelta 1.1.3 for Win32 you can here:
http://www.eng.uwaterloo.ca/~ejones/software/xdelta-win32.html

Comment 2

[Jesse Ruderman]

Comment 0 says that the delta between 1.5 and 1.6 is 5.25 MB when you run it on
compressed tarballs.  How big is it when you run it on uncompressed tarballs or
on trees of files?  I would expect the delta to be smaller when run on
uncompressed files, but I don't know enough about rsync/xdelta or bz2 to be sure.

Comment 3

[Mark Stier]

Alternatively, provide rsync access to the unpacked nightly source tarball
release. That would be much more flexible.

Comment 4

[Alexey Molchanov]

(In reply to comment #2)
> Comment 0 says that the delta between 1.5 and 1.6 is 5.25 MB when you run it on
> compressed tarballs.  How big is it when you run it on uncompressed tarballs or
> on trees of files?  I would expect the delta to be smaller when run on
> uncompressed files, but I don't know enough about rsync/xdelta or bz2 to be sure.

1. xdelta program implicitly unpacks gzip archives (but not bzip2) before
calculating delta between files. So in my example in comment 0 we already have
5.25 Mb xdelta file created by comparing uncompressed (implicitly!) tar files.
2. xdelta can't generate delta between trees of files. I works only with two files.

Can we hope that mozilla.org will serve xdelta patches for its products in the
near future?

Comment 5

[Jesse Ruderman]

The new update system for Firefox binaries uses bsdiff, a tool similar to
xdelta. It was turned on in bug 305642.

This bug seems to be about source diffs rather than binary diffs.

Comment 6

[Benjamin Smedberg]

The work/reward factor here is silly. We're not going to optimize for developers.