987267 - use gzip compression for hg bundles

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[(dormant account)]

hg operations are gonna take as long or longer than builds once we implement caching on windows. According to glandium's stats, hg unbundling only happens on ~3% of the builds, but it does lead to some surprisingly-slow build times.

Simply zipping up the whole repo ends in a 3x speed up, switching to gzip bundles is a 1.4x speedup. Lets swap to gzip bundles asap(or just hg clone --uncompressed on windows which would be fastest)
 
++ mkdir mozilla-central
++ cd mozilla-central
++ hg init .
++ hg unbundle ../mozilla-central.hg
adding changesets
adding manifests
adding file changes
added 174831 changesets with 983111 changes to 153611 files
(run 'hg update' to get a working copy)

real    31m59.826s
user    0m0.015s
sys     0m0.015s

++ cd mozilla-central
++ hg init .
++ hg unbundle ../mozilla-central.gz.bundle
adding changesets
adding manifests
adding file changes
added 174831 changesets with 983111 changes to 153611 files
(run 'hg update' to get a working copy)

real    22m59.116s
user    0m0.015s
sys     0m0.030s


++ rm -fR mozilla-central
++ unzip -q mozilla-central.zip

real    11m19.762s
user    0m0.015s
sys     0m0.000s

1.1G mozilla-central.zip
914M mozilla-central.7z
790M mozilla-central.gz.bundle
631M mozilla-central.hg

Comment 1

[Rail Aliiev [:rail]]

Attachment: gzip.bundle.diff

Switching to gzip is easy. We'll need to regenerate the bundles (see https://wiki.mozilla.org/ReleaseEngineering/Merge_Duty/Steps#Create_new_hg_bundles)

zipping .hg would require changes in hg-tool.py

Comment 2

[Rail Aliiev [:rail]]

Attachment: gzip.bundle.diff

err, wrong position

Comment 3

[Rail Aliiev [:rail]]

Comment on attachment 8395828 [details] [diff] [review]
gzip.bundle.diff

https://hg.mozilla.org/build/tools/rev/44faaa1a3937

Comment 4

[Rail Aliiev [:rail]]

Regenerated the bundles.

Comment 5

[(dormant account)]

(In reply to Rail Aliiev [:rail] from comment #4)
> Regenerated the bundles.

<3 

Any reason to keep this open? We can pursue other options in another bug another time.

Comment 6

[Gregory Szorc [:gps]]

23 minutes to do an unbundle from a gzipped bundle and 11:20 to unzip?! My computer can do a gzip unbundle in 8 minutes. And 4 of those should go away with bundle2 (I'd like to add cache data into bundles so clients don't need to regenerate it).