Closed Bug 987267 Opened 6 years ago Closed 6 years ago

use gzip compression for hg bundles

Categories

(Release Engineering :: General, defect)

x86_64
Windows 8.1
defect
Not set

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: taras.mozilla, Assigned: rail)

References

Details

Attachments

(1 file, 1 obsolete file)

hg operations are gonna take as long or longer than builds once we implement caching on windows. According to glandium's stats, hg unbundling only happens on ~3% of the builds, but it does lead to some surprisingly-slow build times.

Simply zipping up the whole repo ends in a 3x speed up, switching to gzip bundles is a 1.4x speedup. Lets swap to gzip bundles asap(or just hg clone --uncompressed on windows which would be fastest)
 
++ mkdir mozilla-central
++ cd mozilla-central
++ hg init .
++ hg unbundle ../mozilla-central.hg
adding changesets
adding manifests
adding file changes
added 174831 changesets with 983111 changes to 153611 files
(run 'hg update' to get a working copy)

real    31m59.826s
user    0m0.015s
sys     0m0.015s

++ cd mozilla-central
++ hg init .
++ hg unbundle ../mozilla-central.gz.bundle
adding changesets
adding manifests
adding file changes
added 174831 changesets with 983111 changes to 153611 files
(run 'hg update' to get a working copy)

real    22m59.116s
user    0m0.015s
sys     0m0.030s


++ rm -fR mozilla-central
++ unzip -q mozilla-central.zip

real    11m19.762s
user    0m0.015s
sys     0m0.000s

1.1G mozilla-central.zip
914M mozilla-central.7z
790M mozilla-central.gz.bundle
631M mozilla-central.hg
Blocks: 940789
Attached patch gzip.bundle.diff (obsolete) — Splinter Review
Switching to gzip is easy. We'll need to regenerate the bundles (see https://wiki.mozilla.org/ReleaseEngineering/Merge_Duty/Steps#Create_new_hg_bundles)

zipping .hg would require changes in hg-tool.py
Attachment #8395825 - Flags: review?(catlee)
Attached patch gzip.bundle.diffSplinter Review
err, wrong position
Attachment #8395825 - Attachment is obsolete: true
Attachment #8395825 - Flags: review?(catlee)
Attachment #8395828 - Flags: review?(catlee)
Attachment #8395828 - Flags: review?(catlee) → review+
Regenerated the bundles.
(In reply to Rail Aliiev [:rail] from comment #4)
> Regenerated the bundles.

<3 

Any reason to keep this open? We can pursue other options in another bug another time.
Assignee: nobody → rail
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
Summary: hg bundles suck → use gzip compression for hg bundles
23 minutes to do an unbundle from a gzipped bundle and 11:20 to unzip?! My computer can do a gzip unbundle in 8 minutes. And 4 of those should go away with bundle2 (I'd like to add cache data into bundles so clients don't need to regenerate it).
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.