The default bug view has changed. See this FAQ.

use gzip compression for hg bundles

RESOLVED FIXED

Status

Release Engineering
General Automation
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: (dormant account), Assigned: rail)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment, 1 obsolete attachment)

(Reporter)

Description

3 years ago
hg operations are gonna take as long or longer than builds once we implement caching on windows. According to glandium's stats, hg unbundling only happens on ~3% of the builds, but it does lead to some surprisingly-slow build times.

Simply zipping up the whole repo ends in a 3x speed up, switching to gzip bundles is a 1.4x speedup. Lets swap to gzip bundles asap(or just hg clone --uncompressed on windows which would be fastest)
 
++ mkdir mozilla-central
++ cd mozilla-central
++ hg init .
++ hg unbundle ../mozilla-central.hg
adding changesets
adding manifests
adding file changes
added 174831 changesets with 983111 changes to 153611 files
(run 'hg update' to get a working copy)

real    31m59.826s
user    0m0.015s
sys     0m0.015s

++ cd mozilla-central
++ hg init .
++ hg unbundle ../mozilla-central.gz.bundle
adding changesets
adding manifests
adding file changes
added 174831 changesets with 983111 changes to 153611 files
(run 'hg update' to get a working copy)

real    22m59.116s
user    0m0.015s
sys     0m0.030s


++ rm -fR mozilla-central
++ unzip -q mozilla-central.zip

real    11m19.762s
user    0m0.015s
sys     0m0.000s

1.1G mozilla-central.zip
914M mozilla-central.7z
790M mozilla-central.gz.bundle
631M mozilla-central.hg
(Reporter)

Updated

3 years ago
Blocks: 940789
(Assignee)

Comment 1

3 years ago
Created attachment 8395825 [details] [diff] [review]
gzip.bundle.diff

Switching to gzip is easy. We'll need to regenerate the bundles (see https://wiki.mozilla.org/ReleaseEngineering/Merge_Duty/Steps#Create_new_hg_bundles)

zipping .hg would require changes in hg-tool.py
Attachment #8395825 - Flags: review?(catlee)
(Assignee)

Comment 2

3 years ago
Created attachment 8395828 [details] [diff] [review]
gzip.bundle.diff

err, wrong position
Attachment #8395825 - Attachment is obsolete: true
Attachment #8395825 - Flags: review?(catlee)
Attachment #8395828 - Flags: review?(catlee)

Updated

3 years ago
Attachment #8395828 - Flags: review?(catlee) → review+
(Assignee)

Comment 3

3 years ago
Comment on attachment 8395828 [details] [diff] [review]
gzip.bundle.diff

https://hg.mozilla.org/build/tools/rev/44faaa1a3937
Attachment #8395828 - Flags: checked-in+
(Assignee)

Comment 4

3 years ago
Regenerated the bundles.
(Reporter)

Comment 5

3 years ago
(In reply to Rail Aliiev [:rail] from comment #4)
> Regenerated the bundles.

<3 

Any reason to keep this open? We can pursue other options in another bug another time.

Updated

3 years ago
Assignee: nobody → rail
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
Summary: hg bundles suck → use gzip compression for hg bundles

Comment 6

3 years ago
23 minutes to do an unbundle from a gzipped bundle and 11:20 to unzip?! My computer can do a gzip unbundle in 8 minutes. And 4 of those should go away with bundle2 (I'd like to add cache data into bundles so clients don't need to regenerate it).
You need to log in before you can comment on or make changes to this bug.