This probably doesn't belong to this module, and maybe I should even submit it to Mercurial rather than here, but... So currently we promote using Mercurial Bundles, and Mercurial does this by default. However, there are several caveats. Firstly, it's extremely slow. Unbundle the whole mozilla-central takes 30~40min, and it doesn't scale to multicore, so powerful CPU doesn't help here. Second, because of its being so slow, if you `hg clone` directly and have Mercurial use bundles by default, the connection may simply be dropped during the processing before the whole bundle is downloaded. I've observed this once when I tried to clone m-c on a GCP instance. I did some experiments, and found that packing the whole .hg directory seems to be actually a pretty good idea. A tar package of .hg of a newly cloned m-c repo takes 3.2GB, and if I xz it, it takes only 2.2GB, which is even smaller than the "stream" type for m-c listed on [Mercurial Bundles page](https://hg.cdn.mozilla.net/). Untarring such xz-ed tar package of .hg takes only 2.5min (plain tar takes just ~1m), and `hg update --clean` from it takes probably 2~3min on a 8 CPU instance, and since it scales to multicore, I would expect it to translate to 4~6min on a reasonable 4-core laptop, or 1min something for a more powerful desktop workstation. 2.2GB is about double size from the zstd bundle, but on a 10Mbps bandwidth, that difference translates to just ~15min longer download time. Even with such slow connection, using .hg package is still faster than using Mercurial Bundle due to the slowness and unscalability of unbundling, not to mention many people have much faster connection nowadays, especially if you download it on cloud. BTW, creating such xz-ed tar package is quite fast as well. On the 8 CPU instance, it takes ~4min to create (plain tar takes ~3.5min), and since xz is scalable to more CPU, this is likely to be IO-bound when there are enough cores. Given these, I suggest that we should probably switch to simply packing .hg directory and develop tools around it instead. Having a faster development setup experience would encourage more people to contribute, and also enable casual contributors (like me currently) to facilitate cloud computing resources for contributing. The experiments are done on GCP, with two different types of instances: * n1-standard-2 (2 vCPUs, 7.5 GB memory) * n1-highcpu-8 (8 vCPUs, 7.2 GB memory) The memory doesn't seem to matter much here, fwiw.
Bug 1535793 Comment 0 Edit History
Note: The actual edited comment in the bug view page will always show the original commenter’s name and original timestamp.
This probably doesn't belong to this module, and maybe I should even submit it to Mercurial rather than here, but... So currently we promote using Mercurial Bundles, and Mercurial does this by default. However, there are several caveats. Firstly, it's extremely slow. Unbundle the whole mozilla-central takes 30-40min, and it doesn't scale to multicore, so powerful CPU doesn't help here. Second, because of its being so slow, if you `hg clone` directly and have Mercurial use bundles by default, the connection may simply be dropped during the processing before the whole bundle is downloaded. I've observed this once when I tried to clone m-c on a GCP instance. I did some experiments, and found that packing the whole .hg directory seems to be actually a pretty good idea. A tar package of .hg of a newly cloned m-c repo takes 3.2GB, and if I xz it, it takes only 2.2GB, which is even smaller than the "stream" type for m-c listed on [Mercurial Bundles page](https://hg.cdn.mozilla.net/). Untarring such xz-ed tar package of .hg takes only 2.5min (plain tar takes just about 1m), and `hg update --clean` from it takes probably 2-3min on a 8 CPU instance, and since it scales to multicore, I would expect it to translate to 4-6min on a reasonable 4-core laptop, or 1min something for a more powerful desktop workstation. 2.2GB is about double size from the zstd bundle, but on a 10Mbps bandwidth, that difference translates to just about 15min longer download time. Even with such slow connection, using .hg package is still faster than using Mercurial Bundle due to the slowness and unscalability of unbundling, not to mention many people have much faster connection nowadays, especially if you download it on cloud. BTW, creating such xz-ed tar package is quite fast as well. On the 8 CPU instance, it takes about 4min to create (plain tar takes about 3.5min), and since xz is scalable to more CPU, this is likely to be IO-bound when there are enough cores. Given these, I suggest that we should probably switch to simply packing .hg directory and develop tools around it instead. Having a faster development setup experience would encourage more people to contribute, and also enable casual contributors (like me currently) to facilitate cloud computing resources for contributing. The experiments are done on GCP, with two different types of instances: * n1-standard-2 (2 vCPUs, 7.5 GB memory) * n1-highcpu-8 (8 vCPUs, 7.2 GB memory) The memory doesn't seem to matter much here, fwiw.