Closed Bug 489338 Opened 11 years ago Closed 10 years ago
Investigate source package based Scheduler/Triggered builds
For full clobber builds, like try builds and nightly builds, it seems like it could save a lot of time to just have a single scheduled builder on Linux that hg clone's the needed sources, runs client.py and assembles a source tarball (without .hg*) It would then upload that source tarball somewhere centrally (ftp.m.o) and triggers all the other builders with it. These builders would not need to run mercurial at all, and instead just download the source tarball and crack it open. This seems to have many possible advantages, limiting the CPU/IO intensive hg clone operation to a single builder, limiting the bandwidth usage, and insuring that all triggered builds are of the exact same source version.
Results of some Linux testing: Linux VM: wget mozilla-central = 1.025s wget comm-central = 0.348s tar jxf comm-central = 4.601s tar jxf mozilla-central = 38.683s du -sh . = 429M hg clone comm-central = 31.1s hg clone mozilla-central = 291.858s (4m51sec) du -sh . = 807M Linux tar: 44.657s (9.60Mb/s) Linux hg : 322.958s (2.49Mb/s) EC2 Windows (external wget): wget comm-central = 14.5s wget mozilla-central = 56.531s tar jxf comm-central = 34.828s tar jxf mozilla-central = 225.65s (3m45s) du -sh . = 429M hg clone comm-central = 514.33s (8m34sec) hg clone mozilla-central = 3844.94s (1h4m4) du -sh . = 807M Windows tar: 331.509s (1.29Mb/s) (5m30s) Windows hg : 4359.270s (0.18Mb/s) (1h12m) Windows tar/hg: tar is 13.1x faster EC2 Windows (s3 wget): wget comm-central = 1.84s wget mozilla-central = 27.90s tar jxf comm-central = 34.828s tar jxf mozilla-central = 225.65s (3m45s) du -sh . = 429M hg clone comm-central = 514.33s (8m34sec) hg clone mozilla-central = 3844.94s (1h4m4) du -sh . = 807M Windows tar: 290.218s (1.47Mb/s) (4m49s) Windows hg : 4359.270s (0.18Mb/s) (1h12m) Windows tar/hg: s3+tar is 15x faster
Mac Leopard VM: wget mozilla-central = 7.352s wget comm-central = 0.946s tar jxf comm-central = 31.429s tar jxf mozilla-central = 143.943s (2m13s) du -sh . = 429M hg clone comm-central = 117.626s hg clone mozilla-central = 1044.804s (17m24s) du -sh . = 807M Mac tar: 183.670s (2.33Mb/s) Mac hg : 1162.430 (.69Mb/s) Mac Leopard tar/hg: tar is 6.3x faster
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.