489338 - Investigate source package based Scheduler/Triggered builds

Status: RESOLVED WONTFIX

(Mozilla Messaging Graveyard :: Release Engineering, defect)

Description

[Philippe M. Chiasson (:gozer)]

For full clobber builds, like try builds and nightly builds, it seems like it could save a lot of time to just have a single scheduled builder on Linux that hg clone's the needed sources, runs client.py and assembles a source tarball (without .hg*)

It would then upload that source tarball somewhere centrally (ftp.m.o) and triggers all the other builders with it. These builders would not need to run mercurial at all, and instead just download the source tarball and crack it open.

This seems to have many possible advantages, limiting the CPU/IO intensive  hg clone operation to a single builder, limiting the bandwidth usage, and insuring that all triggered builds are of the exact same source version.

Comment 1

[Philippe M. Chiasson (:gozer)]

Results of some Linux testing:

Linux VM:

 wget mozilla-central     = 1.025s
 wget comm-central        = 0.348s
 tar jxf comm-central     = 4.601s
 tar jxf mozilla-central  = 38.683s
 du -sh .                 = 429M

 hg clone comm-central    = 31.1s
 hg clone mozilla-central = 291.858s (4m51sec)
 du -sh .                 = 807M

Linux tar: 44.657s  (9.60Mb/s)
Linux hg : 322.958s (2.49Mb/s)

EC2 Windows (external wget):

 wget comm-central        = 14.5s
 wget mozilla-central     = 56.531s
 tar jxf comm-central     = 34.828s
 tar jxf mozilla-central  = 225.65s  (3m45s)
 du -sh .                 = 429M

 hg clone comm-central    = 514.33s  (8m34sec)
 hg clone mozilla-central = 3844.94s (1h4m4)
 du -sh .                 = 807M

Windows tar: 331.509s   (1.29Mb/s) (5m30s)
Windows hg : 4359.270s  (0.18Mb/s) (1h12m)

Windows tar/hg: tar is 13.1x faster

EC2 Windows (s3 wget):

 wget comm-central        = 1.84s
 wget mozilla-central     = 27.90s
 tar jxf comm-central     = 34.828s
 tar jxf mozilla-central  = 225.65s  (3m45s)
 du -sh .                 = 429M

 hg clone comm-central    = 514.33s  (8m34sec)
 hg clone mozilla-central = 3844.94s (1h4m4)
 du -sh .                 = 807M

Windows tar: 290.218s   (1.47Mb/s) (4m49s)
Windows hg : 4359.270s  (0.18Mb/s) (1h12m)

Windows tar/hg: s3+tar is 15x faster

Comment 2

[Philippe M. Chiasson (:gozer)]

Mac Leopard VM:

 wget mozilla-central     = 7.352s
 wget comm-central        = 0.946s
 tar jxf comm-central     = 31.429s
 tar jxf mozilla-central  = 143.943s  (2m13s)
 du -sh .                 = 429M

 hg clone comm-central    = 117.626s
 hg clone mozilla-central = 1044.804s (17m24s)
 du -sh .                 = 807M

Mac tar: 183.670s  (2.33Mb/s)
Mac hg : 1162.430  (.69Mb/s)

Mac Leopard tar/hg: tar is 6.3x faster