623609 - Buildbot short-term speed improvements

Status: VERIFIED FIXED

(Tamarin Graveyard :: Build Config, defect)

Description

[Brent Baker]

Action items out of a whiteboard session:

1) Move windows mobile compilation to the windows-deep build
2) Move the windows32 build slave to the vacant winmo build slave hardware
3) Move the PPC build slaves out of phase 1 and into the deep phase
4) Move the linux32 build slave onto new HP server with Ubuntu 10.10 (x64)
5) Move linux-deep build slave to old lin32 phase 1 hardware and retire machine
6) Drop windows mobile performance build slave and retire old hardware

Comment 1

[Brent Baker]

Attachment: 3) Move the PPC build slaves out of phase 1 and into the deep phase

Comment 2

[Brent Baker]

Attachment: 3) Move the PPC build slaves out of phase 1 and into the deep phase

Updated patch to include build slave rename to mac-ppc*-deep

Comment 3

[Brent Baker]

Comment on attachment 501713 [details] [diff] [review]
3) Move the PPC build slaves out of phase 1 and into the deep phase

Chris when I push this patch I will be dropping the following from the deep slave config:

    mac_ppc_104a_deep_factory.addStep(test_smoke)
    mac_ppc_104a_deep_factory.addStep(util_process_clean)

It is not possible to run smokes on these machines since not all of the shells will be available (ONLY shells that are compiled on this machine are guaranteed to be available)

Comment 4

[Brent Baker]

Comment on attachment 501713 [details] [diff] [review]
3) Move the PPC build slaves out of phase 1 and into the deep phase

Removing review request. I need to rework the current ppc-deep build steps into the moved over slaves since they are the builders that will actually be compiling the binaries. Previous configuration assumed that all of the necessary binaries were all available by the time that the deep phase started, which is no longer true.

I will repost an updated patch for review shortly.

Comment 5

[Brent Baker]

Attachment: 3) Move the PPC build slaves out of phase 1 and into the deep phase

Move PPC into the deep phase, also had to juggle the current ppc-deep build steps around so that the step is run on the machine that is compiling the binary.

A follow-up bug/patch could attempt to rebalance the build steps to take advantage of the now under used mac-ppc-deep build slave

Comment 6

[Brent Baker]

Attachment: Drop windows mobile performance build slave and retire old hardware

This patch builds on attachment 501749 [details] [diff] [review] (move PPC to deep)

Comment 7

[Brent Baker]

Attachment: Move windows mobile compilation to the windows-deep build

Move winmo compilation to the windows-deep build slave, this builds on attachment 501751 [details] [diff] [review]

NOTE: A follow up patch will be adding the windows64 compilation build steps back

Comment 8

[Tamarin Bot]

changeset: 5740:288db49ca52e
user:      Brent Baker <[e-mail]>
summary:   Bug 623609: move PPC build slaves to the deep phase of the build (r+cpeyer)

http://hg.mozilla.org/tamarin-redux/rev/288db49ca52e

Comment 9

[Tamarin Bot]

changeset: 5741:d463158c3322
user:      Brent Baker <[e-mail]>
summary:   Bug 623609: remove the windows mobile performance build slave (r+cpeyer)

http://hg.mozilla.org/tamarin-redux/rev/d463158c3322

Comment 10

[Tamarin Bot]

changeset: 5742:f1375a9e78bc
user:      Brent Baker <[e-mail]>
summary:   Bug 623609: remove windows mobile compilation from phase 1 of the build and move it to the deep phase of the build (r+cpeyer)

http://hg.mozilla.org/tamarin-redux/rev/f1375a9e78bc

Comment 11

[Tamarin Bot]

changeset: 5743:6b6f12be7ac9
user:      Brent Baker <[e-mail]>
summary:   Bug 623609: remove unused scripts now that PPC binaries are no longer built on the intel slaves (r+cpeyer)

http://hg.mozilla.org/tamarin-redux/rev/6b6f12be7ac9

Comment 12

[Tamarin Bot]

changeset: 5744:93398e36011c
user:      Brent Baker <[e-mail]>
summary:   Bug 623609: fix a couple of tweaks with moving PPC to the deep phase that were found when trying to deploy changes (r+brbaker)

http://hg.mozilla.org/tamarin-redux/rev/93398e36011c

Comment 13

[Brent Baker]

Attachment: Move the windows32 build slave to the vacant winmo build slave hardware

Move windows build slave to asteamwin19 and also add compilation of win64 to this builder

Comment 14

[Tamarin Bot]

changeset: 5745:be6ec39a0659
user:      Brent Baker <[e-mail]>
summary:   Bug 623609: move windows builder to a new machine and add compiling of win64 binaries to this machine (r+dschaffe)

http://hg.mozilla.org/tamarin-redux/rev/be6ec39a0659

Comment 15

[Tamarin Bot]

changeset: 5746:8ca89d8412b8
user:      Brent Baker <[e-mail]>
summary:   Bug 623609: remove the compilation build steps on windows64, binaries are compiled on a faster machine, matches previous setup  (r+brbaker)

http://hg.mozilla.org/tamarin-redux/rev/8ca89d8412b8

Comment 16

[Brent Baker]

Items that still need to be addressed are:

4) Move the linux32 build slave onto new HP server with Ubuntu 10.10 (x64)
5) Move linux-deep build slave to old lin32 phase 1 hardware and retire machine

Comment 17

[Brent Baker]

Attachment: Move the linux32 build slave onto new HP server with Ubuntu 10.10 (x64)

Comment 18

[Tamarin Bot]

changeset: 5747:5aca6297e78e
user:      Brent Baker <[e-mail]>
summary:   Bug 623609: move linux build slave to new hardware (r+cpeyer)

http://hg.mozilla.org/tamarin-redux/rev/5aca6297e78e

Comment 19

[Tamarin Bot]

changeset: 5748:b75eb04b85d0
user:      Brent Baker <[e-mail]>
summary:   Bug 623609: do not set the compiler in the build script and use all available cores to compile (r+cpeyer)

http://hg.mozilla.org/tamarin-redux/rev/b75eb04b85d0

Comment 20

[Brent Baker]

Attachment: Move sparc debug compilation to deep phase

Move the compilation of the solaris debug shell to the deep phase. This binary is not used in the first phase of the build so there is no need to compile the shell. Dropping the compilation shaves 4:30 off of the compile phase of the build. Solaris-sparc compile is the current long pole in compile phase by 11 minutes, nearly 50% of the compile phase time is spent waiting for solaris to complete.

There is also a minor tweak for the solaris builder to use the debug-debugger build in the smoke phase instead of the debug build. This will still trigger any asserts that may happen in the smoke phase.

NOTE: This patch will most likely be reverted once new sparc hardware is acquired that is not 10yrs old.

Comment 21

[Tamarin Bot]

changeset: 5756:8d1cc0d76193
user:      Brent Baker <[e-mail]>
summary:   Bug 623609: follow up fix for moving PPC build slaves to the deep phase of the build. Incorrect slave name was used causing deep build to not be triggered (r+brbaker)

http://hg.mozilla.org/tamarin-redux/rev/8d1cc0d76193

Comment 22

[Tamarin Bot]

changeset: 5757:ae8f3c6b6631
user:      Brent Baker <[e-mail]>
summary:   Bug 623609: Move sparc debug compilation to deep phase (r+dschaffe)

http://hg.mozilla.org/tamarin-redux/rev/ae8f3c6b6631

Comment 23

[Brent Baker]

As of 2011-01-11 all listed items in this bug have been implemented. The following is an overview of the changes and possible next steps. If next steps are approved I will post patches with implemented changes, otherwise this bug will be closed as fixed.

Overview:
Compile phase sped up by roughly 20%, currently at 20 minutes
Test phase sped up by roughly 35%, currently at 62 minutes


Details:

Compile Phase:
- Moved windows mobile compilation to the deep phase of the build
- Move PPC compilation to the deep phase of the build
- Reduced windows build slave compilation time from 22m to 12m by moving to different hardware (the 12 minutes also includes windows64 binary compilation)
- Reduced linux32 compilation from 20m to 6m by moving to new hardware
- Reduced SPARC compilation step from 24m to 20m by moving the compilation of the debug binary to the deep phase (debug binary is only ran in the deep phase since it is so slow)
- Compile time is gated by the SPARC compilation which is at 20m, the next slowest compile slave is at 13m. Really need new/fast SPARC hardware in order to make any more improvement.

Smoke Phase:
- No changes

Test Phase:
- Moved PPC testing to the deep phase of the build
- Moved SPARC debug shell testing to the deep phase (1h 36m -> 53m)
- Windows32 build slave moving to new hardware decreased test time from 53m to 26m
- Linux32 build slave moving to new hardware decreased test time from 58m to 23m
- Test time is currently gated by SPARC testing at 62m, closely followed by:
   linux-arm:  60m
   linux-arm2: 57m
   solaris-sparc: 53m
   android: 52m
   windows64: 40m


Next Steps:
COMPILE: (reduce compile phase to 16m (-4m)) 
A) Move wordcode compilation and testing to the deep phase of the build. This is not a shipping feature. (reduce compilation time by almost 4 minutes to 16 minutes)

TEST: (reduce test phase to 52m (-10m))
A) Reduce amount of SPARC testing done in the first phase of the build by running acceptance for a specified amount of time. Do a COMPLETE acceptance run in the deep phase of the build. (time reduction based on how much time we want this to run)
B) Move linux-arm softfloat testing to the deep phase (drop 29m from one linux-arm build slave)
C) Add an additional beagleboard to the test phase as linux-arm3, reduce current linux-arm testing time of 60 minutes to 32 minutes

Comment 24

[Brent Baker]

Attachment: Move wordcode compilation and testing to the deep phase

Stuffing patch here. No review requested until this has been approved

Comment 25

[Brent Baker]

Attachment: Limit SPARC testing in phase one, but do complete testing in deep

Stuffing patch here. No review requested until this has been approved

Comment 26

[Brent Baker]

Attachment: Move arm softfloat testing to the deep phase of the build

Stuffing patch here. No review requested until this has been approved

Comment 27

[Dan Schaffer]

the panda board reduced current linux-arm test runs from 28m to 7m.  Trevor is trying to get more.  If we can use panda20 instead of adding a 3rd beagleboard it could reduce the test time.

Comment 28

[Brent Baker]

(In reply to comment #27)
> the panda board reduced current linux-arm test runs from 28m to 7m.  Trevor is
> trying to get more.  If we can use panda20 instead of adding a 3rd beagleboard
> it could reduce the test time.

I thought that the panda board was just on loan from the CE team.

Comment 29

[Brent Baker]

Comment on attachment 502835 [details] [diff] [review]
Move wordcode compilation and testing to the deep phase

Requesting review:
I checked with Lars (primary wordcode dev) and he was fine with moving this into the deep phase of the build.

Comment 30

[Tamarin Bot]

changeset: 5781:7aafb0439a74
user:      Brent Baker <[e-mail]>
summary:   Bug 623609: move wordcode compilation and testing into the deep phase of the build (r+cpeyer)

http://hg.mozilla.org/tamarin-redux/rev/7aafb0439a74

Comment 31

[Brent Baker]

(In reply to comment #30)
> changeset: 5781:7aafb0439a74
> user:      Brent Baker <[e-mail]>
> summary:   Bug 623609: move wordcode compilation and testing into the deep
> phase of the build (r+cpeyer)
> 
> http://hg.mozilla.org/tamarin-redux/rev/7aafb0439a74

This reduced the compilation phase from 20 minutes to 16 minutes

Comment 32

[Brent Baker]

Attachment: v2. Limit SPARC testing in phase one, but do complete testing in deep

Rebase patch to apply to current tip

Comment 33

[Brent Baker]

Attachment: Add another beagle board to the first phase of the build

Comment 34

[Tamarin Bot]

changeset: 5782:4f146b39fc38
user:      Brent Baker <[e-mail]>
summary:   Bug 623609: limit SPARC testing in phase one, but do complete testing in deep (r+dschaffe)

http://hg.mozilla.org/tamarin-redux/rev/4f146b39fc38

Comment 35

[Tamarin Bot]

changeset: 5783:6a312cd5e0f9
user:      Brent Baker <[e-mail]>
summary:   Bug 623609: move linux-arm softfloat testing to the deep phase of the build (r+dschaffe)

http://hg.mozilla.org/tamarin-redux/rev/6a312cd5e0f9

Comment 36

[Tamarin Bot]

changeset: 5784:8907566f08e3
user:      Brent Baker <[e-mail]>
summary:   Bug 623609: move linux-arm softfloat testing to the deep phase of the build (r+dschaffe)
Follow up fixes for moving sparc2 build slave to the deep phase of the build

http://hg.mozilla.org/tamarin-redux/rev/8907566f08e3

Comment 37

[Tamarin Bot]

changeset: 5785:6d6ee8c63146
user:      Brent Baker <[e-mail]>
summary:   Bug 623609: Follow up fixes for moving sparc2 build slave to the deep phase of the build

http://hg.mozilla.org/tamarin-redux/rev/6d6ee8c63146

Comment 38

[Brent Baker]

Status updated:

All patches have been pushed and there are no pending changes at this time. The following is a high level review of how the build time has changed:

- Reduced compile phase from 24 minutes to 15 minutes
- Reduced test phase from 96 minutes to 47 minutes
- Overall build time dropped from 2h 3m to 1h 5m, almost cutting build turn around in half.

Comment 39

[Tamarin Bot]

changeset: 5787:28c035dfe59c
user:      Dan Schaffer <[e-mail]>
summary:   bug 623609: fix for linux-deep to verify 64BIT and AMD64 features (r=dschaffe)

http://hg.mozilla.org/tamarin-redux/rev/28c035dfe59c

Comment 40

[Tamarin Bot]

changeset: 5798:5512254f7584
user:      Chris Peyer <[e-mail]>
summary:   Bug 623609 Addendum: name linux realease wordcode correctly (p=brbaker, r=cpeyer)

http://hg.mozilla.org/tamarin-redux/rev/5512254f7584

Comment 41

[Tamarin Bot]

changeset: 5802:c82ece99d8e0
user:      Brent Baker <[e-mail]>
summary:   Bug 623609: reduce the allotted amount of time for running acceptance on SPARC (r+brbaker)

http://hg.mozilla.org/tamarin-redux/rev/c82ece99d8e0

Comment 42

[Tamarin Bot]

changeset: 5814:e270f9eeca30
user:      Brent Baker <[e-mail]>
summary:   Bug 623609: rebalance the SPARC deep testing between the 2 deep build slaves (r+brbaker)

http://hg.mozilla.org/tamarin-redux/rev/e270f9eeca30

Comment 43

[Brent Baker]

(In reply to comment #42)
> changeset: 5814:e270f9eeca30
> user:      Brent Baker <[e-mail]>
> summary:   Bug 623609: rebalance the SPARC deep testing between the 2 deep
> build slaves (r+brbaker)
> 
> http://hg.mozilla.org/tamarin-redux/rev/e270f9eeca30

This changed the deep phase build from 13+ hours to 6 hours.

Comment 44

[Brent Baker]

marking as fixed since all current items have been implemented to speed up the build.

Comment 45

[Tamarin Bot]

changeset: 5830:4f14f30ff90e
user:      Brent Baker <[e-mail]>
summary:   Bug 623609: need to make sure that the solaris deep build slave that runs the selftests on debug builds is also the machine that compiles the binaries (r+brbaker)

http://hg.mozilla.org/tamarin-redux/rev/4f14f30ff90e

Comment 46

[Tamarin Bot]

changeset: 5993:7c122aa3ffed
user:      Chris Peyer <[e-mail]>
summary:   Bug 623609: Move linux compiler args back to environment.sh so that both 32 and 64 bit builds can be run on a single slave (r=brbaker)

http://hg.mozilla.org/tamarin-redux/rev/7c122aa3ffed

Comment 47

[Tamarin Bot]

changeset: 5994:dc144b714e90
user:      Chris Peyer <[e-mail]>
summary:   Bug 623609: Only set CXX & CC vars if they are not defined (r=cpeyer)

http://hg.mozilla.org/tamarin-redux/rev/dc144b714e90