429670 - mozilla2 builds need a BuildFactory

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[[e-mail] (:bhearsum)]

With all the recent activity on the Mozilla2 Buildbot (symbols, updates, debug builds, etc) a lot of conditional logic has crept into the config. It's getting a little confusing to read, IMHO. I think it'd be great if we had a factory that we could call like this:
linuxBuilder = Mozilla2Factory(
  uploadBuild=True,
  uploadMar=True,
  createSnippet=True,
  uploadSymbols=True,
  mozconfig=path/to/some/file,
)

There would probably need to be more arguments, but that's basically it.

What do you guys think?

Comment 1

[[e-mail] (:bhearsum)]

Been doing some more thinking about this. It might be nice to break this down further to generalize it some more. I'm thinking something like...:

moz2LinuxBuilder = Mozilla2BuildFactory(...)
moz2LinuxBuilder.steps.append(UploadFactory(...))

I'm looking at creating the following factories:
Mozilla2BuildFactory
UploadFactory
UpdateFactory
SymbolFactory


I'm not entirely sure what this would do to the moz2 config - it could make it unprogrammatic, but maybe not. I think either way would work...

CC'ing ted since he was chatting with us on IRC ;)

Comment 2

[[e-mail] (:bhearsum)]

Rather:
moz2LinuxBuilder = Mozilla2BuildFactory(...)
moz2LinuxBuilder.steps.append(UploadFactory(...).steps)

Comment 3

[[e-mail] (:bhearsum)]

Attachment: [untested] moz2, upload, update, and symbol build factories.

I'll give these a good 'ol test next week, after the production master gets moved to production-master and we have a better staging environment.

Comment 4

[[e-mail] (:bhearsum)]

Attachment: [wip] more build factories, more tested

Comment 5

[[e-mail] (:bhearsum)]

Attachment: [wip] moz2 master.cfg using build factories

still needs more testing

Comment 6

[[e-mail] (:bhearsum)]

Attachment: build factories suitable for use by mozilla2 buildbot

These are updated build factories for building, creating updates, uploading builds, symbol generation/upload, and basic tests (codesighs/leaktest).

The idea here is that all of the platform specific logic is factored out to these factories and we're left with something much simpler/shorter in the master.cfg. When the mozilla2-staging config is using these factories it is roughly half the length. Other benefits:
* Allows others to more easily replicate what we're doing
* UploadFactory, UpdateFactory, SymbolFactory, and BuildTestFactory are all useful for any project/build with the same structure. e.g., Seamonkey or Thunderbird could easily benefit from these.
* buildbotcustom is a much more sensible place to store our build process, IMHO.

I'm interested to hear others' thoughts.

Comment 7

[[e-mail] (:bhearsum)]

I'm not going to get to this anytime soon.

Comment 8

[[e-mail] (:bhearsum)]

Attachment: BuildFactory for dep/nightly builds

This patch simply adds all the steps we already do on dep/nightly builds to a big BuildFactory. Ideally, I would've wanted to split the different tasks into separate factories, but there isn't an easy way to combine multiple factories into one Builder.

Comment 9

[[e-mail] (:bhearsum)]

Attachment: [checked in] remove unnecessary crud from the patch

(CVS sucks for developing multiple patches at once)

Comment 10

[[e-mail] (:bhearsum)]

Attachment: [checked in] get the dep/nightly buildbot using the new factory

I've tested this out for quite a bit on the staging environment and AFAICT there is no behaviour change.

Comment 11

[Chris Cooper [:coop] (he/him)]

Comment on attachment 332927 [details] [diff] [review]
[checked in] get the dep/nightly buildbot using the new factory

We seriously need 'fake' update_platforms in there? Ugh.

Comment 12

[[e-mail] (:bhearsum)]

Comment on attachment 332926 [details] [diff] [review]
[checked in] remove unnecessary crud from the patch

Checking in factory.py;
/cvsroot/mozilla/tools/buildbotcustom/process/factory.py,v  <--  factory.py
new revision: 1.5; previous revision: 1.4
done

Comment 13

[[e-mail] (:bhearsum)]

Comment on attachment 332927 [details] [diff] [review]
[checked in] get the dep/nightly buildbot using the new factory

changeset:   225:76dd6b5bbcf8

Comment 14

[[e-mail] (:bhearsum)]

I add to add hgToolsRepo to the staging config to do a test-run:
diff -u -r1.29 fx-moz18-staging-bootstrap.cfg
--- fx-moz18-staging-bootstrap.cfg	25 Jul 2008 13:31:07 -0000	1.29
+++ fx-moz18-staging-bootstrap.cfg	12 Aug 2008 16:20:10 -0000
@@ -80,3 +80,4 @@
 symbolServerKey  = /home/cltbld/.ssh/id_rsa
 useTarGz        = 1
 useBetaChannel  = 1
+hgToolsRepo     = http://hg.mozilla.org/build/tools
ca-245:configs bhearsum$ cvs commit -m "bug 449208: add hgToolsRepo to staging 1.8 bootstrap.cfg so we can do a test run of the new Bump.pm code."
cvs commit: Examining .
Checking in fx-moz18-staging-bootstrap.cfg;
/cvsroot/mozilla/tools/release/configs/fx-moz18-staging-bootstrap.cfg,v  <--  fx-moz18-staging-bootstrap.cfg
new revision: 1.30; previous revision: 1.29
done

Comment 15

[[e-mail] (:bhearsum)]

Whoops, wrong bug on that last comment.

Comment 16

[[e-mail] (:bhearsum)]

Things here were mostly OK - I had to check-in a bustage fix to disable codesighs on win32. Other than that, we look good!

Comment 17

[[e-mail] (:bhearsum)]

Attachment: [checked in] make sure upload symbols can be disabled on certain trees/builders

Comment 18

[[e-mail] (:bhearsum)]

Attachment: disable uploadSymbols for actionmonkey and tracemonkey

Comment 19

[[e-mail] (:bhearsum)]

Attachment: [checked in] again, fixing linux codesighs, too

Ugh, I can't believe how many things this bug regressed :(.

Comment 20

[[e-mail] (:bhearsum)]

Comment on attachment 333543 [details] [diff] [review]
[checked in] make sure upload symbols can be disabled on certain trees/builders

Checking in factory.py;
/cvsroot/mozilla/tools/buildbotcustom/process/factory.py,v  <--  factory.py
new revision: 1.8; previous revision: 1.7
done

Comment 21

[[e-mail] (:bhearsum)]

Comment on attachment 333608 [details] [diff] [review]
[checked in] again, fixing linux codesighs, too

changeset:   230:b43bde1fd0b5

Comment 22

[[e-mail] (:bhearsum)]

I think this one is finally done.