486567 - unify the try server code with the existing BuildFactory's

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[[e-mail] (:bhearsum)]

Almost every single time a feature or bugfix is deployed to moz2-master the try server does not get it for some time. Sometimes days, weeks, or even months later. Right now, a prime example of this is crashstacks.

These days, it should be relatively easy to unify the try code with the existing factories we have. Try will probably need a bit of subclassing to deal with properly displaying information and working with patches via the web interface, but on the whole, it should be mostly the same.

Comment 1

[[e-mail] (:bhearsum)]

Getting started on this today.

Comment 2

[[e-mail] (:bhearsum)]

Attachment: TryBuildFactory (for hg only)

This patch adds a TryBuildFactory. Mostly it's a dump of logic from tryserver/master.cfg to the factory, but wherever possible we share the MercurialBuildFactory code. To accommodate that, I've had to do a few things:
* Rename MozillaBuildFactory.addPreBuildSteps to addInitialSteps
* Move self.configRepo and self.mozconfig assignment to MercurialBuildFactory.addConfigSteps
* Add a 'pkgArgs' to MercurialBuildFactory.addUploadSteps to allow for PKG_BASENAME overriding

I think tryserverfactory.py is pretty straightforward, it's the bare minimum we need to override for try builds (which is a lot, sadly). Most of the overriding is necessary to support web submitted builds - CustomPatch, PatchDownload, etc. I haven't added sendchange support just yet, I'll do that before this goes live in production, though.

The last part of this patch is just the tryserver/steps.py -> steps/tryserver.py change. It's a straight copy, no changes made.

buildbot-configs patch will be posted soon.

Comment 3

[[e-mail] (:bhearsum)]

Attachment: master side patch

Alright, this diff is huge, but there's not a whole lot going on here. Here's a summary:
* Move variables to config.py
* Move CVS/unittest steps to factories.py
* Rename builders to be consistent with other Buildbots
* Switch regular hg builders to TryBuildFactory
* Move mac mozconfigs to mozconfig-macosx*

That's it!

Note that the unittest builds aren't using a buildbotcustom factory since we're going to switch to packaged unittest runs soon enough - it didn't seem worth the effort to do it.

Comment 4

[Chris AtLee [:catlee]]

Comment on attachment 376744 [details] [diff] [review]
master side patch

It would be nice to get rid of the repeated calls to TryBuildFactory eventually, and have the per-platform factories constructed in a loop, similar to how our build/unittest master works.

Comment 5

[[e-mail] (:bhearsum)]

Attachment: buildbotcustom patch, tryserver tryserver/* removed

Only change in this patch is the removal of the win32 tryserver environment - which has been moved to the buildbot-configs repo. Updated master-side patch is incoming. (Carrying review forward as per IRC.)

Comment 6

[[e-mail] (:bhearsum)]

Attachment: updated master side patch

Here's a summary of the changes:
* Added config.WIN32_ENVIRONMENT
* Rename mac mozconfig to macosx, for consistency
* Pass the correct 'objdir' for Mac hg builds

That's it!

I still need to implement sendchange, but I'm going to do that in a patch on top of these ones.

Comment 7

[[e-mail] (:bhearsum)]

Comment on attachment 376913 [details] [diff] [review]
buildbotcustom patch, tryserver tryserver/* removed

changeset:   279:c245fff1048b

Holding off on the master side patch until the sendchange part is done.

Comment 8

[[e-mail] (:bhearsum)]

Comment on attachment 376913 [details] [diff] [review]
buildbotcustom patch, tryserver tryserver/* removed

I had to back this out because it broke the moz2-master config.

Comment 9

[[e-mail] (:bhearsum)]

Attachment: followup buildbotcustom patch with fixes, sendchange

This patch is based on the previous one and has a few things in it:
* Bugfixes to make dep/nightly/leaktest work again (self.mozconfig/config* stuff)
* Support sendchange to talos, unittest

Because of the way the configs are being done now I had to use the self.super_class technique to make sure reconfigs don't hose anything.

Comment 10

[[e-mail] (:bhearsum)]

Attachment: master side followup

Same deal here - this is based on the previous patch. Here's what's happening:
* Finish renaming 'mac' to 'macosx'
* s/TALOS_TRY_MASTER/TALOS_TRY_MASTERS/
* Change PACKAGE_URL to accommodate the new factory

Comment 11

[[e-mail] (:bhearsum)]

Attachment: same as before, with better upload directories

My previous patch ended up with crappy upload directories: they didn't have the username in them and they were per platform. This patch fixes that on the buildbotcustom side.

Comment 12

[[e-mail] (:bhearsum)]

Attachment: master side followup

Same thing on the master side - the metadiff should make it pretty clear.

Comment 13

[[e-mail] (:bhearsum)]

Comment on attachment 377530 [details] [diff] [review]
master side followup

changeset:   1146:6a021eec4d50

Comment 14

[[e-mail] (:bhearsum)]

Comment on attachment 377528 [details] [diff] [review]
same as before, with better upload directories

changeset:   285:ea5c9d8f38e4

Comment 15

[[e-mail] (:bhearsum)]

Comment on attachment 376914 [details] [diff] [review]
updated master side patch

changeset:   1145:c5d4d2a880a3

Comment 16

[[e-mail] (:bhearsum)]

Comment on attachment 376913 [details] [diff] [review]
buildbotcustom patch, tryserver tryserver/* removed

changeset:   284:f96c012a30be

Comment 17

[[e-mail] (:bhearsum)]

This bug is FIXED. I've filed a couple of followups for additional cleanup: bug 493227 - separate BuildFactorys into different files and bug 493228 - run packaged unittest builds on the try server.