Create package structure for custom buildbot steps

RESOLVED FIXED

Status

P3
normal
RESOLVED FIXED
11 years ago
5 years ago

People

(Reporter: rcampbell, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(2 attachments, 2 obsolete attachments)

(Reporter)

Description

11 years ago
Currently we have a number of buildbot masters running with custom versions of mozbuild.py or completely different code in the case of the performance machines. As we grow more of these special purpose machines, it would make reuse much easier to setup a package structure for containing this different files.

I propose we use mozilla/tools/buildbot-steps as the top-level directory for housing these, with subdirectories for testing, build and l10n, for starters.
(Reporter)

Updated

11 years ago
OS: Mac OS X → All
Hardware: Macintosh → All

Comment 1

11 years ago
l10n and testing are going to intersect for me. I actually intend to have the buildbot logic for testing/tests/l10n inside testing/tests/l10n, too, that sounded like the logical place for me to put it.

The stuff I would like to land is in bug 393127, and consists of steps, schedulers, and build subclasses. I used to have my buildbot code locally in tools/buildbot/mozilla, that is, in parallel to the buildbot package. A separate dir might be easier, not sure.

One thing I'd love would be a more agile review policy for code not in production yet. I'm heavily coding on my buildbot steps, but in my own private hidden rep where nobody can see it, just because it's been not review-worthy so far. Still, bhearsum could have grabbed parts of it almost two weeks ago. We could just add an experimental directory for those, too.
(Reporter)

Comment 2

11 years ago
(In reply to comment #1)
> l10n and testing are going to intersect for me. I actually intend to have the
> buildbot logic for testing/tests/l10n inside testing/tests/l10n, too, that
> sounded like the logical place for me to put it.

Right, you've still got your comparelocales.py stuff over there and it should probably stay there for now.

> The stuff I would like to land is in bug 393127, and consists of steps,
> schedulers, and build subclasses. I used to have my buildbot code locally in
> tools/buildbot/mozilla, that is, in parallel to the buildbot package. A
> separate dir might be easier, not sure.

I'm thinking we'll want to keep our buildbot steps separated from the buildbot source. An adjacent directory makes sense to me there in case we ever decide to blow away the current buildbot source with a new copy from buildbot trunk.

I think wherever we end up putting our buildbot build steps, the stuff in 393127 would fit nicely in there, likely under an l10n subdir or factored into separate packages if there's other stuff we can repurpose elsewhere.

> One thing I'd love would be a more agile review policy for code not in
> production yet. I'm heavily coding on my buildbot steps, but in my own private
> hidden rep where nobody can see it, just because it's been not review-worthy so
> far. Still, bhearsum could have grabbed parts of it almost two weeks ago. We
> could just add an experimental directory for those, too.

Yeah, we'll need to figure out some other mechanism for dealing with that. A separate Hg repo or something might work better for that style of development. I know I've been bitten (and caused biting to occur) by developing locally and then moving files over to the server, only to end up tweaking those files on the production machine and seeing them drift from checkout. I'm hoping the current snapshot in CVS matches reality now.
I'd like to get this ball rolling. It sounds like we need more than just an area for Buildbot steps. For now, how about we create "mozilla/tools/buildbot-custom". We could or could not create subdirectories there, I'm not picky. I think it's important to start getting these things into CVS though. I'm doing improvements on a lot of the steps right now (particularly Buildbot 0.7.6 support), and this would make it much easier for everyone else to get them.
Another thought that comes to mind is to copy the Buildbot package structure. mozilla/tools/buildbot-custom/steps/*.py

I also think that we should stay away from tying things to l10n/testing/build/etc. when possible. Hopefully, a lot of our work can crossover. The unit test running steps, for example. We definitely want to avoid having two steps/schedulers/whatever that do the same thing.
(Reporter)

Comment 5

11 years ago
I agree with Ben about not tying things to specific purposes. A lot of our customizations are useful across the board (TinderboxPoller for example).

I'm ok with mozilla/tools/buildbot-custom/steps, etc. We should probably mirror the buildbot directory structure so it'll be easier to send changes upstream when appropriate.
Rob/Axel, any thoughts?

Comment 7

11 years ago
Actually, I'm torn. I used to put the buildbot steps for compare-locales into my copies of testing/tests/l10n, so that I can update them when I change compare-locales. On other trees, I have put them in buildbot-custom, so that I can change them when I change my buildbot stuff. Right now, I'm hoping that they should eventually move back to testing/tests/l10n, as there's work to do over there as well, and my buildbot hackery might become more stable.

For the actual l10n buildbot infrastructure, that shouldn't be in testing/tests/l10n for sure, yeah. 

Apart from that I'm all in favor of mimicking buildbot's directory structure, I used to do that locally, too.
OK. So unless someone voices an objection I'm going to create mozilla/tools/buildbot-custom and a 'steps' subdirectory. We may want to create 'changes' and 'status' subdirectories at some point. I think all of our status and change plugins have gotten upstream though, so I won't create them right now. (We should only use buildbot-custom for non-upstream changes. Things that will get accepted upstream can just go right into mozilla/tools/buildbot.)

Comment 9

11 years ago
I've been chatting with Ben on #build on how closely to match the buildbot dir structure.

Should we do a buildbot-custom/steps and copy buildbot-custom over to the master dir (and slaves, in my case), or should we do a buildbot-custom/buildbost-custom/steps with a buildbot-custom/setup.py to install our modules?

Note, if we do the latter, there'd be obvious places to put docs and/or scripts, too.

Comment 10

11 years ago
Moving this over to mozilla.org, Build & Release, where all the other buildbot bugs are.
Component: Testing → Build & Release
Product: Core → mozilla.org
QA Contact: testing → build
Version: unspecified → other

Comment 11

11 years ago
Note, buildbot-custom as a package name is verboten. I guess that's http://docs.python.org/ref/identifiers.html, but I'm not sure.
You are right, Axel. OK, how about buildbotcustom, then. Can someone (rob?) give their a+ to that?
Created attachment 293723 [details] [diff] [review]
a possible solution

I spent some time today working up an example of how to do this. A patch is attached, but here's how it works:
1) Checkout a buildbot-configs directory. For example, 'cvs co -d tryserver mozilla/tools/buildbot-configs/tryserver'
2) Checkout the buildbotcustom directory inside of that directory. For example, 'cvs co -d buildbotcustom mozilla/tools/buildbotcustom'.
3) Start buildbot.

Because Twisted chdir()'s to the master directory before Buildbot loads the config file there isn't even a need to set PYTHONPATH.

In this patch, tryserver's master.cfg has been adjusted to import from buildbotcustom for all the custom steps it uses. The thing I don't like about this is needing to import/reload a bunch of modules. reload() doesn't work recursively so I think that mass of import/reload statements at the top is necessary to make 'buildbot reconfig' work.

I've mirrored the Buildbot directory structure. I think this type of organization makes as much sense as anything.

Axel, I just noticed that you're doing things differently over in bug 393127, particularly installing buildbotcustom as a package. I hadn't really thought about doing it that way...that could be useful on machines with many masters, though. Personally, I like just being able to check out and use it, I'm not sure if that's the best solution overall though.

Rob/Rob/Axel/anyone else interested, I'd appreciate comments/questions/thoughts.
Priority: -- → P3
Attachment #293723 - Attachment is patch: true
Attachment #293723 - Attachment mime type: application/octet-stream → text/plain
Created attachment 294673 [details] [diff] [review]
minor fix in master.cfg
Attachment #293723 - Attachment is obsolete: true
(Reporter)

Comment 15

11 years ago
I'm ok with buildbotcustom. Consider this my a+!

@bhearsum: Your approach for the tryserver sounds good at first, post-holiday skimming.

Trivia: How many new machines have we brought up that we'll have to migrate to a new step structure since this bug was created?

(no, I don't know the answer) :)
(In reply to comment #15)
> I'm ok with buildbotcustom. Consider this my a+!
> 
> @bhearsum: Your approach for the tryserver sounds good at first, post-holiday
> skimming.
> 

Cool. I'm going to give it a try on the production try server, then.

> Trivia: How many new machines have we brought up that we'll have to migrate to
> a new step structure since this bug was created?
> 
> (no, I don't know the answer) :)
> 

Lots :)

One thing that I noticed is that this structure doesn't work so well for things that don't get imported into master.cfg. Ex, a new SlaveCommand. For these I think we need to set PYTHONPATH=".:$PYTHONPATH"; that's something I need to test out, though.
Attachment #294673 - Flags: review?(rcampbell)
(Reporter)

Comment 17

11 years ago
Comment on attachment 294673 [details] [diff] [review]
minor fix in master.cfg

great start on this, ben.

A couple of suggestions:

1) split out env.py into task-specific subdirs, i.e., buildbotcustom.tryserver - use env.py and import the buildbotcustom.env module from there to include base profiles, creating task-specific versions from full copies of the base and adding updates to the new version.

see http://pastebin.mozilla.org/285457 for a dirty example.

2) buildbotcustom/steps/misc.py should be moved under a project-specific directory, e.g., buildbotcustom/tryserver and given a more descriptive name, i.e., stepsupport.py or just steps.py.
Attachment #294673 - Flags: review?(rcampbell) → review-
Created attachment 295121 [details] [diff] [review]
[checked in] try #2

Rob, this patch addresses your concerns about the last one. I've moved _all_ of the steps into buildbotcustom/tryserver/steps.py since they currently only work on the Try Server (I've filed bug 410510 about this though, since some of them should be more generic).
Assignee: nobody → bhearsum
Attachment #294673 - Attachment is obsolete: true
Status: NEW → ASSIGNED
Attachment #295121 - Flags: review?(rcampbell)
(Reporter)

Comment 19

11 years ago
Comment on attachment 295121 [details] [diff] [review]
[checked in] try #2

not a whole lot of reuse here, but I think MozillaClientMk could likely be subclassed from a reusable version of that. Also, it looks like you're just using the base env here, which is fine if you don't need to make any changes to it. This looks good!
(Reporter)

Comment 20

11 years ago
Comment on attachment 295121 [details] [diff] [review]
[checked in] try #2

(and the stamp)
Attachment #295121 - Flags: review?(rcampbell) → review+
Comment on attachment 295121 [details] [diff] [review]
[checked in] try #2

Checking in buildbot-configs/tryserver/master.cfg;
/cvsroot/mozilla/tools/buildbot-configs/tryserver/master.cfg,v  <--  master.cfg
new revision: 1.4; previous revision: 1.3
done
Removing buildbot-configs/tryserver/mozbuild.py;
/cvsroot/mozilla/tools/buildbot-configs/tryserver/mozbuild.py,v  <--  mozbuild.py
new revision: delete; previous revision: 1.3
done
RCS file: /cvsroot/mozilla/tools/buildbotcustom/__init__.py,v
done
Checking in buildbotcustom/__init__.py;
/cvsroot/mozilla/tools/buildbotcustom/__init__.py,v  <--  __init__.py
initial revision: 1.1
done
RCS file: /cvsroot/mozilla/tools/buildbotcustom/env.py,v
done
Checking in buildbotcustom/env.py;
/cvsroot/mozilla/tools/buildbotcustom/env.py,v  <--  env.py
initial revision: 1.1
done
RCS file: /cvsroot/mozilla/tools/buildbotcustom/tryserver/__init__.py,v
done
Checking in buildbotcustom/tryserver/__init__.py;
/cvsroot/mozilla/tools/buildbotcustom/tryserver/__init__.py,v  <--  __init__.py
initial revision: 1.1
done
RCS file: /cvsroot/mozilla/tools/buildbotcustom/tryserver/steps.py,v
done
Checking in buildbotcustom/tryserver/steps.py;
/cvsroot/mozilla/tools/buildbotcustom/tryserver/steps.py,v  <--  steps.py
initial revision: 1.1
done
Attachment #295121 - Attachment description: try #2 → [checked in] try #2
Priority: P3 → P2
Created attachment 295268 [details] [diff] [review]
[checked in] add missing imports, actually start UploadTryBuild

This is just a supplemental patch to fix a couple copy/paste errors.
Attachment #295268 - Flags: review?(rcampbell)
(Reporter)

Updated

11 years ago
Attachment #295268 - Flags: review?(rcampbell) → review+
Comment on attachment 295268 [details] [diff] [review]
[checked in] add missing imports, actually start UploadTryBuild

Checking in tools/buildbotcustom/tryserver/steps.py;
/cvsroot/mozilla/tools/buildbotcustom/tryserver/steps.py,v  <--  steps.py
new revision: 1.2; previous revision: 1.1
done
Attachment #295268 - Attachment description: add missing imports, actually start UploadTryBuild → [checked in] add missing imports, actually start UploadTryBuild
Priority: P2 → P3
I'm going to be caught up in other things for the next couple of weeks, tossing these back into the pool.
Assignee: bhearsum → nobody
Status: ASSIGNED → NEW
Status: NEW → RESOLVED
Last Resolved: 11 years ago
Resolution: --- → FIXED
(Assignee)

Updated

5 years ago
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.