393127 - Evaluate buildbot infrastructure for l10n automated builds

Status: RESOLVED INCOMPLETE

(Release Engineering :: General, defect, P3)

Description

[Axel Hecht [:Pike]]

Attachment: master.cfg, mozl10n.py and mozlib package

We should be able to use buildbot's infrastructure to be more clever about how we do l10n builds, and in particular when. It should also be pretty easy to reuse the build setup for different branches.

Initial patch for my second design is attached. The only real hack is the EnsureDirectory step, which needs slave code. We could fix that on the side of app/locales/Makefile.in, it just needs to add an nsinstall dist-foo/install to the langpacks target. For now, this works, though.

I'll walk rhelmer through the patch in a one-on-one.

One place to put this code could be mozilla/tools/buildbot-configs/l10n, right now they're in my own cvs rep.

Comment 1

[Robert Helmer [:rhelmer]]

Comment on attachment 277637 [details] [diff] [review]
master.cfg, mozl10n.py and mozlib package

mozilla/tools/buildbot-configs/l10n sounds fine. Sorry for taking so long.

This looks fine to me, it helped very much to have you walk through it with me, thanks :)

This way of doing parallelization has a nice result without much code change, but I find it hard to keep my head wrapped around.

Do you have any thoughts on adding this feature in a more general way, that upstream buildbot would take?

If Changes were more generic (for example) would that work for you instead of doing the tree-dependent config variables?

Comment 2

[Axel Hecht [:Pike]]

Attachment: buildbot-config and buildbot-custom for l10n, bb 0.7.6+

This is the state of the art for the l10n buildbot infrastructure on our buildbot 0.7.6+ version. Compared to attachment 277637 [details] [diff] [review], there are a few changes:

All modules and packages are now in buildbot-custom, in a module name buildbotcustom. master.cfg is in mozilla-config/l10n, together with a public_html dir, which as of now doesn't contain modifications, but there should be.

I added a intermediate class to make regular python logging go into twisted logging. This way, I can control the logging from master.cfg, which is cool. Not all parts of the code use this yet.

I did some partial clean up, too, but mostly this is badly shuffling code around to map to the new structure.

buildbotcustom/l10n.py maps to scheduler.py, broadly, thus it's where it is.

For the html output, I'll probably have to subclass WebParts to disable some parts of it, and add other parts that make more sense to us, but I'd like to do that in a second cycle.

For the compare-locales part, this is now in the buildbotcustom module, but it should eventually move over to testing/tests/l10n, I think.

The build side changed a bit, I removed the EnsureDir stuff, and use l10n-checkout again.

I removed my subclass of TinderboxMailNotifier, as bhearsum added WithProperties support for that, so I don't need that no more.

Comment 3

[Axel Hecht [:Pike]]

I rambled a bit about how the things glue together and trees and dispatchers and that on http://wiki.mozilla.org/User:AxelHecht/Build#Continuous_l10n_builds, including some flow diagram. Late, but better late than never. This should map the information that I dumped in rhelmers head at our last face to face.

Comment 4

[Axel Hecht [:Pike]]

Hi, I was wondering if it'd get us ahead faster if we'd push bigger items into follow up patches, I'm thinking about the source class stuff in particular.

Comment 5

[Robert Helmer [:rhelmer]]

(In reply to comment #4)
> Hi, I was wondering if it'd get us ahead faster if we'd push bigger items into
> follow up patches, I'm thinking about the source class stuff in particular.

I've taken a look at this stuff, and it's a lot :) I don't think I can give it a fair review until I have time to really focus on it, which probably isn't going to be until next year.

If you just want to get it landed I am fine with that.. like I said I think you're doing too much weird under-the-cover stuff but if it works for you then that's fine with me.

If you can give me smaller patches to look at that'd make it much easier for me. Were you wanting to get anything into mozilla/tools/buildbot directly or just into your own buildbot-configs area?

Comment 6

[Axel Hecht [:Pike]]

I'm only having changes to buildbot-configs and buildbot-custom.

The trickiest thing right now is the directory structure inside buildbot-custom, I guess. Mostly because the top dir was supposed to be buildbot-custom, and '-' is not allowed in package names. And that I use setup.py in there.

Here's the list of files I'd add under mozilla/tools:

buildbot-configs/l10n/master.cfg
buildbot-configs/l10n/public_html/buildbot.css
buildbot-configs/l10n/public_html/index.html
buildbot-configs/l10n/public_html/robots.txt
buildbot-custom/setup.py
buildbot-custom/buildbotcustom/__init__.py
buildbot-custom/buildbotcustom/log.py
buildbot-custom/buildbotcustom/slave/comparestep.py
buildbot-custom/buildbotcustom/slave/__init__.py
buildbot-custom/buildbotcustom/status/__init__.py
buildbot-custom/buildbotcustom/status/create.py
buildbot-custom/buildbotcustom/status/enum.py
buildbot-custom/buildbotcustom/steps/comparestep.py
buildbot-custom/buildbotcustom/steps/shell.py
buildbot-custom/buildbotcustom/steps/__init__.py
buildbot-custom/buildbotcustom/l10n.py

Comment 7

[[e-mail] (:bhearsum)]

(In reply to comment #6)
> I'm only having changes to buildbot-configs and buildbot-custom.
> 
> The trickiest thing right now is the directory structure inside
> buildbot-custom, I guess. Mostly because the top dir was supposed to be
> buildbot-custom, and '-' is not allowed in package names. And that I use
> setup.py in there.
> 
> Here's the list of files I'd add under mozilla/tools:
> 
> buildbot-configs/l10n/master.cfg
> buildbot-configs/l10n/public_html/buildbot.css
> buildbot-configs/l10n/public_html/index.html
> buildbot-configs/l10n/public_html/robots.txt
> buildbot-custom/setup.py
> buildbot-custom/buildbotcustom/__init__.py
> buildbot-custom/buildbotcustom/log.py
> buildbot-custom/buildbotcustom/slave/comparestep.py
> buildbot-custom/buildbotcustom/slave/__init__.py
> buildbot-custom/buildbotcustom/status/__init__.py
> buildbot-custom/buildbotcustom/status/create.py
> buildbot-custom/buildbotcustom/status/enum.py
> buildbot-custom/buildbotcustom/steps/comparestep.py
> buildbot-custom/buildbotcustom/steps/shell.py
> buildbot-custom/buildbotcustom/steps/__init__.py
> buildbot-custom/buildbotcustom/l10n.py
> 

That naming is my mistake, I'll correct it.

Comment 8

[[e-mail] (:bhearsum)]

Nevermind my last comment!

Comment 9

[Robert Helmer [:rhelmer]]

Comment on attachment 288769 [details]
buildbot-config and buildbot-custom for l10n, bb 0.7.6+

Is this landed? Anything else to do here?

For the record, I think this is all great and totally fine for incubator, I don't think it's what we want for the current depend/nightly boxes, but that's ok as they serve different purposes.

I'd love to have the same system able to take on everything, but I think it's serving too many masters (QA, depend/nightly, releases, l10n, etc).

Comment 10

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Axel - can we close this or is there something specific to do here?

Comment 11

[Axel Hecht [:Pike]]

Resolving this bug INCOMPLETE.

Basically, it WORKSFORME, but really only for me.

The fundamental bug in this is that I can't do reconfig, that is, any configuration change needs a restart, and thus an empty build queue. Or, to put the story the other way around, the build queue is in the scheduler, not the builder, and the scheduler doesn't maintain state across reconfigs, just the builder would.

The only way I see to fix this is to use custom build properties, a patch that's in a not-so-good state, see http://buildbot.net/trac/ticket/87 and http://thread.gmane.org/gmane.comp.python.buildbot.devel/2822/focus=2825