438315 - disambiguate buildbot changes from multiple hg repositories

Status: RESOLVED DUPLICATE of bug 434289

(Release Engineering :: General, defect, P3)

Description

[Axel Hecht [:Pike]]

When processing changes in buildbot schedulers, we need to be able to tell different hg repositories apart.

This comes up whenever you want automated builds for something non-mozilla-central, I guess.

l10n is one example, too. As we dropped the leading 'mozilla' in the filenames in mozilla-central, too, looking at the files doesn't seem to suffice.

I haven't come up with anything that's not a deep-core buildbot patch like partitioning the changesource/scheduler universe, or adding the repository url to the changes.

Using multiple masters for partitioning doesn't seem to be a working option, as at least for l10n we'll need controlled leaks between the partitions to build l10n on en-US check-ins. Doesn't seem to scale very well either, and is probably tough to maintain.

Comment 1

[Benjamin Smedberg]

Axel, this bug sounds like gibberish to me... what are you trying to accomplish, in plain English?

Comment 2

[Axel Hecht [:Pike]]

Make the scheduler know which change belongs to which hg repo.

Comment 3

[[e-mail] (:bhearsum)]

You have to specify branch, no? ChangeSource and Scheduler are related in that way. I don't understand the problem here either.

Comment 4

[Axel Hecht [:Pike]]

Hrm. http://mxr.mozilla.org/mozilla/source/tools/buildbotcustom/changes/hgpoller.py hard-codes a branch on there, I don't know enough about hg branches to comment on whether that's a bug or a feature.

Comment 5

[[e-mail] (:bhearsum)]

Are you talking about in-repo branches?

Comment 6

[Axel Hecht [:Pike]]

yep.

Comment 7

[Benjamin Smedberg]

It sounds like there are at least two different questions here:

Q: How do I track multiple repositories with a single scheduler?
A: Why do you need to? Can't you use a different scheduler for each locale?

Q: How do I track multiple branches in a single repository?
A: We currently don't use multiple branches in a single repo, and I don't think we want to.

Did I answer the right questions?

Comment 8

[Axel Hecht [:Pike]]

The second one, yes. At one point, I'd like to understand how we're going to port relbranches and stuff like that to the new world, but that's probably a better topic for a public post in the newsgroup.

For the first one, schedulers are, like builders and changesources, assumed to be static at buildmaster runtime. Doing a reconfig of the master each time we add a locale isn't something that joduinn wants to do. 

For reference: Schedulers are not tied to change sources. Each change source sends all its changes to the buildmaster singleton, which then forwards each change to all schedulers. Branches are merely one feature of those changes that schedulers can use to drop changes or not.

So the question isn't really if I'm using one scheduler or many, I need to tell them apart within one master.

If we continue to code up the branch in the poller on the buildbot side, that'd be a strong argument for having only one l10n repo, I wouldn't know how to mangle the data back and forth for a single poller for l10n. Again, adding changesources to the master dynamically for new locales sounds bad and risky, and bad-perf for both the master and the hg server. That'd put our faith on the SoC project for partial clones, mostly.

Comment 9

[(not currently active) Ted Mielczarek]

Couldn't you just throw together an RSS aggregator to aggregate all the l10n pushlog feeds, and then point buildbot at that?

Comment 10

[Axel Hecht [:Pike]]

That wouldn't really solve the load and configuration problem, you'd still hit the individual feeds for each cycle, and you had to separate out the repo information out again, possibly faking branches from a url like 
http://hg.mozilla.org/l10n-central/de/index.cgi/rev/377f2910d180a0fefb30af1c1ad6809b3315d3db. And you had to reconfigure the aggregator to poll additional repos on adding new locales.

It doesn't (at least ad-hoc) allow for things like setting start and end dates or cut-off revisions, as I mumbled about in bug 437429 comment 2.

Comment 11

[[e-mail] (:bhearsum)]

Axel, are you working on this?

Comment 12

[Axel Hecht [:Pike]]

I'm not working on this.

Comment 13

[Axel Hecht [:Pike]]

Fixing deps. I'm resolving this as DUPE of bug 434289, as we're mugging around with the changes.Change objects to carry the information of the repo forward that we need in the schedulers.

The aggregator story is now in bug 443600.

Comment 15

[Chris Cooper [:coop] (he/him)]

Moving closed Future bugs into Release Engineering in preparation for removing the Future component.