539588 - Tracking bug for getting schedulerdb working

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P5)

Description

[Chris AtLee [:catlee]]

Objectives:
* Get buildbot with schedulerdb working on all our masters
* Ability to have a master crash/go down, and not have the list of pending jobs lost
* Ability to have multiple masters handle changes for the same branch without duplicating effort

Comment 1

[Chris AtLee [:catlee]]

Attachment: buildbot-configs for staging scheduler/builder masters

This is the beginning of the new single-directory layout.

Several symlinks need to be set up, depending on the master being created.

builder/scheduler_master.cfg -> master.cfg
staging/production_config.py -> localconfig.py
staging_builder_master_localconfig.py / staging_scheduler_mater_localconfig.py -> master_localconfig.py

Comment 2

[Chris AtLee [:catlee]]

Comment on attachment 439599 [details] [diff] [review]
buildbot-configs for staging scheduler/builder masters

changeset:   2327:77d10fffd0ae
branch:      buildbot-0.8.0

Comment 3

[Chris AtLee [:catlee]]

Attachment: Production configs for scheduler/builder master

Comment 4

[Chris AtLee [:catlee]]

Comment on attachment 442684 [details] [diff] [review]
Production configs for scheduler/builder master

changeset:   2341:c4649fd8a33d

Comment 5

[Chris AtLee [:catlee]]

Attachment: Sync over config changes to mozilla/

Comment 6

[[e-mail] (:bhearsum)]

Comment on attachment 444981 [details] [diff] [review]
Sync over config changes to mozilla/

You should probably remove the talos-master.m.o:9012 from unittest_masters, as Nick suggested I do in that bug.

Comment 7

[Chris AtLee [:catlee]]

Comment on attachment 444981 [details] [diff] [review]
Sync over config changes to mozilla/

changeset:   2444:842e9f53c3e8

Comment 8

[Chris AtLee [:catlee]]

Comment on attachment 444981 [details] [diff] [review]
Sync over config changes to mozilla/

whoops.  this was checked in, but not as that changeset

Comment 9

[Chris AtLee [:catlee]]

Attachment: Enable mozilla-1.9.1, 1.9.2 on schedulerdb

Comment 10

[Chris AtLee [:catlee]]

Attachment: Enable trunk on schedulerdb

Comment 11

[Chris AtLee [:catlee]]

Attachment: Disable mozilla-central on production-master

Comment 12

[Chris AtLee [:catlee]]

Attachment: Disable mozilla-1.9.2, 1.9.1 on pm,pm02

Comment 13

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 449777 [details] [diff] [review]
Enable mozilla-1.9.1, 1.9.2 on schedulerdb

OK, but lets watch out for 75 l10n builds grabbing all the slaves and overloading hg.m.o each time a nightly finishes.

Comment 14

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 449780 [details] [diff] [review]
Disable mozilla-1.9.2, 1.9.1 on pm,pm02

r+ if you remove SchedulerL10n which will have no builders to drive after this.

Comment 15

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 449779 [details] [diff] [review]
Disable mozilla-central on production-master

r+ with the l10n scheduler removed.

Comment 16

[Chris AtLee [:catlee]]

Attachment: Limit the number of l10n jobs we do to 20 per branch per master

Comment 17

[Chris AtLee [:catlee]]

Attachment: Add properties for our branch and platform to builders

Comment 18

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 449932 [details] [diff] [review]
Limit the number of l10n jobs we do to 20 per branch per master

>diff --git a/misc.py b/misc.py
>+    # Limit us to doing 20 l10n jobs per branch per master
>+    l10nLock = locks.MasterLock("%s-l10n" % name, maxCount=20)

The code changes make sense, but we could get 40 total jobs per branch once we migrate m-1.9.1/m-1.9.2/m-c to buildbot 0.8.0 ? (assuming two masters is enough). There seems to be overlap between branches in the current up-to-8-slaves setup, given nightlies fire off at 30 minute intervals, so that could go even higher. And that seems fairly hard on hg.m.o.

Are you expressing a preference for lowering the maxCount if we hit issues ?

Comment 19

[Chris AtLee [:catlee]]

(In reply to comment #18)
> (From update of attachment 449932 [details] [diff] [review])
> >diff --git a/misc.py b/misc.py
> >+    # Limit us to doing 20 l10n jobs per branch per master
> >+    l10nLock = locks.MasterLock("%s-l10n" % name, maxCount=20)
> 
> The code changes make sense, but we could get 40 total jobs per branch once we
> migrate m-1.9.1/m-1.9.2/m-c to buildbot 0.8.0 ? (assuming two masters is
> enough). There seems to be overlap between branches in the current
> up-to-8-slaves setup, given nightlies fire off at 30 minute intervals, so that
> could go even higher. And that seems fairly hard on hg.m.o.
> 
> Are you expressing a preference for lowering the maxCount if we hit issues ?

Will lowering this to 10 help?

Another thing we could do is to have a global variable so we limit to N per master rather than N per branch.

Taking this even further, we could create an hg lock that everything that touches hg needs to acquire before running, and then wouldn't need to limit l10n in particular.

Comment 20

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 449932 [details] [diff] [review]
Limit the number of l10n jobs we do to 20 per branch per master

(In reply to comment #19)
> Will lowering this to 10 help?
> Another thing we could do is to have a global variable so we limit to N per
> master rather than N per branch.

Some lower value combined with a per-master limit would be better if the other approaches we discussed on irc don't work out (eg first N slaves connected to master may do l10n)

> Taking this even further, we could create an hg lock that everything that
> touches hg needs to acquire before running, and then wouldn't need to limit
> l10n in particular.

Are all hg operations of equal weight ? I suspect cloning m-c > updating m-c > cloning tools but I can't back that up </drevil>.

Comment 21

[Chris AtLee [:catlee]]

Attachment: Limit ourselves to using the first 8 connected slaves for l10n jobs

Comment 22

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 450233 [details] [diff] [review]
Limit ourselves to using the first 8 connected slaves for l10n jobs

Looks good, thanks for fixing this.

Comment 23

[Armen [:armenzg]]

Can we have no lock for L10n release repackages and use the whole pool? (once we move that to schedulerdb)

Comment 24

[Chris AtLee [:catlee]]

Comment on attachment 449935 [details] [diff] [review]
Add properties for our branch and platform to builders

768:e355a4b0422b

Comment 25

[Chris AtLee [:catlee]]

Comment on attachment 450233 [details] [diff] [review]
Limit ourselves to using the first 8 connected slaves for l10n jobs

767:8fea408513ee

Comment 26

[Chris AtLee [:catlee]]

Comment on attachment 449780 [details] [diff] [review]
Disable mozilla-1.9.2, 1.9.1 on pm,pm02

2512:ab150c39bacd

Comment 27

[Chris AtLee [:catlee]]

Comment on attachment 449777 [details] [diff] [review]
Enable mozilla-1.9.1, 1.9.2 on schedulerdb

2511:512ea8b96cf4

Comment 28

[Chris AtLee [:catlee]]

Comment on attachment 449778 [details] [diff] [review]
Enable trunk on schedulerdb

2520:af400edd659e

Comment 29

[Chris AtLee [:catlee]]

Comment on attachment 449779 [details] [diff] [review]
Disable mozilla-central on production-master

2521:ed1073a3ce34

Comment 31

[Chris AtLee [:catlee]]

Tracking bug -> P5

Comment 32

[Chris AtLee [:catlee]]

Only things left are mobile and releases, which have their own bugs.