If you think a bug might affect users in the 57 release, please set the correct tracking and status flags for Release Management.

Tracking bug for getting schedulerdb working

RESOLVED FIXED

Status

Release Engineering
General
P5
normal
RESOLVED FIXED
8 years ago
4 years ago

People

(Reporter: catlee, Assigned: catlee)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [q2goal][automation])

Attachments

(9 attachments, 1 obsolete attachment)

49.52 KB, patch
bhearsum
: review+
catlee
: checked-in+
Details | Diff | Splinter Review
3.20 KB, patch
bhearsum
: review+
catlee
: checked-in+
Details | Diff | Splinter Review
3.80 KB, patch
bhearsum
: review+
catlee
: checked-in+
Details | Diff | Splinter Review
3.43 KB, patch
nthomas
: review+
catlee
: checked-in+
Details | Diff | Splinter Review
1.71 KB, patch
nthomas
: review+
catlee
: checked-in+
Details | Diff | Splinter Review
553 bytes, patch
nthomas
: review+
catlee
: checked-in+
Details | Diff | Splinter Review
1.17 KB, patch
nthomas
: review+
catlee
: checked-in+
Details | Diff | Splinter Review
9.31 KB, patch
nthomas
: review+
catlee
: checked-in+
Details | Diff | Splinter Review
4.19 KB, patch
nthomas
: review+
catlee
: checked-in+
Details | Diff | Splinter Review
(Assignee)

Description

8 years ago
Objectives:
* Get buildbot with schedulerdb working on all our masters
* Ability to have a master crash/go down, and not have the list of pending jobs lost
* Ability to have multiple masters handle changes for the same branch without duplicating effort
(Assignee)

Updated

8 years ago
Depends on: 508672
(Assignee)

Updated

8 years ago
Depends on: 539589
(Assignee)

Updated

8 years ago
Depends on: 553300
(Assignee)

Updated

8 years ago
Depends on: 556391, 556390
(Assignee)

Updated

8 years ago
Blocks: 557613
Blocks: 421895
(Assignee)

Updated

8 years ago
Whiteboard: [q2goal][automation]
(Assignee)

Updated

8 years ago
Blocks: 559880
(Assignee)

Updated

8 years ago
Blocks: 559882
(Assignee)

Updated

8 years ago
Blocks: 559885
(Assignee)

Updated

8 years ago
Blocks: 559886
(Assignee)

Comment 1

8 years ago
Created attachment 439599 [details] [diff] [review]
buildbot-configs for staging scheduler/builder masters

This is the beginning of the new single-directory layout.

Several symlinks need to be set up, depending on the master being created.

builder/scheduler_master.cfg -> master.cfg
staging/production_config.py -> localconfig.py
staging_builder_master_localconfig.py / staging_scheduler_mater_localconfig.py -> master_localconfig.py
Attachment #439599 - Flags: review?(bhearsum)
Attachment #439599 - Flags: review?(bhearsum) → review+
(Assignee)

Comment 2

8 years ago
Comment on attachment 439599 [details] [diff] [review]
buildbot-configs for staging scheduler/builder masters

changeset:   2327:77d10fffd0ae
branch:      buildbot-0.8.0
Attachment #439599 - Flags: checked-in+
(Assignee)

Comment 3

8 years ago
Created attachment 442684 [details] [diff] [review]
Production configs for scheduler/builder master
Attachment #442684 - Flags: review?(bhearsum)
Attachment #442684 - Flags: review?(bhearsum) → review+
(Assignee)

Comment 4

8 years ago
Comment on attachment 442684 [details] [diff] [review]
Production configs for scheduler/builder master

changeset:   2341:c4649fd8a33d
Attachment #442684 - Flags: checked-in+
(Assignee)

Updated

8 years ago
Depends on: 565427
(Assignee)

Comment 5

8 years ago
Created attachment 444981 [details] [diff] [review]
Sync over config changes to mozilla/
Attachment #444981 - Flags: review?(bhearsum)
Comment on attachment 444981 [details] [diff] [review]
Sync over config changes to mozilla/

You should probably remove the talos-master.m.o:9012 from unittest_masters, as Nick suggested I do in that bug.
Attachment #444981 - Flags: review?(bhearsum) → review+
(Assignee)

Updated

7 years ago
Depends on: 567192
(Assignee)

Comment 7

7 years ago
Comment on attachment 444981 [details] [diff] [review]
Sync over config changes to mozilla/

changeset:   2444:842e9f53c3e8
Attachment #444981 - Flags: checked-in+
(Assignee)

Comment 8

7 years ago
Comment on attachment 444981 [details] [diff] [review]
Sync over config changes to mozilla/

whoops.  this was checked in, but not as that changeset
(Assignee)

Updated

7 years ago
Depends on: 568568
(Assignee)

Updated

7 years ago
Depends on: 568570
(Assignee)

Updated

7 years ago
Depends on: 568617
(Assignee)

Updated

7 years ago
Depends on: 568848

Updated

7 years ago
Depends on: 569551
(Assignee)

Updated

7 years ago
Depends on: 569696
(Assignee)

Updated

7 years ago
No longer depends on: 568617
(Assignee)

Comment 9

7 years ago
Created attachment 449777 [details] [diff] [review]
Enable mozilla-1.9.1, 1.9.2 on schedulerdb
Attachment #449777 - Flags: review?(nrthomas)
(Assignee)

Comment 10

7 years ago
Created attachment 449778 [details] [diff] [review]
Enable trunk on schedulerdb
Attachment #449778 - Flags: review?(nrthomas)
(Assignee)

Comment 11

7 years ago
Created attachment 449779 [details] [diff] [review]
Disable mozilla-central on production-master
Attachment #449779 - Flags: review?(nrthomas)
(Assignee)

Comment 12

7 years ago
Created attachment 449780 [details] [diff] [review]
Disable mozilla-1.9.2, 1.9.1 on pm,pm02
Attachment #449780 - Flags: review?(nrthomas)
Comment on attachment 449777 [details] [diff] [review]
Enable mozilla-1.9.1, 1.9.2 on schedulerdb

OK, but lets watch out for 75 l10n builds grabbing all the slaves and overloading hg.m.o each time a nightly finishes.
Attachment #449777 - Flags: review?(nrthomas) → review+

Updated

7 years ago
Attachment #449778 - Flags: review?(nrthomas) → review+
Comment on attachment 449780 [details] [diff] [review]
Disable mozilla-1.9.2, 1.9.1 on pm,pm02

r+ if you remove SchedulerL10n which will have no builders to drive after this.
Attachment #449780 - Flags: review?(nrthomas) → review+
Comment on attachment 449779 [details] [diff] [review]
Disable mozilla-central on production-master

r+ with the l10n scheduler removed.
Attachment #449779 - Flags: review?(nrthomas) → review+
(Assignee)

Comment 16

7 years ago
Created attachment 449932 [details] [diff] [review]
Limit the number of l10n jobs we do to 20 per branch per master
Attachment #449932 - Flags: review?(nrthomas)
(Assignee)

Comment 17

7 years ago
Created attachment 449935 [details] [diff] [review]
Add properties for our branch and platform to builders
Attachment #449935 - Flags: review?(nrthomas)
Comment on attachment 449932 [details] [diff] [review]
Limit the number of l10n jobs we do to 20 per branch per master

>diff --git a/misc.py b/misc.py
>+    # Limit us to doing 20 l10n jobs per branch per master
>+    l10nLock = locks.MasterLock("%s-l10n" % name, maxCount=20)

The code changes make sense, but we could get 40 total jobs per branch once we migrate m-1.9.1/m-1.9.2/m-c to buildbot 0.8.0 ? (assuming two masters is enough). There seems to be overlap between branches in the current up-to-8-slaves setup, given nightlies fire off at 30 minute intervals, so that could go even higher. And that seems fairly hard on hg.m.o.

Are you expressing a preference for lowering the maxCount if we hit issues ?
(Assignee)

Comment 19

7 years ago
(In reply to comment #18)
> (From update of attachment 449932 [details] [diff] [review])
> >diff --git a/misc.py b/misc.py
> >+    # Limit us to doing 20 l10n jobs per branch per master
> >+    l10nLock = locks.MasterLock("%s-l10n" % name, maxCount=20)
> 
> The code changes make sense, but we could get 40 total jobs per branch once we
> migrate m-1.9.1/m-1.9.2/m-c to buildbot 0.8.0 ? (assuming two masters is
> enough). There seems to be overlap between branches in the current
> up-to-8-slaves setup, given nightlies fire off at 30 minute intervals, so that
> could go even higher. And that seems fairly hard on hg.m.o.
> 
> Are you expressing a preference for lowering the maxCount if we hit issues ?

Will lowering this to 10 help?

Another thing we could do is to have a global variable so we limit to N per master rather than N per branch.

Taking this even further, we could create an hg lock that everything that touches hg needs to acquire before running, and then wouldn't need to limit l10n in particular.

Updated

7 years ago
Attachment #449935 - Flags: review?(nrthomas) → review+

Updated

7 years ago
Attachment #449932 - Flags: review?(nrthomas)
Comment on attachment 449932 [details] [diff] [review]
Limit the number of l10n jobs we do to 20 per branch per master

(In reply to comment #19)
> Will lowering this to 10 help?
> Another thing we could do is to have a global variable so we limit to N per
> master rather than N per branch.

Some lower value combined with a per-master limit would be better if the other approaches we discussed on irc don't work out (eg first N slaves connected to master may do l10n)

> Taking this even further, we could create an hg lock that everything that
> touches hg needs to acquire before running, and then wouldn't need to limit
> l10n in particular.

Are all hg operations of equal weight ? I suspect cloning m-c > updating m-c > cloning tools but I can't back that up </drevil>.
(Assignee)

Comment 21

7 years ago
Created attachment 450233 [details] [diff] [review]
Limit ourselves to using the first 8 connected slaves for l10n jobs
Attachment #449932 - Attachment is obsolete: true
Attachment #450233 - Flags: review?(nrthomas)
Comment on attachment 450233 [details] [diff] [review]
Limit ourselves to using the first 8 connected slaves for l10n jobs

Looks good, thanks for fixing this.
Attachment #450233 - Flags: review?(nrthomas) → review+
Depends on: 523946

Comment 23

7 years ago
Can we have no lock for L10n release repackages and use the whole pool? (once we move that to schedulerdb)
(Assignee)

Updated

7 years ago
Depends on: 572188
(Assignee)

Comment 24

7 years ago
Comment on attachment 449935 [details] [diff] [review]
Add properties for our branch and platform to builders

768:e355a4b0422b
Attachment #449935 - Flags: checked-in+
(Assignee)

Comment 25

7 years ago
Comment on attachment 450233 [details] [diff] [review]
Limit ourselves to using the first 8 connected slaves for l10n jobs

767:8fea408513ee
Attachment #450233 - Flags: checked-in+
(Assignee)

Comment 26

7 years ago
Comment on attachment 449780 [details] [diff] [review]
Disable mozilla-1.9.2, 1.9.1 on pm,pm02

2512:ab150c39bacd
Attachment #449780 - Flags: checked-in+
(Assignee)

Comment 27

7 years ago
Comment on attachment 449777 [details] [diff] [review]
Enable mozilla-1.9.1, 1.9.2 on schedulerdb

2511:512ea8b96cf4
Attachment #449777 - Flags: checked-in+
(Assignee)

Updated

7 years ago
Blocks: 571571
(Assignee)

Comment 28

7 years ago
Comment on attachment 449778 [details] [diff] [review]
Enable trunk on schedulerdb

2520:af400edd659e
Attachment #449778 - Flags: checked-in+
(Assignee)

Comment 29

7 years ago
Comment on attachment 449779 [details] [diff] [review]
Disable mozilla-central on production-master

2521:ed1073a3ce34
Attachment #449779 - Flags: checked-in+
(Assignee)

Updated

7 years ago
No longer blocks: 571571
(Assignee)

Updated

7 years ago
Duplicate of this bug: 552723
Depends on: 574729
(Assignee)

Updated

7 years ago
OS: Linux → All
Hardware: x86 → All
(Assignee)

Comment 31

7 years ago
Tracking bug -> P5
Priority: -- → P5
No longer blocks: 559880
No longer blocks: 559885
(Assignee)

Comment 32

7 years ago
Only things left are mobile and releases, which have their own bugs.
Status: NEW → RESOLVED
Last Resolved: 7 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.