bugzilla.mozilla.org has resumed normal operation. Attachments prior to 2014 will be unavailable for a few days. This is tracked in Bug 1475801.
Please report any other irregularities here.

Add more capacity in AWS for tests (tst-linux64-spot)

RESOLVED FIXED

Status

Release Engineering
General
RESOLVED FIXED
3 years ago
2 months ago

People

(Reporter: nthomas, Assigned: massimo)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(3 attachments)

(Reporter)

Description

3 years ago
Currently 1500 linux64 test jobs running, and 1300+ pending (mostly in try). Time to add more masters and testers to meet the scaling up of push load.

Bug 1090139 for the last time we did this.
(Reporter)

Updated

3 years ago
Summary: Add more capacity in AWS for tests → Add more capacity in AWS for tests (tst-linux64-spot)
our master lag metrics look ok so far, so maybe we could add a few hundred slaves first before worrying about masters?

Comment 2

3 years ago
I also filed bug 1143681 yesterday to make sure we're using the capacity we already have.
bug 1090568 was the last time we added more slaves
(Assignee)

Comment 4

3 years ago
Created attachment 8578759 [details] [review]
Add more tst-linux64 machines (+200 per region)
Assignee: nobody → mgervasini
Attachment #8578759 - Flags: review?(rail)
(Assignee)

Comment 5

3 years ago
Created attachment 8578782 [details] [diff] [review]
[buildbot-configs] Bug 1143901 - Increase the number of tst-linux64-spot instances; pep8 fixes.patch

buildbot-config changes
Attachment #8578782 - Flags: review?(rail)
(Assignee)

Comment 6

3 years ago
Created attachment 8578812 [details]
slavealloc.csv

slavealloc changes
Attachment #8578812 - Flags: review?(rail)
Attachment #8578759 - Flags: review?(rail) → review+
Attachment #8578782 - Flags: review?(rail) → review+
Attachment #8578812 - Attachment mime type: text/csv → text/plain
Comment on attachment 8578812 [details]
slavealloc.csv

Make sure you add this only after the reconfig for the buildbot-configs patch happened. Otherwise cloud-tools will start using these names and get unauthorized login errors from the masters.
Attachment #8578812 - Flags: review?(rail) → review+
(Assignee)

Comment 8

3 years ago
Comment on attachment 8578782 [details] [diff] [review]
[buildbot-configs] Bug 1143901 - Increase the number of tst-linux64-spot instances; pep8 fixes.patch

thanks rail,

landed: https://hg.mozilla.org/build/buildbot-configs/rev/6e758f0c91ca
Attachment #8578782 - Flags: checked-in+
(Assignee)

Comment 10

3 years ago
Comment on attachment 8578812 [details]
slavealloc.csv

imported into slavealloc
Attachment #8578812 - Flags: checked-in+
(Reporter)

Comment 11

3 years ago
The cloud-tools change landed several hours ago, but the number of running jobs hasn't risen above our previous limit of ~1500 (and there are pending jobs which should drive it higher). I would have expected the combined effect to be enabling an extra 400 slaves. Are there some more steps to complete here ?
I think we forgot to enable the slaves in slavealloc (--enable during initial import would do that).
(Assignee)

Comment 13

3 years ago
Enabled in slavealloc:
* tst-linux64-spot-{1700-1719} us-west2
* tst-linux64-spot-{1900-1919} us-east-1

If there are no issues, I am going to enable more instances tomorrow morning (Europe Time)
(Assignee)

Comment 14

3 years ago
Enabled in slavealloc:
* tst-linux64-spot-{1720-1739} us-west2
* tst-linux64-spot-{1920-1939} us-east-1
(Assignee)

Comment 15

3 years ago
Enabled in slavealloc:
* tst-linux64-spot-{1740-1769} us-west2
* tst-linux64-spot-{1940-1969} us-east-1
(Assignee)

Comment 16

3 years ago
Enabled in slavealloc:
* tst-linux64-spot-{1770-1799} us-west2
* tst-linux64-spot-{1970-1999} us-east-1

Adding more in the next hour
(Assignee)

Comment 17

3 years ago
All the new tst-linux64-spot-{1700-2099} instances are now enabled in slavealloc
(Assignee)

Updated

3 years ago
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
Component: General Automation → General
Product: Release Engineering → Release Engineering
You need to log in before you can comment on or make changes to this bug.