The default bug view has changed. See this FAQ.

increase disk size for AWS buildbot masters

RESOLVED FIXED

Status

Release Engineering
Buildduty
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: arr, Assigned: coop)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

(Reporter)

Description

3 years ago
I've noticed that buildbot masters in AWS frequently alert on low disk space (115 and 116 in particular). Can we increase the disk size to stop the checks alerting?
(Assignee)

Comment 1

3 years ago
These are alerting because of the increase in the # of 2.3 emulator jobs being thrown at them. 

I've fixed the alerting for now by removing all the double-digit twistd.log files from bm115 and bm116.

As an alternative to increasing the disk space, perhaps we should consider limiting the number of logs retained by these masters to 50 instead of 100?
(Assignee)

Updated

3 years ago
Duplicate of this bug: 1030881
(Assignee)

Comment 3

3 years ago
Created attachment 8450246 [details] [diff] [review]
Keep only 50 twistd.log files per master
Assignee: nobody → coop
Status: NEW → ASSIGNED
Attachment #8450246 - Flags: review?(bugspam.Callek)

Updated

3 years ago
Attachment #8450246 - Flags: review?(bugspam.Callek) → review+
(Assignee)

Comment 4

3 years ago
Comment on attachment 8450246 [details] [diff] [review]
Keep only 50 twistd.log files per master

Review of attachment 8450246 [details] [diff] [review]:
-----------------------------------------------------------------

https://hg.mozilla.org/build/puppet/rev/13e690097f26
Attachment #8450246 - Flags: checked-in+
See action_set_logging in https://github.com/catlee/tools/compare/master...fabric if you are looking for a way to deploy this change without a master restart.
(Assignee)

Comment 6

3 years ago
(In reply to Nick Thomas [:nthomas] from comment #5)
> See action_set_logging in
> https://github.com/catlee/tools/compare/master...fabric if you are looking
> for a way to deploy this change without a master restart.

Fantastic. Thanks, Nick.
(Assignee)

Comment 7

3 years ago
The script failed on a bunch of slow-running masters where it couldn't establish a timely connection, but I patched those ones us by hand via the manhole.
Status: ASSIGNED → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
merged to production
You need to log in before you can comment on or make changes to this bug.