The default bug view has changed. See this FAQ.

Monitor aws_stop_idle.py hungs

RESOLVED FIXED

Status

Release Engineering
General Automation
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: rail, Assigned: rail)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(2 attachments)

(Assignee)

Description

3 years ago
aws_stop_idle.py was hung this morning what caused a lot of builders running. We should monitor aws_stop_idle.py and kill it/alert if it stays alive too long.
(Assignee)

Comment 1

3 years ago
Created attachment 8434944 [details] [diff] [review]
nagios-puppet.diff
Attachment #8434944 - Flags: review?(dustin)
Comment on attachment 8434944 [details] [diff] [review]
nagios-puppet.diff

you do fine work!
Attachment #8434944 - Flags: review?(dustin) → review+
(Assignee)

Comment 3

3 years ago
Comment on attachment 8434944 [details] [diff] [review]
nagios-puppet.diff

remote:   https://hg.mozilla.org/build/puppet/rev/51607523ebff
remote:   https://hg.mozilla.org/build/puppet/rev/68b291568001


(In reply to Dustin J. Mitchell [:dustin] from comment #2)
> you do fine work!

Thank you! :)
Attachment #8434944 - Flags: checked-in+
(Assignee)

Updated

3 years ago
Depends on: 1021003
(Assignee)

Comment 4

3 years ago
Sounds like this has some impact on yesterday's tree closure - I just checked the log and the latest timestamp was 2014-06-18 13:33:30,084...
(Assignee)

Updated

3 years ago
Blocks: 1027437
(Assignee)

Comment 5

3 years ago
Created attachment 8443426 [details] [diff] [review]
signal-cloud-tools.diff

This should help with long running stop idle processes...
Attachment #8443426 - Flags: review?(catlee)
(Assignee)

Comment 6

3 years ago
Comment on attachment 8443426 [details] [diff] [review]
signal-cloud-tools.diff

On IRC we decided to use something external to watch this.
Attachment #8443426 - Flags: review?(catlee)
(Assignee)

Updated

3 years ago
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
(Assignee)

Updated

3 years ago
See Also: → bug 1055600
You need to log in before you can comment on or make changes to this bug.