b-2008-ec2-golden: lockfile

RESOLVED FIXED

Status

Release Engineering
Buildduty
RESOLVED FIXED
16 days ago
15 days ago

People

(Reporter: danutlabici, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

16 days ago
While monitoring the #buildduty irc channel this alert came up, after repeated Nagios alerts about golden-ami. 

Bug ticket here:  
https://bugzilla.mozilla.org/show_bug.cgi?id=1428344


[sns alert] Jan 05 01:38:01 aws-manager2.srv.releng.scl3.mozilla.com b-2008-ec2-golden: lockfile: Sorry, giving up on "/builds/aws_manager/b-2008-ec2-golden.lock"
(Reporter)

Comment 1

16 days ago
Could you please take a look at this?
Thank you.
Flags: needinfo?(jlund)
Update: there have been multiple alarms today (these are only some of them):

nagios-releng> Sat 07:32:32 UTC [7448] [] aws-manager2.srv.releng.scl3.mozilla.com:procs age - golden AMI is CRITICAL: ELAPSED CRITICAL: 5 crit, 0 warn out of 5 processes with args 'ec2-golden' (http://m.mozilla.org/procs+age+-+golden+AMI)
10:32 AM Sat 08:32:32 UTC [7450] [] aws-manager2.srv.releng.scl3.mozilla.com:procs age - golden AMI is CRITICAL: ELAPSED CRITICAL: 5 crit, 0 warn out of 5 processes with args 'ec2-golden' (http://m.mozilla.org/procs+age+-+golden+AMI)
11:32 AM Sat 09:32:33 UTC [7452] [] aws-manager2.srv.releng.scl3.mozilla.com:procs age - golden AMI is CRITICAL: ELAPSED CRITICAL: 5 crit, 0 warn out of 35 processes with args 'ec2-golden' (http://m.mozilla.org/procs+age+-+golden+AMI)
12:32 PM Sat 10:32:32 UTC [7454] [] aws-manager2.srv.releng.scl3.mozilla.com:procs age - golden AMI is CRITICAL: ELAPSED CRITICAL: 5 crit, 0 warn out of 25 processes with args 'ec2-golden' (http://m.mozilla.org/procs+age+-+golden+AMI)
1:32 PM Sat 11:32:33 UTC [7456] [] aws-manager2.srv.releng.scl3.mozilla.com:procs age - golden AMI is CRITICAL: ELAPSED CRITICAL: 5 crit, 0 warn out of 5 processes with args 'ec2-golden' (http://m.mozilla.org/procs+age+-+golden+AMI)

<nagios-releng> Sat 12:32:32 UTC [7459] [] aws-manager2.srv.releng.scl3.mozilla.com:procs age - golden AMI is CRITICAL: ELAPSED CRITICAL: 5 crit, 0 warn out of 5 processes with args 'ec2-golden' (http://m.mozilla.org/procs+age+-+golden+AMI)
3:05 PM 

<relengbot> [sns alert] Jan 06 01:38:01 aws-manager2.srv.releng.scl3.mozilla.com b-2008-ec2-golden: lockfile: Sorry, giving up on "/builds/aws_manager/b-2008-ec2-golden.lock"
3:32 PM 

<nagios-releng> Sat 13:32:33 UTC [7462] [] aws-manager2.srv.releng.scl3.mozilla.com:procs age - golden AMI is CRITICAL: ELAPSED CRITICAL: 5 crit, 0 warn out of 5 processes with args 'ec2-golden' (http://m.mozilla.org/procs+age+-+golden+AMI)
4:32 PM Sat 14:32:33 UTC [7467] [] aws-manager2.srv.releng.scl3.mozilla.com:procs age - golden AMI is CRITICAL: ELAPSED CRITICAL: 5 crit, 0 warn out of 5 processes with args 'ec2-golden' (http://m.mozilla.org/procs+age+-+golden+AMI)
4:53 PM ⇐ safwan quit (uid93576@moz-e1sk3h.brockwell.irccloud.com) Quit: Connection closed for inactivity
5:32 PM 

<nagios-releng> Sat 15:32:33 UTC [7470] [] aws-manager2.srv.releng.scl3.mozilla.com:procs age - golden AMI is CRITICAL: ELAPSED CRITICAL: 5 crit, 0 warn out of 5 processes with args 'ec2-golden' (http://m.mozilla.org/procs+age+-+golden+AMI)
6:32 PM Sat 16:32:33 UTC [7472] [] aws-manager2.srv.releng.scl3.mozilla.com:procs age - golden AMI is CRITICAL: ELAPSED CRITICAL: 5 crit, 0 warn out of 5 processes with args 'ec2-golden' (http://m.mozilla.org/procs+age+-+golden+AMI)

Can someone take a look ? thank you
Flags: needinfo?(catlee)

Comment 3

15 days ago
ty Andreea,

I poked at this. Let's see if this helps: https://bugzilla.mozilla.org/show_bug.cgi?id=1428344#c2
Flags: needinfo?(jlund)
Flags: needinfo?(catlee)

Comment 4

15 days ago
nagios-releng> Sat 18:32:32 UTC [7480] [] aws-manager2.srv.releng.scl3.mozilla.com:procs age - golden AMI is OK: ELAPSED OK: 0 processes with args 'ec2-golden' (http://m.mozilla.org/procs+age+-+golden+AMI)

\o/
Status: NEW → RESOLVED
Last Resolved: 15 days ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.