Closed Bug 1127699 Opened 11 years ago Closed 11 years ago

http://builddata.pub.build.mozilla.org//buildjson/builds-4hr.js.gz last updated at 20:36, all trees closed

Categories

(Infrastructure & Operations Graveyard :: CIDuty, task)

Product:
Infrastructure & Operations Graveyard
Component:
CIDuty
Type:
task
Priority:
Not set
Severity:
blocker

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: philor, Assigned: Callek)

Details

(Whiteboard: [TreeClosure])

Sigh
Per: https://wiki.mozilla.org/ReleaseEngineering/BuildAPI#report_crontask [root@relengwebadm.private.scl3 ~]# ps auxwww | grep 4hr buildapi 6310 0.0 0.0 9232 1176 ? Ss 20:36 0:00 /bin/sh /var/lib/buildapi/report-4hr.sh buildapi 6311 0.0 0.0 9232 760 ? S 20:36 0:00 /bin/sh /var/lib/buildapi/report-4hr.sh buildapi 6317 0.2 6.3 565556 378464 ? S 20:36 0:07 python2.7 /data/releng/www/buildapi/run-reporter --config /data/ releng/www/buildapi/reporter.cfg -z -o /mnt/netapp/relengweb/builddata/buildjson/builds-4hr.js.gz --starttime 1422578161 root 12066 0.0 0.0 103244 872 pts/0 R+ 21:32 0:00 grep 4hr [root@relengwebadm.private.scl3 ~]# date Thu Jan 29 21:32:27 PST 2015 sooo... thats been almost an hour since it launched :/ [root@relengwebadm.private.scl3 ~]# kill 6317; rm -f /var/lock/buildapi/builds-4hr And it just launched, awaiting confirmation that it clears... buildapi 12210 0.0 0.0 9232 1180 ? Ss 21:35 0:00 /bin/sh /var/lib/buildapi/report-4hr.sh buildapi 12216 0.0 0.0 9232 764 ? S 21:35 0:00 /bin/sh /var/lib/buildapi/report-4hr.sh buildapi 12221 19.1 1.5 280284 93140 ? S 21:35 0:01 python2.7 /data/releng/www/buildapi/run-reporter --config /data/ releng/www/buildapi/reporter.cfg -z -o /mnt/netapp/relengweb/builddata/buildjson/builds-4hr.js.gz --starttime 1422581701
Assignee: nobody → bugspam.Callek
Fwiw, I see e-mail alerts from all masters with notes about "MySQL Server has gone away" with timestamps lining up to 20:36 as well, and within the last hour I got e-mail from MOC saying that due to GHOST among services restarted was load balancers. I suspect this may have hung due to MySQL going away as well, because of the MySQL load balancer being restarted.
Whiteboard: [TreeClosure]
[00:43:30] nagios-releng Thu 21:43:38 PST [4465] builddata.pub.build.mozilla.org:http file age - /buildjson/builds-4hr.js.gz is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.047 second response time (http://m.mozilla.org/http+file+age+-+/buildjson/builds-4hr.js.gz) Tree's are reopenable as soon as sheriffs validate there is no code-related bustage.
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Product: Release Engineering → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.