Closed Bug 1127699 Opened 9 years ago Closed 9 years ago

http://builddata.pub.build.mozilla.org//buildjson/builds-4hr.js.gz last updated at 20:36, all trees closed

Categories

(Infrastructure & Operations Graveyard :: CIDuty, task)

Product:
Infrastructure & Operations Graveyard
Component:
CIDuty
Type:
task
Priority:
Not set
Severity:
blocker

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: philor, Assigned: Callek)

Details

(Whiteboard: [TreeClosure]) 

Sigh
Per:

https://wiki.mozilla.org/ReleaseEngineering/BuildAPI#report_crontask

[root@relengwebadm.private.scl3 ~]# ps auxwww | grep 4hr
buildapi  6310  0.0  0.0   9232  1176 ?        Ss   20:36   0:00 /bin/sh /var/lib/buildapi/report-4hr.sh
buildapi  6311  0.0  0.0   9232   760 ?        S    20:36   0:00 /bin/sh /var/lib/buildapi/report-4hr.sh
buildapi  6317  0.2  6.3 565556 378464 ?       S    20:36   0:07 python2.7 /data/releng/www/buildapi/run-reporter --config /data/
releng/www/buildapi/reporter.cfg -z -o /mnt/netapp/relengweb/builddata/buildjson/builds-4hr.js.gz --starttime 1422578161
root     12066  0.0  0.0 103244   872 pts/0    R+   21:32   0:00 grep 4hr
[root@relengwebadm.private.scl3 ~]# date
Thu Jan 29 21:32:27 PST 2015

sooo...

thats been almost an hour since it launched :/

[root@relengwebadm.private.scl3 ~]# kill 6317; rm -f /var/lock/buildapi/builds-4hr

And it just launched, awaiting confirmation that it clears...

buildapi 12210  0.0  0.0   9232  1180 ?        Ss   21:35   0:00 /bin/sh /var/lib/buildapi/report-4hr.sh
buildapi 12216  0.0  0.0   9232   764 ?        S    21:35   0:00 /bin/sh /var/lib/buildapi/report-4hr.sh
buildapi 12221 19.1  1.5 280284 93140 ?        S    21:35   0:01 python2.7 /data/releng/www/buildapi/run-reporter --config /data/
releng/www/buildapi/reporter.cfg -z -o /mnt/netapp/relengweb/builddata/buildjson/builds-4hr.js.gz --starttime 1422581701
Assignee: nobody → bugspam.Callek
Fwiw, I see e-mail alerts from all masters with notes about "MySQL Server has gone away" with timestamps lining up to 20:36 as well, and within the last hour I got e-mail from MOC saying that due to GHOST among services restarted was load balancers. I suspect this may have hung due to MySQL going away as well, because of the MySQL load balancer being restarted.
Whiteboard: [TreeClosure]
[00:43:30]	nagios-releng	Thu 21:43:38 PST [4465] builddata.pub.build.mozilla.org:http file age - /buildjson/builds-4hr.js.gz is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.047 second response time (http://m.mozilla.org/http+file+age+-+/buildjson/builds-4hr.js.gz)

Tree's are reopenable as soon as sheriffs validate there is no code-related bustage.
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Product: Release Engineering → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.