1076694 - All tree closed due to builddata.pub.build.mozilla.org/http file age - /buildjson/builds-4hr.js.gz is CRITICAL **

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Carsten Book [:Tomcat]]

***** Nagios  *****

Notification Type: PROBLEM

Service: http file age - /buildjson/builds-4hr.js.gz
Host: builddata.pub.build.mozilla.org
Address: 63.245.215.57
State: CRITICAL

Date/Time: 10-01-2014 23:13:32

Additional Info:
HTTP CRITICAL: HTTP/1.1 200 OK - Last modified 1:11:53 ago - 8191 bytes in 0.006 second response time

and trees closed now

Comment 1

[Carsten Book [:Tomcat]]

and reopened now that we got the nagios recovery notice

Comment 2

[Justin Wood (:Callek)]

using https://wiki.mozilla.org/ReleaseEngineering/BuildAPI#report_crontask I killed the proc at ~ 23:20

Confirmed it started a new run at 23:23 which finished at 

Wed 23:30:52 PDT [4034] builddata.pub.build.mozilla.org:http file age - /buildjson/builds-4hr.js.gz is OK: HTTP OK: HTTP/1.1 200 OK - 4091 bytes in 0.024 second response time (http://m.mozilla.org/http+file+age+-+/buildjson/builds-4hr.js.gz)

Marking this bug fixed.

Trees Reopened.


[NOTE: One of the reasons this took longer to notice was that nagios-releng was unrelatedly not reporting in #buildduty, the irc bot was hung or something]

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

That's twice we've had this since the upgrades/reboots today, something may have changed for the worse. The first time the jobs were trying to read from a socket to buildbot-ro-vip, and weren't getting anything.