Closed Bug 912428 Opened 6 years ago Closed 6 years ago

buildapi/scripts/reporter.py hangs generating builds-4hr.js.gz, resulting in tbpl.m.o not showing completed builds

Categories

(Infrastructure & Operations :: CIDuty, task, P1, blocker)

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: emorley, Assigned: nthomas)

References

Details

(Keywords: buildapi)

+++ This bug was initially created as a clone of Bug #905554 +++

Completed builds are not showing up on TBPL, due to builds-4hr.js.gz having last been updated just over an hour ago. See timestamps at:
https://secure.pub.build.mozilla.org/builddata/buildjson/

Guessing the redis service needs a kick again, similar to bug 905554.

Trees are closed (shortly) as a result.
Note to self: We have until 13:44 UK before we'll need to start manually backfilling job data, since we'll be outside the 4hr window.
Assignee: nobody → nthomas
You were right, the redis service needed bouncing. builds-4hr.js.gz is back up to date, and builds-2013-09-04.js.gz will follow in a few mins.
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
Thank you - jobs are showing up fine now that the TBPL cron has picked those up too.

Will file a bug for nagios alert for builds-4hr.js.gz being out of date (though seem to recall that already being done? maybe we can email sheriffs@ too).
There's one in #buildduty already:

Wed 03:04:10 PDT [4342] builddata.pub.build.mozilla.org:http file age - /buildjson/builds-4hr.js.gz is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - Last modified 1:19:40 ago - 1210708 bytes in 0.072 second response time (http://m.allizom.org/http+file+age+-+/buildjson/builds-4hr.js.gz)

Wed 03:34:11 PDT [4348] builddata.pub.build.mozilla.org:http file age - /buildjson/builds-4hr.js.gz is OK: HTTP OK: HTTP/1.1 200 OK - 1212185 bytes in 0.069 second response time (http://m.allizom.org/http+file+age+-+/buildjson/builds-4hr.js.gz)
Ah, I just tend not to be in that channel since it's pretty noisy.

I've filed bug 912463 for getting sheriffs@ emailed when the alert triggers (presuming that's possible).
Blocks: 914574
Blocks: 926246
Product: Release Engineering → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.