922594 - Build API returning 503 & builds-4hr.js.gz not updating

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Ed Morley [:emorley]]

eg:
https://secure.pub.build.mozilla.org/buildapi/self-serve/mozilla-inbound/rev/0a76209b85d2

-> all trees closed.

Comment 1

[Ed Morley [:emorley]]

Also:
***** Nagios  *****

Notification Type: PROBLEM

Service: http file age - /buildjson/builds-4hr.js.gz
Host: builddata.pub.build.mozilla.org
Address: 63.245.215.57
State: CRITICAL

Date/Time: 10-01-2013 05:22:54

Additional Info:
HTTP CRITICAL: HTTP/1.1 200 OK - Last modified 0:11:31 ago - 1207011 bytes in 0.053 second response time

Comment 2

[Ed Morley [:emorley]]

Might be ok now, will leave it a moment longer to see if things are appearing on TBPL:

***** Nagios  *****

Notification Type: RECOVERY

Service: http file age - /buildjson/builds-4hr.js.gz
Host: builddata.pub.build.mozilla.org
Address: 63.245.215.57
State: OK

Date/Time: 10-01-2013 05:32:54

Additional Info:
HTTP OK: HTTP/1.1 200 OK - 1278924 bytes in 0.668 second response time

Comment 3

[[e-mail] (:bhearsum)]

I suspect that the lag in builds-4hr being updated is a separate problem than the 503s, but let's confirm that before we move it out to a separate bug. I filed bug 922606 to get someone to look at the load balancer in front of buildapi, as we're not seeing the 503s on the backend server.

Comment 4

[Ed Morley [:emorley]]

I haven't seen any more 503s for a while now, and builds-4hr has recovered - I'm inclined to reopen now :-)

Comment 5

[Ed Morley [:emorley]]

Trees reopened

Comment 6

[[e-mail] (:bhearsum)]

From bug 922606:
(In reply to Corey Shields [:cshields] from comment #3)
> I enabled logging on the releng and releng-https vips so hopefully we can
> coorelate these missing hits next time.

Comment 7

[Armen [:armenzg]]

We recovered from this issue.