648783 - self-serve returning proxy errors for retriggers

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Phil Ringnalda (:philor)]

STR:

1. https://build.mozilla.org/buildapi/self-serve/mozilla-central/rev/1edfae12015b
2. type "andr" in the builds section's search box
3. click rebuild for the Android build

Expected:
Retrigger it

Actual:
502 Proxy Error
Proxy Error

The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request POST /buildapi/self-serve/mozilla-central/build.

Reason: Error reading from remote server

Apache/2.2.3 (Red Hat) Server at cruncher.build.mozilla.org Port 80

Comment 1

[Daniel Holbert [:dholbert]]

This is also happening for me when I try to trigger any additional reftest jobs (any platform*) on my tryserver push: https://build.mozilla.org/buildapi/self-serve/try/rev/e1e4c67862d8
* (only linux debug reftest & linux opt reftest are available, since I TryChooser'd those in particular)

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

I restarted buildapi on cruncher, by impersonating catlee. There's all sorts of things we could be doing better here, many of which are filed already.

Comment 3

[Phil Ringnalda (:philor)]

Didn't quite survive for 24 hours, which is considerably worse than the usual (though it's not up to the point of nagios bitching about the age of pending and running, the way it was yesterday).

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

I've restarted both parts this time (cruncher and the agent on pm03). 

The last thing that agent.log logged was:
2011-04-10 11:43:30,963 cancelling build by [e-mail] of 3058530
2011-04-10 11:43:30,970 build is complete, nothing to do
[snipped the channel shutdown]

and it failed to do this (from access.log):
2011-04-10 15:58:45,704 INFO  [buildapi.access] [worker 19] [e-mail] accessing POST /self-serve/try/build
2011-04-10 15:58:45,861 INFO  [buildapi.access] [worker 19] [e-mail] rebuild_build try 3059570 0

philor successfully did a rebuild after that.

Comment 5

[[e-mail] (:bhearsum)]

Hitting this again

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

Looks like it's working at the moment. Did someone restart without commenting ?

Comment 7

[Lukas Blakk [:lsblakk] use ?needinfo]

We have steps in our Releng Sheriff notes on how to kick the buildapi when needed so this bug can be closed now and if there are proxy errors in the future that require a kick, ping *-buildduty in #build or #developers