self-serve returning proxy errors for retriggers

RESOLVED FIXED

Status

Release Engineering
General
--
critical
RESOLVED FIXED
7 years ago
4 years ago

People

(Reporter: philor, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

7 years ago
STR:

1. https://build.mozilla.org/buildapi/self-serve/mozilla-central/rev/1edfae12015b
2. type "andr" in the builds section's search box
3. click rebuild for the Android build

Expected:
Retrigger it

Actual:
502 Proxy Error
Proxy Error

The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request POST /buildapi/self-serve/mozilla-central/build.

Reason: Error reading from remote server

Apache/2.2.3 (Red Hat) Server at cruncher.build.mozilla.org Port 80
This is also happening for me when I try to trigger any additional reftest jobs (any platform*) on my tryserver push: https://build.mozilla.org/buildapi/self-serve/try/rev/e1e4c67862d8
* (only linux debug reftest & linux opt reftest are available, since I TryChooser'd those in particular)
I restarted buildapi on cruncher, by impersonating catlee. There's all sorts of things we could be doing better here, many of which are filed already.
Status: NEW → RESOLVED
Last Resolved: 7 years ago
Resolution: --- → FIXED
(Reporter)

Comment 3

7 years ago
Didn't quite survive for 24 hours, which is considerably worse than the usual (though it's not up to the point of nagios bitching about the age of pending and running, the way it was yesterday).
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
I've restarted both parts this time (cruncher and the agent on pm03). 

The last thing that agent.log logged was:
2011-04-10 11:43:30,963 cancelling build by dbaron@mozilla.com of 3058530
2011-04-10 11:43:30,970 build is complete, nothing to do
[snipped the channel shutdown]

and it failed to do this (from access.log):
2011-04-10 15:58:45,704 INFO  [buildapi.access] [worker 19] cpearce@mozilla.com accessing POST /self-serve/try/build
2011-04-10 15:58:45,861 INFO  [buildapi.access] [worker 19] cpearce@mozilla.com rebuild_build try 3059570 0

philor successfully did a rebuild after that.
Status: REOPENED → RESOLVED
Last Resolved: 7 years ago7 years ago
Resolution: --- → FIXED
Hitting this again
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Looks like it's working at the moment. Did someone restart without commenting ?
We have steps in our Releng Sheriff notes on how to kick the buildapi when needed so this bug can be closed now and if there are proxy errors in the future that require a kick, ping *-buildduty in #build or #developers
Status: REOPENED → RESOLVED
Last Resolved: 7 years ago7 years ago
Resolution: --- → FIXED
(Assignee)

Updated

4 years ago
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.