buildapi returning "HTTP Error 500: Internal Server Error"

RESOLVED FIXED

Status

Release Engineering
General
RESOLVED FIXED
2 years ago
a year ago

People

(Reporter: coop, Assigned: aselagea)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

2 years ago
Buildapi has been returning 500 errors for the past little while. Here's the SNS alert we get in IRC:

09:04 <relengbot> [sns alert] Aug 09 06:02:42 relengwebadm.private.scl3.mozilla.com buildapi_waittimes: Error: fetching wait times from location http://buildapi.pvt.build.mozilla.org/buildapi/reports/waittimes/trybuildpool?maxb=480&endtime=1470726000&mpb=15&format=json : HTTP Error 500: Internal Server Error
09:06 <relengbot> [sns alert] Aug 09 06:04:23 relengwebadm.private.scl3.mozilla.com buildapi_waittimes: Error: fetching wait times from location http://buildapi.pvt.build.mozilla.org/buildapi/reports/waittimes/testpool?maxb=480&endtime=1470726000&mpb=15&format=json : HTTP Error 500: Internal Server Error
09:06 <relengbot> [sns alert] Aug 09 06:06:04 relengwebadm.private.scl3.mozilla.com buildapi_waittimes: Error: fetching wait times from location http://buildapi.pvt.build.mozilla.org/buildapi/reports/waittimes/buildpool?maxb=480&endtime=1470726000&mpb=15&format=json : HTTP Error 500: Internal Server Error

SNS was done for a few months, so it's hard to tell when this started. Despite our desire to turn off the wait times emails, the rest of buildapi is still required.
(Assignee)

Updated

2 years ago
Assignee: nobody → aselagea
(Assignee)

Comment 1

2 years ago
@coop: any thoughts on how this application is deployed? I'm not able to find out where the reports from https://secure.pub.build.mozilla.org/buildapi/ should get data from.
Flags: needinfo?(coop)
(Reporter)

Comment 2

2 years ago
(In reply to Alin Selagea [:aselagea][:buildduty] from comment #1)
> @coop: any thoughts on how this application is deployed? I'm not able to
> find out where the reports from
> https://secure.pub.build.mozilla.org/buildapi/ should get data from.

Have you looked at the following docs?

https://wiki.mozilla.org/ReleaseEngineering/How_To/Update_a_Releng_Cluster_Site
https://wiki.mozilla.org/ReleaseEngineering/BuildAPI
https://wiki.mozilla.org/ReleaseEngineering/How_To/Restart_BuildAPI

I can't attest to whether those docs are up-to-date or not, but it's a starting point that I hope you've seen.

:catlee - you and I had talked about turning off the wait times reports, but it seems all the buildapi reports are missing, i.e.
https://secure.pub.build.mozilla.org/buildapi/reports/* 

Did we actually turn something off, or is this 500 error unrelated?
Flags: needinfo?(coop) → needinfo?(catlee)
No, I haven't turned anything off yet, I think this is unrelated.
Flags: needinfo?(catlee)
This is failing with:
[Thu Sep 01 11:36:00 2016] [error] [client 10.22.81.211] Error - <type 'exceptions.ImportError'>: No module named gviz_api

From /var/log/httpd/buildapi.pvt.build.mozilla.org/error_log on web[12].releng.webapp.scl3.mozilla.com
(Assignee)

Comment 5

2 years ago
Thanks for spotting this! I somehow missed the import error as it being logged only in the early PDT morning (when the check for the waittimes report is done) and then zipped.

Did a bit more investigation on this and found the following:
    -  gviz_api is the google visualization library and it *is* used by the buildapi reports
    -  according to https://bugzilla.mozilla.org/show_bug.cgi?id=1271661#c6, upgrading the python version from 2.7.3 to 2.7.11 caused issues with the virtual envs on relengwebadmin, so they needed to be rebuilt
    - gviz_api module is not present on buildapi virtual environments from relengwebadmin nor webheads. We can install it using something like:
       pip install -U -f https://github.com/google/google-visualization-python/ gviz-api-py
(Assignee)

Comment 6

2 years ago
Okay, so installing the module on relengwebadmin and then restarting buildapi fixed the issue and all buildapi reports except https://secure.pub.build.mozilla.org/buildapi/pushes are available again.

Note: I used "pip install -U -f https://github.com/google/google-visualization-python/ gviz-api-py", as the command above did not work.
(Assignee)

Updated

2 years ago
Duplicate of this bug: 1279556
(Assignee)

Updated

2 years ago
See Also: → bug 1271661
(Assignee)

Updated

a year ago
Status: NEW → RESOLVED
Last Resolved: a year ago
Resolution: --- → FIXED
Component: Tools → General
Product: Release Engineering → Release Engineering
You need to log in before you can comment on or make changes to this bug.