Closed
Bug 1293659
Opened 8 years ago
Closed 7 years ago
buildapi returning "HTTP Error 500: Internal Server Error"
Categories
(Release Engineering :: General, defect)
Release Engineering
General
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: coop, Assigned: aselagea)
References
Details
Buildapi has been returning 500 errors for the past little while. Here's the SNS alert we get in IRC: 09:04 <relengbot> [sns alert] Aug 09 06:02:42 relengwebadm.private.scl3.mozilla.com buildapi_waittimes: Error: fetching wait times from location http://buildapi.pvt.build.mozilla.org/buildapi/reports/waittimes/trybuildpool?maxb=480&endtime=1470726000&mpb=15&format=json : HTTP Error 500: Internal Server Error 09:06 <relengbot> [sns alert] Aug 09 06:04:23 relengwebadm.private.scl3.mozilla.com buildapi_waittimes: Error: fetching wait times from location http://buildapi.pvt.build.mozilla.org/buildapi/reports/waittimes/testpool?maxb=480&endtime=1470726000&mpb=15&format=json : HTTP Error 500: Internal Server Error 09:06 <relengbot> [sns alert] Aug 09 06:06:04 relengwebadm.private.scl3.mozilla.com buildapi_waittimes: Error: fetching wait times from location http://buildapi.pvt.build.mozilla.org/buildapi/reports/waittimes/buildpool?maxb=480&endtime=1470726000&mpb=15&format=json : HTTP Error 500: Internal Server Error SNS was done for a few months, so it's hard to tell when this started. Despite our desire to turn off the wait times emails, the rest of buildapi is still required.
Assignee | ||
Updated•8 years ago
|
Assignee: nobody → aselagea
Assignee | ||
Comment 1•8 years ago
|
||
@coop: any thoughts on how this application is deployed? I'm not able to find out where the reports from https://secure.pub.build.mozilla.org/buildapi/ should get data from.
Flags: needinfo?(coop)
Reporter | ||
Comment 2•8 years ago
|
||
(In reply to Alin Selagea [:aselagea][:buildduty] from comment #1) > @coop: any thoughts on how this application is deployed? I'm not able to > find out where the reports from > https://secure.pub.build.mozilla.org/buildapi/ should get data from. Have you looked at the following docs? https://wiki.mozilla.org/ReleaseEngineering/How_To/Update_a_Releng_Cluster_Site https://wiki.mozilla.org/ReleaseEngineering/BuildAPI https://wiki.mozilla.org/ReleaseEngineering/How_To/Restart_BuildAPI I can't attest to whether those docs are up-to-date or not, but it's a starting point that I hope you've seen. :catlee - you and I had talked about turning off the wait times reports, but it seems all the buildapi reports are missing, i.e. https://secure.pub.build.mozilla.org/buildapi/reports/* Did we actually turn something off, or is this 500 error unrelated?
Flags: needinfo?(coop) → needinfo?(catlee)
Comment 3•8 years ago
|
||
No, I haven't turned anything off yet, I think this is unrelated.
Flags: needinfo?(catlee)
Comment 4•8 years ago
|
||
This is failing with: [Thu Sep 01 11:36:00 2016] [error] [client 10.22.81.211] Error - <type 'exceptions.ImportError'>: No module named gviz_api From /var/log/httpd/buildapi.pvt.build.mozilla.org/error_log on web[12].releng.webapp.scl3.mozilla.com
Assignee | ||
Comment 5•8 years ago
|
||
Thanks for spotting this! I somehow missed the import error as it being logged only in the early PDT morning (when the check for the waittimes report is done) and then zipped. Did a bit more investigation on this and found the following: - gviz_api is the google visualization library and it *is* used by the buildapi reports - according to https://bugzilla.mozilla.org/show_bug.cgi?id=1271661#c6, upgrading the python version from 2.7.3 to 2.7.11 caused issues with the virtual envs on relengwebadmin, so they needed to be rebuilt - gviz_api module is not present on buildapi virtual environments from relengwebadmin nor webheads. We can install it using something like: pip install -U -f https://github.com/google/google-visualization-python/ gviz-api-py
Assignee | ||
Comment 6•8 years ago
|
||
Okay, so installing the module on relengwebadmin and then restarting buildapi fixed the issue and all buildapi reports except https://secure.pub.build.mozilla.org/buildapi/pushes are available again. Note: I used "pip install -U -f https://github.com/google/google-visualization-python/ gviz-api-py", as the command above did not work.
Assignee | ||
Updated•7 years ago
|
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Updated•7 years ago
|
Component: Tools → General
You need to log in
before you can comment on or make changes to this bug.
Description
•