1293659 - buildapi returning "HTTP Error 500: Internal Server Error"

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Chris Cooper [:coop] (he/him)]

Buildapi has been returning 500 errors for the past little while. Here's the SNS alert we get in IRC:

09:04 <relengbot> [sns alert] Aug 09 06:02:42 relengwebadm.private.scl3.mozilla.com buildapi_waittimes: Error: fetching wait times from location http://buildapi.pvt.build.mozilla.org/buildapi/reports/waittimes/trybuildpool?maxb=480&endtime=1470726000&mpb=15&format=json : HTTP Error 500: Internal Server Error
09:06 <relengbot> [sns alert] Aug 09 06:04:23 relengwebadm.private.scl3.mozilla.com buildapi_waittimes: Error: fetching wait times from location http://buildapi.pvt.build.mozilla.org/buildapi/reports/waittimes/testpool?maxb=480&endtime=1470726000&mpb=15&format=json : HTTP Error 500: Internal Server Error
09:06 <relengbot> [sns alert] Aug 09 06:06:04 relengwebadm.private.scl3.mozilla.com buildapi_waittimes: Error: fetching wait times from location http://buildapi.pvt.build.mozilla.org/buildapi/reports/waittimes/buildpool?maxb=480&endtime=1470726000&mpb=15&format=json : HTTP Error 500: Internal Server Error

SNS was done for a few months, so it's hard to tell when this started. Despite our desire to turn off the wait times emails, the rest of buildapi is still required.

Comment 1

[Alin Selagea [:aselagea]]

@coop: any thoughts on how this application is deployed? I'm not able to find out where the reports from https://secure.pub.build.mozilla.org/buildapi/ should get data from.

Comment 2

[Chris Cooper [:coop] (he/him)]

(In reply to Alin Selagea [:aselagea][:buildduty] from comment #1)
> @coop: any thoughts on how this application is deployed? I'm not able to
> find out where the reports from
> https://secure.pub.build.mozilla.org/buildapi/ should get data from.

Have you looked at the following docs?

https://wiki.mozilla.org/ReleaseEngineering/How_To/Update_a_Releng_Cluster_Site
https://wiki.mozilla.org/ReleaseEngineering/BuildAPI
https://wiki.mozilla.org/ReleaseEngineering/How_To/Restart_BuildAPI

I can't attest to whether those docs are up-to-date or not, but it's a starting point that I hope you've seen.

:catlee - you and I had talked about turning off the wait times reports, but it seems all the buildapi reports are missing, i.e.
https://secure.pub.build.mozilla.org/buildapi/reports/* 

Did we actually turn something off, or is this 500 error unrelated?

Comment 3

[Chris AtLee [:catlee]]

No, I haven't turned anything off yet, I think this is unrelated.

Comment 4

[Chris AtLee [:catlee]]

This is failing with:
[Thu Sep 01 11:36:00 2016] [error] [client 10.22.81.211] Error - <type 'exceptions.ImportError'>: No module named gviz_api

From /var/log/httpd/buildapi.pvt.build.mozilla.org/error_log on web[12].releng.webapp.scl3.mozilla.com

Comment 5

[Alin Selagea [:aselagea]]

Thanks for spotting this! I somehow missed the import error as it being logged only in the early PDT morning (when the check for the waittimes report is done) and then zipped.

Did a bit more investigation on this and found the following:
    -  gviz_api is the google visualization library and it *is* used by the buildapi reports
    -  according to https://bugzilla.mozilla.org/show_bug.cgi?id=1271661#c6, upgrading the python version from 2.7.3 to 2.7.11 caused issues with the virtual envs on relengwebadmin, so they needed to be rebuilt
    - gviz_api module is not present on buildapi virtual environments from relengwebadmin nor webheads. We can install it using something like:
       pip install -U -f https://github.com/google/google-visualization-python/ gviz-api-py

Comment 6

[Alin Selagea [:aselagea]]

Okay, so installing the module on relengwebadmin and then restarting buildapi fixed the issue and all buildapi reports except https://secure.pub.build.mozilla.org/buildapi/pushes are available again.

Note: I used "pip install -U -f https://github.com/google/google-visualization-python/ gviz-api-py", as the command above did not work.