1088226 - Drain/undrain the treeherder webheads when deploying, to avoid 503s being seen

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Infrastructure, defect, P2)

Description

[Ed Morley [:emorley]]

Currently if someone visits treeherder.m.o during a deploy, they get a server error (iirc; Ryan - could you confirm what you saw the other day?).

We should so some or all of:
* Have a fallback maintenance page if the processes are being restarted
* Rework the deploy script so the amount of time where prod is disrupted is kept to a minimum.

This is partly related to bug 1088204, in that if we stop displaying confusing errors (and so long as the UI recovers/retries), then fixing this bug will mean we can maybe wontfix bug 1088204 (unless we decide that messaging people to let them know that a change occurred, is a good idea so that they can keep an eye out for regressions etc).

Comment 1

[Ed Morley [:emorley]]

Few things we can do:

1) Catch the 503s and display a "Treeherder API unavailable - a deployment may be occurring" instead of "Error retrieving job data"

11:20:09.176 GET 
https://treeherder.mozilla.org/browserid/info/ [HTTP/1.1 503 Service Temporarily Unavailable 169ms]
11:20:09.176 GET 
https://treeherder.mozilla.org/api/repository/ [HTTP/1.1 503 Service Temporarily Unavailable 174ms]
11:20:09.177 GET 
https://treeherder.mozilla.org/api/project/mozilla-central/resultset/ [HTTP/1.1 503 Service Temporarily Unavailable 172ms]
11:20:09.177 GET 
https://treeherder.mozilla.org/api/buildplatform/ [HTTP/1.1 503 Service Temporarily Unavailable 359ms]
11:20:09.178 GET 
https://treeherder.mozilla.org/api/jobtype/ [HTTP/1.1 503 Service Temporarily Unavailable 502ms]
11:20:09.178 GET 
https://treeherder.mozilla.org/api/option/ [HTTP/1.1 503 Service Temporarily Unavailable 511ms]
11:20:09.178 GET 
https://treeherder.mozilla.org/api/optioncollection/ [HTTP/1.1 503 Service Temporarily Unavailable 663ms]

2) Fix:

11:20:09.240 "Error: JSON.parse: unexpected character at line 1 column 1 of the JSON data
fromJson@https://treeherder.mozilla.org/js/index.min-6ad613f4a436a3f228bbde48c01349b0.js:5:5246
resultSetResponseTransformer@https://treeherder.mozilla.org/js/index.min-6ad613f4a436a3f228bbde48c01349b0.js:18:1243
transformData@https://treeherder.mozilla.org/js/index.min-6ad613f4a436a3f228bbde48c01349b0.js:7:8627
transformResponse@https://treeherder.mozilla.org/js/index.min-6ad613f4a436a3f228bbde48c01349b0.js:7:10089
processQueue@https://treeherder.mozilla.org/js/index.min-6ad613f4a436a3f228bbde48c01349b0.js:8:9278
scheduleProcessQueue/<@https://treeherder.mozilla.org/js/index.min-6ad613f4a436a3f228bbde48c01349b0.js:8:9542
$RootScopeProvider/this.$get</Scope.prototype.$eval@https://treeherder.mozilla.org/js/index.min-6ad613f4a436a3f228bbde48c01349b0.js:8:21683
$RootScopeProvider/this.$get</Scope.prototype.$digest@https://treeherder.mozilla.org/js/index.min-6ad613f4a436a3f228bbde48c01349b0.js:8:19506
$RootScopeProvider/this.$get</Scope.prototype.$apply@https://treeherder.mozilla.org/js/index.min-6ad613f4a436a3f228bbde48c01349b0.js:8:22068
done@https://treeherder.mozilla.org/js/index.min-6ad613f4a436a3f228bbde48c01349b0.js:7:13155
completeRequest@https://treeherder.mozilla.org/js/index.min-6ad613f4a436a3f228bbde48c01349b0.js:7:16714
createHttpBackend/</xhr.onload@https://treeherder.mozilla.org/js/index.min-6ad613f4a436a3f228bbde48c01349b0.js:7:17636
"

and

11:20:09.241 "ThResultSetModel" "Error: JSON.parse: unexpected character at line 1 column 1 of the JSON data
fromJson@https://treeherder.mozilla.org/js/index.min-6ad613f4a436a3f228bbde48c01349b0.js:5:5246
resultSetResponseTransformer@https://treeherder.mozilla.org/js/index.min-6ad613f4a436a3f228bbde48c01349b0.js:18:1243
transformData@https://treeherder.mozilla.org/js/index.min-6ad613f4a436a3f228bbde48c01349b0.js:7:8627
transformResponse@https://treeherder.mozilla.org/js/index.min-6ad613f4a436a3f228bbde48c01349b0.js:7:10089
processQueue@https://treeherder.mozilla.org/js/index.min-6ad613f4a436a3f228bbde48c01349b0.js:8:9278
scheduleProcessQueue/<@https://treeherder.mozilla.org/js/index.min-6ad613f4a436a3f228bbde48c01349b0.js:8:9542
$RootScopeProvider/this.$get</Scope.prototype.$eval@https://treeherder.mozilla.org/js/index.min-6ad613f4a436a3f228bbde48c01349b0.js:8:21683
$RootScopeProvider/this.$get</Scope.prototype.$digest@https://treeherder.mozilla.org/js/index.min-6ad613f4a436a3f228bbde48c01349b0.js:8:19506
$RootScopeProvider/this.$get</Scope.prototype.$apply@https://treeherder.mozilla.org/js/index.min-6ad613f4a436a3f228bbde48c01349b0.js:8:22068
done@https://treeherder.mozilla.org/js/index.min-6ad613f4a436a3f228bbde48c01349b0.js:7:13155
completeRequest@https://treeherder.mozilla.org/js/index.min-6ad613f4a436a3f228bbde48c01349b0.js:7:16714
createHttpBackend/</xhr.onload@https://treeherder.mozilla.org/js/index.min-6ad613f4a436a3f228bbde48c01349b0.js:7:17636
"

and 

11:20:09.242 "Error: resultSets is undefined
.getResultSetJobs@https://treeherder.mozilla.org/js/index.min-6ad613f4a436a3f228bbde48c01349b0.js:18:3237
fetchResultSets/<@https://treeherder.mozilla.org/js/index.min-6ad613f4a436a3f228bbde48c01349b0.js:18:17996
processQueue@https://treeherder.mozilla.org/js/index.min-6ad613f4a436a3f228bbde48c01349b0.js:8:9278
scheduleProcessQueue/<@https://treeherder.mozilla.org/js/index.min-6ad613f4a436a3f228bbde48c01349b0.js:8:9542
$RootScopeProvider/this.$get</Scope.prototype.$eval@https://treeherder.mozilla.org/js/index.min-6ad613f4a436a3f228bbde48c01349b0.js:8:21683
$RootScopeProvider/this.$get</Scope.prototype.$digest@https://treeherder.mozilla.org/js/index.min-6ad613f4a436a3f228bbde48c01349b0.js:8:19506
$RootScopeProvider/this.$get</Scope.prototype.$apply@https://treeherder.mozilla.org/js/index.min-6ad613f4a436a3f228bbde48c01349b0.js:8:22068
done@https://treeherder.mozilla.org/js/index.min-6ad613f4a436a3f228bbde48c01349b0.js:7:13155
completeRequest@https://treeherder.mozilla.org/js/index.min-6ad613f4a436a3f228bbde48c01349b0.js:7:16714
createHttpBackend/</xhr.onload@https://treeherder.mozilla.org/js/index.min-6ad613f4a436a3f228bbde48c01349b0.js:7:17636
"

Comment 2

[Ed Morley [:emorley]]

fubar has already partly implemented this in /root/bin/th_functions.sh

@fubar: I did spot one typo - in restartweb() in that file, there is:
ssh -i ${SSHKEY} ${host} supervisorctl restart gunicorn

This needs a s/gunicorn/run_gunicorn/

Comment 3

[Ed Morley [:emorley]]

(In reply to Ed Morley [:edmorley] from comment #2)
> fubar has already partly implemented this in /root/bin/th_functions.sh

I should add: but the code isn't used at present (in case you wondered why we see 503s on next deploy, Ryan)

Comment 4

[Kendall Libby [:fubar] (he/him)]

fixed the typo, thanks for catching it! (was going to add the flow bug but you beat me to it ;-P)

Comment 5

[Kendall Libby [:fubar] (he/him)]

Updated apache config to allow queries from admin node; restart-jobs script now using it.

Comment 6

[Ed Morley [:emorley]]

Awesome - thank you :-)