1394128 - Can't deploy code to Treeherder while in Maintenance mode

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Infrastructure, enhancement, P1)

Description

[Cameron Dawson [:camd]]

The bin/pre_deploy script includes a call to curl to get the revision:

OLD_REVISION="$(curl --silent --show-error --fail --retry 5 --retry-max-time 15 $SITE_URL/revision.txt)"

However, when the site is in maintenance mode, this call fails, which prevents the code from deploying, but happens AFTER data migrations.  This can cause old code to be running against the new database when maint mode is turned off.

In my case (with bug 1215587) this meant there was a window where API data submission was failing due to the db/code discrepancy.  So data was lost.

We should make it possible to deploy code in maint mode as we'll almost surely need to do this again.

Comment 1

[Ed Morley [:emorley]]

Making the New Relic deploy reporting fatal was intentional when implemented, since we wanted to catch cases were the reporting wasn't working, and the `curl` call uses `--retry 5` so isn't likely going to be affected by transient network issues.

However that is definitely not compatible with maintenance mode, so if it's something we're now going to start using (I think this may be the first time we've used it), then we should allow failures.

Comment 2

[Cameron Dawson [:camd]]

Yeah, I believe this is the first time we've used Heroku maint mode.  I don't know that we'd need to use it often, but it seems like it would be handy to be able to use it.  It would be great if there was a trigger we could loop into when we go out of maintenance mode that we could do that curl call.  Then we could try it during pre_deploy and let if gracefully fail.  Then when main mode ends, try it once more so the value is accurately set.

Perhaps the easiest thing would be to have a command we could run by hand after deploying in maint mode, and document that it's part of the procedure.  

I say all this knowing that you will likely come up with a better idea.  :)

Comment 3

[Ed Morley [:emorley]]

There's `IGNORE_PREDEPLOY_ERRORS` which would have allowed the deploy in this case without having to turn off maintenance mode fwiw.

Comment 4

[GitHub Autolander Bot]

Attachment: [treeherder] mozilla:allow-revision-txt-deploy-failures > mozilla:master

Comment 5

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/bbbaf01c6c7945908a644d03f329e5a68f64f9a3
Bug 1394128 - Heroku: Fail gracefully if revision.txt inaccessible (#2754)

This allows deploys to succeed even if the live version of revision.txt
is not currently accessible, such as when maintenance mode is enabled.