Closed Bug 1394128 Opened 7 years ago Closed 7 years ago

Can't deploy code to Treeherder while in Maintenance mode


(Tree Management :: Treeherder: Infrastructure, enhancement, P1)



(Not tracked)



(Reporter: camd, Assigned: emorley)



(1 file)

The bin/pre_deploy script includes a call to curl to get the revision:

OLD_REVISION="$(curl --silent --show-error --fail --retry 5 --retry-max-time 15 $SITE_URL/revision.txt)"

However, when the site is in maintenance mode, this call fails, which prevents the code from deploying, but happens AFTER data migrations.  This can cause old code to be running against the new database when maint mode is turned off.

In my case (with bug 1215587) this meant there was a window where API data submission was failing due to the db/code discrepancy.  So data was lost.

We should make it possible to deploy code in maint mode as we'll almost surely need to do this again.
Making the New Relic deploy reporting fatal was intentional when implemented, since we wanted to catch cases were the reporting wasn't working, and the `curl` call uses `--retry 5` so isn't likely going to be affected by transient network issues.

However that is definitely not compatible with maintenance mode, so if it's something we're now going to start using (I think this may be the first time we've used it), then we should allow failures.
Assignee: nobody → emorley
Priority: -- → P1
Yeah, I believe this is the first time we've used Heroku maint mode.  I don't know that we'd need to use it often, but it seems like it would be handy to be able to use it.  It would be great if there was a trigger we could loop into when we go out of maintenance mode that we could do that curl call.  Then we could try it during pre_deploy and let if gracefully fail.  Then when main mode ends, try it once more so the value is accurately set.

Perhaps the easiest thing would be to have a command we could run by hand after deploying in maint mode, and document that it's part of the procedure.  

I say all this knowing that you will likely come up with a better idea.  :)
There's `IGNORE_PREDEPLOY_ERRORS` which would have allowed the deploy in this case without having to turn off maintenance mode fwiw.
Attachment #8904308 - Flags: review?(cdawson)
Attachment #8904308 - Flags: review?(cdawson) → review+
Commit pushed to master at
Bug 1394128 - Heroku: Fail gracefully if revision.txt inaccessible (#2754)

This allows deploys to succeed even if the live version of revision.txt
is not currently accessible, such as when maintenance mode is enabled.
Closed: 7 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.