1286546 - deploy rbbz merge (bug 1262548)

Status: RESOLVED FIXED

(MozReview Graveyard :: Infrastructure, defect)

Description

[:glob ✱]

we need to deploy the patch in bug 1262548 to remove rbbz.

Comment 1

[:glob ✱]

to allow me to deploy this outside of peak times, i'll hardhat the service via .htaccess.

note to self: use a rewritecond to allow access from my ip address, and from the zeus status check.
rewrite to a static outage page.

Comment 2

[:glob ✱]

things haven't gone to plan.

i've committed the rebased rbbz changes (after ensuring all tests pass).

1. hardhat review board

my original plan for hardhatting was to use a .htaccess rewriterule to only allow access from my ip address; returning a static html "offline" page to everyone else (this would keep the load balancer from thinking the site of offline).  while this worked without issue in my dev environment, on production this only blocked the static assets; the wsgi app continued to serve requests (even after an |apache graceful|).

so i let moc know that i'm going to break things, and broke reviewboard.wsgi so it returned http/500 (https://reviewboard.mozilla.org/errordocs/500.html).

2. use rb-site to disable the rbbz extension

seemed simple enough, and again something that worked in my dev environment.

here's the log from production:

> # /data/reviewboard/src/reviewboard.mozilla.org/venv/bin/rb-site manage /data/reviewboard/src/reviewboard.mozilla.org/reviewboard list-extensions
> * Name: rbbz
>   Status: enabled
>   ID: rbbz.extension.BugzillaExtension
> (snip)
> # /data/reviewboard/src/reviewboard.mozilla.org/venv/bin/rb-site manage /data/reviewboard/src/reviewboard.mozilla.org/reviewboard disable-extension rbbz
> # /data/reviewboard/src/reviewboard.mozilla.org/venv/bin/rb-site manage /data/reviewboard/src/reviewboard.mozilla.org/reviewboard list-extensions
> * Name: rbbz
>   Status: enabled
>   ID: rbbz.extension.BugzillaExtension
> (snip)

so the "disable-extension" command ran without reporting any errors, yet it didn't disable the extension.  as i hadn't tested the upgrade process with rb in this state (api down, rbbz enabled), i decided not to proceed without further investigation.

Comment 3

[:glob ✱]

disable-extension wants the class, not the name.
ie. rb-site manage /path/ disable-extension rbbz.extension.BugzillaExtension
looks like errors generated by disable-extension are never echoed back to the user.

"install-extension-media" and "collectstatic" both appear to work when the wsgi app is down.

i'll try this again at about the same time tomorrow.  in the unlikely event that we need to do a production deployment today, we can back out the rbbz changes from the tree.

Comment 4

[Gregory Szorc [:gps]]

I believe you also need to run the rb-site commands on both web heads.

In the past where we've had to take the server offline, I stop httpd on all but 1 webhead, do maintenance on the "main" server until things work, repeat on the other webheads, put the main webhead in service, verify everything is working, the put the other webhead in service.

Yes, I've even had to do db migrations on each web head. Why, I don't know. RB can be wonky.

Comment 5

[:glob ✱]

(In reply to Gregory Szorc [:gps] from comment #4)
> I believe you also need to run the rb-site commands on both web heads.

i disabled it on the admin node, they are showing up as disabled when using rb-site on the webheads, so this wasn't required.

after a false start (the deploy code fails if it can't contact httpd to ensure it is drained of all requests), this has been deployed.