900375 - downtime required to deploy the UserProfile and RequestNagging extension to BMO

Status: RESOLVED FIXED

(bugzilla.mozilla.org :: Infrastructure, defect)

Description

[:glob ✱]

during the last push (bug 900328) a change which adds a column to the profiles table failed.  see my comment on that bug for details.

we'll need to schedule some downtime to repair and deploy the extension:

0. block access to bmo
1. drop the profiles.last_activity_ts column
2. disable the database script/setting which kills long running queries
3. re-enable the UserProfiles extension
4. run checksetup.pl to add last_activity_ts and last_statistics_ts to the profiles table (this should complete quickly)
5. run migration.pl (this will not complete quickly)
6. run update.pl (i hope you like waiting)
7. re-enable long database query killing
8. test
9. unblock access

note - having the profiles.last_activity_ts column in the database won't be causing any issues right now, as bugzilla doesn't know it exists.

Comment 1

[Kendall Libby [:fubar] (he/him)]

Presumably we'll want to run this by CAB, yes?  :ashish, I know you usually handle the bmo pushes; do you want to stick with this one, or have me take it?  Or both, since it'd be my first go around. :)

Comment 2

[:glob ✱]

RequestNagging (bug 892615) will also touch the profiles table, so this should be deployed at the same time.

Comment 3

[Ashish Vijayaram [:ashish]]

(In reply to Kendall Libby [:fubar] from comment #1)
> Presumably we'll want to run this by CAB, yes?  :ashish, I know you usually
> handle the bmo pushes; do you want to stick with this one, or have me take
> it?  Or both, since it'd be my first go around. :)

I don't think we can get this through when :glob is in PST (given he's required to be around). :glob - what is a good time for you? Trying to find an overlap in time with :fubar.

Comment 4

[:glob ✱]

(In reply to Ashish Vijayaram [:ashish] from comment #3)
> I don't think we can get this through when :glob is in PST (given he's
> required to be around). :glob - what is a good time for you? Trying to find
> an overlap in time with :fubar.

my general availability:
gmt+8 : noon - 5pm, 9.30pm -  1am
pdt   :  9pm - 2am, 6.30am - 10am

Comment 5

[Daniel Einspanjer [:dre] [:deinspanjer]]

What will be the impact for a process which accesses a slave copy of the DB directly (i.e. Metrics Bugzilla DWH ETL).  Do we need to stop this process ahead of the maintenance window or will it be okay if it tries to run its standard queries throughout the window?

Comment 6

[:glob ✱]

(In reply to Daniel Einspanjer :dre [:deinspanjer] from comment #5)
> What will be the impact for a process which accesses a slave copy of the DB
> directly (i.e. Metrics Bugzilla DWH ETL).  Do we need to stop this process
> ahead of the maintenance window or will it be okay if it tries to run its
> standard queries throughout the window?

it will be ok to run those queries during this downtime.

Comment 7

[Daniel Einspanjer [:dre] [:deinspanjer]]

Awesome, that takes care of all the worries on my end then.

Comment 8

[Ashish Vijayaram [:ashish]]

(In reply to Byron Jones ‹:glob› from comment #4)
> (In reply to Ashish Vijayaram [:ashish] from comment #3)
> > I don't think we can get this through when :glob is in PST (given he's
> > required to be around). :glob - what is a good time for you? Trying to find
> > an overlap in time with :fubar.
> 
> my general availability:
> gmt+8 : noon - 5pm, 9.30pm -  1am
> pdt   :  9pm - 2am, 6.30am - 10am

I will not be around for this push since I'm on PTO my 25th (24th Pacific). Somebody else will need to cover for me :(

Comment 9

[Sheeri Cabral [:sheeri]]

I'm fine with it all except I don't actually know how to disable the database script/setting which kills long running queries and then re-enable it. I'm sure it's a cron somewhere, but it's not on the db servers as far as I can tell.

Comment 10

[Sheeri Cabral [:sheeri]]

There is no longer a query killer. It has been replaced by a 500 (I think, maybe it's 5 min = 300 seconds?) -second Zeus timeout.

Comment 11

[David Lawrence [:dkl]]

(In reply to Sheeri Cabral [:sheeri] from comment #10)
> There is no longer a query killer. It has been replaced by a 500 (I think,
> maybe it's 5 min = 300 seconds?) -second Zeus timeout.

Thanks. Will there be a way to temporarily disable the Zeus timeout or raise it to some large value during the push?

dkl

Comment 12

[Sheeri Cabral [:sheeri]]

Yes.

Comment 13

[Kendall Libby [:fubar] (he/him)]

pushed!