Last Comment Bug 587824 - Push SUMO 2.2.2 and SUMO mobile on Thursday, August 19
: Push SUMO 2.2.2 and SUMO mobile on Thursday, August 19
Status: VERIFIED FIXED
08/19/2010 @ 4pm
:
Product: Infrastructure & Operations
Classification: Other
Component: WebOps: Other (show other bugs)
: other
: All Other
: -- minor (vote)
: ---
Assigned To: Jeremy Orem [:oremj]
: matthew zeier [:mrz]
Mentors:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2010-08-16 15:16 PDT by Paul Craciunoiu [:paulc]
Modified: 2013-10-09 10:29 PDT (History)
8 users (show)
mzeier: needs‑downtime+
See Also:
Due Date:
QA Whiteboard:
Iteration: ---
Points: ---
Cab Review: ServiceNow Change Request (use flag)


Attachments

Description Paul Craciunoiu [:paulc] 2010-08-16 15:16:03 PDT
This is a smaller release than 2.2.1, with the important difference that we also need to push a minor mobile SUMO change, so I figured it's OK to have it in the same push bug.

Steps (we'll post if this is missing any):

mobile.sumo:
* svn up
* flush caches

desktop.sumo:
* git checkout 2.2.2
* schematic migrations/
Comment 1 Rebecca Billings [:rbillings] 2010-08-17 09:21:34 PDT
The push meeting is scheduled for 1pm West Coast time- is it possible to move this back from 7pm [what I see on the push bug whiteboard]? I am not available for a 7pm push Thurs night as I'm not going to be home- but I could do it in the afternoon.
Comment 2 matthew zeier [:mrz] 2010-08-17 09:28:17 PDT
No.  The downtime windows are Tues & Thurs from 7-11pm.  On occasion we've started as early as 4pm but 1pm affects to many time zones.
Comment 3 Paul Craciunoiu [:paulc] 2010-08-17 09:46:16 PDT
We don't expect any downtime for this push. The schematic migrations we have to run are adding some permissions, but those are not required for the site to run.

If 1pm is too early, Rebecca: does 4pm work?
Comment 4 Paul Craciunoiu [:paulc] 2010-08-17 09:48:00 PDT
(In reply to comment #3)
> If 1pm is too early, Rebecca: does 4pm work?
Nevermind. 4pm is AMO :)
Comment 5 matthew zeier [:mrz] 2010-08-17 09:59:57 PDT
Last SUMO push didn't go smooth so I'm not willing to do this outside of a normal push window.  Also need to make sure we have the right people online incase something does go wrong.
Comment 6 Paul Craciunoiu [:paulc] 2010-08-17 10:09:01 PDT
(In reply to comment #5)
> Last SUMO push didn't go smooth so I'm not willing to do this outside of a
> normal push window.  Also need to make sure we have the right people online
> incase something does go wrong.
Fair point. Is there any way we can do tomorrow (Wed) instead? Anytime after 2pm?
Comment 7 Paul Craciunoiu [:paulc] 2010-08-17 14:09:16 PDT
Talked to mrz, moved to tomorrow @ 4pm.
Comment 8 matthew zeier [:mrz] 2010-08-17 21:28:55 PDT
Paul, we talked but why can't this wait till next Tuesday's window?
Comment 9 Paul Craciunoiu [:paulc] 2010-08-18 10:12:32 PDT
James scheduled this push for tomorrow at 1pm a week ago. Pushing next week puts us behind schedule. We plan to have another push this month.

We've discussed our availability and we cannot do tomorrow at 7pm because neither Rebecca (our QA lead) nor I can be here. We can do today, or tomorrow at 1pm (as originally scheduled). We would much rather push this week, so - if neither of those work, how can we get this out this week?

If it helps - as opposed to our previous pushes, this one doesn't have any new requirements to install (which were usually the problem), so the risks are lower.


As a side note, is pushing today a problem because of beta 4? (and isn't that going out next week?)
Comment 10 matthew zeier [:mrz] 2010-08-18 10:26:49 PDT
Regardless of what was originally requested, downtime windows are Tuesday/Thursday with any other window an emergency push.
Comment 11 Rebecca Billings [:rbillings] 2010-08-18 10:40:34 PDT
We released 2.2 on Wed Aug 4th- and it wasn't an emergency push. Is the schedule for when releases are available written somewhere?

It seems like there is a better process possible. If the Thurs at 1pm window wasn't acceptable it should have come up earlier. Pushing out until next week just puts everyone behind.
Comment 12 David Tenser [:djst] 2010-08-18 10:54:15 PDT
mrz, moving this to next week would really slow us down -- if at all possible, can we please do this push tomorrow at 4pm like Paul is requesting?

As discussed in the 1.5.4/2.0 postmortem, we want to do SUMO pushes earlier on the day to ensure we don't end up in similar situations again. As long as push windows are in the evenings, we can't really use them for our pushes.
Comment 13 matthew zeier [:mrz] 2010-08-18 11:23:13 PDT
(In reply to comment #12)
> mrz, moving this to next week would really slow us down -- if at all possible,
> can we please do this push tomorrow at 4pm like Paul is requesting?

Thursday @ 4pm is fine. 

> As discussed in the 1.5.4/2.0 postmortem, we want to do SUMO pushes earlier on
> the day to ensure we don't end up in similar situations again. As long as push
> windows are in the evenings, we can't really use them for our pushes.

1pm is too early.  The trough -starts- at 2pm Pacific and isn't at the bottom until 6pm which is why the normal windows are 7-11pm.

1pm affects too many users.
Comment 14 James Socol [:jsocol, :james] 2010-08-18 11:40:23 PDT
1pm was my fault: I inadvertently specified 4pm Eastern instead of 4pm Pacific.

mrz, w/r/t having the right people online, 7pm PDT windows start at 10pm my time, making it difficult to be coherent and useful. Both AMO and SUMO have been pushing at 4pm Pacific for the past couple of months and it seems to be working much better.

I'll email infra@ and webdev@.
Comment 15 Paul Craciunoiu [:paulc] 2010-08-18 12:30:33 PDT
So is this happening today or tomorrow? (either would have to be at 4pm)

Looks like AMO's not pushing tomorrow?
Comment 16 Paul Craciunoiu [:paulc] 2010-08-18 17:05:23 PDT
Adding this to the steps to save us from going through bug 588469 again:

* restart celery
Comment 17 David Tenser [:djst] 2010-08-19 04:31:01 PDT
(In reply to comment #15)
> So is this happening today or tomorrow? (either would have to be at 4pm)

Thursday 19 August @ 4pm PDT.
Comment 18 Jeremy Orem [:oremj] 2010-08-19 16:11:14 PDT
[root@mradm02 mobile.support.mozilla.com]# svn up
U    webroot/tiki-register.php
U    webroot/lib/userslib.php
U    webroot/templates/styles/mozkb/tiki-register.tpl


[root@mradm02 prod]# git fetch
remote: Counting objects: 221, done.
remote: Compressing objects: 100% (145/145), done.
remote: Total 152 (delta 94), reused 1 (delta 0)
Receiving objects: 100% (152/152), 39.16 KiB, done.
Resolving deltas: 100% (94/94), completed with 39 local objects.
From http://github.com/jsocol/kitsune
   5536aca..65bd879  development -> origin/development
   b76cd10..4197f60  master     -> origin/master
 * [new branch]      wiki       -> origin/wiki
From http://github.com/jsocol/kitsune
 * [new tag]         2.2.2      -> 2.2.2
[root@mradm02 prod]# git checkout 2.2.2
Previous HEAD position was b76cd10... Merge branch 'development'
HEAD is now at 67a4192... [583106] Introduce 2 new permissions--view_in_forum and post_in_forum--making it possible to have hidden-to-the-public and read-only forums.



Running migration 30:
INSERT INTO authority_permission
    VALUES 
        (NULL,'forums_forum.thread_move_forum',
            (select id from django_content_type where app_label='forums' and model='forum'),1,NULL,1,47963,1,
            '2010-08-10 10:37:22','2010-08-10 10:39:57'),
        (NULL,'forums_forum.thread_move_forum',
            (select id from django_content_type where app_label='forums' and model='forum'),2,NULL,1,47963,1,
            '2010-08-10 10:37:22','2010-08-10 10:39:57'),
        (NULL,'forums_forum.thread_move_forum',
            (select id from django_content_type where app_label='forums' and model='forum'),3,NULL,1,47963,1,
            '2010-08-10 10:37:22','2010-08-10 10:39:57');

That took 0.07 seconds
################################################## 



Running migration 31:
SET @ct = (SELECT id FROM django_content_type WHERE app_label='forums' AND model='forum');
INSERT INTO auth_permission (name, content_type_id, codename) VALUES
    ('Can view restricted forums', @ct, 'view_in_forum'),
    ('Can post in restricted forums', @ct, 'post_in_forum');

That took 0.17 seconds
##################################################

Note You need to log in before you can comment on or make changes to this bug.