617168 - Push SUMO 2.4 on Tuesday, December 14th @ 4pm PST

Status: VERIFIED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[James Socol [:jsocol, :james]]

SUMO 2.4 is our last "moving-functionality-into-python" milestone. It involves a small svn update to pick up some .htaccess changes and a solid (though tiny compared to 2.3) git update with schema changes.

There is a data migration that we will try to precalculate what we can, but will largely need to run on production to pick up users who register up to the push. So far it looks like it takes around 45 minutes (hopefully less if we can get parts done early).

Full steps coming.

Comment 1

[Shyam Mani [:fox2mike]]

Alright, I'll run this. Put up steps on the bug and we'll take it from there.

Comment 2

[James Socol [:jsocol, :james]]

Steps!

git/python stuff:
* git checkout 2.4
* schematic migrations
* git submodule update --init
* pushd vendor && git pull origin master && git submodule update --init && popd
* pushd locale && ./compile-mo.sh . && popd
* manage.py compress_assets
* unzip khan:/home/jsocol/avatars.??? into /media/uploads/avatars
* manage.py migrate_avatars   (maybe 45 minutes)

svn/php stuff:
* svn switch to https://svn.mozilla.org/projects/sumo/tags/1.5/1.5.8_r79069_20101214
** Tagged from trunk/, not trunk/webroot/
* webroot/htaccess.sh


We believe that's it. We'll post any new things we think of.

SUMO 2.4, \o/

Comment 3

[Matthew Middleton (:zzxc)]

After the push, the following needs to happen to unbreak Live Chat:

1) Ensure that (only) the version of the sumo auth plugin containing updated queries is installed, by installing the RPM on dm-sumotools01 at /home/zzxc/sumo24/openfire-3.6.4-1.i386.rpm
2) Change migration status in the openfire administration panel to "2".  (This will permanently deactivate all non-migrated accounts)
3) Restart Openfire
4) Test that helpers can log in again

Comment 4

[Matthew Middleton (:zzxc)]

Openfire login is now working, but bug 619243 was discovered that is causing some accounts to be rejected.  The RPM has been updated to include the fix on that bug, which should be installed once the accounts have been re-tested with the stage database:

sm-chat01.mozilla.org:/home/zzxc/sumo24/openfire-3.6.4-1.i386.rpm

Comment 5

[Shyam Mani [:fox2mike]]

(In reply to comment #2)
> Steps!
> 
> git/python stuff:
> * git checkout 2.4

[root@mradm02 prod]# git checkout 2.4
Previous HEAD position was f6b0a38... Updating locale.
HEAD is now at 3844d67... [bug 618601] Fix image deletion error in gallery.

> * schematic migrations

[root@mradm02 prod]# schematic migrations/
Running migration 63:
-- Set user passwords from Tiki.
UPDATE auth_user a SET password = (SELECT hash FROM users_users u WHERE u.userId = a.id);

That took 5.95 seconds
################################################## 

Running migration 64:
CREATE TABLE `users_profile` (
  `user_id` int(11) NOT NULL,
  `name` varchar(255) NOT NULL,
  `public_email` tinyint(1) NOT NULL,
  `avatar` varchar(250) DEFAULT NULL,
  `bio` longtext,
  `website` varchar(255) DEFAULT NULL,
  `twitter` varchar(255) DEFAULT NULL,
  `facebook` varchar(255) DEFAULT NULL,
  `irc_handle` varchar(255) DEFAULT NULL,
  `timezone` varchar(100) DEFAULT NULL,
  `country` varchar(2) DEFAULT NULL,
  `city` varchar(255) DEFAULT NULL,
  `livechat_id` varchar(255) DEFAULT NULL,
  PRIMARY KEY (`user_id`),
  CONSTRAINT `user_id_refs_id_21617f27` FOREIGN KEY (`user_id`) REFERENCES `auth_user` (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

That took 0.04 seconds
################################################## 

Running migration 65:
CREATE TABLE `users_registrationprofile` (
    `id` integer AUTO_INCREMENT NOT NULL PRIMARY KEY,
    `user_id` integer NOT NULL UNIQUE,
    `activation_key` varchar(40) NOT NULL
);
ALTER TABLE `users_registrationprofile` ADD CONSTRAINT `user_id_refs_id_e9e30776` FOREIGN KEY (`user_id`) REFERENCES `auth_user` (`id`);

That took 0.07 seconds
################################################## 

Running migration 66:
-- Display or nickname isn't required.
ALTER TABLE `users_profile`
        MODIFY `name` varchar(255);

That took 0.11 seconds
################################################## 

Running migration 67:
UPDATE wiki_document SET is_localizable=FALSE WHERE html LIKE '<p>REDIRECT <a %';

That took 0.07 seconds
################################################## 

Running migration 68:
ALTER TABLE users_users
    MODIFY login varchar(30) COLLATE utf8_unicode_ci;

INSERT IGNORE INTO auth_group(id, name)
    VALUES (NULL, 'Live Chat helpers');

INSERT IGNORE INTO auth_user_groups (user_id, group_id)
  SELECT auth_user.id, (
        SELECT id FROM auth_group
        WHERE name = 'Live Chat helpers'
    )
  FROM auth_user JOIN users_users ON
          (auth_user.username = users_users.login),
       users_usergroups
  WHERE users_users.userId = users_usergroups.userId
        AND users_usergroups.groupName = 'Live Chat helpers';

That took 23.24 seconds
################################################## 

Running migration 69:
ALTER TABLE customercare_cannedcategory ENGINE=InnoDB;
ALTER TABLE customercare_cannedresponse ENGINE=InnoDB;
ALTER TABLE customercare_categorymembership ENGINE=InnoDB;
ALTER TABLE customercare_tweet ENGINE=InnoDB;
ALTER TABLE flagit_flaggedobject ENGINE=InnoDB;
ALTER TABLE schema_version ENGINE=InnoDB;
ALTER TABLE upload_imageattachment ENGINE=InnoDB;

That took 0.68 seconds
##################################################

> * git submodule update --init

[root@mradm02 vendor]# git submodule update --init
Submodule 'src/django-timezones' (git://github.com/brosner/django-timezones.git) registered for path 'src/django-timezones'
Cloning into src/django-timezones...
remote: Counting objects: 269, done.
remote: Compressing objects: 100% (143/143), done.
remote: Total 269 (delta 140), reused 233 (delta 112)
Receiving objects: 100% (269/269), 35.15 KiB, done.
Resolving deltas: 100% (140/140), done.
Submodule path 'src/django-timezones': checked out '2b903a38da1ff9df4b2aba8e4f5429d967f73881'

> * pushd locale && ./compile-mo.sh . && popd

Done

> * manage.py compress_assets

Also ran :

[root@mradm02 prod]# git submodule update --init media/global
Cloning into media/global...
remote: Counting objects: 51, done.
remote: Compressing objects: 100% (51/51), done.
remote: Total 51 (delta 26), reused 0 (delta 0)
Receiving objects: 100% (51/51), 18.54 KiB, done.
Resolving deltas: 100% (26/26), done.
Submodule path 'media/global': checked out '956819fe801b525b751d092ad9351234a55262db'
[root@mradm02 prod]# git submodule update --init templates/global
Cloning into templates/global...
remote: Counting objects: 51, done.
remote: Compressing objects: 100% (51/51), done.
remote: Total 51 (delta 26), reused 0 (delta 0)
Receiving objects: 100% (51/51), 18.54 KiB, done.
Resolving deltas: 100% (26/26), done.
Submodule path 'templates/global': checked out '956819fe801b525b751d092ad9351234a55262db'

to prevent missing file errors.

> * unzip khan:/home/jsocol/avatars.??? into /media/uploads/avatars

[root@mradm02 uploads]# time tar jxf /root/avatars.tar.bz2 

real    8m53.488s
user    0m14.333s
sys     0m13.855s

> * manage.py migrate_avatars   (maybe 45 minutes)

Done!

Took

real    113m35.320s
user    28m55.790s
sys     1m28.667s

> 
> svn/php stuff:
> * svn switch to
> https://svn.mozilla.org/projects/sumo/tags/1.5/1.5.8_r79069_20101214
> ** Tagged from trunk/, not trunk/webroot/

[root@mradm02 support.mozilla.com]# svn switch https://svn.mozilla.org/projects/sumo/tags/1.5/1.5.8_r79069_20101214
U    webroot/htaccess.dist
Updated to revision 79410.

> * webroot/htaccess.sh

[root@mradm02 webroot]# ./htaccess.sh 
activating ./pics/.htaccess
activating ./templates_c/.htaccess
activating ./styles/.htaccess
activating ./whelp/.htaccess
activating ./db/.htaccess
activating ./.htaccess
activating ./css/.htaccess
activating ./lib/fckeditor_tiki/.htaccess
activating ./lib/.htaccess
activating ./lib/fckeditor/.htaccess
activating ./dump/.htaccess
activating ./popups/.htaccess
activating ./backups/.htaccess
activating ./doc/.htaccess
activating ./js/.htaccess
activating ./images/.htaccess
activating ./files/.htaccess
activating ./temp/.htaccess
activating ./lang/.htaccess
activating ./templates/.htaccess
activating ./components/.htaccess
activating ./modules/.htaccess
activating ./tikimovies/.htaccess
activating ./games/.htaccess
activating ./bin/.htaccess
 
> 
> We believe that's it. We'll post any new things we think of.
> 
> SUMO 2.4, \o/

Comment 6

[Shyam Mani [:fox2mike]]

(In reply to comment #4)
> Openfire login is now working, but bug 619243 was discovered that is causing
> some accounts to be rejected.  The RPM has been updated to include the fix on
> that bug, which should be installed once the accounts have been re-tested with
> the stage database:
> 
> sm-chat01.mozilla.org:/home/zzxc/sumo24/openfire-3.6.4-1.i386.rpm

Pulled the RPM and pushed it into dm-chat01.

Calling this done.

Comment 7

[James Socol [:jsocol, :james]]

Going to go ahead and fix this now: http://jsocol.pastebin.mozilla.org/888513

Comment 8

[Shyam Mani [:fox2mike]]

mysql> UPDATE users_profile SET avatar = REPLACE(avatar, '/data/sumo_python/src/prod/media/', '');
Query OK, 153276 rows affected (5.79 sec)
Rows matched: 155579  Changed: 153276  Warnings: 0

Done.

Comment 9

[Rebecca Billings [:rbillings]]

Verified this completed last night.