Last Comment Bug 614034 - Push SUMO 2.3 on Tuesday, 30 November @ 4pm PT
: Push SUMO 2.3 on Tuesday, 30 November @ 4pm PT
Status: VERIFIED FIXED
11/30/2010 @ 4pm
:
Product: mozilla.org Graveyard
Classification: Graveyard
Component: Server Operations (show other bugs)
: other
: All Other
: -- normal (vote)
: ---
Assigned To: Jeremy Orem [:oremj]
: matthew zeier [:mrz]
Mentors:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2010-11-22 11:59 PST by James Socol [:jsocol, :james]
Modified: 2015-03-12 08:17 PDT (History)
8 users (show)
mzeier: needs‑downtime+
See Also:
QA Whiteboard:
Iteration: ---
Points: ---


Attachments

Description James Socol [:jsocol, :james] 2010-11-22 11:59:32 PST
Please push SUMO 2.3 on Tuesday, 30 Nov, @ 4pm PT.

If everything goes well we're looking at around 1 hr of downtime. We should plan for 2 hours in case there are issues and we need to roll back.

Full steps coming but very similar to bug 612670 comment 0.

We're making database changes. We should stop replication on a slave so we have a backup in case of emergency.
Comment 1 Shyam Mani [:fox2mike] 2010-11-22 22:08:55 PST
zomgggggg I can do this :p I'll be there!
Comment 2 Paul Craciunoiu [:paulc] 2010-11-24 10:21:22 PST
Adding a note here with only the stuff I'll need for data migration:

* a copy of the folder webroot/img/wiki_up from svn (lots of images should be there). You can zip this up and put it in my home folder on khan, as images.zip should be fine.
* a copy of the production videos from our CDN (I can get this myself though)
* a database dump from after Sunday night (Nov 28th) with at least these tables:
auth_user
django_content_type
tiki_category_objects
tiki_content
tiki_freetagged_objects
tiki_objects
tiki_pages
tiki_programmed_content
tiki_translated_objects


That should cover everything pre-push. Thanks Shyam!
Comment 3 James Socol [:jsocol, :james] 2010-11-30 07:59:46 PST
Here we go here we go! Push tasks.

Prelim stuff:
* Get the video CDN set up. (all set, aiui)
* Make sure the server that will run the crons can talk to this host/port:
  https://chat-support.mozilla.com:9091

Network stuff:
* Not sure when, but we want to flip over to the ZLB for the DB connections. Justdave knows more.

Outage page!

Catastrophe plan!:
* Stop replication on a slave so we can use it as a backup in case of emergency.

SVN:
* switch to svn.mozilla.org/projects/sumo/tags/1.5/1.5.7_r77930_20101129/
* run webroot/htaccess.sh

Git:
* Kill celery dead.
* switch to tag 2.3
* find . -name "*.pyc" -exec rm -rf {} \;
* pushd vendor && git pull origin master && git submodule update --init && popd
* git submodule update --init
* pushd locale && git pull origin master && ./compile-mo.sh . && popd
* `schematic migrations`
** `schematic -v migrations` should say "36" before and "61" after
* `./manage.py compress_assets`
* `./manage.py update_product_details`

Wiki import:
* After `schematic migrations`!
* Import khan:/home/pcraciunuio/migration.sql
* Unzip /home/pcraciunoiu/gallery.zip into /media/uploads (should end up with a /media/uploads/gallery/ with images and videos subdirectories).

Crons:
* `./manage.py update_product_details` maybe every 6 hours
* `./manage.py cron collect_tweets` every 1 minute
* `./manage.py cron get_queue_status` every 1 minute
* `./manage.py cron calculate_related_documents` every 4 hours
* (We don't care about normal output of any of these.)

Settings:
* GALLERY_VIDEO_URL = (based on CDN)
* WIKI_REBUILD_ON_DEMAND = True (we'll disable if we have to)

Sphinx:
* Update configs from 2.3 tag.
* Reindex.

Celeryd:
* Start 'er up!

I think that's everything, including the two issues we ran into during our dry run with Jeremy. Hopefully he'll be around to provide sage advice from that experience.

Post-script: If everything's good we should let that slave catch up and get back in the pool.
Comment 4 Kadir Topal [:atopal] 2010-11-30 08:15:03 PST
Woohooo!

Okay, sorry for the bugspam, but this is the culmination of 9 month of work, basically since I started at Mozilla. So awesome to see it all come together, huge thanks to everyone contributing to it! Here's hoping for a smooth migration :)
Comment 5 Paul Craciunoiu [:paulc] 2010-11-30 09:18:28 PST
(In reply to comment #3)
> * Import khan:/home/pcraciunuio/migration.sql
Typo here, it's /home/pcraciunoiu/migration.sql
> * Unzip /home/pcraciunoiu/gallery.zip into /media/uploads (should end up with a
> /media/uploads/gallery/ with images and videos subdirectories).
This one is correct though :)

> Crons:
> * `./manage.py update_product_details` maybe every 6 hours
> * `./manage.py cron collect_tweets` every 1 minute
> * `./manage.py cron get_queue_status` every 1 minute
> * `./manage.py cron calculate_related_documents` every 4 hours
> * (We don't care about normal output of any of these.)
But it would be nice to be notified if they fail, right?

Looking good!
Comment 6 James Socol [:jsocol, :james] 2010-11-30 10:55:43 PST
(In reply to comment #5)
> > * (We don't care about normal output of any of these.)
> But it would be nice to be notified if they fail, right?

Jabba just set up cron-sumo@m.c as an alias for these things. Easily filterable. Yeah, stdout we don't care but stderr we do.
Comment 7 Shyam Mani [:fox2mike] 2010-11-30 11:07:41 PST
Jeremy will be running this.
Comment 8 James Socol [:jsocol, :james] 2010-11-30 12:20:55 PST
After unzipping the gallery images we need to make sure that all the relevant folders are still writable by the web server.
Comment 9 Jeremy Orem [:oremj] 2010-11-30 16:28:29 PST
[root@mradm02 support.mozilla.com]# svn info
Path: .
URL: https://svn.mozilla.org/projects/sumo/tags/1.5/1.5.7_r77930_20101129
Repository Root: https://svn.mozilla.org
Repository UUID: 4eb1ac78-321c-0410-a911-ec516a8615a5
Revision: 78497
Node Kind: directory
Schedule: normal
Last Changed Author: jsocol@mozilla.com
Last Changed Rev: 78375
Last Changed Date: 2010-11-29 09:20:40 -0800 (Mon, 29 Nov 2010)

activating ./webroot/pics/.htaccess
activating ./webroot/templates_c/.htaccess
activating ./webroot/styles/.htaccess
activating ./webroot/whelp/.htaccess
activating ./webroot/db/.htaccess
activating ./webroot/.htaccess
activating ./webroot/css/.htaccess
activating ./webroot/lib/fckeditor_tiki/.htaccess
activating ./webroot/lib/.htaccess
activating ./webroot/lib/fckeditor/.htaccess
activating ./webroot/dump.svn/.htaccess
activating ./webroot/img.svn/.htaccess
activating ./webroot/popups/.htaccess
activating ./webroot/backups/.htaccess
activating ./webroot/doc/.htaccess
activating ./webroot/img-local/.htaccess
activating ./webroot/js/.htaccess
activating ./webroot/images/.htaccess
activating ./webroot/files/.htaccess
activating ./webroot/temp/.htaccess
activating ./webroot/lang/.htaccess
activating ./webroot/templates/.htaccess
activating ./webroot/components/.htaccess
activating ./webroot/img.old/.htaccess
activating ./webroot/modules/.htaccess
activating ./webroot/tikimovies/.htaccess
activating ./webroot/games/.htaccess
activating ./webroot/bin/.htaccess
Comment 10 Jeremy Orem [:oremj] 2010-11-30 16:33:14 PST
[root@mradm02 prod]# git fetch
remote: Counting objects: 2014, done.
remote: Compressing objects: 100% (628/628), done.
remote: Total 1848 (delta 1346), reused 1564 (delta 1128)
Receiving objects: 100% (1848/1848), 489.78 KiB | 538 KiB/s, done.
Resolving deltas: 100% (1346/1346), completed with 104 local objects.
From http://github.com/jsocol/kitsune
 * [new branch]      2.3.x      -> origin/2.3.x
 * [new branch]      fix-rebuild -> origin/fix-rebuild
   2c54c56..cad9ffa  master     -> origin/master
 * [new tag]         2.3        -> 2.3
[root@mradm02 prod]# git checkout 2.3
M	locale
Previous HEAD position was 8d9501e... Make Army of Awesome page validate. Bug 608086.
HEAD is now at fa1eb9f... Update locales.

[root@mradm02 vendor]# git pull origin master && git submodule update --init 
remote: Counting objects: 23, done.
remote: Compressing objects: 100% (21/21), done.
remote: Total 21 (delta 14), reused 0 (delta 0)
Unpacking objects: 100% (21/21), done.
From git://github.com/jsocol/kitsune-lib
 * branch            master     -> FETCH_HEAD
Updating 382a9f2..0c81413
Fast-forward
 src/bleach        |    2 +-
 src/django-csp    |    2 +-
 src/jingo-minify  |    2 +-
 src/py-wikimarkup |    2 +-
 src/schematic     |    2 +-
 5 files changed, 5 insertions(+), 5 deletions(-)
remote: Counting objects: 15, done.
remote: Compressing objects: 100% (9/9), done.
remote: Total 9 (delta 6), reused 0 (delta 0)
Unpacking objects: 100% (9/9), done.
From git://github.com/jsocol/bleach
   d1b2a25..b856582  master     -> origin/master
Submodule path 'src/bleach': checked out 'b85658206b93cc928f32e90bfcc2dfae3711ea95'
Submodule path 'src/django-csp': checked out '5c5f5a6b55fb78e99db939c79f0f0d8003a62c98'
remote: Counting objects: 11, done.
remote: Compressing objects: 100% (6/6), done.
remote: Total 6 (delta 3), reused 0 (delta 0)
Unpacking objects: 100% (6/6), done.
From git://github.com/jsocol/jingo-minify
   126e1d6..95f24e9  master     -> origin/master
Submodule path 'src/jingo-minify': checked out '95f24e9ecdbeb5d1d88029896b3d53ae8828e533'
remote: Counting objects: 21, done.
remote: Compressing objects: 100% (17/17), done.
remote: Total 17 (delta 9), reused 0 (delta 0)
Unpacking objects: 100% (17/17), done.
From git://github.com/pcraciunoiu/py-wikimarkup
 * [new branch]      add-section -> origin/add-section
   cc06e6d..4189367  master     -> origin/master
Submodule path 'src/py-wikimarkup': checked out '418936741d9bba1e6326716428a25e7223e58709'
remote: Counting objects: 7, done.
remote: Compressing objects: 100% (4/4), done.
Unpacking objects: 100% (4/4), done.
remote: Total 4 (delta 2), reused 0 (delta 0)
From git://github.com/jbalogh/schematic
   144134c..e749978  master     -> origin/master
Submodule path 'src/schematic': checked out 'e749978cc06bda8ba655faf8d672ae270e25b5f5'
[root@mradm02 vendor]# cd ..
[root@mradm02 prod]# git submodule update --init
cdremote: Counting objects: 1697, done.
remote: Compressing objects: 100% (797/797), done.
remote: Total 1604 (delta 805), reused 933 (delta 421)
Receiving objects: 100% (1604/1604), 1.09 MiB | 1.29 MiB/s, done.
Resolving deltas: 100% (805/805), completed with 1 local objects.
From git://github.com/jsocol/sumo-locales
   07aa01a..c177774  master     -> origin/master
Submodule path 'locale': checked out '3f9c89fe6a91bcc21cfca8bceeb58b9bd23b7eb0'
[root@mradm02 prod]# cd locale/
[root@mradm02 locale]# git pull origin master && git submodule update --init 
From git://github.com/jsocol/sumo-locales
 * branch            master     -> FETCH_HEAD
Updating 3f9c89f..c177774
Fast-forward
 nl/LC_MESSAGES/messages.po |    6 +++---
 1 files changed, 3 insertions(+), 3 deletions(-)
[root@mradm02 locale]# ./compile-mo.sh .
[root@mradm02 locale]#
Comment 11 Jeremy Orem [:oremj] 2010-11-30 16:36:50 PST
[root@mradm02 prod]# python26 ./vendor/src/schematic/schematic -v migrations/
61
[root@mradm02 prod]# python26 manage.py compress_assets
[root@mradm02 prod]# python26 manage.py update_product_details
Grabbing list of JSON files from the server.
No last update timestamp found.
Updating firefox_primary_builds.json from server
Writing new copy of firefox_primary_builds.json to /mnt/netapp/support.mozilla.com-kitsune/product_details_json.
Updating firefox_history_major_releases.json from server
Writing new copy of firefox_history_major_releases.json to /mnt/netapp/support.mozilla.com-kitsune/product_details_json.
Updating mobile_history_development_releases.json from server
Writing new copy of mobile_history_development_releases.json to /mnt/netapp/support.mozilla.com-kitsune/product_details_json.
Updating firefox_history_stability_releases.json from server
Writing new copy of firefox_history_stability_releases.json to /mnt/netapp/support.mozilla.com-kitsune/product_details_json.
Updating thunderbird_history_major_releases.json from server
Writing new copy of thunderbird_history_major_releases.json to /mnt/netapp/support.mozilla.com-kitsune/product_details_json.
Updating mobile_history_stability_releases.json from server
Writing new copy of mobile_history_stability_releases.json to /mnt/netapp/support.mozilla.com-kitsune/product_details_json.
Updating thunderbird_primary_builds.json from server
Writing new copy of thunderbird_primary_builds.json to /mnt/netapp/support.mozilla.com-kitsune/product_details_json.
Updating thunderbird_beta_builds.json from server
Writing new copy of thunderbird_beta_builds.json to /mnt/netapp/support.mozilla.com-kitsune/product_details_json.
Updating thunderbird_history_development_releases.json from server
Writing new copy of thunderbird_history_development_releases.json to /mnt/netapp/support.mozilla.com-kitsune/product_details_json.
Updating firefox_history_development_releases.json from server
Writing new copy of firefox_history_development_releases.json to /mnt/netapp/support.mozilla.com-kitsune/product_details_json.
Updating mobile_details.json from server
Writing new copy of mobile_details.json to /mnt/netapp/support.mozilla.com-kitsune/product_details_json.
Updating languages.json from server
Writing new copy of languages.json to /mnt/netapp/support.mozilla.com-kitsune/product_details_json.
Updating thunderbird_versions.json from server
Writing new copy of thunderbird_versions.json to /mnt/netapp/support.mozilla.com-kitsune/product_details_json.
Updating firefox_versions.json from server
Writing new copy of firefox_versions.json to /mnt/netapp/support.mozilla.com-kitsune/product_details_json.
Updating mobile_history_major_releases.json from server
Writing new copy of mobile_history_major_releases.json to /mnt/netapp/support.mozilla.com-kitsune/product_details_json.
Updating firefox_beta_builds.json from server
Writing new copy of firefox_beta_builds.json to /mnt/netapp/support.mozilla.com-kitsune/product_details_json.
Updating thunderbird_history_stability_releases.json from server
Writing new copy of thunderbird_history_stability_releases.json to /mnt/netapp/support.mozilla.com-kitsune/product_details_json.
Writing last-updated timestamp (Wed, 10 Nov 2010 20:05:30 GMT).
Product Details update run complete.
Comment 12 Jeremy Orem [:oremj] 2010-11-30 17:05:04 PST
All done.
Comment 13 Stephen Donner [:stephend] 2010-12-09 17:06:36 PST
Pushed.

Note You need to log in before you can comment on or make changes to this bug.