Closed
Bug 614034
Opened 14 years ago
Closed 14 years ago
Push SUMO 2.3 on Tuesday, 30 November @ 4pm PT
Categories
(mozilla.org Graveyard :: Server Operations, task)
Tracking
(Not tracked)
VERIFIED
FIXED
People
(Reporter: jsocol, Assigned: oremj)
Details
(Whiteboard: 11/30/2010 @ 4pm)
Please push SUMO 2.3 on Tuesday, 30 Nov, @ 4pm PT. If everything goes well we're looking at around 1 hr of downtime. We should plan for 2 hours in case there are issues and we need to roll back. Full steps coming but very similar to bug 612670 comment 0. We're making database changes. We should stop replication on a slave so we have a backup in case of emergency.
Updated•14 years ago
|
Flags: needs-downtime+
Whiteboard: 11/30/2010 @ 4pm
Comment 2•14 years ago
|
||
Adding a note here with only the stuff I'll need for data migration: * a copy of the folder webroot/img/wiki_up from svn (lots of images should be there). You can zip this up and put it in my home folder on khan, as images.zip should be fine. * a copy of the production videos from our CDN (I can get this myself though) * a database dump from after Sunday night (Nov 28th) with at least these tables: auth_user django_content_type tiki_category_objects tiki_content tiki_freetagged_objects tiki_objects tiki_pages tiki_programmed_content tiki_translated_objects That should cover everything pre-push. Thanks Shyam!
Reporter | ||
Comment 3•14 years ago
|
||
Here we go here we go! Push tasks. Prelim stuff: * Get the video CDN set up. (all set, aiui) * Make sure the server that will run the crons can talk to this host/port: https://chat-support.mozilla.com:9091 Network stuff: * Not sure when, but we want to flip over to the ZLB for the DB connections. Justdave knows more. Outage page! Catastrophe plan!: * Stop replication on a slave so we can use it as a backup in case of emergency. SVN: * switch to svn.mozilla.org/projects/sumo/tags/1.5/1.5.7_r77930_20101129/ * run webroot/htaccess.sh Git: * Kill celery dead. * switch to tag 2.3 * find . -name "*.pyc" -exec rm -rf {} \; * pushd vendor && git pull origin master && git submodule update --init && popd * git submodule update --init * pushd locale && git pull origin master && ./compile-mo.sh . && popd * `schematic migrations` ** `schematic -v migrations` should say "36" before and "61" after * `./manage.py compress_assets` * `./manage.py update_product_details` Wiki import: * After `schematic migrations`! * Import khan:/home/pcraciunuio/migration.sql * Unzip /home/pcraciunoiu/gallery.zip into /media/uploads (should end up with a /media/uploads/gallery/ with images and videos subdirectories). Crons: * `./manage.py update_product_details` maybe every 6 hours * `./manage.py cron collect_tweets` every 1 minute * `./manage.py cron get_queue_status` every 1 minute * `./manage.py cron calculate_related_documents` every 4 hours * (We don't care about normal output of any of these.) Settings: * GALLERY_VIDEO_URL = (based on CDN) * WIKI_REBUILD_ON_DEMAND = True (we'll disable if we have to) Sphinx: * Update configs from 2.3 tag. * Reindex. Celeryd: * Start 'er up! I think that's everything, including the two issues we ran into during our dry run with Jeremy. Hopefully he'll be around to provide sage advice from that experience. Post-script: If everything's good we should let that slave catch up and get back in the pool.
Comment 4•14 years ago
|
||
Woohooo! Okay, sorry for the bugspam, but this is the culmination of 9 month of work, basically since I started at Mozilla. So awesome to see it all come together, huge thanks to everyone contributing to it! Here's hoping for a smooth migration :)
Comment 5•14 years ago
|
||
(In reply to comment #3) > * Import khan:/home/pcraciunuio/migration.sql Typo here, it's /home/pcraciunoiu/migration.sql > * Unzip /home/pcraciunoiu/gallery.zip into /media/uploads (should end up with a > /media/uploads/gallery/ with images and videos subdirectories). This one is correct though :) > Crons: > * `./manage.py update_product_details` maybe every 6 hours > * `./manage.py cron collect_tweets` every 1 minute > * `./manage.py cron get_queue_status` every 1 minute > * `./manage.py cron calculate_related_documents` every 4 hours > * (We don't care about normal output of any of these.) But it would be nice to be notified if they fail, right? Looking good!
Reporter | ||
Comment 6•14 years ago
|
||
(In reply to comment #5) > > * (We don't care about normal output of any of these.) > But it would be nice to be notified if they fail, right? Jabba just set up cron-sumo@m.c as an alias for these things. Easily filterable. Yeah, stdout we don't care but stderr we do.
Reporter | ||
Comment 8•14 years ago
|
||
After unzipping the gallery images we need to make sure that all the relevant folders are still writable by the web server.
Assignee | ||
Comment 9•14 years ago
|
||
[root@mradm02 support.mozilla.com]# svn info Path: . URL: https://svn.mozilla.org/projects/sumo/tags/1.5/1.5.7_r77930_20101129 Repository Root: https://svn.mozilla.org Repository UUID: 4eb1ac78-321c-0410-a911-ec516a8615a5 Revision: 78497 Node Kind: directory Schedule: normal Last Changed Author: jsocol@mozilla.com Last Changed Rev: 78375 Last Changed Date: 2010-11-29 09:20:40 -0800 (Mon, 29 Nov 2010) activating ./webroot/pics/.htaccess activating ./webroot/templates_c/.htaccess activating ./webroot/styles/.htaccess activating ./webroot/whelp/.htaccess activating ./webroot/db/.htaccess activating ./webroot/.htaccess activating ./webroot/css/.htaccess activating ./webroot/lib/fckeditor_tiki/.htaccess activating ./webroot/lib/.htaccess activating ./webroot/lib/fckeditor/.htaccess activating ./webroot/dump.svn/.htaccess activating ./webroot/img.svn/.htaccess activating ./webroot/popups/.htaccess activating ./webroot/backups/.htaccess activating ./webroot/doc/.htaccess activating ./webroot/img-local/.htaccess activating ./webroot/js/.htaccess activating ./webroot/images/.htaccess activating ./webroot/files/.htaccess activating ./webroot/temp/.htaccess activating ./webroot/lang/.htaccess activating ./webroot/templates/.htaccess activating ./webroot/components/.htaccess activating ./webroot/img.old/.htaccess activating ./webroot/modules/.htaccess activating ./webroot/tikimovies/.htaccess activating ./webroot/games/.htaccess activating ./webroot/bin/.htaccess
Assignee | ||
Comment 10•14 years ago
|
||
[root@mradm02 prod]# git fetch remote: Counting objects: 2014, done. remote: Compressing objects: 100% (628/628), done. remote: Total 1848 (delta 1346), reused 1564 (delta 1128) Receiving objects: 100% (1848/1848), 489.78 KiB | 538 KiB/s, done. Resolving deltas: 100% (1346/1346), completed with 104 local objects. From http://github.com/jsocol/kitsune * [new branch] 2.3.x -> origin/2.3.x * [new branch] fix-rebuild -> origin/fix-rebuild 2c54c56..cad9ffa master -> origin/master * [new tag] 2.3 -> 2.3 [root@mradm02 prod]# git checkout 2.3 M locale Previous HEAD position was 8d9501e... Make Army of Awesome page validate. Bug 608086. HEAD is now at fa1eb9f... Update locales. [root@mradm02 vendor]# git pull origin master && git submodule update --init remote: Counting objects: 23, done. remote: Compressing objects: 100% (21/21), done. remote: Total 21 (delta 14), reused 0 (delta 0) Unpacking objects: 100% (21/21), done. From git://github.com/jsocol/kitsune-lib * branch master -> FETCH_HEAD Updating 382a9f2..0c81413 Fast-forward src/bleach | 2 +- src/django-csp | 2 +- src/jingo-minify | 2 +- src/py-wikimarkup | 2 +- src/schematic | 2 +- 5 files changed, 5 insertions(+), 5 deletions(-) remote: Counting objects: 15, done. remote: Compressing objects: 100% (9/9), done. remote: Total 9 (delta 6), reused 0 (delta 0) Unpacking objects: 100% (9/9), done. From git://github.com/jsocol/bleach d1b2a25..b856582 master -> origin/master Submodule path 'src/bleach': checked out 'b85658206b93cc928f32e90bfcc2dfae3711ea95' Submodule path 'src/django-csp': checked out '5c5f5a6b55fb78e99db939c79f0f0d8003a62c98' remote: Counting objects: 11, done. remote: Compressing objects: 100% (6/6), done. remote: Total 6 (delta 3), reused 0 (delta 0) Unpacking objects: 100% (6/6), done. From git://github.com/jsocol/jingo-minify 126e1d6..95f24e9 master -> origin/master Submodule path 'src/jingo-minify': checked out '95f24e9ecdbeb5d1d88029896b3d53ae8828e533' remote: Counting objects: 21, done. remote: Compressing objects: 100% (17/17), done. remote: Total 17 (delta 9), reused 0 (delta 0) Unpacking objects: 100% (17/17), done. From git://github.com/pcraciunoiu/py-wikimarkup * [new branch] add-section -> origin/add-section cc06e6d..4189367 master -> origin/master Submodule path 'src/py-wikimarkup': checked out '418936741d9bba1e6326716428a25e7223e58709' remote: Counting objects: 7, done. remote: Compressing objects: 100% (4/4), done. Unpacking objects: 100% (4/4), done. remote: Total 4 (delta 2), reused 0 (delta 0) From git://github.com/jbalogh/schematic 144134c..e749978 master -> origin/master Submodule path 'src/schematic': checked out 'e749978cc06bda8ba655faf8d672ae270e25b5f5' [root@mradm02 vendor]# cd .. [root@mradm02 prod]# git submodule update --init cdremote: Counting objects: 1697, done. remote: Compressing objects: 100% (797/797), done. remote: Total 1604 (delta 805), reused 933 (delta 421) Receiving objects: 100% (1604/1604), 1.09 MiB | 1.29 MiB/s, done. Resolving deltas: 100% (805/805), completed with 1 local objects. From git://github.com/jsocol/sumo-locales 07aa01a..c177774 master -> origin/master Submodule path 'locale': checked out '3f9c89fe6a91bcc21cfca8bceeb58b9bd23b7eb0' [root@mradm02 prod]# cd locale/ [root@mradm02 locale]# git pull origin master && git submodule update --init From git://github.com/jsocol/sumo-locales * branch master -> FETCH_HEAD Updating 3f9c89f..c177774 Fast-forward nl/LC_MESSAGES/messages.po | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) [root@mradm02 locale]# ./compile-mo.sh . [root@mradm02 locale]#
Assignee | ||
Comment 11•14 years ago
|
||
[root@mradm02 prod]# python26 ./vendor/src/schematic/schematic -v migrations/ 61 [root@mradm02 prod]# python26 manage.py compress_assets [root@mradm02 prod]# python26 manage.py update_product_details Grabbing list of JSON files from the server. No last update timestamp found. Updating firefox_primary_builds.json from server Writing new copy of firefox_primary_builds.json to /mnt/netapp/support.mozilla.com-kitsune/product_details_json. Updating firefox_history_major_releases.json from server Writing new copy of firefox_history_major_releases.json to /mnt/netapp/support.mozilla.com-kitsune/product_details_json. Updating mobile_history_development_releases.json from server Writing new copy of mobile_history_development_releases.json to /mnt/netapp/support.mozilla.com-kitsune/product_details_json. Updating firefox_history_stability_releases.json from server Writing new copy of firefox_history_stability_releases.json to /mnt/netapp/support.mozilla.com-kitsune/product_details_json. Updating thunderbird_history_major_releases.json from server Writing new copy of thunderbird_history_major_releases.json to /mnt/netapp/support.mozilla.com-kitsune/product_details_json. Updating mobile_history_stability_releases.json from server Writing new copy of mobile_history_stability_releases.json to /mnt/netapp/support.mozilla.com-kitsune/product_details_json. Updating thunderbird_primary_builds.json from server Writing new copy of thunderbird_primary_builds.json to /mnt/netapp/support.mozilla.com-kitsune/product_details_json. Updating thunderbird_beta_builds.json from server Writing new copy of thunderbird_beta_builds.json to /mnt/netapp/support.mozilla.com-kitsune/product_details_json. Updating thunderbird_history_development_releases.json from server Writing new copy of thunderbird_history_development_releases.json to /mnt/netapp/support.mozilla.com-kitsune/product_details_json. Updating firefox_history_development_releases.json from server Writing new copy of firefox_history_development_releases.json to /mnt/netapp/support.mozilla.com-kitsune/product_details_json. Updating mobile_details.json from server Writing new copy of mobile_details.json to /mnt/netapp/support.mozilla.com-kitsune/product_details_json. Updating languages.json from server Writing new copy of languages.json to /mnt/netapp/support.mozilla.com-kitsune/product_details_json. Updating thunderbird_versions.json from server Writing new copy of thunderbird_versions.json to /mnt/netapp/support.mozilla.com-kitsune/product_details_json. Updating firefox_versions.json from server Writing new copy of firefox_versions.json to /mnt/netapp/support.mozilla.com-kitsune/product_details_json. Updating mobile_history_major_releases.json from server Writing new copy of mobile_history_major_releases.json to /mnt/netapp/support.mozilla.com-kitsune/product_details_json. Updating firefox_beta_builds.json from server Writing new copy of firefox_beta_builds.json to /mnt/netapp/support.mozilla.com-kitsune/product_details_json. Updating thunderbird_history_stability_releases.json from server Writing new copy of thunderbird_history_stability_releases.json to /mnt/netapp/support.mozilla.com-kitsune/product_details_json. Writing last-updated timestamp (Wed, 10 Nov 2010 20:05:30 GMT). Product Details update run complete.
Assignee | ||
Comment 12•14 years ago
|
||
All done.
Status: NEW → RESOLVED
Closed: 14 years ago
Resolution: --- → FIXED
Pushed.
Status: RESOLVED → VERIFIED
Updated•9 years ago
|
Product: mozilla.org → mozilla.org Graveyard
You need to log in
before you can comment on or make changes to this bug.
Description
•