Closed Bug 622898 Opened 13 years ago Closed 13 years ago

Push SUMO 2.4.1 Tuesday, 11 January @ 4pm PST

Categories

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

All
Other
task
Not set
normal

Tracking

(Not tracked)

VERIFIED FIXED

People

(Reporter: jsocol, Assigned: fox2mike)

References

Details

(Whiteboard: 01/11/2011 @ 4pm)

Per the webdev releases calendar, we have a fairly small release, SUMO 2.4.1, ready for Thursday.

The git tag will be 2.4.1. There will be an SVN tag, I'll post it soon.

We're *dropping all the unused tiki tables* in this release. It may introduce some replication lag. It's probably safest to put up the outage page and keep replication on a slave stopped.

I think the only git steps are:

git checkout 2.4.1
git submodule update --init
pushd locale && ./compile-mo.sh . && popd
manage.py compress_assets

set ENGAGE_ROBOTS = True in settings_local.py.

I believe that's it. We'll add anything else if we forgot it.
can we piggy back the RAM upgrade here before the push?
Flags: needs-downtime+
Whiteboard: [1/6 @ 4PM PST]
I would rather isolate these and reduce entropy rather than increase it.
Also need to run:
schematic migrations
to drop the tiki tables.
(In reply to comment #2)
> I would rather isolate these and reduce entropy rather than increase it.

It's a simple shutdown and and installing more RAM.
I know, but it's also at the same time as a code change, which means that if anything goes wrong with either, determining the cause is that much more difficult.

Wasn't the RAM upgrade scheduled for this afternoon, anyway? Can we just do it as planned, or next Tuesday.
It is planned for today, but mrz is concerned about having it down for more than 15 minutes.
Assignee: server-ops → shyam
Combining this along with bug 622674.  Since we're taking a site hit anyways, I'd rather do the memory along with it.
Depends on: 622674
SVN tag: https://svn.mozilla.org/projects/sumo/tags/1.5/1.5.9_r80088_20110106
Git tag: 2.4.1

Recap: steps are in comment 0 and comment 3, in that order.

Looks like it's all ready. I'll post again if anything changes.
Ah, one last thing: after the svn switch, webroot/htaccess.sh
Shyam, Phong: We need to postpone this code update. We discovered a number of issues last minute and we're just not 100% confident in the release.

We may try to push something next week, though the slots are full, or else we'll wait for our scheduled 2.4.2 push.

Phong, it's up to you and mrz if you want to do the hardware upgrades as planned.

Sorry, everyone. Discovering issues this late sucks. We're post-morteming this tomorrow.
Summary: Push SUMO 2.4.1 on Thursday, 6 January at 4pm PST → Push SUMO 2.4.1
Whiteboard: [1/6 @ 4PM PST]
pushing off memory too
boo-hoo! I woke up for this :p
Talked to Shyam, talked to the team. We believe all the issues are already solved and should be good to go for Tuesday.
Summary: Push SUMO 2.4.1 → Push SUMO 2.4.1 Tuesday, 11 January @ 4pm PST
Phong/mrz, are we piggy-backing RAM on this too?
Whiteboard: 01/11/2011 @ 4pm
Shyam, if you wouldn't mind, could we also `svn up` mobile.support.mozilla.com while we've got your attention? (Well, `svn up` and `webroot/htaccess.sh`.)
The new tag is 2.4.1.1, including the fixes from last week.
(In reply to comment #0)

> I think the only git steps are:
> 
> git checkout 2.4.1

[root@mradm02 prod]# git fetch
remote: Counting objects: 845, done.
remote: Compressing objects: 100% (409/409), done.
remote: Total 695 (delta 518), reused 440 (delta 270)
Receiving objects: 100% (695/695), 135.30 KiB, done.
Resolving deltas: 100% (518/518), completed with 120 local objects.
From http://github.com/jsocol/kitsune
 + d1767ca...6cb06b5 2.4.x      -> origin/2.4.x  (forced update)
   910d95d..e887c25  master     -> origin/master
 * [new branch]      next       -> origin/next
 * [new branch]      opensearch-616149 -> origin/opensearch-616149
 + 4522ade...9b8dd81 services-monitor -> origin/services-monitor  (forced update)
 * [new branch]      unicode-summary-622502 -> origin/unicode-summary-622502
 * [new tag]         2.4.1.1    -> 2.4.1.1
From http://github.com/jsocol/kitsune
 * [new tag]         2.4.1      -> 2.4.1
[root@mradm02 prod]# git checkout 2.4.1
M       locale
M       media/global
M       templates/global
Previous HEAD position was d1767ca... [bug 619746] Tell ForParser about the <section> element, which we recently informed wikimarkup of. wikimarkup preserves occurrences of the <section> tag in input, so the ForParser needs to take it into account when choosing whether a {for} will become a span or a div.
HEAD is now at 97e9423... Updating the locale submodule before tagging 2.4.1

> git submodule update --init

[root@mradm02 prod]# git submodule update --init
remote: Counting objects: 804, done.
remote: Compressing objects: 100% (268/268), done.
remote: Total 709 (delta 370), reused 560 (delta 281)
Receiving objects: 100% (709/709), 801.50 KiB | 1.39 MiB/s, done.
Resolving deltas: 100% (370/370), completed with 4 local objects.
From git://github.com/jsocol/sumo-locales
   8e7098e..4dbdbf8  master     -> origin/master
error: You have local changes to 'tr/LC_MESSAGES/javascript.po'; cannot switch branches.
Unable to checkout '93918595975618edfd0e247b2b891224c2cfa6fc' in submodule path 'locale'
[root@mradm02 prod]# cd locale/
[root@mradm02 locale]# git checkout -- .
[root@mradm02 locale]# cd ..
[root@mradm02 prod]# git submodule update --init
Submodule path 'locale': checked out '93918595975618edfd0e247b2b891224c2cfa6fc'
remote: Counting objects: 5, done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 3 (delta 2), reused 0 (delta 0)
Unpacking objects: 100% (3/3), done.
From git://github.com/mozilla/django-moz-header
   956819f..6affd20  master     -> origin/master
Submodule path 'media/global': checked out '6affd2086b4e5580901fbc5831f19f5f1959d90c'
remote: Counting objects: 5, done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 3 (delta 2), reused 0 (delta 0)
Unpacking objects: 100% (3/3), done.
From git://github.com/mozilla/django-moz-header
   956819f..6affd20  master     -> origin/master
Submodule path 'templates/global': checked out '6affd2086b4e5580901fbc5831f19f5f1959d90c'

> pushd locale && ./compile-mo.sh . && popd

[root@mradm02 prod]# pushd locale && ./compile-mo.sh . && popd
/data/sumo_python/src/prod/locale /data/sumo_python/src/prod
/data/sumo_python/src/prod

> manage.py compress_assets

[root@mradm02 prod]# python26 manage.py compress_assets

> set ENGAGE_ROBOTS = True in settings_local.py.

Done.


(In reply to comment #3)
> Also need to run:
> schematic migrations
> to drop the tiki tables.

[root@mradm02 prod]# schematic migrations/
Running migration 70:
-- tiki_%
DROP TABLE tiki_actionlog;
DROP TABLE tiki_actionlog_conf;
DROP TABLE tiki_actionlog_params;
DROP TABLE tiki_article_types;
DROP TABLE tiki_articles;
DROP TABLE tiki_banners;
DROP TABLE tiki_banning;
DROP TABLE tiki_banning_sections;
DROP TABLE tiki_blog_activity;
DROP TABLE tiki_blog_posts;
DROP TABLE tiki_blog_posts_images;
DROP TABLE tiki_blogs;
DROP TABLE tiki_calendar_categories;
DROP TABLE tiki_calendar_items;
DROP TABLE tiki_calendar_locations;
DROP TABLE tiki_calendar_options;
DROP TABLE tiki_calendar_roles;
DROP TABLE tiki_calendars;
DROP TABLE tiki_categories;
DROP TABLE tiki_categorized_objects;
DROP TABLE tiki_category_objects;
DROP TABLE tiki_category_sites;
DROP TABLE tiki_chart_items;
DROP TABLE tiki_charts;
DROP TABLE tiki_charts_rankings;
DROP TABLE tiki_charts_votes;
DROP TABLE tiki_chat_channels;
DROP TABLE tiki_chat_messages;
DROP TABLE tiki_chat_users;
DROP TABLE tiki_comments;
DROP TABLE tiki_comments_metadata;
DROP TABLE tiki_content;
DROP TABLE tiki_content_templates;
DROP TABLE tiki_content_templates_sections;
DROP TABLE tiki_contributions;
DROP TABLE tiki_contributions_assigned;
DROP TABLE tiki_cookies;
DROP TABLE tiki_copyrights;
DROP TABLE tiki_directory_categories;
DROP TABLE tiki_directory_search;
DROP TABLE tiki_directory_sites;
DROP TABLE tiki_download;
DROP TABLE tiki_drawings;
DROP TABLE tiki_dsn;
DROP TABLE tiki_dynamic_variables;
DROP TABLE tiki_eph;
DROP TABLE tiki_events;
DROP TABLE tiki_extwiki;
DROP TABLE tiki_faq_questions;
DROP TABLE tiki_faqs;
DROP TABLE tiki_featured_links;
DROP TABLE tiki_file_galleries;
DROP TABLE tiki_file_handlers;
DROP TABLE tiki_files;
DROP TABLE tiki_forum_attachments;
DROP TABLE tiki_forum_reads;
DROP TABLE tiki_forums;
DROP TABLE tiki_forums_queue;
DROP TABLE tiki_forums_reported;
DROP TABLE tiki_freetagged_objects;
DROP TABLE tiki_freetags;
DROP TABLE tiki_friends;
DROP TABLE tiki_friendship_requests;
DROP TABLE tiki_galleries;
DROP TABLE tiki_galleries_scales;
DROP TABLE tiki_games;
DROP TABLE tiki_group_inclusion;
DROP TABLE tiki_history;
DROP TABLE tiki_hotwords;
DROP TABLE tiki_html_pages;
DROP TABLE tiki_html_pages_dynamic_zones;
DROP TABLE tiki_images;
DROP TABLE tiki_images_data;
DROP TABLE tiki_integrator_reps;
DROP TABLE tiki_integrator_rules;
DROP TABLE tiki_language;
DROP TABLE tiki_languages;
DROP TABLE tiki_link_cache;
DROP TABLE tiki_links;
DROP TABLE tiki_live_support_events;
DROP TABLE tiki_live_support_message_comments;
DROP TABLE tiki_live_support_messages;
DROP TABLE tiki_live_support_modules;
DROP TABLE tiki_live_support_operators;
DROP TABLE tiki_live_support_requests;
DROP TABLE tiki_logs;
DROP TABLE tiki_mail_events;
DROP TABLE tiki_mailin_accounts;
DROP TABLE tiki_menu_languages;
DROP TABLE tiki_menu_options;
DROP TABLE tiki_menus;
DROP TABLE tiki_minical_events;
DROP TABLE tiki_minical_topics;
DROP TABLE tiki_modules;
DROP TABLE tiki_mypage;
DROP TABLE tiki_mypage_types;
DROP TABLE tiki_mypage_types_components;
DROP TABLE tiki_mypagewin;
DROP TABLE tiki_newsletter_groups;
DROP TABLE tiki_newsletter_included;
DROP TABLE tiki_newsletter_subscriptions;
DROP TABLE tiki_newsletters;
DROP TABLE tiki_newsreader_marks;
DROP TABLE tiki_newsreader_servers;
DROP TABLE tiki_object_ratings;
DROP TABLE tiki_objects;
DROP TABLE tiki_page_drafts;
DROP TABLE tiki_page_footnotes;
DROP TABLE tiki_page_list_types;
DROP TABLE tiki_page_lists;
DROP TABLE tiki_pages;
DROP TABLE tiki_pages_translation_bits;
DROP TABLE tiki_pageviews;
DROP TABLE tiki_poll_objects;
DROP TABLE tiki_poll_options;
DROP TABLE tiki_poll_questions;
DROP TABLE tiki_poll_votes;
DROP TABLE tiki_polls;
DROP TABLE tiki_preferences;
DROP TABLE tiki_private_messages;
DROP TABLE tiki_programmed_content;
DROP TABLE tiki_question_fields;
DROP TABLE tiki_question_sections;
DROP TABLE tiki_quicktags;
DROP TABLE tiki_quiz_question_options;
DROP TABLE tiki_quiz_questions;
DROP TABLE tiki_quiz_results;
DROP TABLE tiki_quiz_stats;
DROP TABLE tiki_quiz_stats_sum;
DROP TABLE tiki_quizzes;
DROP TABLE tiki_received_articles;
DROP TABLE tiki_received_pages;
DROP TABLE tiki_referer_stats;
DROP TABLE tiki_registration_fields;
DROP TABLE tiki_related_categories;
DROP TABLE tiki_rss_feeds;
DROP TABLE tiki_rss_modules;
DROP TABLE tiki_score;
DROP TABLE tiki_search_stats;
DROP TABLE tiki_searchindex;
DROP TABLE tiki_searchsyllable;
DROP TABLE tiki_searchwords;
DROP TABLE tiki_secdb;
DROP TABLE tiki_semaphores;
DROP TABLE tiki_sent_newsletters;
DROP TABLE tiki_sent_newsletters_errors;
DROP TABLE tiki_sessions;
DROP TABLE tiki_sheet_layout;
DROP TABLE tiki_sheet_values;
DROP TABLE tiki_sheets;
DROP TABLE tiki_shoutbox;
DROP TABLE tiki_shoutbox_words;
DROP TABLE tiki_stats;
DROP TABLE tiki_structure_versions;
DROP TABLE tiki_structures;
DROP TABLE tiki_submissions;
DROP TABLE tiki_suggested_faq_questions;
DROP TABLE tiki_survey_question_options;
DROP TABLE tiki_survey_questions;
DROP TABLE tiki_surveys;
DROP TABLE tiki_tags;
DROP TABLE tiki_theme_control_categs;
DROP TABLE tiki_theme_control_objects;
DROP TABLE tiki_theme_control_sections;
DROP TABLE tiki_topics;
DROP TABLE tiki_tracker_fields;
DROP TABLE tiki_tracker_item_attachments;
DROP TABLE tiki_tracker_item_comments;
DROP TABLE tiki_tracker_item_fields;
DROP TABLE tiki_tracker_items;
DROP TABLE tiki_tracker_options;
DROP TABLE tiki_trackers;
DROP TABLE tiki_translated_objects;
DROP TABLE tiki_untranslated;
DROP TABLE tiki_urlt_config;
DROP TABLE tiki_urlt_regex_in;
DROP TABLE tiki_urlt_regex_out;
DROP TABLE tiki_user_answers;
DROP TABLE tiki_user_answers_uploads;
DROP TABLE tiki_user_assigned_modules;
DROP TABLE tiki_user_bookmarks_folders;
DROP TABLE tiki_user_bookmarks_urls;
DROP TABLE tiki_user_mail_accounts;
DROP TABLE tiki_user_menus;
DROP TABLE tiki_user_modules;
DROP TABLE tiki_user_notes;
DROP TABLE tiki_user_postings;
DROP TABLE tiki_user_preferences;
DROP TABLE tiki_user_quizzes;
DROP TABLE tiki_user_taken_quizzes;
DROP TABLE tiki_user_tasks;
DROP TABLE tiki_user_tasks_history;
DROP TABLE tiki_user_votings;
DROP TABLE tiki_user_watches;
DROP TABLE tiki_userfiles;
DROP TABLE tiki_userpoints;
DROP TABLE tiki_users;
DROP TABLE tiki_users_score;
DROP TABLE tiki_webmail_contacts;
DROP TABLE tiki_webmail_contacts_ext;
DROP TABLE tiki_webmail_contacts_fields;
DROP TABLE tiki_webmail_contacts_groups;
DROP TABLE tiki_webmail_messages;
DROP TABLE tiki_wiki_attachments;
DROP TABLE tiki_zones;

-- Events
DROP TABLE events_event;
DROP TABLE events_signup;

-- Galaxia
DROP TABLE galaxia_activities;
DROP TABLE galaxia_activity_roles;
DROP TABLE galaxia_instance_activities;
DROP TABLE galaxia_instance_comments;
DROP TABLE galaxia_instances;
DROP TABLE galaxia_processes;
DROP TABLE galaxia_roles;
DROP TABLE galaxia_transitions;
DROP TABLE galaxia_user_roles;
DROP TABLE galaxia_workitems;

-- Users (excluding users_profile)
DROP TABLE users_grouppermissions;
DROP TABLE users_groups;
DROP TABLE users_objectpermissions;
DROP TABLE users_permissions;
DROP TABLE users_usergroups;
DROP TABLE users_users;

-- Others
DROP TABLE adodb_logsql;
DROP TABLE messu_archive;
DROP TABLE messu_messages;
DROP TABLE messu_sent;
DROP TABLE metrics_assigned;
DROP TABLE metrics_metric;
DROP TABLE metrics_tab;
DROP TABLE se_weights;
DROP TABLE sessions;

That took 9.72 seconds
################################################## 

Running migration 71:
CREATE TABLE `users_emailchange` (
    `id` integer AUTO_INCREMENT NOT NULL PRIMARY KEY,
    `user_id` integer NOT NULL UNIQUE,
    `activation_key` varchar(40) NOT NULL,
    `email` varchar(75)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
ALTER TABLE `users_emailchange`
    ADD CONSTRAINT `user_id_refs_id_7c0fddb0` FOREIGN KEY (`user_id`)
        REFERENCES `auth_user` (`id`);
CREATE INDEX `users_emailchange_email`
    ON `users_emailchange` (`email`);

That took 0.23 seconds
##################################################

(In reply to comment #16)
> The new tag is 2.4.1.1, including the fixes from last week.

[root@mradm02 prod]# git fetch
remote: Counting objects: 17, done.
remote: Compressing objects: 100% (8/8), done.
remote: Total 10 (delta 5), reused 4 (delta 2)
Unpacking objects: 100% (10/10), done.
From http://github.com/jsocol/kitsune
 * [new branch]      notifications-623641 -> origin/notifications-623641
[root@mradm02 prod]# git checkout 2.4.1.1
Previous HEAD position was 97e9423... Updating the locale submodule before tagging 2.4.1
HEAD is now at 6cb06b5... Updated for new URLs without trailing slash. [bug 623724] Fixes /gallery->/gallery/images redirect and /home->/mobile and back based on Help With selection.
[root@mradm02 prod]# python26 manage.py compress_assets

(In reply to comment #8)
> SVN tag: https://svn.mozilla.org/projects/sumo/tags/1.5/1.5.9_r80088_20110106

[root@mradm02 support.mozilla.com]# svn switch https://svn.mozilla.org/projects/sumo/tags/1.5/1.5.9_r80088_20110106
U    webroot/htaccess.dist
Updated to revision 80611.
[root@mradm02 support.mozilla.com]# cd webroot/
[root@mradm02 webroot]# ./htaccess.sh 
activating ./pics/.htaccess
activating ./templates_c/.htaccess
activating ./styles/.htaccess
activating ./whelp/.htaccess
activating ./db/.htaccess
activating ./.htaccess
activating ./css/.htaccess
activating ./lib/fckeditor_tiki/.htaccess
activating ./lib/.htaccess
activating ./lib/fckeditor/.htaccess
activating ./dump/.htaccess
activating ./popups/.htaccess
activating ./backups/.htaccess
activating ./doc/.htaccess
activating ./js/.htaccess
activating ./images/.htaccess
activating ./files/.htaccess
activating ./temp/.htaccess
activating ./lang/.htaccess
activating ./templates/.htaccess
activating ./components/.htaccess
activating ./modules/.htaccess
activating ./tikimovies/.htaccess
activating ./games/.htaccess
activating ./bin/.htaccess

(In reply to comment #15)
> Shyam, if you wouldn't mind, could we also `svn up` mobile.support.mozilla.com
> while we've got your attention? (Well, `svn up` and `webroot/htaccess.sh`.)

Nothing to svn up, ran :

[root@mradm02 webroot]# ./htaccess.sh 
activating ./pics/.htaccess
activating ./templates_c/.htaccess
activating ./styles/.htaccess
activating ./whelp/.htaccess
activating ./db/.htaccess
activating ./.htaccess
activating ./css/.htaccess
activating ./lib/fckeditor_tiki/.htaccess
activating ./lib/.htaccess
activating ./lib/fckeditor/.htaccess
activating ./popups/.htaccess
activating ./backups/.htaccess
activating ./doc/.htaccess
activating ./js/.htaccess
activating ./images/.htaccess
activating ./files/.htaccess
activating ./temp/.htaccess
activating ./lang/.htaccess
activating ./templates/.htaccess
activating ./components/.htaccess
activating ./modules/.htaccess
activating ./tikimovies/.htaccess
activating ./games/.htaccess
activating ./img/.htaccess
activating ./bin/.htaccess

All done.

[root@mradm02 sumo_python]# /data/bin/omg_push_sumo_live.sh .
Do you really want to commit /data/sumo/www/./ ? [y/n] y
[master 5b0a7e2] forcing push
 25 files changed, 852 insertions(+), 0 deletions(-)
 create mode 100644 support.mozilla.com/kitsune/apps/questions/templates/questions/confirm_email.html
 create mode 100644 support.mozilla.com/kitsune/apps/questions/tests/test_tasks.py
 create mode 100644 support.mozilla.com/kitsune/apps/search/forms.py
 create mode 100644 support.mozilla.com/kitsune/apps/sumo/monkeypatch.py
 create mode 100644 support.mozilla.com/kitsune/apps/sumo/templates/sumo/robots.html
 create mode 100644 support.mozilla.com/kitsune/apps/sumo/tests/test__utils.py
 create mode 100644 support.mozilla.com/kitsune/apps/sumo/tests/test_forms.py
 create mode 100644 support.mozilla.com/kitsune/apps/sumo/urls.py
 create mode 100644 support.mozilla.com/kitsune/apps/upload/storage.py
 create mode 100644 "support.mozilla.com/kitsune/apps/upload/tests/media/123ascii\346\234\211\345\212\271.jpg"
 create mode 100644 support.mozilla.com/kitsune/apps/users/templates/users/change_email.html
 create mode 100644 support.mozilla.com/kitsune/apps/users/templates/users/change_email_complete.html
 create mode 100644 support.mozilla.com/kitsune/apps/users/templates/users/change_email_done.html
 create mode 100644 support.mozilla.com/kitsune/apps/users/templates/users/email/confirm_email.ltxt
 create mode 100644 support.mozilla.com/kitsune/apps/users/templates/users/resend_confirmation.html
 create mode 100644 support.mozilla.com/kitsune/apps/users/templates/users/resend_confirmation_done.html
 create mode 100644 support.mozilla.com/kitsune/apps/users/widgets.py
 create mode 100644 support.mozilla.com/kitsune/media/img/markup.png
 create mode 100644 support.mozilla.com/kitsune/media/img/old_markup.png
 create mode 100644 support.mozilla.com/kitsune/media/js/users-all.js
 create mode 100644 support.mozilla.com/kitsune/media/js/users-min.js
 create mode 100755 support.mozilla.com/kitsune/media/js/users.js
 create mode 100644 support.mozilla.com/kitsune/migrations/70-drop-tiki-tables.sql
 create mode 100644 support.mozilla.com/kitsune/migrations/71-change-email.sql
 create mode 100644 support.mozilla.com/kitsune/scripts/anonymize.sql

Added 15 01 * * * apache cd /data/sumo/www/support.mozilla.com/kitsune; python26 ./manage.py cron update_weekly_votes > /dev/null to cron as well

Kicked Apache and Celery, removed the outage page.
Think we're all done.
Status: NEW → RESOLVED
Closed: 13 years ago
Resolution: --- → FIXED
Released and verified
Status: RESOLVED → VERIFIED
Component: Server Operations: Web Operations → WebOps: Other
Product: mozilla.org → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.