Closed Bug 872102 Opened 12 years ago Closed 12 years ago

[basket] Push new version to staging and prod

Categories

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Product:
Infrastructure & Operations Graveyard
Component:
WebOps: Other
Platform:
All
Other
Type:
task
Priority:
Not set
Severity:
critical

Tracking

(Not tracked)

Status:
VERIFIED FIXED

People

(Reporter: pmac, Assigned: cturra)

Details

(Whiteboard: [push interrupt])

Subscriptions are in many cases broken right now. This push should fix it. Urgent because of errors and this being Fx release time it's a big subscription time. Please push the tag "v2013.6" to stage and prod.
push to stage complete. [root@genericadm.private.phx1 basket.allizom.org]# bash -x update v2013.6 + CODE_DIR=/data/genericrhel6-stage/src/basket.allizom.org/basket + VENDOR_DIR=/data/genericrhel6-stage/src/basket.allizom.org/basket/vendor + LOCALE_DIR=/data/genericrhel6-stage/src/basket.allizom.org/basket/locale + techo 'Updating code.' ++ date -u '+%F %T' + STAMP='[2013-05-14 17:39:22]' + echo '[2013-05-14 17:39:22] Updating code.' [2013-05-14 17:39:22] Updating code. + cd /data/genericrhel6-stage/src/basket.allizom.org/basket + git fetch origin -q + checkretval + retval=0 + [[ 0 -gt 0 ]] + git checkout -f origin/master -q + git submodule sync -q + git submodule update --init -q + checkretval + retval=0 + [[ 0 -gt 0 ]] + git log -3 + tee /data/genericrhel6-stage/src/basket.allizom.org/basket/revision_info.txt commit aec5292e5ca49ae7c4e2687b4b2d7d58f6611e22 Merge: 71767fb 088b307 Author: Paul McLanahan <pmac@mozilla.com> Date: Tue May 14 10:29:00 2013 -0700 Merge pull request #41 from pmclanahan/fix-error-on-get-user-data-bug-871764 Bug 871764: Fix errors causing subscriptions to fail. commit 088b3072f5d30603ee512be13d9ae293b5074334 Author: Paul McLanahan <pmac@mozilla.com> Date: Tue May 14 13:13:19 2013 -0400 Bug 871764: Fix errors causing subscriptions to fail. commit 71767fb37fc4790b89a37a5d65bfe7bbafe785c5 Author: Paul McLanahan <pmac@mozilla.com> Date: Fri May 10 00:49:28 2013 -0400 Remove manual sentry capture. Seems to be causing problems in reporting. + techo 'Updating vendor.' ++ date -u '+%F %T' + STAMP='[2013-05-14 17:39:23]' + echo '[2013-05-14 17:39:23] Updating vendor.' [2013-05-14 17:39:23] Updating vendor. + cd /data/genericrhel6-stage/src/basket.allizom.org/basket/vendor + git submodule sync -q + git submodule update --init -q + checkretval + retval=0 + [[ 0 -gt 0 ]] + techo 'Sync & migrate database.' ++ date -u '+%F %T' + STAMP='[2013-05-14 17:39:25]' + echo '[2013-05-14 17:39:25] Sync & migrate database.' [2013-05-14 17:39:25] Sync & migrate database. + cd /data/genericrhel6-stage/src/basket.allizom.org/basket + python manage.py syncdb --noinput Syncing... Creating tables ... Installing custom SQL ... Installing indexes ... No fixtures found. Synced: > basketauth > emailer > subscriptions > vars > piston > south > raven.contrib.django.raven_compat > django.contrib.auth > django.contrib.contenttypes > django.contrib.sessions > django.contrib.sites > django.contrib.messages > django.contrib.admin Not synced (use migrations): - news - djcelery (use ./manage.py migrate to migrate these) + checkretval + retval=0 + [[ 0 -gt 0 ]] + python manage.py migrate --noinput Running migrations for news: - Nothing to migrate. - Loading initial data for news. No fixtures found. Running migrations for djcelery: - Nothing to migrate. - Loading initial data for djcelery. No fixtures found. + checkretval + retval=0 + [[ 0 -gt 0 ]] + techo 'Clean up old deploy.' ++ date -u '+%F %T' + STAMP='[2013-05-14 17:39:34]' + echo '[2013-05-14 17:39:34] Clean up old deploy.' [2013-05-14 17:39:34] Clean up old deploy. + cd /data/genericrhel6-stage/src/basket.allizom.org/basket + find . -type f -name .gitignore -or -name '*.pyc' -delete + checkretval + retval=0 + [[ 0 -gt 0 ]] + cd /data/genericrhel6-stage/src/basket.allizom.org/basket + touch wsgi/basket.wsgi + update_celery + techo 'Update celery.' ++ date -u '+%F %T' + STAMP='[2013-05-14 17:39:36]' + echo '[2013-05-14 17:39:36] Update celery.' [2013-05-14 17:39:36] Update celery. + ISSUE_MULTI='issue-multi-command -i /root/.ssh/id_rsa genericrhel6-celery-stage' + echo 'sh /data/bin/update-www.sh' + issue-multi-command -i /root/.ssh/id_rsa genericrhel6-celery-stage [2013-05-14 10:39:36] [generic-celery1.stage.seamicro.phx1.mozilla.com] running: sh /data/bin/update-www.sh [2013-05-14 10:40:25] [generic-celery1.stage.seamicro.phx1.mozilla.com] finished: sh /data/bin/update-www.sh (48.918s) [generic-celery1.stage.seamicro.phx1.mozilla.com] out: Not removing blog.allizom.org/wp-content/plugins/simple-embed-code/ [generic-celery1.stage.seamicro.phx1.mozilla.com] out: Not removing ship-it.allizom.org/release-kickoff/logs/ + echo '/etc/init.d/celeryd-basket-stage restart' + issue-multi-command -i /root/.ssh/id_rsa genericrhel6-celery-stage [2013-05-14 10:40:25] [generic-celery1.stage.seamicro.phx1.mozilla.com] running: /etc/init.d/celeryd-basket-stage restart [2013-05-14 10:40:40] [generic-celery1.stage.seamicro.phx1.mozilla.com] finished: /etc/init.d/celeryd-basket-stage restart (15.014s) [generic-celery1.stage.seamicro.phx1.mozilla.com] out: Restarting celery-basket-stage: celery-basket-stage: stopped [generic-celery1.stage.seamicro.phx1.mozilla.com] out: celery-basket-stage: started [generic-celery1.stage.seamicro.phx1.mozilla.com] out: [ OK ] + techo Deploying. ++ date -u '+%F %T' + STAMP='[2013-05-14 17:40:40]' + echo '[2013-05-14 17:40:40] Deploying.' [2013-05-14 17:40:40] Deploying. + '[' xterm == dumb ']' + /data/genericrhel6-stage/deploy basket.allizom.org [2013-05-14 10:40:40] Running rsync_project [2013-05-14 10:40:40] [localhost] running: /usr/bin/rsync -aq --include '.gitkeep' --exclude '.git*' --exclude '.hg*' --exclude '.svn*' --exclude 'CVS' --exclude '.bzr*' --delete /data/genericrhel6-stage/src/basket.allizom.org/ /data/genericrhel6-stage/www/basket.allizom.org/ [2013-05-14 10:41:06] [localhost] finished: /usr/bin/rsync -aq --include '.gitkeep' --exclude '.git*' --exclude '.hg*' --exclude '.svn*' --exclude 'CVS' --exclude '.bzr*' --delete /data/genericrhel6-stage/src/basket.allizom.org/ /data/genericrhel6-stage/www/basket.allizom.org/ (25.789s) [2013-05-14 10:41:06] Finished rsync_project (25.790s) [2013-05-14 10:41:06] Running commit_www [2013-05-14 10:41:06] [localhost] running: cd /data/genericrhel6-stage/www && /usr/bin/git add .; /usr/bin/git commit -a -m 'deploy ['basket.allizom.org']' [2013-05-14 10:48:14] [localhost] finished: cd /data/genericrhel6-stage/www && /usr/bin/git add .; /usr/bin/git commit -a -m 'deploy ['basket.allizom.org']' (428.631s) [localhost] out: [master c90285e] deploy [basket.allizom.org] [localhost] out: 3 files changed, 163 insertions(+), 36 deletions(-) [localhost] out: rewrite basket.allizom.org/basket/revision_info.txt (86%) [2013-05-14 10:48:14] Finished commit_www (428.632s)
Assignee: server-ops-webops → cturra
Whiteboard: [push interrupt]
as discussed on irc, i have completed the push to prod. we found that some of the new code hadn't been deployed to the celery node in stage, so i have updated the order of operations in our update script (in both dev/stage/prod) to address this. [root@genericadm.private.phx1 basket.mozilla.org]# bash -x update v2013.6 + CODE_DIR=/data/genericrhel6/src/basket.mozilla.org/basket + VENDOR_DIR=/data/genericrhel6/src/basket.mozilla.org/basket/vendor + LOCALE_DIR=/data/genericrhel6/src/basket.mozilla.org/basket/locale + techo 'Updating code.' ++ date -u '+%F %T' + STAMP='[2013-05-14 19:03:14]' + echo '[2013-05-14 19:03:14] Updating code.' [2013-05-14 19:03:14] Updating code. + cd /data/genericrhel6/src/basket.mozilla.org/basket + git fetch origin -q + checkretval + retval=0 + [[ 0 -gt 0 ]] + git checkout -f origin/master -q + git submodule sync -q + git submodule update --init -q + checkretval + retval=0 + [[ 0 -gt 0 ]] + git log -3 + tee /data/genericrhel6/src/basket.mozilla.org/basket/revision_info.txt commit aec5292e5ca49ae7c4e2687b4b2d7d58f6611e22 Merge: 71767fb 088b307 Author: Paul McLanahan <pmac@mozilla.com> Date: Tue May 14 10:29:00 2013 -0700 Merge pull request #41 from pmclanahan/fix-error-on-get-user-data-bug-871764 Bug 871764: Fix errors causing subscriptions to fail. commit 088b3072f5d30603ee512be13d9ae293b5074334 Author: Paul McLanahan <pmac@mozilla.com> Date: Tue May 14 13:13:19 2013 -0400 Bug 871764: Fix errors causing subscriptions to fail. commit 71767fb37fc4790b89a37a5d65bfe7bbafe785c5 Author: Paul McLanahan <pmac@mozilla.com> Date: Fri May 10 00:49:28 2013 -0400 Remove manual sentry capture. Seems to be causing problems in reporting. + techo 'Updating vendor.' ++ date -u '+%F %T' + STAMP='[2013-05-14 19:03:14]' + echo '[2013-05-14 19:03:14] Updating vendor.' [2013-05-14 19:03:14] Updating vendor. + cd /data/genericrhel6/src/basket.mozilla.org/basket/vendor + git submodule sync -q + git submodule update --init -q + checkretval + retval=0 + [[ 0 -gt 0 ]] + techo 'Sync & migrate database.' ++ date -u '+%F %T' + STAMP='[2013-05-14 19:03:16]' + echo '[2013-05-14 19:03:16] Sync & migrate database.' [2013-05-14 19:03:16] Sync & migrate database. + cd /data/genericrhel6/src/basket.mozilla.org/basket + python manage.py syncdb --noinput Syncing... Creating tables ... Installing custom SQL ... Installing indexes ... No fixtures found. Synced: > basketauth > emailer > subscriptions > vars > piston > south > raven.contrib.django.raven_compat > django.contrib.auth > django.contrib.contenttypes > django.contrib.sessions > django.contrib.sites > django.contrib.messages > django.contrib.admin Not synced (use migrations): - news - djcelery (use ./manage.py migrate to migrate these) + checkretval + retval=0 + [[ 0 -gt 0 ]] + python manage.py migrate --noinput Running migrations for news: - Nothing to migrate. - Loading initial data for news. No fixtures found. Running migrations for djcelery: - Nothing to migrate. - Loading initial data for djcelery. No fixtures found. + checkretval + retval=0 + [[ 0 -gt 0 ]] + techo 'Clean up old deploy.' ++ date -u '+%F %T' + STAMP='[2013-05-14 19:03:24]' + echo '[2013-05-14 19:03:24] Clean up old deploy.' [2013-05-14 19:03:24] Clean up old deploy. + cd /data/genericrhel6/src/basket.mozilla.org/basket + find . -type f -name .gitignore -or -name '*.pyc' -delete + checkretval + retval=0 + [[ 0 -gt 0 ]] + cd /data/genericrhel6/src/basket.mozilla.org/basket + touch wsgi/basket.wsgi + techo Deploying. ++ date -u '+%F %T' + STAMP='[2013-05-14 19:03:26]' + echo '[2013-05-14 19:03:26] Deploying.' [2013-05-14 19:03:26] Deploying. + '[' xterm == dumb ']' + /data/genericrhel6/deploy basket.mozilla.org [2013-05-14 12:03:26] Running rsync_project [2013-05-14 12:03:26] [localhost] running: /usr/bin/rsync -aq --include '.gitkeep' --exclude '.git*' --exclude '.hg*' --exclude '.svn*' --exclude 'CVS' --exclude '.bzr*' --delete /data/genericrhel6/src/basket.mozilla.org/ /data/genericrhel6/www/basket.mozilla.org/ [2013-05-14 12:03:27] [localhost] finished: /usr/bin/rsync -aq --include '.gitkeep' --exclude '.git*' --exclude '.hg*' --exclude '.svn*' --exclude 'CVS' --exclude '.bzr*' --delete /data/genericrhel6/src/basket.mozilla.org/ /data/genericrhel6/www/basket.mozilla.org/ (0.452s) [2013-05-14 12:03:27] Finished rsync_project (0.453s) [2013-05-14 12:03:27] Running commit_www [2013-05-14 12:03:27] [localhost] running: cd /data/genericrhel6/www && /usr/bin/git add .; /usr/bin/git commit -a -m 'deploy ['basket.mozilla.org']' [2013-05-14 12:03:32] [localhost] finished: cd /data/genericrhel6/www && /usr/bin/git add .; /usr/bin/git commit -a -m 'deploy ['basket.mozilla.org']' (4.952s) [localhost] out: [master bf6b68e] deploy [basket.mozilla.org] [localhost] out: 4 files changed, 166 insertions(+), 38 deletions(-) [localhost] out: rewrite basket.mozilla.org/basket/revision_info.txt (86%) [2013-05-14 12:03:32] Finished commit_www (4.953s) [2013-05-14 12:03:32] Running push_www [2013-05-14 12:03:32] [generic1.webapp.phx1.mozilla.com] running: /data/bin/update-www.sh basket.mozilla.org [2013-05-14 12:03:32] [generic2.webapp.phx1.mozilla.com] running: /data/bin/update-www.sh basket.mozilla.org [2013-05-14 12:03:32] [generic3.webapp.phx1.mozilla.com] running: /data/bin/update-www.sh basket.mozilla.org [2013-05-14 12:03:32] [generic4.webapp.phx1.mozilla.com] running: /data/bin/update-www.sh basket.mozilla.org [2013-05-14 12:03:32] [generic5.webapp.phx1.mozilla.com] running: /data/bin/update-www.sh basket.mozilla.org [2013-05-14 12:03:32] [generic102.webapp.phx1.mozilla.com] running: /data/bin/update-www.sh basket.mozilla.org [2013-05-14 12:03:38] [generic4.webapp.phx1.mozilla.com] finished: /data/bin/update-www.sh basket.mozilla.org (6.743s) [generic4.webapp.phx1.mozilla.com] out: Not removing ship-it.mozilla.org/release-kickoff/logs/ [2013-05-14 12:03:39] [generic3.webapp.phx1.mozilla.com] finished: /data/bin/update-www.sh basket.mozilla.org (6.787s) [generic3.webapp.phx1.mozilla.com] out: Not removing ship-it.mozilla.org/release-kickoff/logs/ [2013-05-14 12:03:39] [generic2.webapp.phx1.mozilla.com] finished: /data/bin/update-www.sh basket.mozilla.org (6.820s) [generic2.webapp.phx1.mozilla.com] out: Not removing ship-it.mozilla.org/release-kickoff/logs/ [2013-05-14 12:03:39] [generic1.webapp.phx1.mozilla.com] finished: /data/bin/update-www.sh basket.mozilla.org (6.838s) [generic1.webapp.phx1.mozilla.com] out: Not removing fhr.cdn.mozilla.net/web-output.orig/ [generic1.webapp.phx1.mozilla.com] out: Not removing ship-it.mozilla.org/release-kickoff/logs/ [2013-05-14 12:03:39] [generic5.webapp.phx1.mozilla.com] finished: /data/bin/update-www.sh basket.mozilla.org (7.371s) [generic5.webapp.phx1.mozilla.com] out: Not removing ship-it.mozilla.org/release-kickoff/logs/ [2013-05-14 12:03:40] [generic102.webapp.phx1.mozilla.com] finished: /data/bin/update-www.sh basket.mozilla.org (8.196s) [generic102.webapp.phx1.mozilla.com] out: Not removing ship-it.mozilla.org/release-kickoff/logs/ [2013-05-14 12:03:40] Finished push_www (8.208s) + issue-multi-command genericrhel6 touch /data/www/basket.mozilla.org/basket/wsgi/basket.wsgi [2013-05-14 12:03:40] [generic1.webapp.phx1.mozilla.com] running: touch /data/www/basket.mozilla.org/basket/wsgi/basket.wsgi [2013-05-14 12:03:40] [generic2.webapp.phx1.mozilla.com] running: touch /data/www/basket.mozilla.org/basket/wsgi/basket.wsgi [2013-05-14 12:03:40] [generic3.webapp.phx1.mozilla.com] running: touch /data/www/basket.mozilla.org/basket/wsgi/basket.wsgi [2013-05-14 12:03:40] [generic4.webapp.phx1.mozilla.com] running: touch /data/www/basket.mozilla.org/basket/wsgi/basket.wsgi [2013-05-14 12:03:40] [generic5.webapp.phx1.mozilla.com] running: touch /data/www/basket.mozilla.org/basket/wsgi/basket.wsgi [2013-05-14 12:03:40] [generic102.webapp.phx1.mozilla.com] running: touch /data/www/basket.mozilla.org/basket/wsgi/basket.wsgi [2013-05-14 12:03:40] [generic1.webapp.phx1.mozilla.com] finished: touch /data/www/basket.mozilla.org/basket/wsgi/basket.wsgi (0.180s) [2013-05-14 12:03:40] [generic3.webapp.phx1.mozilla.com] finished: touch /data/www/basket.mozilla.org/basket/wsgi/basket.wsgi (0.182s) [2013-05-14 12:03:40] [generic2.webapp.phx1.mozilla.com] finished: touch /data/www/basket.mozilla.org/basket/wsgi/basket.wsgi (0.194s) [2013-05-14 12:03:40] [generic4.webapp.phx1.mozilla.com] finished: touch /data/www/basket.mozilla.org/basket/wsgi/basket.wsgi (0.198s) [2013-05-14 12:03:40] [generic5.webapp.phx1.mozilla.com] finished: touch /data/www/basket.mozilla.org/basket/wsgi/basket.wsgi (0.194s) [2013-05-14 12:03:40] [generic102.webapp.phx1.mozilla.com] finished: touch /data/www/basket.mozilla.org/basket/wsgi/basket.wsgi (0.240s) + update_celery + techo 'Update celery.' ++ date -u '+%F %T' + STAMP='[2013-05-14 19:03:40]' + echo '[2013-05-14 19:03:40] Update celery.' [2013-05-14 19:03:40] Update celery. + ISSUE_MULTI='issue-multi-command -i /root/.ssh/id_rsa genericrhel6-celery' + echo 'sh /data/bin/update-www.sh' + issue-multi-command -i /root/.ssh/id_rsa genericrhel6-celery [2013-05-14 12:03:40] [generic-celery1.webapp.phx1.mozilla.com] running: sh /data/bin/update-www.sh [2013-05-14 12:03:40] [generic-celery2.webapp.phx1.mozilla.com] running: sh /data/bin/update-www.sh [2013-05-14 12:03:40] [generic-celery3.webapp.phx1.mozilla.com] running: sh /data/bin/update-www.sh [2013-05-14 12:03:46] [generic-celery2.webapp.phx1.mozilla.com] finished: sh /data/bin/update-www.sh (5.445s) [generic-celery2.webapp.phx1.mozilla.com] out: Not removing ship-it.mozilla.org/release-kickoff/logs/ [2013-05-14 12:03:46] [generic-celery1.webapp.phx1.mozilla.com] finished: sh /data/bin/update-www.sh (5.961s) [generic-celery1.webapp.phx1.mozilla.com] out: Not removing ship-it.mozilla.org/release-kickoff/logs/ [2013-05-14 12:03:46] [generic-celery3.webapp.phx1.mozilla.com] finished: sh /data/bin/update-www.sh (6.000s) [generic-celery3.webapp.phx1.mozilla.com] out: Not removing ship-it.mozilla.org/release-kickoff/logs/ + echo '/etc/init.d/celery-basket restart' + issue-multi-command -i /root/.ssh/id_rsa genericrhel6-celery [2013-05-14 12:03:46] [generic-celery1.webapp.phx1.mozilla.com] running: /etc/init.d/celery-basket restart [2013-05-14 12:03:46] [generic-celery2.webapp.phx1.mozilla.com] running: /etc/init.d/celery-basket restart [2013-05-14 12:03:46] [generic-celery3.webapp.phx1.mozilla.com] running: /etc/init.d/celery-basket restart [2013-05-14 12:03:58] [generic-celery1.webapp.phx1.mozilla.com] finished: /etc/init.d/celery-basket restart (11.122s) [generic-celery1.webapp.phx1.mozilla.com] out: Restarting celery-basket: celery-basket: stopped [generic-celery1.webapp.phx1.mozilla.com] out: celery-basket: started [generic-celery1.webapp.phx1.mozilla.com] out: [ OK ] [2013-05-14 12:03:58] [generic-celery3.webapp.phx1.mozilla.com] finished: /etc/init.d/celery-basket restart (11.325s) [generic-celery3.webapp.phx1.mozilla.com] out: Restarting celery-basket: celery-basket: stopped [generic-celery3.webapp.phx1.mozilla.com] out: celery-basket: started [generic-celery3.webapp.phx1.mozilla.com] out: [ OK ] [2013-05-14 12:04:04] [generic-celery2.webapp.phx1.mozilla.com] finished: /etc/init.d/celery-basket restart (17.113s) [generic-celery2.webapp.phx1.mozilla.com] out: Restarting celery-basket: celery-basket: stopped [generic-celery2.webapp.phx1.mozilla.com] out: celery-basket: started [generic-celery2.webapp.phx1.mozilla.com] out: [ OK ]
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Working again!! Thanks cturra.
Status: RESOLVED → VERIFIED
Component: Server Operations: Web Operations → WebOps: Other
Product: mozilla.org → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.