Closed Bug 910850 Opened 12 years ago Closed 12 years ago

push Socorro 57.1 release

Categories

(Infrastructure & Operations Graveyard :: WebOps: Socorro, task, P3)

Product:
Infrastructure & Operations Graveyard
Component:
WebOps: Socorro
Type:
task

Tracking

(Not tracked)

Status:
VERIFIED FIXED

People

(Reporter: lonnen, Assigned: bburton)

Details

(Whiteboard: [push interrupt])

Build: https://ci.mozilla.org/job/socorro-release/486/artifact/socorro.tar.gz In order to clear a regression affecting our nagios alerts on prod, we're pushing a second time this week with a handful of stability fixes that have landed since yesterday. These fixes will be targeted to 58, which will be tagged next week.
Assignee: server-ops-webops → bburton
Django pushed [root@socorroadm.private.phx1 crash-stats.mozilla.org]# bash -x update https://ci.mozilla.org/job/socorro-release/486/artifact/socorro.tar.gz +++ dirname update ++ cd . ++ pwd + DIR=/data/crashstats/src/crash-stats.mozilla.org ++ echo /data/crashstats/src/crash-stats.mozilla.org ++ cut -f3 -d/ + ENV=crashstats ++ echo /data/crashstats/src/crash-stats.mozilla.org ++ cut -f5 -d/ + SITE=crash-stats.mozilla.org + APP=socorro + echo 'DIR: /data/crashstats/src/crash-stats.mozilla.org' DIR: /data/crashstats/src/crash-stats.mozilla.org + echo 'ENV: crashstats' ENV: crashstats + echo 'SITE: crash-stats.mozilla.org' SITE: crash-stats.mozilla.org + echo 'APP: socorro' APP: socorro + echo 'Upgrading socorro-crashstats (django) prod' Upgrading socorro-crashstats (django) prod + '[' 1 '!=' 1 ']' + url=https://ci.mozilla.org/job/socorro-release/486/artifact/socorro.tar.gz + echo 'Upgrading socorro to https://ci.mozilla.org/job/socorro-release/486/artifact/socorro.tar.gz' Upgrading socorro to https://ci.mozilla.org/job/socorro-release/486/artifact/socorro.tar.gz + pushd /data/crashstats/src/crash-stats.mozilla.org /data/crashstats/src/crash-stats.mozilla.org /data/crashstats/src/crash-stats.mozilla.org + curl -s -O https://ci.mozilla.org/job/socorro-release/486/artifact/socorro.tar.gz + tar xzf socorro.tar.gz + cp -f /data/crashstats/src/crash-stats.mozilla.org/local.py /data/crashstats/src/crash-stats.mozilla.org/socorro/webapp-django/crashstats/settings/ + python socorro/webapp-django/manage.py syncdb Creating tables ... Installing custom SQL ... Installing indexes ... Installed 0 object(s) from 0 fixture(s) + /data/crashstats/deploy crash-stats.mozilla.org [2013-08-29 19:40:19] Running rsync_project [2013-08-29 19:40:19] [localhost] running: /usr/bin/rsync -aq --include '.gitkeep' --exclude '.git*' --exclude '.hg*' --exclude '.svn*' --exclude 'CVS' --exclude '.bzr*' --delete /data/crashstats/src/crash-stats.mozilla.org/ /data/crashstats/www/crash-stats.mozilla.org/ [2013-08-29 19:40:20] [localhost] finished: /usr/bin/rsync -aq --include '.gitkeep' --exclude '.git*' --exclude '.hg*' --exclude '.svn*' --exclude 'CVS' --exclude '.bzr*' --delete /data/crashstats/src/crash-stats.mozilla.org/ /data/crashstats/www/crash-stats.mozilla.org/ (1.266s) [2013-08-29 19:40:20] Finished rsync_project (1.267s) [2013-08-29 19:40:20] Running commit_www [2013-08-29 19:40:20] [localhost] running: cd /data/crashstats/www && /usr/bin/git add .; /usr/bin/git commit -a -m 'deploy ['crash-stats.mozilla.org']' [2013-08-29 19:40:23] [localhost] finished: cd /data/crashstats/www && /usr/bin/git add .; /usr/bin/git commit -a -m 'deploy ['crash-stats.mozilla.org']' (2.862s) [localhost] out: [master 17352ce] deploy [crash-stats.mozilla.org] [localhost] out: 85 files changed, 2442 insertions(+), 1057 deletions(-) [localhost] out: create mode 100755 crash-stats.mozilla.org/socorro/socorro-virtualenv/lib/python2.6/site-packages/configman-1.1.9-py2.6.egg-info/PKG-INFO [localhost] out: create mode 100755 crash-stats.mozilla.org/socorro/socorro-virtualenv/lib/python2.6/site-packages/configman-1.1.9-py2.6.egg-info/SOURCES.txt [localhost] out: create mode 100755 crash-stats.mozilla.org/socorro/socorro-virtualenv/lib/python2.6/site-packages/configman-1.1.9-py2.6.egg-info/dependency_links.txt [localhost] out: create mode 100755 crash-stats.mozilla.org/socorro/socorro-virtualenv/lib/python2.6/site-packages/configman-1.1.9-py2.6.egg-info/installed-files.txt [localhost] out: create mode 100755 crash-stats.mozilla.org/socorro/socorro-virtualenv/lib/python2.6/site-packages/configman-1.1.9-py2.6.egg-info/not-zip-safe [localhost] out: create mode 100755 crash-stats.mozilla.org/socorro/socorro-virtualenv/lib/python2.6/site-packages/configman-1.1.9-py2.6.egg-info/top_level.txt [localhost] out: create mode 100755 crash-stats.mozilla.org/socorro/webapp-django/static/CACHE/js/afa5794d4ad9.js [localhost] out: create mode 100755 crash-stats.mozilla.org/socorro/webapp-django/static/CACHE/js/cd44bf51f206.js [2013-08-29 19:40:23] Finished commit_www (2.863s) [2013-08-29 19:40:23] Running push_www [2013-08-29 19:40:23] [socorro1.webapp.phx1.mozilla.com] running: /data/bin/update-www.sh crash-stats.mozilla.org [2013-08-29 19:40:23] [socorro2.webapp.phx1.mozilla.com] running: /data/bin/update-www.sh crash-stats.mozilla.org [2013-08-29 19:40:23] [socorro3.webapp.phx1.mozilla.com] running: /data/bin/update-www.sh crash-stats.mozilla.org [2013-08-29 19:40:23] [socorro4.webapp.phx1.mozilla.com] running: /data/bin/update-www.sh crash-stats.mozilla.org [2013-08-29 19:40:37] [socorro4.webapp.phx1.mozilla.com] finished: /data/bin/update-www.sh crash-stats.mozilla.org (14.135s) [2013-08-29 19:40:37] [socorro2.webapp.phx1.mozilla.com] finished: /data/bin/update-www.sh crash-stats.mozilla.org (14.445s) [2013-08-29 19:40:37] [socorro3.webapp.phx1.mozilla.com] finished: /data/bin/update-www.sh crash-stats.mozilla.org (14.470s) [2013-08-29 19:40:37] [socorro1.webapp.phx1.mozilla.com] finished: /data/bin/update-www.sh crash-stats.mozilla.org (14.611s) [socorro1.webapp.phx1.mozilla.com] out: Not removing crash-stats.mozilla.org/app/ [2013-08-29 19:40:37] Finished push_www (14.612s) + issue-multi-command crashstats service httpd restart [2013-08-29 19:40:37] [socorro1.webapp.phx1.mozilla.com] running: service httpd restart [2013-08-29 19:40:37] [socorro2.webapp.phx1.mozilla.com] running: service httpd restart [2013-08-29 19:40:37] [socorro3.webapp.phx1.mozilla.com] running: service httpd restart [2013-08-29 19:40:37] [socorro4.webapp.phx1.mozilla.com] running: service httpd restart [2013-08-29 19:40:39] [socorro4.webapp.phx1.mozilla.com] finished: service httpd restart (1.717s) [socorro4.webapp.phx1.mozilla.com] out: Stopping httpd: [ OK ] [socorro4.webapp.phx1.mozilla.com] out: Starting httpd: [ OK ] [2013-08-29 19:40:39] [socorro3.webapp.phx1.mozilla.com] finished: service httpd restart (1.800s) [socorro3.webapp.phx1.mozilla.com] out: Stopping httpd: [ OK ] [socorro3.webapp.phx1.mozilla.com] out: Starting httpd: [ OK ] [2013-08-29 19:40:39] [socorro1.webapp.phx1.mozilla.com] finished: service httpd restart (1.836s) [socorro1.webapp.phx1.mozilla.com] out: Stopping httpd: [ OK ] [socorro1.webapp.phx1.mozilla.com] out: Starting httpd: [ OK ] [2013-08-29 19:40:39] [socorro2.webapp.phx1.mozilla.com] finished: service httpd restart (1.838s) [socorro2.webapp.phx1.mozilla.com] out: Stopping httpd: [ OK ] [socorro2.webapp.phx1.mozilla.com] out: Starting httpd: [ OK ] + issue-multi-command memcache service memcached restart [2013-08-29 19:40:39] [socorro-memcache1.webapp.phx1.mozilla.com] running: service memcached restart [2013-08-29 19:40:39] [socorro-memcache2.webapp.phx1.mozilla.com] running: service memcached restart [2013-08-29 19:40:40] [socorro-memcache2.webapp.phx1.mozilla.com] finished: service memcached restart (0.539s) [socorro-memcache2.webapp.phx1.mozilla.com] out: Stopping memcached: [ OK ] [socorro-memcache2.webapp.phx1.mozilla.com] out: Starting memcached: [ OK ] [2013-08-29 19:40:40] [socorro-memcache1.webapp.phx1.mozilla.com] finished: service memcached restart (0.589s) [socorro-memcache1.webapp.phx1.mozilla.com] out: Stopping memcached: [ OK ] [socorro-memcache1.webapp.phx1.mozilla.com] out: Starting memcached: [ OK ]
Status: NEW → ASSIGNED
The rest of production was upgraded successfully [root@sp-admin01.phx1 ~]# bash -x /root/bin/update-socorro-all.sh https://ci.mozilla.org/job/socorro-release/486/artifact/socorro.tar.gz + HOSTS='sp-collector01.phx1.mozilla.com sp-collector02.phx1.mozilla.com sp-collector03.phx1.mozilla.com sp-collector04.phx1.mozilla.com sp-collector05.phx1.mozilla.com sp-collector06.phx1.mozilla.com sp-mware01.phx1.mozilla.com sp-mware02.phx1.mozilla.com sp-mware03.phx1.mozilla.com sp-processor01.phx1.mozilla.com sp-processor02.phx1.mozilla.com sp-processor03.phx1.mozilla.com sp-processor04.phx1.mozilla.com sp-processor05.phx1.mozilla.com sp-processor06.phx1.mozilla.com sp-processor07.phx1.mozilla.com sp-processor08.phx1.mozilla.com sp-processor09.phx1.mozilla.com sp-processor10.phx1.mozilla.com sp-web01.phx1.mozilla.com sp-web02.phx1.mozilla.com sp-web03.phx1.mozilla.com sp-web04.phx1.mozilla.com sp-web05.phx1.mozilla.com sp-admin01.phx1.mozilla.com tp-socorro01-master01.phx1.mozilla.com tp-socorro01-master02.phx1.mozilla.com' + '[' 1 '!=' 1 ']' + url=https://ci.mozilla.org/job/socorro-release/486/artifact/socorro.tar.gz + echo 'Upgrading socorro to https://ci.mozilla.org/job/socorro-release/486/artifact/socorro.tar.gz' Upgrading socorro to https://ci.mozilla.org/job/socorro-release/486/artifact/socorro.tar.gz + for host in '$HOSTS' + echo 'Upgrading socorro on sp-collector01.phx1.mozilla.com' Upgrading socorro on sp-collector01.phx1.mozilla.com + ssh sp-collector01.phx1.mozilla.com 'sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/486/artifact/socorro.tar.gz' Stopping socorro-crashmover: [ OK ] Starting socorro-crashmover: [ OK ] Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/486/artifact/socorro.tar.gz Checksum: c3207884a0deb81179b7a8f4f157c80e Backed up original to /data/socorro.29-08-2013_12_41_24 + fatal 0 'Could not upgrade socorro on sp-collector01.phx1.mozilla.com' + '[' 2 '!=' 2 ']' + exit_code=0 + message='Could not upgrade socorro on sp-collector01.phx1.mozilla.com' + '[' 0 '!=' 0 ']' + for host in '$HOSTS' + echo 'Upgrading socorro on sp-collector02.phx1.mozilla.com' Upgrading socorro on sp-collector02.phx1.mozilla.com + ssh sp-collector02.phx1.mozilla.com 'sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/486/artifact/socorro.tar.gz' Stopping socorro-crashmover: [ OK ] Starting socorro-crashmover: [ OK ] Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/486/artifact/socorro.tar.gz Checksum: c3207884a0deb81179b7a8f4f157c80e Backed up original to /data/socorro.29-08-2013_19_41_31 + fatal 0 'Could not upgrade socorro on sp-collector02.phx1.mozilla.com' + '[' 2 '!=' 2 ']' + exit_code=0 + message='Could not upgrade socorro on sp-collector02.phx1.mozilla.com' + '[' 0 '!=' 0 ']' + for host in '$HOSTS' + echo 'Upgrading socorro on sp-collector03.phx1.mozilla.com' Upgrading socorro on sp-collector03.phx1.mozilla.com + ssh sp-collector03.phx1.mozilla.com 'sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/486/artifact/socorro.tar.gz' Stopping socorro-crashmover: [ OK ] Starting socorro-crashmover: [ OK ] Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/486/artifact/socorro.tar.gz Checksum: c3207884a0deb81179b7a8f4f157c80e Backed up original to /data/socorro.29-08-2013_19_41_38 + fatal 0 'Could not upgrade socorro on sp-collector03.phx1.mozilla.com' + '[' 2 '!=' 2 ']' + exit_code=0 + message='Could not upgrade socorro on sp-collector03.phx1.mozilla.com' + '[' 0 '!=' 0 ']' + for host in '$HOSTS' + echo 'Upgrading socorro on sp-collector04.phx1.mozilla.com' Upgrading socorro on sp-collector04.phx1.mozilla.com + ssh sp-collector04.phx1.mozilla.com 'sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/486/artifact/socorro.tar.gz' Stopping socorro-crashmover: [ OK ] Starting socorro-crashmover: [ OK ] Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/486/artifact/socorro.tar.gz Checksum: c3207884a0deb81179b7a8f4f157c80e Backed up original to /data/socorro.29-08-2013_19_41_55 + fatal 0 'Could not upgrade socorro on sp-collector04.phx1.mozilla.com' + '[' 2 '!=' 2 ']' + exit_code=0 + message='Could not upgrade socorro on sp-collector04.phx1.mozilla.com' + '[' 0 '!=' 0 ']' + for host in '$HOSTS' + echo 'Upgrading socorro on sp-collector05.phx1.mozilla.com' Upgrading socorro on sp-collector05.phx1.mozilla.com + ssh sp-collector05.phx1.mozilla.com 'sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/486/artifact/socorro.tar.gz' Stopping socorro-crashmover: [ OK ] Starting socorro-crashmover: [ OK ] Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/486/artifact/socorro.tar.gz Checksum: c3207884a0deb81179b7a8f4f157c80e Backed up original to /data/socorro.29-08-2013_19_42_01 + fatal 0 'Could not upgrade socorro on sp-collector05.phx1.mozilla.com' + '[' 2 '!=' 2 ']' + exit_code=0 + message='Could not upgrade socorro on sp-collector05.phx1.mozilla.com' + '[' 0 '!=' 0 ']' + for host in '$HOSTS' + echo 'Upgrading socorro on sp-collector06.phx1.mozilla.com' Upgrading socorro on sp-collector06.phx1.mozilla.com + ssh sp-collector06.phx1.mozilla.com 'sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/486/artifact/socorro.tar.gz' Stopping socorro-crashmover: [ OK ] Starting socorro-crashmover: [ OK ] Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/486/artifact/socorro.tar.gz Checksum: c3207884a0deb81179b7a8f4f157c80e Backed up original to /data/socorro.29-08-2013_19_42_07 + fatal 0 'Could not upgrade socorro on sp-collector06.phx1.mozilla.com' + '[' 2 '!=' 2 ']' + exit_code=0 + message='Could not upgrade socorro on sp-collector06.phx1.mozilla.com' + '[' 0 '!=' 0 ']' + for host in '$HOSTS' + echo 'Upgrading socorro on sp-mware01.phx1.mozilla.com' Upgrading socorro on sp-mware01.phx1.mozilla.com + ssh sp-mware01.phx1.mozilla.com 'sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/486/artifact/socorro.tar.gz' Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/486/artifact/socorro.tar.gz Checksum: c3207884a0deb81179b7a8f4f157c80e Backed up original to /data/socorro.29-08-2013_12_42_15 + fatal 0 'Could not upgrade socorro on sp-mware01.phx1.mozilla.com' + '[' 2 '!=' 2 ']' + exit_code=0 + message='Could not upgrade socorro on sp-mware01.phx1.mozilla.com' + '[' 0 '!=' 0 ']' + for host in '$HOSTS' + echo 'Upgrading socorro on sp-mware02.phx1.mozilla.com' Upgrading socorro on sp-mware02.phx1.mozilla.com + ssh sp-mware02.phx1.mozilla.com 'sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/486/artifact/socorro.tar.gz' Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/486/artifact/socorro.tar.gz Checksum: c3207884a0deb81179b7a8f4f157c80e Backed up original to /data/socorro.29-08-2013_12_42_21 + fatal 0 'Could not upgrade socorro on sp-mware02.phx1.mozilla.com' + '[' 2 '!=' 2 ']' + exit_code=0 + message='Could not upgrade socorro on sp-mware02.phx1.mozilla.com' + '[' 0 '!=' 0 ']' + for host in '$HOSTS' + echo 'Upgrading socorro on sp-mware03.phx1.mozilla.com' Upgrading socorro on sp-mware03.phx1.mozilla.com + ssh sp-mware03.phx1.mozilla.com 'sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/486/artifact/socorro.tar.gz' Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/486/artifact/socorro.tar.gz Checksum: c3207884a0deb81179b7a8f4f157c80e Backed up original to /data/socorro.29-08-2013_12_42_25 + fatal 0 'Could not upgrade socorro on sp-mware03.phx1.mozilla.com' + '[' 2 '!=' 2 ']' + exit_code=0 + message='Could not upgrade socorro on sp-mware03.phx1.mozilla.com' + '[' 0 '!=' 0 ']' + for host in '$HOSTS' + echo 'Upgrading socorro on sp-processor01.phx1.mozilla.com' Upgrading socorro on sp-processor01.phx1.mozilla.com + ssh sp-processor01.phx1.mozilla.com 'sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/486/artifact/socorro.tar.gz' Stopping socorro-processor: [ OK ] Starting socorro-processor: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/486/artifact/socorro.tar.gz Checksum: c3207884a0deb81179b7a8f4f157c80e Backed up original to /data/socorro.29-08-2013_12_42_29 + fatal 0 'Could not upgrade socorro on sp-processor01.phx1.mozilla.com' + '[' 2 '!=' 2 ']' + exit_code=0 + message='Could not upgrade socorro on sp-processor01.phx1.mozilla.com' + '[' 0 '!=' 0 ']' + for host in '$HOSTS' + echo 'Upgrading socorro on sp-processor02.phx1.mozilla.com' Upgrading socorro on sp-processor02.phx1.mozilla.com + ssh sp-processor02.phx1.mozilla.com 'sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/486/artifact/socorro.tar.gz' Stopping socorro-processor: [ OK ] Starting socorro-processor: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/486/artifact/socorro.tar.gz Checksum: c3207884a0deb81179b7a8f4f157c80e Backed up original to /data/socorro.29-08-2013_12_42_38 + fatal 0 'Could not upgrade socorro on sp-processor02.phx1.mozilla.com' + '[' 2 '!=' 2 ']' + exit_code=0 + message='Could not upgrade socorro on sp-processor02.phx1.mozilla.com' + '[' 0 '!=' 0 ']' + for host in '$HOSTS' + echo 'Upgrading socorro on sp-processor03.phx1.mozilla.com' Upgrading socorro on sp-processor03.phx1.mozilla.com + ssh sp-processor03.phx1.mozilla.com 'sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/486/artifact/socorro.tar.gz' Stopping socorro-processor: [ OK ] Starting socorro-processor: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/486/artifact/socorro.tar.gz Checksum: c3207884a0deb81179b7a8f4f157c80e Backed up original to /data/socorro.29-08-2013_12_42_48 + fatal 0 'Could not upgrade socorro on sp-processor03.phx1.mozilla.com' + '[' 2 '!=' 2 ']' + exit_code=0 + message='Could not upgrade socorro on sp-processor03.phx1.mozilla.com' + '[' 0 '!=' 0 ']' + for host in '$HOSTS' + echo 'Upgrading socorro on sp-processor04.phx1.mozilla.com' Upgrading socorro on sp-processor04.phx1.mozilla.com + ssh sp-processor04.phx1.mozilla.com 'sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/486/artifact/socorro.tar.gz' Stopping socorro-processor: [ OK ] Starting socorro-processor: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/486/artifact/socorro.tar.gz Checksum: c3207884a0deb81179b7a8f4f157c80e Backed up original to /data/socorro.29-08-2013_12_43_13 + fatal 0 'Could not upgrade socorro on sp-processor04.phx1.mozilla.com' + '[' 2 '!=' 2 ']' + exit_code=0 + message='Could not upgrade socorro on sp-processor04.phx1.mozilla.com' + '[' 0 '!=' 0 ']' + for host in '$HOSTS' + echo 'Upgrading socorro on sp-processor05.phx1.mozilla.com' Upgrading socorro on sp-processor05.phx1.mozilla.com + ssh sp-processor05.phx1.mozilla.com 'sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/486/artifact/socorro.tar.gz' Stopping socorro-processor: [ OK ] Starting socorro-processor: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/486/artifact/socorro.tar.gz Checksum: c3207884a0deb81179b7a8f4f157c80e Backed up original to /data/socorro.29-08-2013_12_43_21 + fatal 0 'Could not upgrade socorro on sp-processor05.phx1.mozilla.com' + '[' 2 '!=' 2 ']' + exit_code=0 + message='Could not upgrade socorro on sp-processor05.phx1.mozilla.com' + '[' 0 '!=' 0 ']' + for host in '$HOSTS' + echo 'Upgrading socorro on sp-processor06.phx1.mozilla.com' Upgrading socorro on sp-processor06.phx1.mozilla.com + ssh sp-processor06.phx1.mozilla.com 'sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/486/artifact/socorro.tar.gz' Stopping socorro-processor: [ OK ] Starting socorro-processor: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/486/artifact/socorro.tar.gz Checksum: c3207884a0deb81179b7a8f4f157c80e Backed up original to /data/socorro.29-08-2013_12_43_30 + fatal 0 'Could not upgrade socorro on sp-processor06.phx1.mozilla.com' + '[' 2 '!=' 2 ']' + exit_code=0 + message='Could not upgrade socorro on sp-processor06.phx1.mozilla.com' + '[' 0 '!=' 0 ']' + for host in '$HOSTS' + echo 'Upgrading socorro on sp-processor07.phx1.mozilla.com' Upgrading socorro on sp-processor07.phx1.mozilla.com + ssh sp-processor07.phx1.mozilla.com 'sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/486/artifact/socorro.tar.gz' Stopping socorro-processor: [ OK ] Starting socorro-processor: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/486/artifact/socorro.tar.gz Checksum: c3207884a0deb81179b7a8f4f157c80e Backed up original to /data/socorro.29-08-2013_12_43_38 + fatal 0 'Could not upgrade socorro on sp-processor07.phx1.mozilla.com' + '[' 2 '!=' 2 ']' + exit_code=0 + message='Could not upgrade socorro on sp-processor07.phx1.mozilla.com' + '[' 0 '!=' 0 ']' + for host in '$HOSTS' + echo 'Upgrading socorro on sp-processor08.phx1.mozilla.com' Upgrading socorro on sp-processor08.phx1.mozilla.com + ssh sp-processor08.phx1.mozilla.com 'sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/486/artifact/socorro.tar.gz' Stopping socorro-processor: [ OK ] Starting socorro-processor: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/486/artifact/socorro.tar.gz Checksum: c3207884a0deb81179b7a8f4f157c80e Backed up original to /data/socorro.29-08-2013_12_44_02 + fatal 0 'Could not upgrade socorro on sp-processor08.phx1.mozilla.com' + '[' 2 '!=' 2 ']' + exit_code=0 + message='Could not upgrade socorro on sp-processor08.phx1.mozilla.com' + '[' 0 '!=' 0 ']' + for host in '$HOSTS' + echo 'Upgrading socorro on sp-processor09.phx1.mozilla.com' Upgrading socorro on sp-processor09.phx1.mozilla.com + ssh sp-processor09.phx1.mozilla.com 'sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/486/artifact/socorro.tar.gz' Stopping socorro-processor: [ OK ] Starting socorro-processor: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/486/artifact/socorro.tar.gz Checksum: c3207884a0deb81179b7a8f4f157c80e Backed up original to /data/socorro.29-08-2013_12_44_27 + fatal 0 'Could not upgrade socorro on sp-processor09.phx1.mozilla.com' + '[' 2 '!=' 2 ']' + exit_code=0 + message='Could not upgrade socorro on sp-processor09.phx1.mozilla.com' + '[' 0 '!=' 0 ']' + for host in '$HOSTS' + echo 'Upgrading socorro on sp-processor10.phx1.mozilla.com' Upgrading socorro on sp-processor10.phx1.mozilla.com + ssh sp-processor10.phx1.mozilla.com 'sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/486/artifact/socorro.tar.gz' Stopping socorro-processor: [ OK ] Starting socorro-processor: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/486/artifact/socorro.tar.gz Checksum: c3207884a0deb81179b7a8f4f157c80e Backed up original to /data/socorro.29-08-2013_12_44_51 + fatal 0 'Could not upgrade socorro on sp-processor10.phx1.mozilla.com' + '[' 2 '!=' 2 ']' + exit_code=0 + message='Could not upgrade socorro on sp-processor10.phx1.mozilla.com' + '[' 0 '!=' 0 ']' + for host in '$HOSTS' + echo 'Upgrading socorro on sp-web01.phx1.mozilla.com' Upgrading socorro on sp-web01.phx1.mozilla.com + ssh sp-web01.phx1.mozilla.com 'sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/486/artifact/socorro.tar.gz' Stopping httpd: [ OK ] Starting httpd: Warning: DocumentRoot [/data/socorro/htdocs] does not exist Warning: DocumentRoot [/data/socorro/htdocs] does not exist [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/486/artifact/socorro.tar.gz Checksum: c3207884a0deb81179b7a8f4f157c80e Backed up original to /data/socorro.29-08-2013_12_45_15 + fatal 0 'Could not upgrade socorro on sp-web01.phx1.mozilla.com' + '[' 2 '!=' 2 ']' + exit_code=0 + message='Could not upgrade socorro on sp-web01.phx1.mozilla.com' + '[' 0 '!=' 0 ']' + for host in '$HOSTS' + echo 'Upgrading socorro on sp-web02.phx1.mozilla.com' Upgrading socorro on sp-web02.phx1.mozilla.com + ssh sp-web02.phx1.mozilla.com 'sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/486/artifact/socorro.tar.gz' Stopping httpd: [ OK ] Starting httpd: Warning: DocumentRoot [/data/socorro/htdocs] does not exist Warning: DocumentRoot [/data/socorro/htdocs] does not exist [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/486/artifact/socorro.tar.gz Checksum: c3207884a0deb81179b7a8f4f157c80e Backed up original to /data/socorro.29-08-2013_12_45_20 + fatal 0 'Could not upgrade socorro on sp-web02.phx1.mozilla.com' + '[' 2 '!=' 2 ']' + exit_code=0 + message='Could not upgrade socorro on sp-web02.phx1.mozilla.com' + '[' 0 '!=' 0 ']' + for host in '$HOSTS' + echo 'Upgrading socorro on sp-web03.phx1.mozilla.com' Upgrading socorro on sp-web03.phx1.mozilla.com + ssh sp-web03.phx1.mozilla.com 'sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/486/artifact/socorro.tar.gz' Stopping httpd: [ OK ] Starting httpd: Warning: DocumentRoot [/data/socorro/htdocs] does not exist Warning: DocumentRoot [/data/socorro/htdocs] does not exist [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/486/artifact/socorro.tar.gz Checksum: c3207884a0deb81179b7a8f4f157c80e Backed up original to /data/socorro.29-08-2013_12_45_26 + fatal 0 'Could not upgrade socorro on sp-web03.phx1.mozilla.com' + '[' 2 '!=' 2 ']' + exit_code=0 + message='Could not upgrade socorro on sp-web03.phx1.mozilla.com' + '[' 0 '!=' 0 ']' + for host in '$HOSTS' + echo 'Upgrading socorro on sp-web04.phx1.mozilla.com' Upgrading socorro on sp-web04.phx1.mozilla.com + ssh sp-web04.phx1.mozilla.com 'sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/486/artifact/socorro.tar.gz' Stopping httpd: [ OK ] Starting httpd: Warning: DocumentRoot [/data/socorro/htdocs] does not exist Warning: DocumentRoot [/data/socorro/htdocs] does not exist [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/486/artifact/socorro.tar.gz Checksum: c3207884a0deb81179b7a8f4f157c80e Backed up original to /data/socorro.29-08-2013_12_45_31 + fatal 0 'Could not upgrade socorro on sp-web04.phx1.mozilla.com' + '[' 2 '!=' 2 ']' + exit_code=0 + message='Could not upgrade socorro on sp-web04.phx1.mozilla.com' + '[' 0 '!=' 0 ']' + for host in '$HOSTS' + echo 'Upgrading socorro on sp-web05.phx1.mozilla.com' Upgrading socorro on sp-web05.phx1.mozilla.com + ssh sp-web05.phx1.mozilla.com 'sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/486/artifact/socorro.tar.gz' Stopping httpd: [ OK ] Starting httpd: Warning: DocumentRoot [/data/socorro/htdocs] does not exist Warning: DocumentRoot [/data/socorro/htdocs] does not exist [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/486/artifact/socorro.tar.gz Checksum: c3207884a0deb81179b7a8f4f157c80e Backed up original to /data/socorro.29-08-2013_12_45_36 + fatal 0 'Could not upgrade socorro on sp-web05.phx1.mozilla.com' + '[' 2 '!=' 2 ']' + exit_code=0 + message='Could not upgrade socorro on sp-web05.phx1.mozilla.com' + '[' 0 '!=' 0 ']' + for host in '$HOSTS' + echo 'Upgrading socorro on sp-admin01.phx1.mozilla.com' Upgrading socorro on sp-admin01.phx1.mozilla.com + ssh sp-admin01.phx1.mozilla.com 'sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/486/artifact/socorro.tar.gz' Stopping socorro-monitor: [ OK ] Starting socorro-monitor: [ OK ] Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/486/artifact/socorro.tar.gz Checksum: c3207884a0deb81179b7a8f4f157c80e Backed up original to /data/socorro.29-08-2013_12_45_39 + fatal 0 'Could not upgrade socorro on sp-admin01.phx1.mozilla.com' + '[' 2 '!=' 2 ']' + exit_code=0 + message='Could not upgrade socorro on sp-admin01.phx1.mozilla.com' + '[' 0 '!=' 0 ']' + for host in '$HOSTS' + echo 'Upgrading socorro on tp-socorro01-master01.phx1.mozilla.com' Upgrading socorro on tp-socorro01-master01.phx1.mozilla.com + ssh tp-socorro01-master01.phx1.mozilla.com 'sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/486/artifact/socorro.tar.gz' Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/486/artifact/socorro.tar.gz Checksum: c3207884a0deb81179b7a8f4f157c80e Backed up original to /data/socorro.29-08-2013_12_45_47 + fatal 0 'Could not upgrade socorro on tp-socorro01-master01.phx1.mozilla.com' + '[' 2 '!=' 2 ']' + exit_code=0 + message='Could not upgrade socorro on tp-socorro01-master01.phx1.mozilla.com' + '[' 0 '!=' 0 ']' + for host in '$HOSTS' + echo 'Upgrading socorro on tp-socorro01-master02.phx1.mozilla.com' Upgrading socorro on tp-socorro01-master02.phx1.mozilla.com + ssh tp-socorro01-master02.phx1.mozilla.com 'sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/486/artifact/socorro.tar.gz' Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/486/artifact/socorro.tar.gz Checksum: c3207884a0deb81179b7a8f4f157c80e Backed up original to /data/socorro.29-08-2013_12_45_54 + fatal 0 'Could not upgrade socorro on tp-socorro01-master02.phx1.mozilla.com' + '[' 2 '!=' 2 ']' + exit_code=0 + message='Could not upgrade socorro on tp-socorro01-master02.phx1.mozilla.com' + '[' 0 '!=' 0 ']' + issue-multi-command web service memcached restart [sp-web01.phx1.mozilla.com] running: service memcached restart [sp-web02.phx1.mozilla.com] running: service memcached restart [sp-web03.phx1.mozilla.com] running: service memcached restart [sp-web04.phx1.mozilla.com] running: service memcached restart [sp-web05.phx1.mozilla.com] running: service memcached restart [sp-web01.phx1.mozilla.com] finished: service memcached restart (0.501s) [sp-web01.phx1.mozilla.com] out: Stopping memcached: [ OK ] [sp-web01.phx1.mozilla.com] out: Starting memcached: [ OK ] [sp-web04.phx1.mozilla.com] finished: service memcached restart (0.599s) [sp-web04.phx1.mozilla.com] out: Stopping memcached: [ OK ] [sp-web04.phx1.mozilla.com] out: Starting memcached: [ OK ] [sp-web05.phx1.mozilla.com] finished: service memcached restart (0.609s) [sp-web05.phx1.mozilla.com] out: Stopping memcached: [ OK ] [sp-web05.phx1.mozilla.com] out: Starting memcached: [ OK ] [sp-web02.phx1.mozilla.com] finished: service memcached restart (1.391s) [sp-web02.phx1.mozilla.com] out: Stopping memcached: [ OK ] [sp-web02.phx1.mozilla.com] out: Starting memcached: [ OK ] [sp-web03.phx1.mozilla.com] finished: service memcached restart (1.482s) [sp-web03.phx1.mozilla.com] out: Stopping memcached: [ OK ] [sp-web03.phx1.mozilla.com] out: Starting memcached: [ OK ]
Status: ASSIGNED → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
socorro.prod: last build: 95 (12 min ago): SUCCESS: http://qa-selenium.mv.mozilla.com:8080/job/socorro.prod/95/ all UI bugs verified in prod
Bumping to QA verified - automation is passing and feature bugs have been manually spot checked. - http://qa-selenium.mv.mozilla.com:8080/job/socorro.prod/96/
Status: RESOLVED → VERIFIED
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.