Status

Infrastructure & Operations Graveyard
WebOps: Socorro
VERIFIED FIXED
4 years ago
a year ago

People

(Reporter: lonnen, Assigned: solarce)

Tracking

Details

(Reporter)

Description

4 years ago
Build: https://ci.mozilla.org/job/socorro-release/602/artifact/socorro.tar.gz

release at 1pm pst
(Assignee)

Updated

4 years ago
Assignee: server-ops-webops → bburton
(Assignee)

Comment 1

4 years ago
Django push complete

[root@socorroadm.private.phx1 ~]# /data/crashstats/src/crash-stats.mozilla.org/update https://ci.mozilla.org/job/socorro-release/602/artifact/socorro.tar.gz
DIR: /data/crashstats/src/crash-stats.mozilla.org
ENV: crashstats
SITE: crash-stats.mozilla.org
APP: socorro
Upgrading socorro-crashstats (django) prod
Upgrading socorro to https://ci.mozilla.org/job/socorro-release/602/artifact/socorro.tar.gz
/data/crashstats/src/crash-stats.mozilla.org ~
Creating tables ...
Installing custom SQL ...
Installing indexes ...
Installed 0 object(s) from 0 fixture(s)
[2013-10-09 20:08:59] Running rsync_project
[2013-10-09 20:08:59] [localhost] running: /usr/bin/rsync -aq --include '.gitkeep' --exclude '.git*' --exclude '.hg*' --exclude '.svn*' --exclude 'CVS' --exclude '.bzr*' --delete /data/crashstats/src/crash-stats.mozilla.org/ /data/crashstats/www/crash-stats.mozilla.org/
[2013-10-09 20:09:04] [localhost] finished: /usr/bin/rsync -aq --include '.gitkeep' --exclude '.git*' --exclude '.hg*' --exclude '.svn*' --exclude 'CVS' --exclude '.bzr*' --delete /data/crashstats/src/crash-stats.mozilla.org/ /data/crashstats/www/crash-stats.mozilla.org/ (5.252s)
[2013-10-09 20:09:04] Finished rsync_project (5.252s)
[2013-10-09 20:09:04] Running commit_www
[2013-10-09 20:09:04] [localhost] running: cd /data/crashstats/www && /usr/bin/git add .; /usr/bin/git commit -a -m 'deploy ['crash-stats.mozilla.org']'
[2013-10-09 20:09:11] [localhost] finished: cd /data/crashstats/www && /usr/bin/git add .; /usr/bin/git commit -a -m 'deploy ['crash-stats.mozilla.org']' (6.743s)
[localhost] out: [master e983815] deploy [crash-stats.mozilla.org]
[localhost] out: 105 files changed, 4940 insertions(+), 2125 deletions(-)
[localhost] out: rewrite crash-stats.mozilla.org/socorro/webapp-django/nosetests.xml (86%)
[localhost] out: create mode 100755 crash-stats.mozilla.org/socorro/webapp-django/static/CACHE/css/96afb03efb3f.css
[localhost] out: create mode 100755 crash-stats.mozilla.org/socorro/webapp-django/static/CACHE/js/4c130b8fe834.js
[localhost] out: create mode 100755 crash-stats.mozilla.org/socorro/webapp-django/static/CACHE/js/549dad23584e.js
[localhost] out: create mode 100755 crash-stats.mozilla.org/socorro/webapp-django/static/browserid/persona-buttons.css
[localhost] out: rewrite crash-stats.mozilla.org/socorro/webapp-django/vendor/lib/python/django_browserid/base.py (83%)
[localhost] out: rewrite crash-stats.mozilla.org/socorro/webapp-django/vendor/lib/python/django_browserid/context_processors.py (83%)
[localhost] out: rewrite crash-stats.mozilla.org/socorro/webapp-django/vendor/lib/python/django_browserid/forms.py (66%)
[localhost] out: create mode 100755 crash-stats.mozilla.org/socorro/webapp-django/vendor/lib/python/django_browserid/helpers.py
[localhost] out: rewrite crash-stats.mozilla.org/socorro/webapp-django/vendor/lib/python/django_browserid/signals.py (80%)
[localhost] out: create mode 100755 crash-stats.mozilla.org/socorro/webapp-django/vendor/lib/python/django_browserid/static/browserid/persona-buttons.css
[localhost] out: create mode 100755 crash-stats.mozilla.org/socorro/webapp-django/vendor/lib/python/django_browserid/templates/browserid/button.html
[localhost] out: create mode 100755 crash-stats.mozilla.org/socorro/webapp-django/vendor/lib/python/django_browserid/templates/browserid/info.html
[localhost] out: create mode 100755 crash-stats.mozilla.org/socorro/webapp-django/vendor/lib/python/django_browserid/templatetags/__init__.py
[localhost] out: create mode 100755 crash-stats.mozilla.org/socorro/webapp-django/vendor/lib/python/django_browserid/templatetags/browserid.py
[localhost] out: create mode 100755 crash-stats.mozilla.org/socorro/webapp-django/vendor/lib/python/django_browserid/tests/models.py
[localhost] out: create mode 100755 crash-stats.mozilla.org/socorro/webapp-django/vendor/lib/python/django_browserid/tests/test_base.py
[localhost] out: create mode 100755 crash-stats.mozilla.org/socorro/webapp-django/vendor/lib/python/django_browserid/tests/test_forms.py
[localhost] out: create mode 100755 crash-stats.mozilla.org/socorro/webapp-django/vendor/lib/python/django_browserid/tests/test_helpers.py
[localhost] out: create mode 100755 crash-stats.mozilla.org/socorro/webapp-django/vendor/lib/python/django_browserid/tests/test_urls.py
[localhost] out: create mode 100755 crash-stats.mozilla.org/socorro/webapp-django/vendor/lib/python/django_browserid/tests/test_util.py
[localhost] out: rewrite crash-stats.mozilla.org/socorro/webapp-django/vendor/lib/python/django_browserid/tests/test_views.py (64%)
[localhost] out: rewrite crash-stats.mozilla.org/socorro/webapp-django/vendor/lib/python/django_browserid/tests/urls.py (74%)
[localhost] out: rewrite crash-stats.mozilla.org/socorro/webapp-django/vendor/lib/python/django_browserid/urls.py (92%)
[localhost] out: create mode 100755 crash-stats.mozilla.org/socorro/webapp-django/vendor/lib/python/django_browserid/util.py
[localhost] out: rewrite crash-stats.mozilla.org/socorro/webapp-django/vendor/lib/python/django_browserid/views.py (64%)
[localhost] out: create mode 100755 crash-stats.mozilla.org/socorro/webapp-django/vendor/lib/python/fancy_tag/__init__.py
[localhost] out: create mode 100755 crash-stats.mozilla.org/socorro/webapp-django/vendor/lib/python/six.py
[localhost] out: create mode 100755 crash-stats.mozilla.org/socorro/webapp-django/vendor/src/funfactory/tests/test_middleware.py
[2013-10-09 20:09:11] Finished commit_www (6.745s)
[2013-10-09 20:09:11] Running push_www
[2013-10-09 20:09:11] [socorro1.webapp.phx1.mozilla.com] running: /data/bin/update-www.sh crash-stats.mozilla.org
[2013-10-09 20:09:11] [socorro2.webapp.phx1.mozilla.com] running: /data/bin/update-www.sh crash-stats.mozilla.org
[2013-10-09 20:09:11] [socorro3.webapp.phx1.mozilla.com] running: /data/bin/update-www.sh crash-stats.mozilla.org
[2013-10-09 20:09:11] [socorro4.webapp.phx1.mozilla.com] running: /data/bin/update-www.sh crash-stats.mozilla.org
[2013-10-09 20:09:38] [socorro2.webapp.phx1.mozilla.com] finished: /data/bin/update-www.sh crash-stats.mozilla.org (27.514s)
[2013-10-09 20:09:38] [socorro3.webapp.phx1.mozilla.com] finished: /data/bin/update-www.sh crash-stats.mozilla.org (27.570s)
[2013-10-09 20:09:39] [socorro1.webapp.phx1.mozilla.com] finished: /data/bin/update-www.sh crash-stats.mozilla.org (27.969s)
[socorro1.webapp.phx1.mozilla.com] out: Not removing crash-stats.mozilla.org/app/
[2013-10-09 20:09:39] [socorro4.webapp.phx1.mozilla.com] finished: /data/bin/update-www.sh crash-stats.mozilla.org (27.973s)
[2013-10-09 20:09:39] Finished push_www (27.982s)
[2013-10-09 20:09:39] [socorro1.webapp.phx1.mozilla.com] running: service httpd restart
[2013-10-09 20:09:39] [socorro2.webapp.phx1.mozilla.com] running: service httpd restart
[2013-10-09 20:09:39] [socorro3.webapp.phx1.mozilla.com] running: service httpd restart
[2013-10-09 20:09:39] [socorro4.webapp.phx1.mozilla.com] running: service httpd restart
[2013-10-09 20:09:40] [socorro1.webapp.phx1.mozilla.com] finished: service httpd restart (1.586s)
[socorro1.webapp.phx1.mozilla.com] out: Stopping httpd: [  OK  ]
[socorro1.webapp.phx1.mozilla.com] out: Starting httpd: [  OK  ]
[2013-10-09 20:09:41] [socorro2.webapp.phx1.mozilla.com] finished: service httpd restart (1.666s)
[socorro2.webapp.phx1.mozilla.com] out: Stopping httpd: [  OK  ]
[socorro2.webapp.phx1.mozilla.com] out: Starting httpd: [  OK  ]
[2013-10-09 20:09:41] [socorro3.webapp.phx1.mozilla.com] finished: service httpd restart (1.680s)
[socorro3.webapp.phx1.mozilla.com] out: Stopping httpd: [  OK  ]
[socorro3.webapp.phx1.mozilla.com] out: Starting httpd: [  OK  ]
[2013-10-09 20:09:41] [socorro4.webapp.phx1.mozilla.com] finished: service httpd restart (1.679s)
[socorro4.webapp.phx1.mozilla.com] out: Stopping httpd: [  OK  ]
[socorro4.webapp.phx1.mozilla.com] out: Starting httpd: [  OK  ]
[2013-10-09 20:09:41] [socorro-memcache1.webapp.phx1.mozilla.com] running: service memcached restart
[2013-10-09 20:09:41] [socorro-memcache2.webapp.phx1.mozilla.com] running: service memcached restart
[2013-10-09 20:09:41] [socorro-memcache1.webapp.phx1.mozilla.com] finished: service memcached restart (0.413s)
[socorro-memcache1.webapp.phx1.mozilla.com] out: Stopping memcached: [  OK  ]
[socorro-memcache1.webapp.phx1.mozilla.com] out: Starting memcached: [  OK  ]
[2013-10-09 20:09:41] [socorro-memcache2.webapp.phx1.mozilla.com] finished: service memcached restart (0.552s)
[socorro-memcache2.webapp.phx1.mozilla.com] out: Stopping memcached: [  OK  ]
[socorro-memcache2.webapp.phx1.mozilla.com] out: Starting memcached: [  OK  ]
Status: NEW → ASSIGNED
(Assignee)

Comment 2

4 years ago
Push complete

[root@sp-admin01.phx1 ~]# issue-multi-command processors service socorro-processor status^C
[root@sp-admin01.phx1 ~]# /root/bin/update-socorro-all.sh https://ci.mozilla.org/job/socorro-release/602/artifact/socorro.tar.gz
Upgrading socorro to https://ci.mozilla.org/job/socorro-release/602/artifact/socorro.tar.gz
Upgrading socorro on sp-collector01.phx1.mozilla.com
Stopping socorro-crashmover: [  OK  ]
Starting socorro-crashmover: [  OK  ]
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/602/artifact/socorro.tar.gz
Checksum: 026ae8fb5d194c573ea3dade01eb93a9
Backed up original to /data/socorro.09-10-2013_13_11_27
Upgrading socorro on sp-collector02.phx1.mozilla.com
Stopping socorro-crashmover: [  OK  ]
Starting socorro-crashmover: [  OK  ]
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/602/artifact/socorro.tar.gz
Checksum: 026ae8fb5d194c573ea3dade01eb93a9
Backed up original to /data/socorro.09-10-2013_20_11_51
Upgrading socorro on sp-collector03.phx1.mozilla.com
Stopping socorro-crashmover: [  OK  ]
Starting socorro-crashmover: [  OK  ]
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/602/artifact/socorro.tar.gz
Checksum: 026ae8fb5d194c573ea3dade01eb93a9
Backed up original to /data/socorro.09-10-2013_20_12_01
Upgrading socorro on sp-collector04.phx1.mozilla.com
Stopping socorro-crashmover: [  OK  ]
Starting socorro-crashmover: [  OK  ]
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/602/artifact/socorro.tar.gz
Checksum: 026ae8fb5d194c573ea3dade01eb93a9
Backed up original to /data/socorro.09-10-2013_20_12_11
Upgrading socorro on sp-collector05.phx1.mozilla.com
Stopping socorro-crashmover: [  OK  ]
Starting socorro-crashmover: [  OK  ]
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/602/artifact/socorro.tar.gz
Checksum: 026ae8fb5d194c573ea3dade01eb93a9
Backed up original to /data/socorro.09-10-2013_20_12_19
Upgrading socorro on sp-collector06.phx1.mozilla.com
Stopping socorro-crashmover: [  OK  ]
Starting socorro-crashmover: [  OK  ]
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/602/artifact/socorro.tar.gz
Checksum: 026ae8fb5d194c573ea3dade01eb93a9
Backed up original to /data/socorro.09-10-2013_20_12_29
Upgrading socorro on sp-mware01.phx1.mozilla.com
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/602/artifact/socorro.tar.gz
Checksum: 026ae8fb5d194c573ea3dade01eb93a9
Backed up original to /data/socorro.09-10-2013_13_12_37
Upgrading socorro on sp-mware02.phx1.mozilla.com
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/602/artifact/socorro.tar.gz
Checksum: 026ae8fb5d194c573ea3dade01eb93a9
Backed up original to /data/socorro.09-10-2013_13_12_48
Upgrading socorro on sp-mware03.phx1.mozilla.com
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/602/artifact/socorro.tar.gz
Checksum: 026ae8fb5d194c573ea3dade01eb93a9
Backed up original to /data/socorro.09-10-2013_13_12_54
Upgrading socorro on sp-processor01.phx1.mozilla.com
Stopping socorro-processor: [  OK  ]
Starting socorro-processor: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/602/artifact/socorro.tar.gz
Checksum: 026ae8fb5d194c573ea3dade01eb93a9
Backed up original to /data/socorro.09-10-2013_13_13_03
Upgrading socorro on sp-processor02.phx1.mozilla.com
Stopping socorro-processor: [  OK  ]
Starting socorro-processor: Can't lock the lock file "/var/lock/socorro-processor.lock". Is another instance running?
[FAILED]
ERROR: could not start socorro-processor
Could not upgrade socorro on sp-processor02.phx1.mozilla.com, exit_code: 1
[root@sp-admin01.phx1 ~]# /root/bin/update-socorro-all.sh https://ci.mozilla.org/job/socorro-release/602/artifact/socorro.tar.gz
Upgrading socorro to https://ci.mozilla.org/job/socorro-release/602/artifact/socorro.tar.gz
Upgrading socorro on sp-collector01.phx1.mozilla.com
Upgrading socorro on sp-collector02.phx1.mozilla.com
Upgrading socorro on sp-collector03.phx1.mozilla.com
Upgrading socorro on sp-collector04.phx1.mozilla.com
Upgrading socorro on sp-collector05.phx1.mozilla.com
Upgrading socorro on sp-collector06.phx1.mozilla.com
Upgrading socorro on sp-mware01.phx1.mozilla.com
Upgrading socorro on sp-mware02.phx1.mozilla.com
Upgrading socorro on sp-mware03.phx1.mozilla.com
Upgrading socorro on sp-processor01.phx1.mozilla.com
Upgrading socorro on sp-processor02.phx1.mozilla.com
Upgrading socorro on sp-processor03.phx1.mozilla.com
Stopping socorro-processor: [  OK  ]
Starting socorro-processor: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/602/artifact/socorro.tar.gz
Checksum: 026ae8fb5d194c573ea3dade01eb93a9
Backed up original to /data/socorro.09-10-2013_13_14_55
Upgrading socorro on sp-processor04.phx1.mozilla.com
Stopping socorro-processor: [  OK  ]
Starting socorro-processor: Can't lock the lock file "/var/lock/socorro-processor.lock". Is another instance running?
[FAILED]
ERROR: could not start socorro-processor
Could not upgrade socorro on sp-processor04.phx1.mozilla.com, exit_code: 1
[root@sp-admin01.phx1 ~]# /root/bin/update-socorro-all.sh https://ci.mozilla.org/job/socorro-release/602/artifact/socorro.tar.gz
Upgrading socorro to https://ci.mozilla.org/job/socorro-release/602/artifact/socorro.tar.gz
Upgrading socorro on sp-collector01.phx1.mozilla.com
Upgrading socorro on sp-collector02.phx1.mozilla.com
Upgrading socorro on sp-collector03.phx1.mozilla.com
Upgrading socorro on sp-collector04.phx1.mozilla.com
Upgrading socorro on sp-collector05.phx1.mozilla.com
Upgrading socorro on sp-collector06.phx1.mozilla.com
Upgrading socorro on sp-mware01.phx1.mozilla.com
Upgrading socorro on sp-mware02.phx1.mozilla.com
Upgrading socorro on sp-mware03.phx1.mozilla.com
Upgrading socorro on sp-processor01.phx1.mozilla.com
Upgrading socorro on sp-processor02.phx1.mozilla.com
Upgrading socorro on sp-processor03.phx1.mozilla.com
Upgrading socorro on sp-processor04.phx1.mozilla.com
Upgrading socorro on sp-processor05.phx1.mozilla.com
Stopping socorro-processor: [  OK  ]
Starting socorro-processor: Can't lock the lock file "/var/lock/socorro-processor.lock". Is another instance running?
[FAILED]
ERROR: could not start socorro-processor
Could not upgrade socorro on sp-processor05.phx1.mozilla.com, exit_code: 1
[root@sp-admin01.phx1 ~]# /root/bin/update-socorro-all.sh https://ci.mozilla.org/job/socorro-release/602/artifact/socorro.tar.gz
Upgrading socorro to https://ci.mozilla.org/job/socorro-release/602/artifact/socorro.tar.gz
Upgrading socorro on sp-collector01.phx1.mozilla.com
Upgrading socorro on sp-collector02.phx1.mozilla.com
Upgrading socorro on sp-collector03.phx1.mozilla.com
Upgrading socorro on sp-collector04.phx1.mozilla.com
Upgrading socorro on sp-collector05.phx1.mozilla.com
Upgrading socorro on sp-collector06.phx1.mozilla.com
Upgrading socorro on sp-mware01.phx1.mozilla.com
Upgrading socorro on sp-mware02.phx1.mozilla.com
Upgrading socorro on sp-mware03.phx1.mozilla.com
Upgrading socorro on sp-processor01.phx1.mozilla.com
Upgrading socorro on sp-processor02.phx1.mozilla.com
Upgrading socorro on sp-processor03.phx1.mozilla.com
Upgrading socorro on sp-processor04.phx1.mozilla.com
Upgrading socorro on sp-processor05.phx1.mozilla.com
Upgrading socorro on sp-processor06.phx1.mozilla.com
Stopping socorro-processor: [  OK  ]
Starting socorro-processor: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/602/artifact/socorro.tar.gz
Checksum: 026ae8fb5d194c573ea3dade01eb93a9
Backed up original to /data/socorro.09-10-2013_13_18_41
Upgrading socorro on sp-processor07.phx1.mozilla.com
Stopping socorro-processor: [  OK  ]
Starting socorro-processor: Can't lock the lock file "/var/lock/socorro-processor.lock". Is another instance running?
[FAILED]
ERROR: could not start socorro-processor
Could not upgrade socorro on sp-processor07.phx1.mozilla.com, exit_code: 1
[root@sp-admin01.phx1 ~]# /root/bin/update-socorro-all.sh https://ci.mozilla.org/job/socorro-release/602/artifact/socorro.tar.gz
Upgrading socorro to https://ci.mozilla.org/job/socorro-release/602/artifact/socorro.tar.gz
Upgrading socorro on sp-collector01.phx1.mozilla.com
Upgrading socorro on sp-collector02.phx1.mozilla.com
Upgrading socorro on sp-collector03.phx1.mozilla.com
Upgrading socorro on sp-collector04.phx1.mozilla.com
Upgrading socorro on sp-collector05.phx1.mozilla.com
Upgrading socorro on sp-collector06.phx1.mozilla.com
Upgrading socorro on sp-mware01.phx1.mozilla.com
Upgrading socorro on sp-mware02.phx1.mozilla.com
Upgrading socorro on sp-mware03.phx1.mozilla.com
Upgrading socorro on sp-processor01.phx1.mozilla.com
Upgrading socorro on sp-processor02.phx1.mozilla.com
Upgrading socorro on sp-processor03.phx1.mozilla.com
Upgrading socorro on sp-processor04.phx1.mozilla.com
Upgrading socorro on sp-processor05.phx1.mozilla.com
Upgrading socorro on sp-processor06.phx1.mozilla.com
Upgrading socorro on sp-processor07.phx1.mozilla.com
Upgrading socorro on sp-processor08.phx1.mozilla.com
Stopping socorro-processor: [  OK  ]
Starting socorro-processor: Can't lock the lock file "/var/lock/socorro-processor.lock". Is another instance running?
[FAILED]
ERROR: could not start socorro-processor
Could not upgrade socorro on sp-processor08.phx1.mozilla.com, exit_code: 1
[root@sp-admin01.phx1 ~]# /root/bin/update-socorro-all.sh https://ci.mozilla.org/job/socorro-release/602/artifact/socorro.tar.gz
Upgrading socorro to https://ci.mozilla.org/job/socorro-release/602/artifact/socorro.tar.gz
Upgrading socorro on sp-collector01.phx1.mozilla.com
Upgrading socorro on sp-collector02.phx1.mozilla.com
Upgrading socorro on sp-collector03.phx1.mozilla.com
Upgrading socorro on sp-collector04.phx1.mozilla.com
Upgrading socorro on sp-collector05.phx1.mozilla.com
Upgrading socorro on sp-collector06.phx1.mozilla.com
Upgrading socorro on sp-mware01.phx1.mozilla.com
Upgrading socorro on sp-mware02.phx1.mozilla.com
Upgrading socorro on sp-mware03.phx1.mozilla.com
Upgrading socorro on sp-processor01.phx1.mozilla.com
Upgrading socorro on sp-processor02.phx1.mozilla.com
Upgrading socorro on sp-processor03.phx1.mozilla.com
Upgrading socorro on sp-processor04.phx1.mozilla.com
Upgrading socorro on sp-processor05.phx1.mozilla.com
Upgrading socorro on sp-processor06.phx1.mozilla.com
Upgrading socorro on sp-processor07.phx1.mozilla.com
Upgrading socorro on sp-processor08.phx1.mozilla.com
Upgrading socorro on sp-processor09.phx1.mozilla.com
Stopping socorro-processor: [  OK  ]
Starting socorro-processor: Can't lock the lock file "/var/lock/socorro-processor.lock". Is another instance running?
[FAILED]
ERROR: could not start socorro-processor
Could not upgrade socorro on sp-processor09.phx1.mozilla.com, exit_code: 1
[root@sp-admin01.phx1 ~]# /root/bin/update-socorro-all.sh https://ci.mozilla.org/job/socorro-release/602/artifact/socorro.tar.gz
Upgrading socorro to https://ci.mozilla.org/job/socorro-release/602/artifact/socorro.tar.gz
Upgrading socorro on sp-collector01.phx1.mozilla.com
Upgrading socorro on sp-collector02.phx1.mozilla.com
Upgrading socorro on sp-collector03.phx1.mozilla.com
Upgrading socorro on sp-collector04.phx1.mozilla.com
Upgrading socorro on sp-collector05.phx1.mozilla.com
Upgrading socorro on sp-collector06.phx1.mozilla.com
Upgrading socorro on sp-mware01.phx1.mozilla.com
Upgrading socorro on sp-mware02.phx1.mozilla.com
Upgrading socorro on sp-mware03.phx1.mozilla.com
Upgrading socorro on sp-processor01.phx1.mozilla.com
Upgrading socorro on sp-processor02.phx1.mozilla.com
Upgrading socorro on sp-processor03.phx1.mozilla.com
Upgrading socorro on sp-processor04.phx1.mozilla.com
Upgrading socorro on sp-processor05.phx1.mozilla.com
Upgrading socorro on sp-processor06.phx1.mozilla.com
Upgrading socorro on sp-processor07.phx1.mozilla.com
Upgrading socorro on sp-processor08.phx1.mozilla.com
Upgrading socorro on sp-processor09.phx1.mozilla.com
Upgrading socorro on sp-processor10.phx1.mozilla.com
Stopping socorro-processor: [  OK  ]
Starting socorro-processor: Can't lock the lock file "/var/lock/socorro-processor.lock". Is another instance running?
[FAILED]
ERROR: could not start socorro-processor
Could not upgrade socorro on sp-processor10.phx1.mozilla.com, exit_code: 1
[root@sp-admin01.phx1 ~]# /root/bin/update-socorro-all.sh https://ci.mozilla.org/job/socorro-release/602/artifact/socorro.tar.gz
Upgrading socorro to https://ci.mozilla.org/job/socorro-release/602/artifact/socorro.tar.gz
Upgrading socorro on sp-collector01.phx1.mozilla.com
Upgrading socorro on sp-collector02.phx1.mozilla.com
Upgrading socorro on sp-collector03.phx1.mozilla.com
Upgrading socorro on sp-collector04.phx1.mozilla.com
Upgrading socorro on sp-collector05.phx1.mozilla.com
Upgrading socorro on sp-collector06.phx1.mozilla.com
Upgrading socorro on sp-mware01.phx1.mozilla.com
Upgrading socorro on sp-mware02.phx1.mozilla.com
Upgrading socorro on sp-mware03.phx1.mozilla.com
Upgrading socorro on sp-processor01.phx1.mozilla.com
Upgrading socorro on sp-processor02.phx1.mozilla.com
Upgrading socorro on sp-processor03.phx1.mozilla.com
Upgrading socorro on sp-processor04.phx1.mozilla.com
Upgrading socorro on sp-processor05.phx1.mozilla.com
Upgrading socorro on sp-processor06.phx1.mozilla.com
Upgrading socorro on sp-processor07.phx1.mozilla.com
Upgrading socorro on sp-processor08.phx1.mozilla.com
Upgrading socorro on sp-processor09.phx1.mozilla.com
Upgrading socorro on sp-processor10.phx1.mozilla.com
Upgrading socorro on sp-web01.phx1.mozilla.com
Stopping httpd: [  OK  ]
Starting httpd: Warning: DocumentRoot [/data/socorro/htdocs] does not exist
Warning: DocumentRoot [/data/socorro/htdocs] does not exist
[  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/602/artifact/socorro.tar.gz
Checksum: 026ae8fb5d194c573ea3dade01eb93a9
Backed up original to /data/socorro.09-10-2013_13_25_36
Upgrading socorro on sp-web02.phx1.mozilla.com
Stopping httpd: [  OK  ]
Starting httpd: Warning: DocumentRoot [/data/socorro/htdocs] does not exist
Warning: DocumentRoot [/data/socorro/htdocs] does not exist
[  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/602/artifact/socorro.tar.gz
Checksum: 026ae8fb5d194c573ea3dade01eb93a9
Backed up original to /data/socorro.09-10-2013_13_25_43
Upgrading socorro on sp-web03.phx1.mozilla.com
Stopping httpd: [  OK  ]
Starting httpd: Warning: DocumentRoot [/data/socorro/htdocs] does not exist
Warning: DocumentRoot [/data/socorro/htdocs] does not exist
[  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/602/artifact/socorro.tar.gz
Checksum: 026ae8fb5d194c573ea3dade01eb93a9
Backed up original to /data/socorro.09-10-2013_13_26_02
Upgrading socorro on sp-web04.phx1.mozilla.com
Stopping httpd: [  OK  ]
Starting httpd: Warning: DocumentRoot [/data/socorro/htdocs] does not exist
Warning: DocumentRoot [/data/socorro/htdocs] does not exist
[  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/602/artifact/socorro.tar.gz
Checksum: 026ae8fb5d194c573ea3dade01eb93a9
Backed up original to /data/socorro.09-10-2013_13_26_12
Upgrading socorro on sp-web05.phx1.mozilla.com
Stopping httpd: [  OK  ]
Starting httpd: Warning: DocumentRoot [/data/socorro/htdocs] does not exist
Warning: DocumentRoot [/data/socorro/htdocs] does not exist
[  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/602/artifact/socorro.tar.gz
Checksum: 026ae8fb5d194c573ea3dade01eb93a9
Backed up original to /data/socorro.09-10-2013_13_26_21
Upgrading socorro on sp-admin01.phx1.mozilla.com
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/602/artifact/socorro.tar.gz
Checksum: 026ae8fb5d194c573ea3dade01eb93a9
Backed up original to /data/socorro.09-10-2013_13_26_29
Upgrading socorro on tp-socorro01-master01.phx1.mozilla.com
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/602/artifact/socorro.tar.gz
Checksum: 026ae8fb5d194c573ea3dade01eb93a9
Backed up original to /data/socorro.09-10-2013_13_26_50
Upgrading socorro on tp-socorro01-master02.phx1.mozilla.com
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/602/artifact/socorro.tar.gz
Checksum: 026ae8fb5d194c573ea3dade01eb93a9
Backed up original to /data/socorro.09-10-2013_13_27_02
(Assignee)

Comment 3

4 years ago
Processor cleanup

bburton@althalus [01:14:26] [~]
-> % ssh sp-processor02.phx1.mozilla.com sudo rm -rf /var/lock/socorro-processor.lock
Connection to sp-processor02.phx1.mozilla.com closed.
bburton@althalus [01:14:40] [~]
-> % ssh sp-processor02.phx1.mozilla.com sudo /sbin/service socorro-processor start
Starting socorro-processor:                                [  OK  ]
Connection to sp-processor02.phx1.mozilla.com closed.
bburton@althalus [01:14:52] [~]
-> % ssh sp-processor04.phx1.mozilla.com sudo rm -rf /var/lock/socorro-processor.lock
Connection to sp-processor04.phx1.mozilla.com closed.
bburton@althalus [01:16:20] [~]
-> % ssh sp-processor04.phx1.mozilla.com sudo /sbin/service socorro-processor start
Starting socorro-processor:                                [  OK  ]
Connection to sp-processor04.phx1.mozilla.com closed.
bburton@althalus [01:16:27] [~]
-> % ssh sp-processor05.phx1.mozilla.com sudo rm -rf /var/lock/socorro-processor.lock
Connection to sp-processor05.phx1.mozilla.com closed.
bburton@althalus [01:17:42] [~]
-> % ssh sp-processor05.phx1.mozilla.com sudo /sbin/service socorro-processor start
Starting socorro-processor:                                [  OK  ]
Connection to sp-processor05.phx1.mozilla.com closed.
bburton@althalus [01:17:58] [~]
-> % ssh sp-processor07.phx1.mozilla.com sudo rm -rf /var/lock/socorro-processor.lock
Connection to sp-processor07.phx1.mozilla.com closed.
bburton@althalus [01:19:31] [~]
-> % ssh sp-processor07.phx1.mozilla.com sudo /sbin/service socorro-processor start
Starting socorro-processor:                                [  OK  ]
Connection to sp-processor07.phx1.mozilla.com closed.
bburton@althalus [01:19:38] [~]
-> % ssh sp-processor08.phx1.mozilla.com sudo rm -rf /var/lock/socorro-processor.lock
Connection to sp-processor08.phx1.mozilla.com closed.
bburton@althalus [01:21:08] [~]
-> % ssh sp-processor08.phx1.mozilla.com sudo /sbin/service socorro-processor start
Starting socorro-processor:                                [  OK  ]
Connection to sp-processor08.phx1.mozilla.com closed.
bburton@althalus [01:21:14] [~]
-> % ssh sp-processor09.phx1.mozilla.com sudo rm -rf /var/lock/socorro-processor.lock
Connection to sp-processor09.phx1.mozilla.com closed.
bburton@althalus [01:22:34] [~]
-> % ssh sp-processor09.phx1.mozilla.com sudo /sbin/service socorro-processor start
Starting socorro-processor:                                [  OK  ]
Connection to sp-processor09.phx1.mozilla.com closed.
bburton@althalus [01:22:40] [~]
-> % ssh sp-processor10.phx1.mozilla.com sudo rm -rf /var/lock/socorro-processor.lock
Connection to sp-processor10.phx1.mozilla.com closed.
bburton@althalus [01:24:38] [~]
-> % ssh sp-processor10.phx1.mozilla.com sudo /sbin/service socorro-processor start
Starting socorro-processor:                                [  OK  ]
Connection to sp-processor10.phx1.mozilla.com closed.
Status: ASSIGNED → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED
QA verified:
- automation passes - http://qa-selenium.mv.mozilla.com:8080/view/Socorro/job/socorro.prod/219/
- manual bfts pass
- all push bugs were [qa-]; nothing to spot check
Status: RESOLVED → VERIFIED
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.