Status

Infrastructure & Operations Graveyard
WebOps: Socorro
P3
normal
VERIFIED FIXED
5 years ago
2 years ago

People

(Reporter: lonnen, Assigned: solarce)

Tracking

Details

(Whiteboard: [push interrupt])

(Assignee)

Comment 1

5 years ago
http://i.imgur.com/pvtEQiy.gif
Assignee: server-ops-webops → bburton
Status: NEW → ASSIGNED
(Assignee)

Comment 2

5 years ago
Part of push complete, with the stopping issue on 9 of 10 processors

[root@sp-admin01.phx1 ~]# /root/bin/update-socorro-all https://ci.mozilla.org/job/socorro-release/567/artifact/socorro.tar.gz
-bash: /root/bin/update-socorro-all: No such file or directory
[root@sp-admin01.phx1 ~]# /root/bin/update-socorro-all.sh https://ci.mozilla.org/job/socorro-release/567/artifact/socorro.tar.gz
Upgrading socorro to https://ci.mozilla.org/job/socorro-release/567/artifact/socorro.tar.gz
Upgrading socorro on sp-collector01.phx1.mozilla.com
Stopping socorro-crashmover: [  OK  ]
Starting socorro-crashmover: [  OK  ]
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/567/artifact/socorro.tar.gz
Checksum: cc38dd364c14769fe74258831e40dc5d
Backed up original to /data/socorro.19-09-2013_14_30_22
Upgrading socorro on sp-collector02.phx1.mozilla.com
Stopping socorro-crashmover: [  OK  ]
Starting socorro-crashmover: [  OK  ]
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/567/artifact/socorro.tar.gz
Checksum: cc38dd364c14769fe74258831e40dc5d
Backed up original to /data/socorro.19-09-2013_21_30_31
Upgrading socorro on sp-collector03.phx1.mozilla.com
Stopping socorro-crashmover: [  OK  ]
Starting socorro-crashmover: [  OK  ]
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/567/artifact/socorro.tar.gz
Checksum: cc38dd364c14769fe74258831e40dc5d
Backed up original to /data/socorro.19-09-2013_21_30_39
Upgrading socorro on sp-collector04.phx1.mozilla.com
Stopping socorro-crashmover: [  OK  ]
Starting socorro-crashmover: [  OK  ]
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/567/artifact/socorro.tar.gz
Checksum: cc38dd364c14769fe74258831e40dc5d
Backed up original to /data/socorro.19-09-2013_21_30_48
Upgrading socorro on sp-collector05.phx1.mozilla.com
Stopping socorro-crashmover: [  OK  ]
Starting socorro-crashmover: [  OK  ]
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/567/artifact/socorro.tar.gz
Checksum: cc38dd364c14769fe74258831e40dc5d
Backed up original to /data/socorro.19-09-2013_21_30_56
Upgrading socorro on sp-collector06.phx1.mozilla.com
Stopping socorro-crashmover: [  OK  ]
Starting socorro-crashmover: [  OK  ]
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/567/artifact/socorro.tar.gz
Checksum: cc38dd364c14769fe74258831e40dc5d
Backed up original to /data/socorro.19-09-2013_21_31_04
Upgrading socorro on sp-mware01.phx1.mozilla.com
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/567/artifact/socorro.tar.gz
Checksum: cc38dd364c14769fe74258831e40dc5d
Backed up original to /data/socorro.19-09-2013_14_31_13
Upgrading socorro on sp-mware02.phx1.mozilla.com
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/567/artifact/socorro.tar.gz
Checksum: cc38dd364c14769fe74258831e40dc5d
Backed up original to /data/socorro.19-09-2013_14_31_24
Upgrading socorro on sp-mware03.phx1.mozilla.com
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/567/artifact/socorro.tar.gz
Checksum: cc38dd364c14769fe74258831e40dc5d
Backed up original to /data/socorro.19-09-2013_14_31_49
Upgrading socorro on sp-processor01.phx1.mozilla.com
Stopping socorro-processor: [  OK  ]
Starting socorro-processor: Can't lock the lock file "/var/lock/socorro-processor.lock". Is another instance running?
[FAILED]
ERROR: could not start socorro-processor
Could not upgrade socorro on sp-processor01.phx1.mozilla.com, exit_code: 1
[root@sp-admin01.phx1 ~]# /root/bin/update-socorro-all.sh https://ci.mozilla.org/job/socorro-release/567/artifact/socorro.tar.gz
Upgrading socorro to https://ci.mozilla.org/job/socorro-release/567/artifact/socorro.tar.gz
Upgrading socorro on sp-collector01.phx1.mozilla.com
Upgrading socorro on sp-collector02.phx1.mozilla.com
Upgrading socorro on sp-collector03.phx1.mozilla.com
Upgrading socorro on sp-collector04.phx1.mozilla.com
Upgrading socorro on sp-collector05.phx1.mozilla.com
Upgrading socorro on sp-collector06.phx1.mozilla.com
Upgrading socorro on sp-mware01.phx1.mozilla.com
Upgrading socorro on sp-mware02.phx1.mozilla.com
Upgrading socorro on sp-mware03.phx1.mozilla.com
Upgrading socorro on sp-processor01.phx1.mozilla.com
Upgrading socorro on sp-processor02.phx1.mozilla.com
Stopping socorro-processor: [  OK  ]
Starting socorro-processor: Can't lock the lock file "/var/lock/socorro-processor.lock". Is another instance running?
[FAILED]
ERROR: could not start socorro-processor
Could not upgrade socorro on sp-processor02.phx1.mozilla.com, exit_code: 1
[root@sp-admin01.phx1 ~]# /root/bin/update-socorro-all.sh https://ci.mozilla.org/job/socorro-release/567/artifact/socorro.tar.gz
Upgrading socorro to https://ci.mozilla.org/job/socorro-release/567/artifact/socorro.tar.gz
Upgrading socorro on sp-collector01.phx1.mozilla.com
Upgrading socorro on sp-collector02.phx1.mozilla.com
Upgrading socorro on sp-collector03.phx1.mozilla.com
Upgrading socorro on sp-collector04.phx1.mozilla.com
Upgrading socorro on sp-collector05.phx1.mozilla.com
Upgrading socorro on sp-collector06.phx1.mozilla.com
Upgrading socorro on sp-mware01.phx1.mozilla.com
Upgrading socorro on sp-mware02.phx1.mozilla.com
Upgrading socorro on sp-mware03.phx1.mozilla.com
Upgrading socorro on sp-processor01.phx1.mozilla.com
Upgrading socorro on sp-processor02.phx1.mozilla.com
Upgrading socorro on sp-processor03.phx1.mozilla.com
Stopping socorro-processor: [  OK  ]
Starting socorro-processor: Can't lock the lock file "/var/lock/socorro-processor.lock". Is another instance running?
[FAILED]
ERROR: could not start socorro-processor
Could not upgrade socorro on sp-processor03.phx1.mozilla.com, exit_code: 1
[root@sp-admin01.phx1 ~]# /root/bin/update-socorro-all.sh https://ci.mozilla.org/job/socorro-release/567/artifact/socorro.tar.gz
Upgrading socorro to https://ci.mozilla.org/job/socorro-release/567/artifact/socorro.tar.gz
Upgrading socorro on sp-collector01.phx1.mozilla.com
Upgrading socorro on sp-collector02.phx1.mozilla.com
Upgrading socorro on sp-collector03.phx1.mozilla.com
Upgrading socorro on sp-collector04.phx1.mozilla.com
Upgrading socorro on sp-collector05.phx1.mozilla.com
Upgrading socorro on sp-collector06.phx1.mozilla.com
Upgrading socorro on sp-mware01.phx1.mozilla.com
Upgrading socorro on sp-mware02.phx1.mozilla.com
Upgrading socorro on sp-mware03.phx1.mozilla.com
Upgrading socorro on sp-processor01.phx1.mozilla.com
Upgrading socorro on sp-processor02.phx1.mozilla.com
Upgrading socorro on sp-processor03.phx1.mozilla.com
Upgrading socorro on sp-processor04.phx1.mozilla.com
Stopping socorro-processor: [  OK  ]
Starting socorro-processor: Can't lock the lock file "/var/lock/socorro-processor.lock". Is another instance running?
[FAILED]
ERROR: could not start socorro-processor
Could not upgrade socorro on sp-processor04.phx1.mozilla.com, exit_code: 1
[root@sp-admin01.phx1 ~]# /root/bin/update-socorro-all.sh https://ci.mozilla.org/job/socorro-release/567/artifact/socorro.tar.gz
Upgrading socorro to https://ci.mozilla.org/job/socorro-release/567/artifact/socorro.tar.gz
Upgrading socorro on sp-collector01.phx1.mozilla.com
Upgrading socorro on sp-collector02.phx1.mozilla.com
Upgrading socorro on sp-collector03.phx1.mozilla.com
Upgrading socorro on sp-collector04.phx1.mozilla.com
Upgrading socorro on sp-collector05.phx1.mozilla.com
Upgrading socorro on sp-collector06.phx1.mozilla.com
Upgrading socorro on sp-mware01.phx1.mozilla.com
Upgrading socorro on sp-mware02.phx1.mozilla.com
Upgrading socorro on sp-mware03.phx1.mozilla.com
Upgrading socorro on sp-processor01.phx1.mozilla.com
Upgrading socorro on sp-processor02.phx1.mozilla.com
Upgrading socorro on sp-processor03.phx1.mozilla.com
Upgrading socorro on sp-processor04.phx1.mozilla.com
Upgrading socorro on sp-processor05.phx1.mozilla.com
Stopping socorro-processor: [  OK  ]
Starting socorro-processor: Can't lock the lock file "/var/lock/socorro-processor.lock". Is another instance running?
[FAILED]
ERROR: could not start socorro-processor
Could not upgrade socorro on sp-processor05.phx1.mozilla.com, exit_code: 1
[root@sp-admin01.phx1 ~]# /root/bin/update-socorro-all.sh https://ci.mozilla.org/job/socorro-release/567/artifact/socorro.tar.gz
Upgrading socorro to https://ci.mozilla.org/job/socorro-release/567/artifact/socorro.tar.gz
Upgrading socorro on sp-collector01.phx1.mozilla.com
Upgrading socorro on sp-collector02.phx1.mozilla.com
Upgrading socorro on sp-collector03.phx1.mozilla.com
Upgrading socorro on sp-collector04.phx1.mozilla.com
Upgrading socorro on sp-collector05.phx1.mozilla.com
Upgrading socorro on sp-collector06.phx1.mozilla.com
Upgrading socorro on sp-mware01.phx1.mozilla.com
Upgrading socorro on sp-mware02.phx1.mozilla.com
Upgrading socorro on sp-mware03.phx1.mozilla.com
Upgrading socorro on sp-processor01.phx1.mozilla.com
Upgrading socorro on sp-processor02.phx1.mozilla.com
Upgrading socorro on sp-processor03.phx1.mozilla.com
Upgrading socorro on sp-processor04.phx1.mozilla.com
Upgrading socorro on sp-processor05.phx1.mozilla.com
Upgrading socorro on sp-processor06.phx1.mozilla.com
Stopping socorro-processor: [  OK  ]
Starting socorro-processor: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/567/artifact/socorro.tar.gz
Checksum: cc38dd364c14769fe74258831e40dc5d
Backed up original to /data/socorro.19-09-2013_14_38_11
Upgrading socorro on sp-processor07.phx1.mozilla.com
Stopping socorro-processor: [  OK  ]
Starting socorro-processor: Can't lock the lock file "/var/lock/socorro-processor.lock". Is another instance running?
[FAILED]
ERROR: could not start socorro-processor
Could not upgrade socorro on sp-processor07.phx1.mozilla.com, exit_code: 1
[root@sp-admin01.phx1 ~]# /root/bin/update-socorro-all.sh https://ci.mozilla.org/job/socorro-release/567/artifact/socorro.tar.gz
Upgrading socorro to https://ci.mozilla.org/job/socorro-release/567/artifact/socorro.tar.gz
Upgrading socorro on sp-collector01.phx1.mozilla.com
Upgrading socorro on sp-collector02.phx1.mozilla.com
Upgrading socorro on sp-collector03.phx1.mozilla.com
Upgrading socorro on sp-collector04.phx1.mozilla.com
Upgrading socorro on sp-collector05.phx1.mozilla.com
Upgrading socorro on sp-collector06.phx1.mozilla.com
Upgrading socorro on sp-mware01.phx1.mozilla.com
Upgrading socorro on sp-mware02.phx1.mozilla.com
Upgrading socorro on sp-mware03.phx1.mozilla.com
Upgrading socorro on sp-processor01.phx1.mozilla.com
Upgrading socorro on sp-processor02.phx1.mozilla.com
Upgrading socorro on sp-processor03.phx1.mozilla.com
Upgrading socorro on sp-processor04.phx1.mozilla.com
Upgrading socorro on sp-processor05.phx1.mozilla.com
Upgrading socorro on sp-processor06.phx1.mozilla.com
Upgrading socorro on sp-processor07.phx1.mozilla.com
Upgrading socorro on sp-processor08.phx1.mozilla.com
Stopping socorro-processor: [  OK  ]
Starting socorro-processor: Can't lock the lock file "/var/lock/socorro-processor.lock". Is another instance running?
[FAILED]
ERROR: could not start socorro-processor
Could not upgrade socorro on sp-processor08.phx1.mozilla.com, exit_code: 1
[root@sp-admin01.phx1 ~]# /root/bin/update-socorro-all.sh https://ci.mozilla.org/job/socorro-release/567/artifact/socorro.tar.gz
Upgrading socorro to https://ci.mozilla.org/job/socorro-release/567/artifact/socorro.tar.gz
Upgrading socorro on sp-collector01.phx1.mozilla.com
Upgrading socorro on sp-collector02.phx1.mozilla.com
Upgrading socorro on sp-collector03.phx1.mozilla.com
Upgrading socorro on sp-collector04.phx1.mozilla.com
Upgrading socorro on sp-collector05.phx1.mozilla.com
Upgrading socorro on sp-collector06.phx1.mozilla.com
Upgrading socorro on sp-mware01.phx1.mozilla.com
Upgrading socorro on sp-mware02.phx1.mozilla.com
Upgrading socorro on sp-mware03.phx1.mozilla.com
Upgrading socorro on sp-processor01.phx1.mozilla.com
Upgrading socorro on sp-processor02.phx1.mozilla.com
Upgrading socorro on sp-processor03.phx1.mozilla.com
Upgrading socorro on sp-processor04.phx1.mozilla.com
Upgrading socorro on sp-processor05.phx1.mozilla.com
Upgrading socorro on sp-processor06.phx1.mozilla.com
Upgrading socorro on sp-processor07.phx1.mozilla.com
Upgrading socorro on sp-processor08.phx1.mozilla.com
Upgrading socorro on sp-processor09.phx1.mozilla.com
Stopping socorro-processor: [  OK  ]
Starting socorro-processor: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/567/artifact/socorro.tar.gz
Checksum: cc38dd364c14769fe74258831e40dc5d
Backed up original to /data/socorro.19-09-2013_14_41_04
Upgrading socorro on sp-processor10.phx1.mozilla.com
Stopping socorro-processor: [  OK  ]
Starting socorro-processor: Can't lock the lock file "/var/lock/socorro-processor.lock". Is another instance running?
[FAILED]
ERROR: could not start socorro-processor
Could not upgrade socorro on sp-processor10.phx1.mozilla.com, exit_code: 1
[root@sp-admin01.phx1 ~]# /root/bin/update-socorro-all.sh https://ci.mozilla.org/job/socorro-release/567/artifact/socorro.tar.gz
Upgrading socorro to https://ci.mozilla.org/job/socorro-release/567/artifact/socorro.tar.gz
Upgrading socorro on sp-collector01.phx1.mozilla.com
Upgrading socorro on sp-collector02.phx1.mozilla.com
Upgrading socorro on sp-collector03.phx1.mozilla.com
Upgrading socorro on sp-collector04.phx1.mozilla.com
Upgrading socorro on sp-collector05.phx1.mozilla.com
Upgrading socorro on sp-collector06.phx1.mozilla.com
Upgrading socorro on sp-mware01.phx1.mozilla.com
Upgrading socorro on sp-mware02.phx1.mozilla.com
Upgrading socorro on sp-mware03.phx1.mozilla.com
Upgrading socorro on sp-processor01.phx1.mozilla.com
Upgrading socorro on sp-processor02.phx1.mozilla.com
Upgrading socorro on sp-processor03.phx1.mozilla.com
Upgrading socorro on sp-processor04.phx1.mozilla.com
Upgrading socorro on sp-processor05.phx1.mozilla.com
Upgrading socorro on sp-processor06.phx1.mozilla.com
Upgrading socorro on sp-processor07.phx1.mozilla.com
Upgrading socorro on sp-processor08.phx1.mozilla.com
Upgrading socorro on sp-processor09.phx1.mozilla.com
Upgrading socorro on sp-processor10.phx1.mozilla.com
Upgrading socorro on sp-web01.phx1.mozilla.com
Stopping httpd: [  OK  ]
Starting httpd: Warning: DocumentRoot [/data/socorro/htdocs] does not exist
Warning: DocumentRoot [/data/socorro/htdocs] does not exist
[  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/567/artifact/socorro.tar.gz
Checksum: cc38dd364c14769fe74258831e40dc5d
Backed up original to /data/socorro.19-09-2013_14_42_55
Upgrading socorro on sp-web02.phx1.mozilla.com
Stopping httpd: [  OK  ]
Starting httpd: Warning: DocumentRoot [/data/socorro/htdocs] does not exist
Warning: DocumentRoot [/data/socorro/htdocs] does not exist
[  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/567/artifact/socorro.tar.gz
Checksum: cc38dd364c14769fe74258831e40dc5d
Backed up original to /data/socorro.19-09-2013_14_43_04
Upgrading socorro on sp-web03.phx1.mozilla.com
Stopping httpd: [  OK  ]
Starting httpd: Warning: DocumentRoot [/data/socorro/htdocs] does not exist
Warning: DocumentRoot [/data/socorro/htdocs] does not exist
[  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/567/artifact/socorro.tar.gz
Checksum: cc38dd364c14769fe74258831e40dc5d
Backed up original to /data/socorro.19-09-2013_14_43_11
Upgrading socorro on sp-web04.phx1.mozilla.com
Stopping httpd: [  OK  ]
Starting httpd: Warning: DocumentRoot [/data/socorro/htdocs] does not exist
Warning: DocumentRoot [/data/socorro/htdocs] does not exist
[  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/567/artifact/socorro.tar.gz
Checksum: cc38dd364c14769fe74258831e40dc5d
Backed up original to /data/socorro.19-09-2013_14_43_18
Upgrading socorro on sp-web05.phx1.mozilla.com
Stopping httpd: [  OK  ]
Starting httpd: Warning: DocumentRoot [/data/socorro/htdocs] does not exist
Warning: DocumentRoot [/data/socorro/htdocs] does not exist
[  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/567/artifact/socorro.tar.gz
Checksum: cc38dd364c14769fe74258831e40dc5d
Backed up original to /data/socorro.19-09-2013_14_43_25
Upgrading socorro on sp-admin01.phx1.mozilla.com
Stopping socorro-monitor: [  OK  ]
Starting socorro-monitor: [  OK  ]
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/567/artifact/socorro.tar.gz
Checksum: cc38dd364c14769fe74258831e40dc5d
Backed up original to /data/socorro.19-09-2013_14_43_33
Upgrading socorro on tp-socorro01-master01.phx1.mozilla.com
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/567/artifact/socorro.tar.gz
Checksum: cc38dd364c14769fe74258831e40dc5d
Backed up original to /data/socorro.19-09-2013_14_43_52
Upgrading socorro on tp-socorro01-master02.phx1.mozilla.com
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/567/artifact/socorro.tar.gz
Checksum: cc38dd364c14769fe74258831e40dc5d
Backed up original to /data/socorro.19-09-2013_14_44_05
[sp-web01.phx1.mozilla.com] running: service memcached restart
[sp-web02.phx1.mozilla.com] running: service memcached restart
[sp-web03.phx1.mozilla.com] running: service memcached restart
[sp-web04.phx1.mozilla.com] running: service memcached restart
[sp-web05.phx1.mozilla.com] running: service memcached restart
[sp-web01.phx1.mozilla.com] finished: service memcached restart (0.641s)
[sp-web01.phx1.mozilla.com] out: Stopping memcached: [  OK  ]
[sp-web01.phx1.mozilla.com] out: Starting memcached: [  OK  ]
[sp-web03.phx1.mozilla.com] finished: service memcached restart (0.639s)
[sp-web03.phx1.mozilla.com] out: Stopping memcached: [  OK  ]
[sp-web03.phx1.mozilla.com] out: Starting memcached: [  OK  ]
[sp-web04.phx1.mozilla.com] finished: service memcached restart (0.746s)
[sp-web04.phx1.mozilla.com] out: Stopping memcached: [  OK  ]
[sp-web04.phx1.mozilla.com] out: Starting memcached: [  OK  ]
[sp-web05.phx1.mozilla.com] finished: service memcached restart (0.746s)
[sp-web05.phx1.mozilla.com] out: Stopping memcached: [  OK  ]
[sp-web05.phx1.mozilla.com] out: Starting memcached: [  OK  ]
[sp-web02.phx1.mozilla.com] finished: service memcached restart (1.991s)
[sp-web02.phx1.mozilla.com] out: Stopping memcached: [  OK  ]
[sp-web02.phx1.mozilla.com] out: Starting memcached: [  OK  ]
(Assignee)

Comment 3

5 years ago
Log of fixing processors

-> % ssh sp-processor01.phx1.mozilla.com sudo rm -rf /var/lock/socorro-processor.lock
Connection to sp-processor01.phx1.mozilla.com closed.
bburton@althalus [02:35:48] [~]
-> % ssh sp-processor01.phx1.mozilla.com sudo /sbin/service socorro-processor start
Starting socorro-processor:                                [  OK  ]
Connection to sp-processor01.phx1.mozilla.com closed.
bburton@althalus [02:36:11] [~]
-> % ssh sp-processor02.phx1.mozilla.com sudo rm -rf /var/lock/socorro-processor.lock
Connection to sp-processor02.phx1.mozilla.com closed.
bburton@althalus [02:36:18] [~]
-> % ssh sp-processor02.phx1.mozilla.com sudo /sbin/service socorro-processor start
Starting socorro-processor:                                [  OK  ]
Connection to sp-processor02.phx1.mozilla.com closed.
bburton@althalus [02:36:31] [~]
-> % ssh sp-processor03.phx1.mozilla.com sudo rm -rf /var/lock/socorro-processor.lock
Connection to sp-processor03.phx1.mozilla.com closed.
bburton@althalus [02:36:38] [~]
-> % ssh sp-processor03.phx1.mozilla.com sudo /sbin/service socorro-processor start
Starting socorro-processor:                                [  OK  ]
Connection to sp-processor03.phx1.mozilla.com closed.
bburton@althalus [02:36:45] [~]
-> % ssh sp-processor04.phx1.mozilla.com sudo rm -rf /var/lock/socorro-processor.lock
Connection to sp-processor04.phx1.mozilla.com closed.
bburton@althalus [02:36:53] [~]
-> % ssh sp-processor04.phx1.mozilla.com sudo /sbin/service socorro-processor start
Starting socorro-processor:                                [  OK  ]
Connection to sp-processor04.phx1.mozilla.com closed.
bburton@althalus [02:37:01] [~]
-> % ssh sp-processor05.phx1.mozilla.com sudo rm -rf /var/lock/socorro-processor.lock
Connection to sp-processor05.phx1.mozilla.com closed.
bburton@althalus [02:37:36] [~]
-> % ssh sp-processor05.phx1.mozilla.com sudo /sbin/service socorro-processor start
Starting socorro-processor:                                [  OK  ]
Connection to sp-processor05.phx1.mozilla.com closed.
bburton@althalus [02:37:43] [~]
-> % ssh sp-processor07.phx1.mozilla.com sudo rm -rf /var/lock/socorro-processor.lock
Connection to sp-processor07.phx1.mozilla.com closed.
bburton@althalus [02:39:00] [~]
-> % ssh sp-processor07.phx1.mozilla.com sudo /sbin/service socorro-processor start
Starting socorro-processor:                                [  OK  ]
Connection to sp-processor07.phx1.mozilla.com closed.
bburton@althalus [02:39:07] [~]
-> % ssh sp-processor08.phx1.mozilla.com sudo rm -rf /var/lock/socorro-processor.lock
Connection to sp-processor08.phx1.mozilla.com closed.
bburton@althalus [02:40:30] [~]
-> % ssh sp-processor08.phx1.mozilla.com sudo /sbin/service socorro-processor start
Starting socorro-processor:                                [  OK  ]
Connection to sp-processor08.phx1.mozilla.com closed.
bburton@althalus [02:40:39] [~]
-> % ssh sp-processor10.phx1.mozilla.com sudo rm -rf /var/lock/socorro-processor.lock
Connection to sp-processor10.phx1.mozilla.com closed.
bburton@althalus [02:42:20] [~]
-> % ssh sp-processor10.phx1.mozilla.com sudo /sbin/service socorro-processor start

Starting socorro-processor:                                [  OK  ]
Connection to sp-processor10.phx1.mozilla.com closed.
(Assignee)

Comment 4

5 years ago
Django push complete

[localhost] out: create mode 100755 crash-stats.mozilla.org/socorro/webapp-django/vendor-local/lib/python/rsa/asn1.py
[localhost] out: create mode 100755 crash-stats.mozilla.org/socorro/webapp-django/vendor/lib/python/django/contrib/gis/gdal/LICENSE
[localhost] out: create mode 100755 crash-stats.mozilla.org/socorro/webapp-django/vendor/lib/python/django/contrib/gis/geos/LICENSE
[localhost] out: create mode 100755 crash-stats.mozilla.org/socorro/webapp-django/vendor/lib/python/django/dispatch/license.txt
[localhost] out: create mode 100755 crash-stats.mozilla.org/socorro/webapp-django/vendor/lib/python/django/utils/simplejson/LICENSE.txt
[2013-09-19 21:46:45] Finished commit_www (14.967s)
[2013-09-19 21:46:45] Running push_www
[2013-09-19 21:46:45] [socorro1.webapp.phx1.mozilla.com] running: /data/bin/update-www.sh crash-stats.mozilla.org
[2013-09-19 21:46:45] [socorro2.webapp.phx1.mozilla.com] running: /data/bin/update-www.sh crash-stats.mozilla.org
[2013-09-19 21:46:45] [socorro3.webapp.phx1.mozilla.com] running: /data/bin/update-www.sh crash-stats.mozilla.org
[2013-09-19 21:46:45] [socorro4.webapp.phx1.mozilla.com] running: /data/bin/update-www.sh crash-stats.mozilla.org
[2013-09-19 21:47:16] [socorro3.webapp.phx1.mozilla.com] finished: /data/bin/update-www.sh crash-stats.mozilla.org (30.887s)
[2013-09-19 21:47:16] [socorro1.webapp.phx1.mozilla.com] finished: /data/bin/update-www.sh crash-stats.mozilla.org (30.970s)
[socorro1.webapp.phx1.mozilla.com] out: Not removing crash-stats.mozilla.org/app/
[2013-09-19 21:47:18] [socorro2.webapp.phx1.mozilla.com] finished: /data/bin/update-www.sh crash-stats.mozilla.org (32.532s)
[2013-09-19 21:47:18] [socorro4.webapp.phx1.mozilla.com] finished: /data/bin/update-www.sh crash-stats.mozilla.org (32.694s)
[2013-09-19 21:47:18] Finished push_www (32.701s)
[2013-09-19 21:47:18] [socorro1.webapp.phx1.mozilla.com] running: service httpd restart
[2013-09-19 21:47:18] [socorro2.webapp.phx1.mozilla.com] running: service httpd restart
[2013-09-19 21:47:18] [socorro3.webapp.phx1.mozilla.com] running: service httpd restart
[2013-09-19 21:47:18] [socorro4.webapp.phx1.mozilla.com] running: service httpd restart
[2013-09-19 21:47:20] [socorro4.webapp.phx1.mozilla.com] finished: service httpd restart (1.822s)
[socorro4.webapp.phx1.mozilla.com] out: Stopping httpd: [  OK  ]
[socorro4.webapp.phx1.mozilla.com] out: Starting httpd: [  OK  ]
[2013-09-19 21:47:20] [socorro2.webapp.phx1.mozilla.com] finished: service httpd restart (1.953s)
[socorro2.webapp.phx1.mozilla.com] out: Stopping httpd: [  OK  ]
[socorro2.webapp.phx1.mozilla.com] out: Starting httpd: [  OK  ]
[2013-09-19 21:47:20] [socorro1.webapp.phx1.mozilla.com] finished: service httpd restart (2.209s)
[socorro1.webapp.phx1.mozilla.com] out: Stopping httpd: [  OK  ]
[socorro1.webapp.phx1.mozilla.com] out: Starting httpd: [  OK  ]
[2013-09-19 21:47:20] [socorro3.webapp.phx1.mozilla.com] finished: service httpd restart (2.220s)
[socorro3.webapp.phx1.mozilla.com] out: Stopping httpd: [  OK  ]
[socorro3.webapp.phx1.mozilla.com] out: Starting httpd: [  OK  ]
[2013-09-19 21:47:20] [socorro-memcache1.webapp.phx1.mozilla.com] running: service memcached restart
[2013-09-19 21:47:20] [socorro-memcache2.webapp.phx1.mozilla.com] running: service memcached restart
[2013-09-19 21:47:21] [socorro-memcache1.webapp.phx1.mozilla.com] finished: service memcached restart (0.691s)
[socorro-memcache1.webapp.phx1.mozilla.com] out: Stopping memcached: [  OK  ]
[socorro-memcache1.webapp.phx1.mozilla.com] out: Starting memcached: [  OK  ]
[2013-09-19 21:47:21] [socorro-memcache2.webapp.phx1.mozilla.com] finished: service memcached restart (0.880s)
[socorro-memcache2.webapp.phx1.mozilla.com] out: Stopping memcached: [  OK  ]
[socorro-memcache2.webapp.phx1.mozilla.com] out: Starting memcached: [  OK  ]
Status: ASSIGNED → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
QA verified -
* prod automation passes
* manual bfts pass
Status: RESOLVED → VERIFIED
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.