Closed Bug 976686 Opened 12 years ago Closed 12 years ago

Push Socorro Release 20140225

Categories

(Socorro :: Infra, task)

Product:
Socorro
Component:
Infra
Platform:
x86
macOS
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED INVALID

People

(Reporter: lonnen, Unassigned)

Details

Initial push had host key verification fails, so I aborted and immediately retried the push. [lonnen@sp-admin01.phx1 ~]$ /usr/local/bin/update-socorro-all.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz Upgrading socorro to https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz upgrading collectors (in parallel) [socorro-collector1.webapp.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz [socorro-collector2.webapp.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz [socorro-collector3.webapp.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz [socorro-collector4.webapp.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz [socorro-collector5.webapp.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz [socorro-collector6.webapp.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz The authenticity of host 'socorro-collector2.webapp.phx1.mozilla.com (10.8.81.23)' can't be established. RSA key fingerprint is 23:46:1f:27:bd:ae:1b:85:40:54:3d:48:11:1a:34:7f. Are you sure you want to continue connecting (yes/no)? The authenticity of host 'socorro-collector3.webapp.phx1.mozilla.com (10.8.81.24)' can't be established. RSA key fingerprint is 52:d0:f4:94:dc:8a:90:c3:62:50:c0:ab:ce:70:0d:3b. Are you sure you want to continue connecting (yes/no)? The authenticity of host 'socorro-collector5.webapp.phx1.mozilla.com (10.8.81.26)' can't be established. RSA key fingerprint is 6d:f8:fa:ee:9c:cc:9f:4c:f2:96:4e:f3:42:c0:61:85. Are you sure you want to continue connecting (yes/no)? The authenticity of host 'socorro-collector1.webapp.phx1.mozilla.com (10.8.81.22)' can't be established. RSA key fingerprint is e1:cb:44:b2:8c:34:c8:44:fe:c6:df:9d:5a:1b:7f:7e. Are you sure you want to continue connecting (yes/no)? The authenticity of host 'socorro-collector6.webapp.phx1.mozilla.com (10.8.81.27)' can't be established. RSA key fingerprint is 84:de:0a:c9:56:2c:16:c1:c4:da:9a:06:7a:4a:8e:0f. Are you sure you want to continue connecting (yes/no)? yes [socorro-collector4.webapp.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz (7.656s) [socorro-collector4.webapp.phx1.mozilla.com] out: Stopping socorro-crashmover: [ OK ] [socorro-collector4.webapp.phx1.mozilla.com] out: Starting socorro-crashmover: [ OK ] [socorro-collector4.webapp.phx1.mozilla.com] out: Stopping httpd: [ OK ] [socorro-collector4.webapp.phx1.mozilla.com] out: Starting httpd: [ OK ] [socorro-collector4.webapp.phx1.mozilla.com] out: Socorro build installed successfully! [socorro-collector4.webapp.phx1.mozilla.com] out: Downloaded from https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz [socorro-collector4.webapp.phx1.mozilla.com] out: Checksum: 9f37c76ddeb799e06366d846eda4208f [socorro-collector4.webapp.phx1.mozilla.com] out: Backed up original to /data/socorro.25-02-2014_19_40_53 [socorro-collector2.webapp.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz (11.973s) [socorro-collector2.webapp.phx1.mozilla.com] out: Stopping socorro-crashmover: [ OK ] [socorro-collector2.webapp.phx1.mozilla.com] out: Starting socorro-crashmover: [ OK ] [socorro-collector2.webapp.phx1.mozilla.com] out: Stopping httpd: [ OK ] [socorro-collector2.webapp.phx1.mozilla.com] out: Starting httpd: [ OK ] [socorro-collector2.webapp.phx1.mozilla.com] out: Socorro build installed successfully! [socorro-collector2.webapp.phx1.mozilla.com] out: Downloaded from https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz [socorro-collector2.webapp.phx1.mozilla.com] out: Checksum: 9f37c76ddeb799e06366d846eda4208f [socorro-collector2.webapp.phx1.mozilla.com] out: Backed up original to /data/socorro.25-02-2014_19_40_57 [socorro-collector2.webapp.phx1.mozilla.com] err: Warning: Permanently added 'socorro-collector2.webapp.phx1.mozilla.com' (RSA) to the list of known hosts. [socorro-collector5.webapp.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz (34.484s) [socorro-collector5.webapp.phx1.mozilla.com] err: Host key verification failed. [socorro-collector3.webapp.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz (37.361s) [socorro-collector3.webapp.phx1.mozilla.com] err: Host key verification failed. [socorro-collector1.webapp.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz (37.364s) [socorro-collector1.webapp.phx1.mozilla.com] err: Host key verification failed. yes [socorro-collector6.webapp.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz (47.707s) [socorro-collector6.webapp.phx1.mozilla.com] out: Stopping socorro-crashmover: [ OK ] [socorro-collector6.webapp.phx1.mozilla.com] out: Starting socorro-crashmover: [ OK ] [socorro-collector6.webapp.phx1.mozilla.com] out: Stopping httpd: [ OK ] [socorro-collector6.webapp.phx1.mozilla.com] out: Starting httpd: [ OK ] [socorro-collector6.webapp.phx1.mozilla.com] out: Socorro build installed successfully! [socorro-collector6.webapp.phx1.mozilla.com] out: Downloaded from https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz [socorro-collector6.webapp.phx1.mozilla.com] out: Checksum: 9f37c76ddeb799e06366d846eda4208f [socorro-collector6.webapp.phx1.mozilla.com] out: Backed up original to /data/socorro.25-02-2014_19_41_33 [socorro-collector6.webapp.phx1.mozilla.com] err: Warning: Permanently added 'socorro-collector6.webapp.phx1.mozilla.com' (RSA) to the list of known hosts. upgrading mware (in parallel) [sp-mware01.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz [sp-mware02.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz [sp-mware03.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz ^X[sp-mware01.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz (6.853s) [sp-mware01.phx1.mozilla.com] out: Stopping httpd: [ OK ] [sp-mware01.phx1.mozilla.com] out: Starting httpd: [ OK ] [sp-mware01.phx1.mozilla.com] out: Socorro build installed successfully! [sp-mware01.phx1.mozilla.com] out: Downloaded from https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz [sp-mware01.phx1.mozilla.com] out: Checksum: 9f37c76ddeb799e06366d846eda4208f [sp-mware01.phx1.mozilla.com] out: Backed up original to /data/socorro.25-02-2014_11_41_41 ^C[sp-mware02.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz (6.928s) [sp-mware02.phx1.mozilla.com] out: Stopping httpd: [sp-mware02.phx1.mozilla.com] err: Killed by signal 2. [sp-mware03.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz (6.927s) [sp-mware03.phx1.mozilla.com] err: Killed by signal 2. Traceback (most recent call last): File "/usr/bin/issue-multi-command", line 45, in <module> remote(hosts, command, remote_limit=int(options.count), ssh_key=options.identity) File "/usr/lib/python2.6/site-packages/commander/commands.py", line 84, in remote t.run_all() File "/usr/lib/python2.6/site-packages/commander/commands.py", line 47, in run_all t.join() File "/usr/lib64/python2.6/threading.py", line 644, in join self.__block.wait() File "/usr/lib64/python2.6/threading.py", line 239, in wait waiter.acquire() KeyboardInterrupt upgrading processors (in parallel) [sp-processor01.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz [sp-processor02.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz [sp-processor03.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz [sp-processor04.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz [sp-processor05.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz [sp-processor06.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz [sp-processor07.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz [sp-processor08.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz [sp-processor09.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz [sp-processor10.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz ^X^C[sp-processor10.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz (3.590s) [sp-processor10.phx1.mozilla.com] err: Killed by signal 2. [sp-processor08.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz (3.594s) [sp-processor08.phx1.mozilla.com] err: Killed by signal 2. [sp-processor05.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz (3.602s) [sp-processor05.phx1.mozilla.com] err: Killed by signal 2. [sp-processor04.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz (3.603s) [sp-processor04.phx1.mozilla.com] err: Killed by signal 2. [sp-processor09.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz (3.593s) [sp-processor09.phx1.mozilla.com] err: Killed by signal 2. [sp-processor02.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz (3.605s) [sp-processor02.phx1.mozilla.com] err: Killed by signal 2. [sp-processor06.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz (3.599s) [sp-processor06.phx1.mozilla.com] err: Killed by signal 2. [sp-processor07.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz (3.599s) [sp-processor07.phx1.mozilla.com] err: Killed by signal 2. [sp-processor03.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz (3.605s) [sp-processor03.phx1.mozilla.com] err: Killed by signal 2. [sp-processor01.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz (3.606s) [sp-processor01.phx1.mozilla.com] err: Killed by signal 2. Traceback (most recent call last): File "/usr/bin/issue-multi-command", line 45, in <module> remote(hosts, command, remote_limit=int(options.count), ssh_key=options.identity) File "/usr/lib/python2.6/site-packages/commander/commands.py", line 84, in remote t.run_all() File "/usr/lib/python2.6/site-packages/commander/commands.py", line 47, in run_all t.join() File "/usr/lib64/python2.6/threading.py", line 644, in join self.__block.wait() File "/usr/lib64/python2.6/threading.py", line 239, in wait waiter.acquire() KeyboardInterrupt upgrading db servers (in parallel) [socorro1.db.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz [socorro2.db.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz ^X^C[socorro1.db.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz (0.729s) [socorro1.db.phx1.mozilla.com] err: Killed by signal 2. Traceback (most recent call last): [socorro2.db.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz (0.729s) File "/usr/bin/issue-multi-command", line 45, in <module> [socorro2.db.phx1.mozilla.com] err: Killed by signal 2. remote(hosts, command, remote_limit=int(options.count), ssh_key=options.identity) File "/usr/lib/python2.6/site-packages/commander/commands.py", line 84, in remote t.run_all() File "/usr/lib/python2.6/site-packages/commander/commands.py", line 47, in run_all t.join() File "/usr/lib64/python2.6/threading.py", line 644, in join self.__block.wait() File "/usr/lib64/python2.6/threading.py", line 239, in wait waiter.acquire() KeyboardInterrupt upgrading admin host ^X^Cupgrading django servers (in parallel) DIR: /data/crashstats/src/crash-stats.mozilla.org ENV: crashstats SITE: crash-stats.mozilla.org APP: socorro Upgrading socorro-crashstats (django) prod Upgrading socorro to https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz /data/crashstats/src/crash-stats.mozilla.org /home/lonnen ^X^CKilled by signal 2. finished upgrading socorro to https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz [lonnen@sp-admin01.phx1 ~]$ # bailing hard because of verification fails [lonnen@sp-admin01.phx1 ~]$ # pushing again [lonnen@sp-admin01.phx1 ~]$ /usr/local/bin/update-socorro-all.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz Upgrading socorro to https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz upgrading collectors (in parallel) [socorro-collector1.webapp.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz [socorro-collector2.webapp.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz [socorro-collector3.webapp.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz [socorro-collector4.webapp.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz [socorro-collector5.webapp.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz [socorro-collector6.webapp.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz The authenticity of host 'socorro-collector3.webapp.phx1.mozilla.com (10.8.81.24)' can't be established. RSA key fingerprint is 52:d0:f4:94:dc:8a:90:c3:62:50:c0:ab:ce:70:0d:3b. Are you sure you want to continue connecting (yes/no)? The authenticity of host 'socorro-collector1.webapp.phx1.mozilla.com (10.8.81.22)' can't be established. RSA key fingerprint is e1:cb:44:b2:8c:34:c8:44:fe:c6:df:9d:5a:1b:7f:7e. Are you sure you want to continue connecting (yes/no)? The authenticity of host 'socorro-collector5.webapp.phx1.mozilla.com (10.8.81.26)' can't be established. RSA key fingerprint is 6d:f8:fa:ee:9c:cc:9f:4c:f2:96:4e:f3:42:c0:61:85. Are you sure you want to continue connecting (yes/no)? yes [socorro-collector2.webapp.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz (2.792s) yes yes [socorro-collector3.webapp.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz (9.648s) [socorro-collector3.webapp.phx1.mozilla.com] out: Stopping socorro-crashmover: [ OK ] [socorro-collector3.webapp.phx1.mozilla.com] out: Starting socorro-crashmover: [ OK ] [socorro-collector3.webapp.phx1.mozilla.com] out: Stopping httpd: [ OK ] [socorro-collector3.webapp.phx1.mozilla.com] out: Starting httpd: [ OK ] [socorro-collector3.webapp.phx1.mozilla.com] out: Socorro build installed successfully! [socorro-collector3.webapp.phx1.mozilla.com] out: Downloaded from https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz [socorro-collector3.webapp.phx1.mozilla.com] out: Checksum: 9f37c76ddeb799e06366d846eda4208f [socorro-collector3.webapp.phx1.mozilla.com] out: Backed up original to /data/socorro.25-02-2014_19_42_12 [socorro-collector3.webapp.phx1.mozilla.com] err: Warning: Permanently added 'socorro-collector3.webapp.phx1.mozilla.com' (RSA) to the list of known hosts. [socorro-collector4.webapp.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz (9.648s) [socorro-collector1.webapp.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz (12.737s) [socorro-collector1.webapp.phx1.mozilla.com] out: Stopping socorro-crashmover: [ OK ] [socorro-collector1.webapp.phx1.mozilla.com] out: Starting socorro-crashmover: [ OK ] [socorro-collector1.webapp.phx1.mozilla.com] out: Stopping httpd: [ OK ] [socorro-collector1.webapp.phx1.mozilla.com] out: Starting httpd: [ OK ] [socorro-collector1.webapp.phx1.mozilla.com] out: Socorro build installed successfully! [socorro-collector1.webapp.phx1.mozilla.com] out: Downloaded from https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz [socorro-collector1.webapp.phx1.mozilla.com] out: Checksum: 9f37c76ddeb799e06366d846eda4208f [socorro-collector1.webapp.phx1.mozilla.com] out: Backed up original to /data/socorro.25-02-2014_11_42_16 [socorro-collector1.webapp.phx1.mozilla.com] err: Warning: Permanently added 'socorro-collector1.webapp.phx1.mozilla.com' (RSA) to the list of known hosts. [socorro-collector5.webapp.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz (13.676s) [socorro-collector5.webapp.phx1.mozilla.com] out: Stopping socorro-crashmover: [ OK ] [socorro-collector5.webapp.phx1.mozilla.com] out: Starting socorro-crashmover: [ OK ] [socorro-collector5.webapp.phx1.mozilla.com] out: Stopping httpd: [ OK ] [socorro-collector5.webapp.phx1.mozilla.com] out: Starting httpd: [ OK ] [socorro-collector5.webapp.phx1.mozilla.com] out: Socorro build installed successfully! [socorro-collector5.webapp.phx1.mozilla.com] out: Downloaded from https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz [socorro-collector5.webapp.phx1.mozilla.com] out: Checksum: 9f37c76ddeb799e06366d846eda4208f [socorro-collector5.webapp.phx1.mozilla.com] out: Backed up original to /data/socorro.25-02-2014_19_42_17 [socorro-collector5.webapp.phx1.mozilla.com] err: Warning: Permanently added 'socorro-collector5.webapp.phx1.mozilla.com' (RSA) to the list of known hosts. [socorro-collector6.webapp.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz (13.675s) upgrading mware (in parallel) [sp-mware01.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz [sp-mware02.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz [sp-mware03.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz [sp-mware03.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz (2.196s) [sp-mware01.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz (3.283s) [sp-mware02.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz (3.283s) upgrading processors (in parallel) [sp-processor01.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz [sp-processor02.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz [sp-processor03.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz [sp-processor04.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz [sp-processor05.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz [sp-processor06.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz [sp-processor07.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz [sp-processor08.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz [sp-processor09.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz [sp-processor10.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz [sp-processor08.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz (3.543s) [sp-processor05.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz (3.825s) [sp-processor04.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz (3.826s) [sp-processor06.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz (5.082s) [sp-processor07.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz (5.080s) [sp-processor01.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz (5.092s) [sp-processor02.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz (5.565s) [sp-processor03.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz (5.765s) [sp-processor09.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz (6.181s) [sp-processor10.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz (6.181s) upgrading db servers (in parallel) [socorro1.db.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz [socorro2.db.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz [socorro2.db.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz (1.805s) [socorro1.db.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz (1.805s) upgrading admin host Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz Checksum: 9f37c76ddeb799e06366d846eda4208f Backed up original to /data/socorro.25-02-2014_11_42_35 upgrading django servers (in parallel) DIR: /data/crashstats/src/crash-stats.mozilla.org ENV: crashstats SITE: crash-stats.mozilla.org APP: socorro Upgrading socorro-crashstats (django) prod Upgrading socorro to https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz /data/crashstats/src/crash-stats.mozilla.org /home/lonnen Creating tables ... Installing custom SQL ... Installing indexes ... Installed 0 object(s) from 0 fixture(s) [2014-02-25 19:42:57] Running rsync_project [2014-02-25 19:42:57] [localhost] running: /usr/bin/rsync -aq --include '.gitkeep' --exclude '.git*' --exclude '.hg*' --exclude '.svn*' --exclude 'CVS' --exclude '.bzr*' --delete /data/crashstats/src/crash-stats.mozilla.org/ /data/crashstats/www/crash-stats.mozilla.org/ [2014-02-25 19:42:58] [localhost] finished: /usr/bin/rsync -aq --include '.gitkeep' --exclude '.git*' --exclude '.hg*' --exclude '.svn*' --exclude 'CVS' --exclude '.bzr*' --delete /data/crashstats/src/crash-stats.mozilla.org/ /data/crashstats/www/crash-stats.mozilla.org/ (1.673s) [2014-02-25 19:42:58] Finished rsync_project (1.673s) [2014-02-25 19:42:58] Running commit_www [2014-02-25 19:42:58] [localhost] running: cd /data/crashstats/www && /usr/bin/git add .; /usr/bin/git commit -a -m 'deploy ['crash-stats.mozilla.org']' [2014-02-25 19:42:59] [localhost] failed: cd /data/crashstats/www && /usr/bin/git add .; /usr/bin/git commit -a -m 'deploy ['crash-stats.mozilla.org']' (0.707s) [localhost] out: # On branch master [localhost] out: nothing to commit (working directory clean) [2014-02-25 19:42:59] [socorro1.webapp.phx1.mozilla.com] running: /data/bin/update-www.sh [2014-02-25 19:42:59] [socorro2.webapp.phx1.mozilla.com] running: /data/bin/update-www.sh [2014-02-25 19:42:59] [socorro3.webapp.phx1.mozilla.com] running: /data/bin/update-www.sh [2014-02-25 19:42:59] [socorro4.webapp.phx1.mozilla.com] running: /data/bin/update-www.sh [2014-02-25 19:43:00] [socorro2.webapp.phx1.mozilla.com] finished: /data/bin/update-www.sh (0.970s) [2014-02-25 19:43:00] [socorro1.webapp.phx1.mozilla.com] finished: /data/bin/update-www.sh (1.394s) [socorro1.webapp.phx1.mozilla.com] out: Not removing crash-stats.mozilla.org/app/ [2014-02-25 19:43:00] [socorro4.webapp.phx1.mozilla.com] finished: /data/bin/update-www.sh (1.390s) [2014-02-25 19:43:00] [socorro3.webapp.phx1.mozilla.com] finished: /data/bin/update-www.sh (1.393s) [2014-02-25 19:43:00] [socorro1.webapp.phx1.mozilla.com] running: service httpd restart [2014-02-25 19:43:00] [socorro2.webapp.phx1.mozilla.com] running: service httpd restart [2014-02-25 19:43:00] [socorro3.webapp.phx1.mozilla.com] running: service httpd restart [2014-02-25 19:43:00] [socorro4.webapp.phx1.mozilla.com] running: service httpd restart [2014-02-25 19:43:01] [socorro3.webapp.phx1.mozilla.com] finished: service httpd restart (0.369s) [socorro3.webapp.phx1.mozilla.com] out: Stopping httpd: [ OK ] [socorro3.webapp.phx1.mozilla.com] out: Starting httpd: [ OK ] [2014-02-25 19:43:01] [socorro2.webapp.phx1.mozilla.com] finished: service httpd restart (0.373s) [socorro2.webapp.phx1.mozilla.com] out: Stopping httpd: [ OK ] [socorro2.webapp.phx1.mozilla.com] out: Starting httpd: [ OK ] [2014-02-25 19:43:01] [socorro1.webapp.phx1.mozilla.com] finished: service httpd restart (0.389s) [socorro1.webapp.phx1.mozilla.com] out: Stopping httpd: [ OK ] [socorro1.webapp.phx1.mozilla.com] out: Starting httpd: [ OK ] [2014-02-25 19:43:01] [socorro4.webapp.phx1.mozilla.com] finished: service httpd restart (0.381s) [socorro4.webapp.phx1.mozilla.com] out: Stopping httpd: [ OK ] [socorro4.webapp.phx1.mozilla.com] out: Starting httpd: [ OK ] [2014-02-25 19:43:01] [socorro-memcache1.webapp.phx1.mozilla.com] running: service memcached restart [2014-02-25 19:43:01] [socorro-memcache2.webapp.phx1.mozilla.com] running: service memcached restart [2014-02-25 19:43:01] [socorro-memcache1.webapp.phx1.mozilla.com] finished: service memcached restart (0.408s) [socorro-memcache1.webapp.phx1.mozilla.com] out: Stopping memcached: [ OK ] [socorro-memcache1.webapp.phx1.mozilla.com] out: Starting memcached: [ OK ] [2014-02-25 19:43:01] [socorro-memcache2.webapp.phx1.mozilla.com] finished: service memcached restart (0.408s) [socorro-memcache2.webapp.phx1.mozilla.com] out: Stopping memcached: [ OK ] [socorro-memcache2.webapp.phx1.mozilla.com] out: Starting memcached: [ OK ] finished upgrading socorro to https://ci.mozilla.org/job/socorro-release/926/artifact/socorro.tar.gz
shortly after the release, everything went bad. pushed last working good release. processor-2 and processor-8 failed to start. rm'd the locks and restarted them. We're back in a good state now. I will try the push again after lunch.
The hostname issue in c1 is due to a mismatch between the current and old hostnames in the docs vs commanderconfig.py I've reviewed all the names and made sure they match I've updated https://mana.mozilla.org/wiki/pages/viewpage.action?pageId=5734601#crash-stats.mozilla.com%28Socorro%29-PrerequisitesforPushing You should go through the prerequisites again before pushing to be sure you have all the hostnames issue-m-c will use
Pushed a second time, no SSH problems this time and prod still began responding with 500s. The middleware servers were returning something unexpected without registering an explicit traceback. I rolled back to last known good release and everything is fine again. We'll try again tomorrow.
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → INVALID
I overlooked the config changes for 948533, thinking they were already pushed to prod when they had not been. We'll be ready to push this tomorrow morning PST.
You need to log in before you can comment on or make changes to this bug.