Closed Bug 974657 Opened 11 years ago Closed 11 years ago

Push Socorro Release 20140219

Categories

(Socorro :: Infra, task)

Product:
Socorro
Component:
Infra
Platform:
x86
macOS
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: lonnen, Unassigned)

Details

The push highlights: * needed to ack fingerprints for new servers even though I ran the precheck script in mana * after acking, presented with a blank screen for a while. needed to hit *enter* a few times to get the buffer to start updating again, not sure what that was about * processor09 had a stuck lock, cleared and restarted it using sudo from the processor09 box The logs: [lonnen@sp-admin01.phx1 ~]$ /usr/local/bin/update-socorro-all.sh https://ci.mozilla.org/job/socorro-release/913/artifact/socorro.tar.gz Upgrading socorro to https://ci.mozilla.org/job/socorro-release/913/artifact/socorro.tar.gz upgrading collectors (in parallel) [socorro-collector1.webapp.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/913/artifact/socorro.tar.gz [socorro-collector2.webapp.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/913/artifact/socorro.tar.gz [socorro-collector3.webapp.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/913/artifact/socorro.tar.gz [socorro-collector4.webapp.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/913/artifact/socorro.tar.gz [socorro-collector5.webapp.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/913/artifact/socorro.tar.gz [socorro-collector6.webapp.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/913/artifact/socorro.tar.gz The authenticity of host 'socorro-collector2.webapp.phx1.mozilla.com (10.8.81.23)' can't be established. RSA key fingerprint is 23:46:1f:27:bd:ae:1b:85:40:54:3d:48:11:1a:34:7f. Are you sure you want to continue connecting (yes/no)? The authenticity of host 'socorro-collector1.webapp.phx1.mozilla.com (10.8.81.22)' can't be established. RSA key fingerprint is e1:cb:44:b2:8c:34:c8:44:fe:c6:df:9d:5a:1b:7f:7e. Are you sure you want to continue connecting (yes/no)? The authenticity of host 'socorro-collector4.webapp.phx1.mozilla.com (10.8.81.25)' can't be established. RSA key fingerprint is 1f:8e:81:14:59:b9:7e:b7:91:58:38:4d:ff:8f:c1:7c. Are you sure you want to continue connecting (yes/no)? The authenticity of host 'socorro-collector3.webapp.phx1.mozilla.com (10.8.81.24)' can't be established. RSA key fingerprint is 52:d0:f4:94:dc:8a:90:c3:62:50:c0:ab:ce:70:0d:3b. Are you sure you want to continue connecting (yes/no)? The authenticity of host 'socorro-collector5.webapp.phx1.mozilla.com (10.8.81.26)' can't be established. RSA key fingerprint is 6d:f8:fa:ee:9c:cc:9f:4c:f2:96:4e:f3:42:c0:61:85. Are you sure you want to continue connecting (yes/no)? The authenticity of host 'socorro-collector6.webapp.phx1.mozilla.com (10.8.81.27)' can't be established. RSA key fingerprint is 84:de:0a:c9:56:2c:16:c1:c4:da:9a:06:7a:4a:8e:0f. Are you sure you want to continue connecting (yes/no)? yes Please type 'yes' or 'no': yes Please type 'yes' or 'no': yes Please type 'yes' or 'no': yes [socorro-collector6.webapp.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/913/artifact/socorro.tar.gz (328.292s) [socorro-collector6.webapp.phx1.mozilla.com] err: Host key verification failed. [socorro-collector2.webapp.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/913/artifact/socorro.tar.gz (328.508s) [socorro-collector2.webapp.phx1.mozilla.com] err: Host key verification failed. [socorro-collector4.webapp.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/913/artifact/socorro.tar.gz (328.506s) [socorro-collector4.webapp.phx1.mozilla.com] out: Stopping socorro-crashmover: [ OK ] [socorro-collector4.webapp.phx1.mozilla.com] out: Starting socorro-crashmover: [ OK ] [socorro-collector4.webapp.phx1.mozilla.com] out: Stopping httpd: [ OK ] [socorro-collector4.webapp.phx1.mozilla.com] out: Starting httpd: [ OK ] [socorro-collector4.webapp.phx1.mozilla.com] out: Socorro build installed successfully! [socorro-collector4.webapp.phx1.mozilla.com] out: Downloaded from https://ci.mozilla.org/job/socorro-release/913/artifact/socorro.tar.gz [socorro-collector4.webapp.phx1.mozilla.com] out: Checksum: 190ea17dbfb343b5dbc06bf97c23e5cd [socorro-collector4.webapp.phx1.mozilla.com] out: Backed up original to /data/socorro.19-02-2014_22_49_24 [socorro-collector4.webapp.phx1.mozilla.com] err: Warning: Permanently added 'socorro-collector4.webapp.phx1.mozilla.com' (RSA) to the list of known hosts. [socorro-collector3.webapp.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/913/artifact/socorro.tar.gz (328.507s) [socorro-collector3.webapp.phx1.mozilla.com] err: Host key verification failed. [socorro-collector1.webapp.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/913/artifact/socorro.tar.gz (329.802s) [socorro-collector1.webapp.phx1.mozilla.com] err: Host key verification failed. [socorro-collector5.webapp.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/913/artifact/socorro.tar.gz (329.798s) [socorro-collector5.webapp.phx1.mozilla.com] err: Host key verification failed. upgrading mware (in parallel) [sp-mware01.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/913/artifact/socorro.tar.gz [sp-mware02.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/913/artifact/socorro.tar.gz [sp-mware03.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/913/artifact/socorro.tar.gz [sp-mware03.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/913/artifact/socorro.tar.gz (5.560s) [sp-mware03.phx1.mozilla.com] out: Stopping httpd: [ OK ] [sp-mware03.phx1.mozilla.com] out: Starting httpd: [ OK ] [sp-mware03.phx1.mozilla.com] out: Socorro build installed successfully! [sp-mware03.phx1.mozilla.com] out: Downloaded from https://ci.mozilla.org/job/socorro-release/913/artifact/socorro.tar.gz [sp-mware03.phx1.mozilla.com] out: Checksum: 190ea17dbfb343b5dbc06bf97c23e5cd [sp-mware03.phx1.mozilla.com] out: Backed up original to /data/socorro.19-02-2014_14_54_47 [sp-mware02.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/913/artifact/socorro.tar.gz (6.794s) [sp-mware02.phx1.mozilla.com] out: Stopping httpd: [ OK ] [sp-mware02.phx1.mozilla.com] out: Starting httpd: [ OK ] [sp-mware02.phx1.mozilla.com] out: Socorro build installed successfully! [sp-mware02.phx1.mozilla.com] out: Downloaded from https://ci.mozilla.org/job/socorro-release/913/artifact/socorro.tar.gz [sp-mware02.phx1.mozilla.com] out: Checksum: 190ea17dbfb343b5dbc06bf97c23e5cd [sp-mware02.phx1.mozilla.com] out: Backed up original to /data/socorro.19-02-2014_14_54_47 [sp-mware01.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/913/artifact/socorro.tar.gz (7.852s) [sp-mware01.phx1.mozilla.com] out: Stopping httpd: [ OK ] [sp-mware01.phx1.mozilla.com] out: Starting httpd: [ OK ] [sp-mware01.phx1.mozilla.com] out: Socorro build installed successfully! [sp-mware01.phx1.mozilla.com] out: Downloaded from https://ci.mozilla.org/job/socorro-release/913/artifact/socorro.tar.gz [sp-mware01.phx1.mozilla.com] out: Checksum: 190ea17dbfb343b5dbc06bf97c23e5cd [sp-mware01.phx1.mozilla.com] out: Backed up original to /data/socorro.19-02-2014_14_54_47 upgrading processors (in parallel) [sp-processor01.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/913/artifact/socorro.tar.gz [sp-processor02.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/913/artifact/socorro.tar.gz [sp-processor03.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/913/artifact/socorro.tar.gz [sp-processor04.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/913/artifact/socorro.tar.gz [sp-processor05.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/913/artifact/socorro.tar.gz [sp-processor06.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/913/artifact/socorro.tar.gz [sp-processor07.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/913/artifact/socorro.tar.gz [sp-processor08.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/913/artifact/socorro.tar.gz [sp-processor09.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/913/artifact/socorro.tar.gz [sp-processor10.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/913/artifact/socorro.tar.gz [sp-processor08.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/913/artifact/socorro.tar.gz (11.807s) [sp-processor08.phx1.mozilla.com] out: Stopping socorro-processor: [ OK ] [sp-processor08.phx1.mozilla.com] out: Starting socorro-processor: [ OK ] [sp-processor08.phx1.mozilla.com] out: Socorro build installed successfully! [sp-processor08.phx1.mozilla.com] out: Downloaded from https://ci.mozilla.org/job/socorro-release/913/artifact/socorro.tar.gz [sp-processor08.phx1.mozilla.com] out: Checksum: 190ea17dbfb343b5dbc06bf97c23e5cd [sp-processor08.phx1.mozilla.com] out: Backed up original to /data/socorro.19-02-2014_14_54_55 [sp-processor04.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/913/artifact/socorro.tar.gz (22.194s) [sp-processor04.phx1.mozilla.com] out: Stopping socorro-processor: [ OK ] [sp-processor04.phx1.mozilla.com] out: Starting socorro-processor: [ OK ] [sp-processor04.phx1.mozilla.com] out: Socorro build installed successfully! [sp-processor04.phx1.mozilla.com] out: Downloaded from https://ci.mozilla.org/job/socorro-release/913/artifact/socorro.tar.gz [sp-processor04.phx1.mozilla.com] out: Checksum: 190ea17dbfb343b5dbc06bf97c23e5cd [sp-processor04.phx1.mozilla.com] out: Backed up original to /data/socorro.19-02-2014_14_54_55 [sp-processor05.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/913/artifact/socorro.tar.gz (22.194s) [sp-processor05.phx1.mozilla.com] out: Stopping socorro-processor: [ OK ] [sp-processor05.phx1.mozilla.com] out: Starting socorro-processor: [ OK ] [sp-processor05.phx1.mozilla.com] out: Socorro build installed successfully! [sp-processor05.phx1.mozilla.com] out: Downloaded from https://ci.mozilla.org/job/socorro-release/913/artifact/socorro.tar.gz [sp-processor05.phx1.mozilla.com] out: Checksum: 190ea17dbfb343b5dbc06bf97c23e5cd [sp-processor05.phx1.mozilla.com] out: Backed up original to /data/socorro.19-02-2014_14_54_55 [sp-processor03.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/913/artifact/socorro.tar.gz (25.248s) [sp-processor03.phx1.mozilla.com] out: Stopping socorro-processor: [ OK ] [sp-processor03.phx1.mozilla.com] out: Starting socorro-processor: [ OK ] [sp-processor03.phx1.mozilla.com] out: Socorro build installed successfully! [sp-processor03.phx1.mozilla.com] out: Downloaded from https://ci.mozilla.org/job/socorro-release/913/artifact/socorro.tar.gz [sp-processor03.phx1.mozilla.com] out: Checksum: 190ea17dbfb343b5dbc06bf97c23e5cd [sp-processor03.phx1.mozilla.com] out: Backed up original to /data/socorro.19-02-2014_14_54_55 [sp-processor01.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/913/artifact/socorro.tar.gz (26.050s) [sp-processor01.phx1.mozilla.com] out: Stopping socorro-processor: [ OK ] [sp-processor01.phx1.mozilla.com] out: Starting socorro-processor: [ OK ] [sp-processor01.phx1.mozilla.com] out: Socorro build installed successfully! [sp-processor01.phx1.mozilla.com] out: Downloaded from https://ci.mozilla.org/job/socorro-release/913/artifact/socorro.tar.gz [sp-processor01.phx1.mozilla.com] out: Checksum: 190ea17dbfb343b5dbc06bf97c23e5cd [sp-processor01.phx1.mozilla.com] out: Backed up original to /data/socorro.19-02-2014_14_54_55 [sp-processor02.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/913/artifact/socorro.tar.gz (26.050s) [sp-processor02.phx1.mozilla.com] out: Stopping socorro-processor: [ OK ] [sp-processor02.phx1.mozilla.com] out: Starting socorro-processor: [ OK ] [sp-processor02.phx1.mozilla.com] out: Socorro build installed successfully! [sp-processor02.phx1.mozilla.com] out: Downloaded from https://ci.mozilla.org/job/socorro-release/913/artifact/socorro.tar.gz [sp-processor02.phx1.mozilla.com] out: Checksum: 190ea17dbfb343b5dbc06bf97c23e5cd [sp-processor02.phx1.mozilla.com] out: Backed up original to /data/socorro.19-02-2014_14_54_55 [sp-processor06.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/913/artifact/socorro.tar.gz (26.159s) [sp-processor06.phx1.mozilla.com] out: Stopping socorro-processor: [ OK ] [sp-processor06.phx1.mozilla.com] out: Starting socorro-processor: [ OK ] [sp-processor06.phx1.mozilla.com] out: Socorro build installed successfully! [sp-processor06.phx1.mozilla.com] out: Downloaded from https://ci.mozilla.org/job/socorro-release/913/artifact/socorro.tar.gz [sp-processor06.phx1.mozilla.com] out: Checksum: 190ea17dbfb343b5dbc06bf97c23e5cd [sp-processor06.phx1.mozilla.com] out: Backed up original to /data/socorro.19-02-2014_14_54_55 [sp-processor10.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/913/artifact/socorro.tar.gz (26.657s) [sp-processor10.phx1.mozilla.com] out: Stopping socorro-processor: [ OK ] [sp-processor10.phx1.mozilla.com] out: Starting socorro-processor: [ OK ] [sp-processor10.phx1.mozilla.com] out: Socorro build installed successfully! [sp-processor10.phx1.mozilla.com] out: Downloaded from https://ci.mozilla.org/job/socorro-release/913/artifact/socorro.tar.gz [sp-processor10.phx1.mozilla.com] out: Checksum: 190ea17dbfb343b5dbc06bf97c23e5cd [sp-processor10.phx1.mozilla.com] out: Backed up original to /data/socorro.19-02-2014_14_54_55 [sp-processor09.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/913/artifact/socorro.tar.gz (26.658s) [sp-processor09.phx1.mozilla.com] out: Stopping socorro-processor: [ OK ] [sp-processor09.phx1.mozilla.com] out: Starting socorro-processor: [FAILED] [sp-processor09.phx1.mozilla.com] out: ERROR: could not start socorro-processor [sp-processor09.phx1.mozilla.com] err: Can't lock the lock file "/var/lock/socorro-processor.lock". Is another instance running? [sp-processor07.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/913/artifact/socorro.tar.gz (27.175s) [sp-processor07.phx1.mozilla.com] out: Stopping socorro-processor: [ OK ] [sp-processor07.phx1.mozilla.com] out: Starting socorro-processor: [ OK ] [sp-processor07.phx1.mozilla.com] out: Socorro build installed successfully! [sp-processor07.phx1.mozilla.com] out: Downloaded from https://ci.mozilla.org/job/socorro-release/913/artifact/socorro.tar.gz [sp-processor07.phx1.mozilla.com] out: Checksum: 190ea17dbfb343b5dbc06bf97c23e5cd [sp-processor07.phx1.mozilla.com] out: Backed up original to /data/socorro.19-02-2014_14_54_55 upgrading db servers (in parallel) [socorro1.db.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/913/artifact/socorro.tar.gz [socorro2.db.phx1.mozilla.com] running: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/913/artifact/socorro.tar.gz [socorro1.db.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/913/artifact/socorro.tar.gz (5.399s) [socorro1.db.phx1.mozilla.com] out: Socorro build installed successfully! [socorro1.db.phx1.mozilla.com] out: Downloaded from https://ci.mozilla.org/job/socorro-release/913/artifact/socorro.tar.gz [socorro1.db.phx1.mozilla.com] out: Checksum: 190ea17dbfb343b5dbc06bf97c23e5cd [socorro1.db.phx1.mozilla.com] out: Backed up original to /data/socorro.19-02-2014_14_55_22 [socorro2.db.phx1.mozilla.com] finished: sudo /data/bin/update-socorro.sh https://ci.mozilla.org/job/socorro-release/913/artifact/socorro.tar.gz (5.638s) [socorro2.db.phx1.mozilla.com] out: Socorro build installed successfully! [socorro2.db.phx1.mozilla.com] out: Downloaded from https://ci.mozilla.org/job/socorro-release/913/artifact/socorro.tar.gz [socorro2.db.phx1.mozilla.com] out: Checksum: 190ea17dbfb343b5dbc06bf97c23e5cd [socorro2.db.phx1.mozilla.com] out: Backed up original to /data/socorro.19-02-2014_14_55_22 upgrading admin host Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/913/artifact/socorro.tar.gz Checksum: 190ea17dbfb343b5dbc06bf97c23e5cd Backed up original to /data/socorro.19-02-2014_14_55_27 upgrading django servers (in parallel) sudo: no tty present and no askpass program specified finished upgrading socorro to https://ci.mozilla.org/job/socorro-release/913/artifact/socorro.tar.gz ... [lonnen@sp-processor09.phx1 ~]$ rm /var/lock/socorro-processor.lock rm: remove write-protected regular empty file `/var/lock/socorro-processor.lock'? y rm: cannot remove `/var/lock/socorro-processor.lock': Permission denied [lonnen@sp-processor09.phx1 ~]$ sudo rm /var/lock/socorro-processor.lock [lonnen@sp-processor09.phx1 ~]$ /etc/init.d/socorro-processor restart Stopping socorro-processor: [FAILED] Starting socorro-processor: Can't create lock file "/var/lock/socorro-processor.lock": Permission denied [FAILED] [lonnen@sp-processor09.phx1 ~]$ ls -la /var/lock/ total 20 drwxrwxr-x. 5 root lock 4096 Feb 19 15:12 . drwxr-xr-x. 20 root root 4096 Nov 11 12:29 .. drwxr-xr-x. 2 root root 4096 Feb 16 2012 dmraid drwx------. 2 root root 4096 Dec 5 2012 lvm drwxr-xr-x. 2 root root 4096 Feb 19 03:09 subsys [lonnen@sp-processor09.phx1 ~]$ sudo /etc/init.d/socorro-processor restart Stopping socorro-processor: [FAILED] Starting socorro-processor: [ OK ]
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.