Closed Bug 857747 Opened 12 years ago Closed 12 years ago

push Socorro 40.2 release

Categories

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Product:
Infrastructure & Operations Graveyard
Component:
WebOps: Other
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: lonnen, Assigned: cturra)

Details

(Whiteboard: [push interrupt])

upgrade was done in two steps b/c sp-processor01.phx1 failed to remove it's lock file the first go around. manually removing the lock file and restarting the processor process resolved this. [cturra@sp-admin01.phx1 ~]$ sudo /root/bin/update-socorro-all.sh https://ci.mozilla.org/job/socorro-release/365/artifact/socorro.tar.gz Upgrading socorro to https://ci.mozilla.org/job/socorro-release/365/artifact/socorro.tar.gz Upgrading socorro on sp-collector01.phx1.mozilla.com Stopping socorro-crashmover: [ OK ] Starting socorro-crashmover: [ OK ] Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/365/artifact/socorro.tar.gz Checksum: 3eb32c65d995362ee69a6d3eb3fdd75e Backed up original to /data/socorro.03-04-2013_13_52_22 Upgrading socorro on sp-collector02.phx1.mozilla.com Stopping socorro-crashmover: [ OK ] Starting socorro-crashmover: [ OK ] Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/365/artifact/socorro.tar.gz Checksum: 3eb32c65d995362ee69a6d3eb3fdd75e Backed up original to /data/socorro.03-04-2013_20_52_26 Upgrading socorro on sp-collector03.phx1.mozilla.com Stopping socorro-crashmover: [ OK ] Starting socorro-crashmover: [ OK ] Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/365/artifact/socorro.tar.gz Checksum: 3eb32c65d995362ee69a6d3eb3fdd75e Backed up original to /data/socorro.03-04-2013_20_52_30 Upgrading socorro on sp-collector04.phx1.mozilla.com Stopping socorro-crashmover: [ OK ] Starting socorro-crashmover: [ OK ] Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/365/artifact/socorro.tar.gz Checksum: 3eb32c65d995362ee69a6d3eb3fdd75e Backed up original to /data/socorro.03-04-2013_20_52_44 Upgrading socorro on sp-collector05.phx1.mozilla.com Stopping socorro-crashmover: [ OK ] Starting socorro-crashmover: [ OK ] Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/365/artifact/socorro.tar.gz Checksum: 3eb32c65d995362ee69a6d3eb3fdd75e Backed up original to /data/socorro.03-04-2013_20_52_48 Upgrading socorro on sp-collector06.phx1.mozilla.com Stopping socorro-crashmover: [ OK ] Starting socorro-crashmover: [ OK ] Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/365/artifact/socorro.tar.gz Checksum: 3eb32c65d995362ee69a6d3eb3fdd75e Backed up original to /data/socorro.03-04-2013_20_52_52 Upgrading socorro on sp-mware01.phx1.mozilla.com Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/365/artifact/socorro.tar.gz Checksum: 3eb32c65d995362ee69a6d3eb3fdd75e Backed up original to /data/socorro.03-04-2013_13_52_56 Upgrading socorro on sp-mware02.phx1.mozilla.com Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/365/artifact/socorro.tar.gz Checksum: 3eb32c65d995362ee69a6d3eb3fdd75e Backed up original to /data/socorro.03-04-2013_13_52_57 Upgrading socorro on sp-mware03.phx1.mozilla.com Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/365/artifact/socorro.tar.gz Checksum: 3eb32c65d995362ee69a6d3eb3fdd75e Backed up original to /data/socorro.03-04-2013_13_52_59 Upgrading socorro on sp-processor01.phx1.mozilla.com Stopping socorro-processor: [ OK ] Starting socorro-processor: Can't lock the lock file "/var/lock/socorro-processor.lock". Is another instance running? [FAILED] ERROR: could not start socorro-processor Could not upgrade socorro on sp-processor01.phx1.mozilla.com, exit_code: 1 ... [root@sp-processor01.phx1 ~]# rm /var/lock/socorro-processor.lock rm: remove regular empty file `/var/lock/socorro-processor.lock'? y [root@sp-processor01.phx1 ~]# /etc/init.d/socorro-processor start Starting socorro-processor: [ OK ] [root@sp-processor01.phx1 ~]# /etc/init.d/socorro-processor status python2.6 (pid 17426) is running... [root@sp-processor01.phx1 ~]# ps aux | grep -i [1]7426 socorro 17426 18.9 0.1 2065444 35016 ? Ssl 13:57 0:15 /usr/bin/python2.6 /data/socorro/application/scripts/startProcessor.py ... [root@sp-admin01.phx1 ~]# /root/bin/update-socorro-tmp.sh https://ci.mozilla.org/job/socorro-release/365/artifact/socorro.tar.gz Upgrading socorro to https://ci.mozilla.org/job/socorro-release/365/artifact/socorro.tar.gz Upgrading socorro on sp-processor02.phx1.mozilla.com Permission denied (publickey,gssapi-keyex,gssapi-with-mic). Could not upgrade socorro on sp-processor02.phx1.mozilla.com, exit_code: 255 [root@sp-admin01.phx1 ~]# logout [cturra@sp-admin01.phx1 ~]$ sudo /root/bin/update-socorro-tmp.sh https://ci.mozilla.org/job/socorro-release/365/artifact/socorro.tar.gz Upgrading socorro to https://ci.mozilla.org/job/socorro-release/365/artifact/socorro.tar.gz Upgrading socorro on sp-processor02.phx1.mozilla.com Stopping socorro-processor: [ OK ] Starting socorro-processor: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/365/artifact/socorro.tar.gz Checksum: 3eb32c65d995362ee69a6d3eb3fdd75e Backed up original to /data/socorro.03-04-2013_14_01_47 Upgrading socorro on sp-processor03.phx1.mozilla.com Stopping socorro-processor: [ OK ] Starting socorro-processor: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/365/artifact/socorro.tar.gz Checksum: 3eb32c65d995362ee69a6d3eb3fdd75e Backed up original to /data/socorro.03-04-2013_14_02_06 Upgrading socorro on sp-processor04.phx1.mozilla.com Stopping socorro-processor: [ OK ] Starting socorro-processor: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/365/artifact/socorro.tar.gz Checksum: 3eb32c65d995362ee69a6d3eb3fdd75e Backed up original to /data/socorro.03-04-2013_14_02_25 Upgrading socorro on sp-processor05.phx1.mozilla.com Stopping socorro-processor: [ OK ] Starting socorro-processor: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/365/artifact/socorro.tar.gz Checksum: 3eb32c65d995362ee69a6d3eb3fdd75e Backed up original to /data/socorro.03-04-2013_14_02_29 Upgrading socorro on sp-processor06.phx1.mozilla.com Stopping socorro-processor: [ OK ] Starting socorro-processor: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/365/artifact/socorro.tar.gz Checksum: 3eb32c65d995362ee69a6d3eb3fdd75e Backed up original to /data/socorro.03-04-2013_14_02_33 Upgrading socorro on sp-processor07.phx1.mozilla.com Stopping socorro-processor: [ OK ] Starting socorro-processor: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/365/artifact/socorro.tar.gz Checksum: 3eb32c65d995362ee69a6d3eb3fdd75e Backed up original to /data/socorro.03-04-2013_14_02_53 Upgrading socorro on sp-processor08.phx1.mozilla.com Stopping socorro-processor: [ OK ] Starting socorro-processor: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/365/artifact/socorro.tar.gz Checksum: 3eb32c65d995362ee69a6d3eb3fdd75e Backed up original to /data/socorro.03-04-2013_14_03_12 Upgrading socorro on sp-processor09.phx1.mozilla.com Stopping socorro-processor: [ OK ] Starting socorro-processor: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/365/artifact/socorro.tar.gz Checksum: 3eb32c65d995362ee69a6d3eb3fdd75e Backed up original to /data/socorro.03-04-2013_14_03_32 Upgrading socorro on sp-processor10.phx1.mozilla.com Stopping socorro-processor: [ OK ] Starting socorro-processor: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/365/artifact/socorro.tar.gz Checksum: 3eb32c65d995362ee69a6d3eb3fdd75e Backed up original to /data/socorro.03-04-2013_14_03_36 Upgrading socorro on sp-web01.phx1.mozilla.com Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/365/artifact/socorro.tar.gz Checksum: 3eb32c65d995362ee69a6d3eb3fdd75e Backed up original to /data/socorro.03-04-2013_14_03_56 Upgrading socorro on sp-web02.phx1.mozilla.com Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/365/artifact/socorro.tar.gz Checksum: 3eb32c65d995362ee69a6d3eb3fdd75e Backed up original to /data/socorro.03-04-2013_14_04_00 Upgrading socorro on sp-web03.phx1.mozilla.com Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/365/artifact/socorro.tar.gz Checksum: 3eb32c65d995362ee69a6d3eb3fdd75e Backed up original to /data/socorro.03-04-2013_14_04_03 Upgrading socorro on sp-web04.phx1.mozilla.com Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/365/artifact/socorro.tar.gz Checksum: 3eb32c65d995362ee69a6d3eb3fdd75e Backed up original to /data/socorro.03-04-2013_14_04_06 Upgrading socorro on sp-web05.phx1.mozilla.com Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/365/artifact/socorro.tar.gz Checksum: 3eb32c65d995362ee69a6d3eb3fdd75e Backed up original to /data/socorro.03-04-2013_14_04_09 Upgrading socorro on sp-admin01.phx1.mozilla.com Stopping socorro-monitor: [ OK ] Starting socorro-monitor: [ OK ] Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/365/artifact/socorro.tar.gz Checksum: 3eb32c65d995362ee69a6d3eb3fdd75e Backed up original to /data/socorro.03-04-2013_14_04_11 Upgrading socorro on tp-socorro01-master01.phx1.mozilla.com Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/365/artifact/socorro.tar.gz Checksum: 3eb32c65d995362ee69a6d3eb3fdd75e Backed up original to /data/socorro.03-04-2013_14_04_41 Upgrading socorro on tp-socorro01-master02.phx1.mozilla.com Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/365/artifact/socorro.tar.gz Checksum: 3eb32c65d995362ee69a6d3eb3fdd75e Backed up original to /data/socorro.03-04-2013_14_04_42 [sp-web01.phx1.mozilla.com] running: service memcached restart [sp-web02.phx1.mozilla.com] running: service memcached restart [sp-web03.phx1.mozilla.com] running: service memcached restart [sp-web04.phx1.mozilla.com] running: service memcached restart [sp-web05.phx1.mozilla.com] running: service memcached restart [sp-web01.phx1.mozilla.com] finished: service memcached restart (0.569s) [sp-web01.phx1.mozilla.com] out: Stopping memcached: [ OK ] [sp-web01.phx1.mozilla.com] out: Starting memcached: [ OK ] [sp-web03.phx1.mozilla.com] finished: service memcached restart (0.592s) [sp-web03.phx1.mozilla.com] out: Stopping memcached: [ OK ] [sp-web03.phx1.mozilla.com] out: Starting memcached: [ OK ] [sp-web04.phx1.mozilla.com] finished: service memcached restart (0.606s) [sp-web04.phx1.mozilla.com] out: Stopping memcached: [ OK ] [sp-web04.phx1.mozilla.com] out: Starting memcached: [ OK ] [sp-web02.phx1.mozilla.com] finished: service memcached restart (0.671s) [sp-web02.phx1.mozilla.com] out: Stopping memcached: [ OK ] [sp-web02.phx1.mozilla.com] out: Starting memcached: [ OK ] [sp-web05.phx1.mozilla.com] finished: service memcached restart (1.752s) [sp-web05.phx1.mozilla.com] out: Stopping memcached: [ OK ] [sp-web05.phx1.mozilla.com] out: Starting memcached: [ OK ]
Assignee: server-ops-webops → cturra
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Component: Server Operations: Web Operations → WebOps: Other
Product: mozilla.org → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.