Closed Bug 860852 Opened 12 years ago Closed 12 years ago

push Socorro 41 release

Categories

(Infrastructure & Operations Graveyard :: WebOps: Other, task, P3)

Product:
Infrastructure & Operations Graveyard
Component:
WebOps: Other
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: lonnen, Assigned: cturra)

References

Details

(Whiteboard: [push interrupt])

Build: https://ci.mozilla.org/job/socorro-release/368/artifact/socorro.tar.gz this release contains a replacement for the processor. We have a plan to manually apply the release to a single processor and verify it works correctly before pushing the code to all servers. Details are in the etherpad tracking the release here: https://etherpad.mozilla.org/socorro-release-checklist-41
just grabbing ownership of this bug since i am actively working on it.
Assignee: server-ops-webops → cturra
push to prod complete. it had a few processor nodes error out with the "lock file" issue, which i manually sorted out, but overall went pretty straight forward. [cturra@sp-admin01.phx1 ~]$ sudo /root/bin/update-socorro-all.sh https://ci.mozilla.org/job/socorro-release/368/artifact/socorro.tar.gz Upgrading socorro to https://ci.mozilla.org/job/socorro-release/368/artifact/socorro.tar.gz Upgrading socorro on sp-collector01.phx1.mozilla.com Stopping socorro-crashmover: [ OK ] Starting socorro-crashmover: [ OK ] Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/368/artifact/socorro.tar.gz Checksum: 75086fd78c425b840a0f4598406dc99b Backed up original to /data/socorro.11-04-2013_13_46_17 Upgrading socorro on sp-collector02.phx1.mozilla.com Stopping socorro-crashmover: [ OK ] Starting socorro-crashmover: [ OK ] Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/368/artifact/socorro.tar.gz Checksum: 75086fd78c425b840a0f4598406dc99b Backed up original to /data/socorro.11-04-2013_20_46_31 Upgrading socorro on sp-collector03.phx1.mozilla.com Stopping socorro-crashmover: [ OK ] Starting socorro-crashmover: [ OK ] Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/368/artifact/socorro.tar.gz Checksum: 75086fd78c425b840a0f4598406dc99b Backed up original to /data/socorro.11-04-2013_20_46_35 Upgrading socorro on sp-collector04.phx1.mozilla.com Stopping socorro-crashmover: [ OK ] Starting socorro-crashmover: [ OK ] Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/368/artifact/socorro.tar.gz Checksum: 75086fd78c425b840a0f4598406dc99b Backed up original to /data/socorro.11-04-2013_20_46_38 Upgrading socorro on sp-collector05.phx1.mozilla.com Stopping socorro-crashmover: [ OK ] Starting socorro-crashmover: [ OK ] Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/368/artifact/socorro.tar.gz Checksum: 75086fd78c425b840a0f4598406dc99b Backed up original to /data/socorro.11-04-2013_20_46_42 Upgrading socorro on sp-collector06.phx1.mozilla.com Stopping socorro-crashmover: [ OK ] Starting socorro-crashmover: [ OK ] Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/368/artifact/socorro.tar.gz Checksum: 75086fd78c425b840a0f4598406dc99b Backed up original to /data/socorro.11-04-2013_20_46_46 Upgrading socorro on sp-mware01.phx1.mozilla.com Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/368/artifact/socorro.tar.gz Checksum: 75086fd78c425b840a0f4598406dc99b Backed up original to /data/socorro.11-04-2013_13_46_50 Upgrading socorro on sp-mware02.phx1.mozilla.com Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/368/artifact/socorro.tar.gz Checksum: 75086fd78c425b840a0f4598406dc99b Backed up original to /data/socorro.11-04-2013_13_46_52 Upgrading socorro on sp-mware03.phx1.mozilla.com Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/368/artifact/socorro.tar.gz Checksum: 75086fd78c425b840a0f4598406dc99b Backed up original to /data/socorro.11-04-2013_13_46_55 Upgrading socorro on sp-processor01.phx1.mozilla.com Stopping socorro-processor: [ OK ] Starting socorro-processor: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/368/artifact/socorro.tar.gz Checksum: 75086fd78c425b840a0f4598406dc99b Backed up original to /data/socorro.11-04-2013_13_46_57 Upgrading socorro on sp-processor02.phx1.mozilla.com Stopping socorro-processor: [ OK ] Starting socorro-processor: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/368/artifact/socorro.tar.gz Checksum: 75086fd78c425b840a0f4598406dc99b Backed up original to /data/socorro.11-04-2013_13_47_02 Upgrading socorro on sp-processor03.phx1.mozilla.com Upgrading socorro on sp-processor04.phx1.mozilla.com Stopping socorro-processor: [ OK ] Starting socorro-processor: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/368/artifact/socorro.tar.gz Checksum: 75086fd78c425b840a0f4598406dc99b Backed up original to /data/socorro.11-04-2013_13_47_07 Upgrading socorro on sp-processor05.phx1.mozilla.com Stopping socorro-processor: [ OK ] Starting socorro-processor: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/368/artifact/socorro.tar.gz Checksum: 75086fd78c425b840a0f4598406dc99b Backed up original to /data/socorro.11-04-2013_13_47_27 Upgrading socorro on sp-processor06.phx1.mozilla.com Stopping socorro-processor: [ OK ] Starting socorro-processor: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/368/artifact/socorro.tar.gz Checksum: 75086fd78c425b840a0f4598406dc99b Backed up original to /data/socorro.11-04-2013_13_47_32 Upgrading socorro on sp-processor07.phx1.mozilla.com Stopping socorro-processor: [ OK ] Starting socorro-processor: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/368/artifact/socorro.tar.gz Checksum: 75086fd78c425b840a0f4598406dc99b Backed up original to /data/socorro.11-04-2013_13_47_52 Upgrading socorro on sp-processor08.phx1.mozilla.com Stopping socorro-processor: [ OK ] Starting socorro-processor: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/368/artifact/socorro.tar.gz Checksum: 75086fd78c425b840a0f4598406dc99b Backed up original to /data/socorro.11-04-2013_13_47_56 Upgrading socorro on sp-processor09.phx1.mozilla.com Stopping socorro-processor: [FAILED] Starting socorro-processor: Can't lock the lock file "/var/lock/socorro-processor.lock". Is another instance running? [FAILED] ERROR: could not start socorro-processor Could not upgrade socorro on sp-processor09.phx1.mozilla.com, exit_code: 1 grr... fix, fix, fix... [cturra@sp-admin01.phx1 ~]$ sudo /root/bin/update-socorro-cturra.sh https://ci.mozilla.org/job/socorro-release/368/artifact/socorro.tar.gz Upgrading socorro to https://ci.mozilla.org/job/socorro-release/368/artifact/socorro.tar.gz Upgrading socorro on sp-processor09.phx1.mozilla.com Upgrading socorro on sp-processor10.phx1.mozilla.com Stopping socorro-processor: [ OK ] Starting socorro-processor: Can't lock the lock file "/var/lock/socorro-processor.lock". Is another instance running? [FAILED] ERROR: could not start socorro-processor Could not upgrade socorro on sp-processor10.phx1.mozilla.com, exit_code: 1 grr... fix, fix, fix (again)... [cturra@sp-admin01.phx1 ~]$ sudo /root/bin/update-socorro-cturra.sh https://ci.mozilla.org/job/socorro-release/368/artifact/socorro.tar.gz Upgrading socorro to https://ci.mozilla.org/job/socorro-release/368/artifact/socorro.tar.gz Upgrading socorro on sp-processor10.phx1.mozilla.com Upgrading socorro on sp-web01.phx1.mozilla.com Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/368/artifact/socorro.tar.gz Checksum: 75086fd78c425b840a0f4598406dc99b Backed up original to /data/socorro.11-04-2013_13_59_07 Upgrading socorro on sp-web02.phx1.mozilla.com Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/368/artifact/socorro.tar.gz Checksum: 75086fd78c425b840a0f4598406dc99b Backed up original to /data/socorro.11-04-2013_13_59_09 Upgrading socorro on sp-web03.phx1.mozilla.com Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/368/artifact/socorro.tar.gz Checksum: 75086fd78c425b840a0f4598406dc99b Backed up original to /data/socorro.11-04-2013_13_59_11 Upgrading socorro on sp-web04.phx1.mozilla.com Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/368/artifact/socorro.tar.gz Checksum: 75086fd78c425b840a0f4598406dc99b Backed up original to /data/socorro.11-04-2013_13_59_27 Upgrading socorro on sp-web05.phx1.mozilla.com Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/368/artifact/socorro.tar.gz Checksum: 75086fd78c425b840a0f4598406dc99b Backed up original to /data/socorro.11-04-2013_13_59_30 Upgrading socorro on sp-admin01.phx1.mozilla.com Stopping socorro-monitor: [ OK ] Starting socorro-monitor: [ OK ] Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/368/artifact/socorro.tar.gz Checksum: 75086fd78c425b840a0f4598406dc99b Backed up original to /data/socorro.11-04-2013_13_59_32 Upgrading socorro on tp-socorro01-master01.phx1.mozilla.com Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/368/artifact/socorro.tar.gz Checksum: 75086fd78c425b840a0f4598406dc99b Backed up original to /data/socorro.11-04-2013_13_59_35 Upgrading socorro on tp-socorro01-master02.phx1.mozilla.com Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/368/artifact/socorro.tar.gz Checksum: 75086fd78c425b840a0f4598406dc99b Backed up original to /data/socorro.11-04-2013_13_59_37 [sp-web01.phx1.mozilla.com] running: service memcached restart [sp-web02.phx1.mozilla.com] running: service memcached restart [sp-web03.phx1.mozilla.com] running: service memcached restart [sp-web04.phx1.mozilla.com] running: service memcached restart [sp-web05.phx1.mozilla.com] running: service memcached restart [sp-web02.phx1.mozilla.com] finished: service memcached restart (0.515s) [sp-web02.phx1.mozilla.com] out: Stopping memcached: [ OK ] [sp-web02.phx1.mozilla.com] out: Starting memcached: [ OK ] [sp-web05.phx1.mozilla.com] finished: service memcached restart (0.532s) [sp-web05.phx1.mozilla.com] out: Stopping memcached: [ OK ] [sp-web05.phx1.mozilla.com] out: Starting memcached: [ OK ] [sp-web04.phx1.mozilla.com] finished: service memcached restart (0.534s) [sp-web04.phx1.mozilla.com] out: Stopping memcached: [ OK ] [sp-web04.phx1.mozilla.com] out: Starting memcached: [ OK ] [sp-web01.phx1.mozilla.com] finished: service memcached restart (0.546s) [sp-web01.phx1.mozilla.com] out: Stopping memcached: [ OK ] [sp-web01.phx1.mozilla.com] out: Starting memcached: [ OK ] [sp-web03.phx1.mozilla.com] finished: service memcached restart (0.622s) [sp-web03.phx1.mozilla.com] out: Stopping memcached: [ OK ] [sp-web03.phx1.mozilla.com] out: Starting memcached: [ OK ]
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Component: Server Operations: Web Operations → WebOps: Other
Product: mozilla.org → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.