Closed Bug 688771 Opened 13 years ago Closed 13 years ago

Push Socorro 2.2.4 release

Categories

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Product:
Infrastructure & Operations Graveyard
Component:
WebOps: Other
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
VERIFIED FIXED

People

(Reporter: laura, Assigned: fox2mike)

Details

Just waiting for QA signoff, but I'd like to get this out sometime this morning.  Will post the build and some needed database changes once we get signoff.
QA has signed off. 

Database changes needed are as follows, and should be done before the push
(via jberkus):
"1) log in to master01 database server.
2) switch to Postgres' shell, with ENV (sudo su - postgres)
3) cd /data/socorro/application/scripts/upgrade/2.2.4/
4) ./upgrade.sh

This will take up to 2 hours to run.  It will produce a lot of feedback
in the shell while it's running.  

The upgrade should not block concurrent usage of Socorro or require
downtime.  However, it would be worth keeping an eye on the database to
be sure that's the case.  QA helped me test on stage that the upgrade would not block concurrent usage.  You should still watch the database, but I'm pretty confident."

Build is 
https://jenkins.mozilla.org/job/socorro-release/12/
The database upgrade.sh is running now, on tp-socorro01-master01.phx1, in a screen (the screen is owned by root).

I also started 'script' before running this, so it should log all output to /tmp/socorro-2.2.4-upgrade. This didn't seem to be writing anything, so we stopped the job and added " | tee /tmp/socorro-2.2.4-upgrade-tee" to the end of it, which is logging as expected.

I suspect 'script' will write out to the file when it exits as well, so in the end we'll have 2 copies. :)

Jabba said he'd probably be around later to finish up the push after this is done.
Taking over from Jabba, since he needs to sleep
Assignee: server-ops → shyam
This just finished :

psql:clear_out_final_betas.sql:36: INFO:  backfilling for 2011-09-26
psql:clear_out_final_betas.sql:36: INFO:  adu
psql:clear_out_final_betas.sql:36: INFO:  tcbs
psql:clear_out_final_betas.sql:36: INFO:  daily crashes
psql:clear_out_final_betas.sql:36: INFO:  signatures
 backfill_matviews 
-------------------
 t
(1 row)

COMMIT
BEGIN
SELECT 5
DELETE 1945
DELETE 0
DELETE 77
DELETE 26
DELETE 5
COMMIT
2.2.4. upgrade done
Mon Sep 26 22:31:32 PDT 2011
bash-4.1$
Push complete :

[root@sp-admin01.phx1 ~]# update-socorro-all.sh https://jenkins.mozilla.org/job/socorro-release/12/artifact/socorro.tar.gz
Upgrading socorro to https://jenkins.mozilla.org/job/socorro-release/12/artifact/socorro.tar.gz
Upgrading socorro on sp-collector01.phx1.mozilla.com
Stopping socorro-crashmover: [  OK  ]
Starting socorro-crashmover: [  OK  ]
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://jenkins.mozilla.org/job/socorro-release/12/artifact/socorro.tar.gz
Checksum: 455f390eda50642efba5e1860f49540b
Backed up original to /data/socorro.26-09-2011_22_38_08
Upgrading socorro on sp-collector02.phx1.mozilla.com
Stopping socorro-crashmover: [  OK  ]
Starting socorro-crashmover: [  OK  ]
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://jenkins.mozilla.org/job/socorro-release/12/artifact/socorro.tar.gz
Checksum: 455f390eda50642efba5e1860f49540b
Backed up original to /data/socorro.26-09-2011_22_38_17
Upgrading socorro on sp-collector03.phx1.mozilla.com
Stopping socorro-crashmover: [  OK  ]
Starting socorro-crashmover: [  OK  ]
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://jenkins.mozilla.org/job/socorro-release/12/artifact/socorro.tar.gz
Checksum: 455f390eda50642efba5e1860f49540b
Backed up original to /data/socorro.26-09-2011_22_38_27
Upgrading socorro on sp-collector04.phx1.mozilla.com
Stopping socorro-crashmover: [  OK  ]
Starting socorro-crashmover: [  OK  ]
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://jenkins.mozilla.org/job/socorro-release/12/artifact/socorro.tar.gz
Checksum: 455f390eda50642efba5e1860f49540b
Backed up original to /data/socorro.26-09-2011_22_38_36
Upgrading socorro on sp-collector05.phx1.mozilla.com
Stopping socorro-crashmover: [  OK  ]
Starting socorro-crashmover: [  OK  ]
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://jenkins.mozilla.org/job/socorro-release/12/artifact/socorro.tar.gz
Checksum: 455f390eda50642efba5e1860f49540b
Backed up original to /data/socorro.26-09-2011_22_38_56
Upgrading socorro on sp-collector06.phx1.mozilla.com
Stopping socorro-crashmover: [  OK  ]
Starting socorro-crashmover: [  OK  ]
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://jenkins.mozilla.org/job/socorro-release/12/artifact/socorro.tar.gz
Checksum: 455f390eda50642efba5e1860f49540b
Backed up original to /data/socorro.26-09-2011_22_39_15
Upgrading socorro on sp-mware01.phx1.mozilla.com
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://jenkins.mozilla.org/job/socorro-release/12/artifact/socorro.tar.gz
Checksum: 455f390eda50642efba5e1860f49540b
Backed up original to /data/socorro.26-09-2011_22_39_34
Upgrading socorro on sp-mware02.phx1.mozilla.com
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://jenkins.mozilla.org/job/socorro-release/12/artifact/socorro.tar.gz
Checksum: 455f390eda50642efba5e1860f49540b
Backed up original to /data/socorro.26-09-2011_22_39_40
Upgrading socorro on sp-mware03.phx1.mozilla.com
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://jenkins.mozilla.org/job/socorro-release/12/artifact/socorro.tar.gz
Checksum: 455f390eda50642efba5e1860f49540b
Backed up original to /data/socorro.26-09-2011_22_39_46
Upgrading socorro on sp-processor02.phx1.mozilla.com
Stopping socorro-processor: [  OK  ]
Starting socorro-processor: [  OK  ]
Socorro build installed successfully!
Downloaded from https://jenkins.mozilla.org/job/socorro-release/12/artifact/socorro.tar.gz
Checksum: 455f390eda50642efba5e1860f49540b
Backed up original to /data/socorro.26-09-2011_22_39_53
Upgrading socorro on sp-processor03.phx1.mozilla.com
Stopping socorro-processor: [  OK  ]
Starting socorro-processor: [  OK  ]
Socorro build installed successfully!
Downloaded from https://jenkins.mozilla.org/job/socorro-release/12/artifact/socorro.tar.gz
Checksum: 455f390eda50642efba5e1860f49540b
Backed up original to /data/socorro.26-09-2011_22_40_01
Upgrading socorro on sp-processor04.phx1.mozilla.com
Stopping socorro-processor: [  OK  ]
Starting socorro-processor: [  OK  ]
Socorro build installed successfully!
Downloaded from https://jenkins.mozilla.org/job/socorro-release/12/artifact/socorro.tar.gz
Checksum: 455f390eda50642efba5e1860f49540b
Backed up original to /data/socorro.26-09-2011_22_40_09
Upgrading socorro on sp-processor05.phx1.mozilla.com
Stopping socorro-processor: [  OK  ]
Starting socorro-processor: [  OK  ]
Socorro build installed successfully!
Downloaded from https://jenkins.mozilla.org/job/socorro-release/12/artifact/socorro.tar.gz
Checksum: 455f390eda50642efba5e1860f49540b
Backed up original to /data/socorro.26-09-2011_22_40_21
Upgrading socorro on sp-processor06.phx1.mozilla.com
Stopping socorro-processor: [  OK  ]
Starting socorro-processor: [  OK  ]
Socorro build installed successfully!
Downloaded from https://jenkins.mozilla.org/job/socorro-release/12/artifact/socorro.tar.gz
Checksum: 455f390eda50642efba5e1860f49540b
Backed up original to /data/socorro.26-09-2011_22_40_30
Upgrading socorro on sp-processor07.phx1.mozilla.com
Stopping socorro-processor: [  OK  ]
Starting socorro-processor: [  OK  ]
Socorro build installed successfully!
Downloaded from https://jenkins.mozilla.org/job/socorro-release/12/artifact/socorro.tar.gz
Checksum: 455f390eda50642efba5e1860f49540b
Backed up original to /data/socorro.26-09-2011_22_40_38
Upgrading socorro on sp-processor08.phx1.mozilla.com
Stopping socorro-processor: [  OK  ]
Starting socorro-processor: [  OK  ]
Socorro build installed successfully!
Downloaded from https://jenkins.mozilla.org/job/socorro-release/12/artifact/socorro.tar.gz
Checksum: 455f390eda50642efba5e1860f49540b
Backed up original to /data/socorro.26-09-2011_22_40_50
Upgrading socorro on sp-processor09.phx1.mozilla.com
Stopping socorro-processor: [  OK  ]
Starting socorro-processor: [  OK  ]
Socorro build installed successfully!
Downloaded from https://jenkins.mozilla.org/job/socorro-release/12/artifact/socorro.tar.gz
Checksum: 455f390eda50642efba5e1860f49540b
Backed up original to /data/socorro.26-09-2011_22_41_01
Upgrading socorro on sp-processor10.phx1.mozilla.com
Stopping socorro-processor: [  OK  ]
Starting socorro-processor: [  OK  ]
Socorro build installed successfully!
Downloaded from https://jenkins.mozilla.org/job/socorro-release/12/artifact/socorro.tar.gz
Checksum: 455f390eda50642efba5e1860f49540b
Backed up original to /data/socorro.26-09-2011_22_41_10
Upgrading socorro on sp-web01.phx1.mozilla.com
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://jenkins.mozilla.org/job/socorro-release/12/artifact/socorro.tar.gz
Checksum: 455f390eda50642efba5e1860f49540b
Backed up original to /data/socorro.26-09-2011_22_41_21
Upgrading socorro on sp-web02.phx1.mozilla.com
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://jenkins.mozilla.org/job/socorro-release/12/artifact/socorro.tar.gz
Checksum: 455f390eda50642efba5e1860f49540b
Backed up original to /data/socorro.26-09-2011_22_41_28
Upgrading socorro on sp-web03.phx1.mozilla.com
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://jenkins.mozilla.org/job/socorro-release/12/artifact/socorro.tar.gz
Checksum: 455f390eda50642efba5e1860f49540b
Backed up original to /data/socorro.26-09-2011_22_41_35
Upgrading socorro on sp-web04.phx1.mozilla.com
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://jenkins.mozilla.org/job/socorro-release/12/artifact/socorro.tar.gz
Checksum: 455f390eda50642efba5e1860f49540b
Backed up original to /data/socorro.26-09-2011_22_41_46
Upgrading socorro on sp-web05.phx1.mozilla.com
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://jenkins.mozilla.org/job/socorro-release/12/artifact/socorro.tar.gz
Checksum: 455f390eda50642efba5e1860f49540b
Backed up original to /data/socorro.26-09-2011_22_41_55
Upgrading socorro on sp-admin01.phx1.mozilla.com
Stopping socorro-monitor: [  OK  ]
Starting socorro-monitor: [  OK  ]
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://jenkins.mozilla.org/job/socorro-release/12/artifact/socorro.tar.gz
Checksum: 455f390eda50642efba5e1860f49540b
Backed up original to /data/socorro.26-09-2011_22_42_16
Upgrading socorro on tp-socorro01-master01.phx1.mozilla.com
Socorro build installed successfully!
Downloaded from https://jenkins.mozilla.org/job/socorro-release/12/artifact/socorro.tar.gz
Checksum: 455f390eda50642efba5e1860f49540b
Backed up original to /data/socorro.26-09-2011_22_42_34
Upgrading socorro on tp-socorro01-master02.phx1.mozilla.com
Socorro build installed successfully!
Downloaded from https://jenkins.mozilla.org/job/socorro-release/12/artifact/socorro.tar.gz
Checksum: 455f390eda50642efba5e1860f49540b
Backed up original to /data/socorro.26-09-2011_22_42_40
Status: NEW → RESOLVED
Closed: 13 years ago
Resolution: --- → FIXED
Also kicked memcached :

[root@sp-admin01.phx1 ~]# issue-multi-command web /etc/init.d/memcached restart
[sp-web01.phx1.mozilla.com] running: /etc/init.d/memcached restart
[sp-web02.phx1.mozilla.com] running: /etc/init.d/memcached restart
[sp-web03.phx1.mozilla.com] running: /etc/init.d/memcached restart
[sp-web04.phx1.mozilla.com] running: /etc/init.d/memcached restart
[sp-web05.phx1.mozilla.com] running: /etc/init.d/memcached restart
[sp-web02.phx1.mozilla.com] finished: /etc/init.d/memcached restart (0.372s)
[sp-web02.phx1.mozilla.com] out: Stopping memcached: [  OK  ]
[sp-web02.phx1.mozilla.com] out: Starting memcached: [  OK  ]
[sp-web03.phx1.mozilla.com] finished: /etc/init.d/memcached restart (0.373s)
[sp-web03.phx1.mozilla.com] out: Stopping memcached: [  OK  ]
[sp-web03.phx1.mozilla.com] out: Starting memcached: [  OK  ]
[sp-web01.phx1.mozilla.com] finished: /etc/init.d/memcached restart (0.391s)
[sp-web01.phx1.mozilla.com] out: Stopping memcached: [  OK  ]
[sp-web01.phx1.mozilla.com] out: Starting memcached: [  OK  ]
[sp-web05.phx1.mozilla.com] finished: /etc/init.d/memcached restart (1.381s)
[sp-web05.phx1.mozilla.com] out: Stopping memcached: [  OK  ]
[sp-web05.phx1.mozilla.com] out: Starting memcached: [  OK  ]
[sp-web04.phx1.mozilla.com] finished: /etc/init.d/memcached restart (1.382s)
[sp-web04.phx1.mozilla.com] out: Stopping memcached: [  OK  ]
[sp-web04.phx1.mozilla.com] out: Starting memcached: [  OK  ]
Verified FIXED; found and filed bug 689448 while testing this deploy.
Status: RESOLVED → VERIFIED
Component: Server Operations: Web Operations → WebOps: Other
Product: mozilla.org → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.