Closed Bug 717055 Opened 14 years ago Closed 14 years ago

Push Socorro 2.4 release

Categories

(mozilla.org Graveyard :: Server Operations, task)

Product:
mozilla.org Graveyard
Component:
Server Operations
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
VERIFIED FIXED

People

(Reporter: rhelmer, Assigned: jason)

References

(
URL
)

Details

(Whiteboard: ETA: Friday, January 13 2011 8AM Pacific)

We need to push Socorro Friday (the 13th) morning (starting at 8 AM). This release is a bit more involved than most, since we need a 2 hour database downtime and will need to have crontab and monitor shut down during this time, and push the code after the DB is back up. Can someone from IT be on-deck to help us do this from 8 AM to around 10 AM this Friday? We're working through the process in https://etherpad.mozilla.org/socorro-2-4-release-process and still testing the code so no final URL yet, but I wanted to file this now to give heads-up.
Assignee: server-ops → jthomas
Sure, I will reach out to you at 8AM PST.
Assignee: jthomas → server-ops
(In reply to Jason Thomas [:jason] from comment #1) > Sure, I will reach out to you at 8AM PST. Oops, mid-aired with you sorry. Thanks!
Assignee: server-ops → jthomas
Is it possible to send traffic bound for https://crash-stats.mozilla.com to a static downtime page for the duration (maybe using zeus)? Since the DB is down, Socorro will be entirely inaccessible for the downtime.
(In reply to Robert Helmer [:rhelmer] from comment #3) > Is it possible to send traffic bound for https://crash-stats.mozilla.com to > a static downtime page for the duration (maybe using zeus)? > > Since the DB is down, Socorro will be entirely inaccessible for the downtime. stephend filed bug 717079 for this, also adding new RPM dep in bug 716247 as blocking this
Depends on: 716247, 717079
Pushed code except to processors. [root@sp-admin01.phx1 ~]# update-socorro-not-processors.sh https://ci.mozilla.org/job/socorro-release/72/artifact/socorro.tar.gz Upgrading socorro to https://ci.mozilla.org/job/socorro-release/72/artifact/socorro.tar.gz Upgrading socorro on sp-collector01.phx1.mozilla.com Stopping socorro-crashmover: [ OK ] Starting socorro-crashmover: [ OK ] Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/72/artifact/socorro.tar.gz Checksum: aec430615b2df8814f85fdc168c88769 Backed up original to /data/socorro.13-01-2012_09_14_42 Upgrading socorro on sp-collector02.phx1.mozilla.com Stopping socorro-crashmover: [ OK ] Starting socorro-crashmover: [ OK ] Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/72/artifact/socorro.tar.gz Checksum: aec430615b2df8814f85fdc168c88769 Backed up original to /data/socorro.13-01-2012_09_14_51 Upgrading socorro on sp-collector03.phx1.mozilla.com Stopping socorro-crashmover: [ OK ] Starting socorro-crashmover: [ OK ] Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/72/artifact/socorro.tar.gz Checksum: aec430615b2df8814f85fdc168c88769 Backed up original to /data/socorro.13-01-2012_09_15_10 Upgrading socorro on sp-collector04.phx1.mozilla.com Stopping socorro-crashmover: [ OK ] Starting socorro-crashmover: [ OK ] Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/72/artifact/socorro.tar.gz Checksum: aec430615b2df8814f85fdc168c88769 Backed up original to /data/socorro.13-01-2012_09_15_18 Upgrading socorro on sp-collector05.phx1.mozilla.com Stopping socorro-crashmover: [ OK ] Starting socorro-crashmover: [ OK ] Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/72/artifact/socorro.tar.gz Checksum: aec430615b2df8814f85fdc168c88769 Backed up original to /data/socorro.13-01-2012_09_15_24 Upgrading socorro on sp-collector06.phx1.mozilla.com Stopping socorro-crashmover: [ OK ] Starting socorro-crashmover: [ OK ] Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/72/artifact/socorro.tar.gz Checksum: aec430615b2df8814f85fdc168c88769 Backed up original to /data/socorro.13-01-2012_09_15_44 Upgrading socorro on sp-mware01.phx1.mozilla.com Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/72/artifact/socorro.tar.gz Checksum: aec430615b2df8814f85fdc168c88769 Backed up original to /data/socorro.13-01-2012_09_16_06 Upgrading socorro on sp-mware02.phx1.mozilla.com Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/72/artifact/socorro.tar.gz Checksum: aec430615b2df8814f85fdc168c88769 Backed up original to /data/socorro.13-01-2012_09_16_11 Upgrading socorro on sp-mware03.phx1.mozilla.com Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/72/artifact/socorro.tar.gz Checksum: aec430615b2df8814f85fdc168c88769 Backed up original to /data/socorro.13-01-2012_09_16_14 Upgrading socorro on sp-web01.phx1.mozilla.com Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/72/artifact/socorro.tar.gz Checksum: aec430615b2df8814f85fdc168c88769 Backed up original to /data/socorro.13-01-2012_09_16_21 Upgrading socorro on sp-web02.phx1.mozilla.com Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/72/artifact/socorro.tar.gz Checksum: aec430615b2df8814f85fdc168c88769 Backed up original to /data/socorro.13-01-2012_09_16_26 Upgrading socorro on sp-web03.phx1.mozilla.com Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/72/artifact/socorro.tar.gz Checksum: aec430615b2df8814f85fdc168c88769 Backed up original to /data/socorro.13-01-2012_09_16_31 Upgrading socorro on sp-web04.phx1.mozilla.com Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/72/artifact/socorro.tar.gz Checksum: aec430615b2df8814f85fdc168c88769 Backed up original to /data/socorro.13-01-2012_09_16_38 Upgrading socorro on sp-web05.phx1.mozilla.com Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/72/artifact/socorro.tar.gz Checksum: aec430615b2df8814f85fdc168c88769 Backed up original to /data/socorro.13-01-2012_09_16_48 Upgrading socorro on sp-admin01.phx1.mozilla.com Stopping socorro-monitor: [FAILED] Starting socorro-monitor: [ OK ] Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/72/artifact/socorro.tar.gz Checksum: aec430615b2df8814f85fdc168c88769 Backed up original to /data/socorro.13-01-2012_09_16_52 Upgrading socorro on tp-socorro01-master01.phx1.mozilla.com Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/72/artifact/socorro.tar.gz Checksum: aec430615b2df8814f85fdc168c88769 Backed up original to /data/socorro.13-01-2012_09_17_08 Upgrading socorro on tp-socorro01-master02.phx1.mozilla.com Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/72/artifact/socorro.tar.gz Checksum: aec430615b2df8814f85fdc168c88769 Backed up original to /data/socorro.13-01-2012_09_17_11 [sp-web01.phx1.mozilla.com] running: service memcached restart [sp-web02.phx1.mozilla.com] running: service memcached restart [sp-web03.phx1.mozilla.com] running: service memcached restart [sp-web04.phx1.mozilla.com] running: service memcached restart [sp-web05.phx1.mozilla.com] running: service memcached restart [sp-web05.phx1.mozilla.com] finished: service memcached restart (1.000s) [sp-web05.phx1.mozilla.com] out: Stopping memcached: [ OK ] [sp-web05.phx1.mozilla.com] out: Starting memcached: [ OK ] [sp-web04.phx1.mozilla.com] finished: service memcached restart (1.656s) [sp-web04.phx1.mozilla.com] out: Stopping memcached: [ OK ] [sp-web04.phx1.mozilla.com] out: Starting memcached: [ OK ] [sp-web03.phx1.mozilla.com] finished: service memcached restart (2.058s) [sp-web03.phx1.mozilla.com] out: Stopping memcached: [ OK ] [sp-web03.phx1.mozilla.com] out: Starting memcached: [ OK ] [sp-web02.phx1.mozilla.com] finished: service memcached restart (2.058s) [sp-web02.phx1.mozilla.com] out: Stopping memcached: [ OK ] [sp-web02.phx1.mozilla.com] out: Starting memcached: [ OK ] [sp-web01.phx1.mozilla.com] finished: service memcached restart (2.059s) [sp-web01.phx1.mozilla.com] out: Stopping memcached: [ OK ] [sp-web01.phx1.mozilla.com] out: Starting memcached: [ OK ]
Pushed https://ci.mozilla.org/job/socorro-release/75/artifact/socorro.tar.gz to prod [root@sp-admin01.phx1 ~]# update-socorro-not-processors.sh https://ci.mozilla.org/job/socorro-release/75/artifact/socorro.tar.gz Upgrading socorro to https://ci.mozilla.org/job/socorro-release/75/artifact/socorro.tar.gz Upgrading socorro on sp-collector01.phx1.mozilla.com Stopping socorro-crashmover: [ OK ] Starting socorro-crashmover: [ OK ] Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/75/artifact/socorro.tar.gz Checksum: 6e53a7647b1926aa781395a4fc2144db Backed up original to /data/socorro.13-01-2012_16_19_53 Upgrading socorro on sp-collector02.phx1.mozilla.com Stopping socorro-crashmover: [ OK ] Starting socorro-crashmover: [ OK ] Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/75/artifact/socorro.tar.gz Checksum: 6e53a7647b1926aa781395a4fc2144db Backed up original to /data/socorro.13-01-2012_16_19_58 Upgrading socorro on sp-collector03.phx1.mozilla.com Stopping socorro-crashmover: [ OK ] Starting socorro-crashmover: [ OK ] Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/75/artifact/socorro.tar.gz Checksum: 6e53a7647b1926aa781395a4fc2144db Backed up original to /data/socorro.13-01-2012_16_20_07 Upgrading socorro on sp-collector04.phx1.mozilla.com Stopping socorro-crashmover: [ OK ] Starting socorro-crashmover: [ OK ] Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/75/artifact/socorro.tar.gz Checksum: 6e53a7647b1926aa781395a4fc2144db Backed up original to /data/socorro.13-01-2012_16_20_12 Upgrading socorro on sp-collector05.phx1.mozilla.com Stopping socorro-crashmover: [ OK ] Starting socorro-crashmover: [ OK ] Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/75/artifact/socorro.tar.gz Checksum: 6e53a7647b1926aa781395a4fc2144db Backed up original to /data/socorro.13-01-2012_16_20_18 Upgrading socorro on sp-collector06.phx1.mozilla.com Stopping socorro-crashmover: [ OK ] Starting socorro-crashmover: [ OK ] Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/75/artifact/socorro.tar.gz Checksum: 6e53a7647b1926aa781395a4fc2144db Backed up original to /data/socorro.13-01-2012_16_20_34 Upgrading socorro on sp-mware01.phx1.mozilla.com Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/75/artifact/socorro.tar.gz Checksum: 6e53a7647b1926aa781395a4fc2144db Backed up original to /data/socorro.13-01-2012_16_20_49 Upgrading socorro on sp-mware02.phx1.mozilla.com Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/75/artifact/socorro.tar.gz Checksum: 6e53a7647b1926aa781395a4fc2144db Backed up original to /data/socorro.13-01-2012_16_20_51 Upgrading socorro on sp-mware03.phx1.mozilla.com Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/75/artifact/socorro.tar.gz Checksum: 6e53a7647b1926aa781395a4fc2144db Backed up original to /data/socorro.13-01-2012_16_20_54 Upgrading socorro on sp-web01.phx1.mozilla.com Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/75/artifact/socorro.tar.gz Checksum: 6e53a7647b1926aa781395a4fc2144db Backed up original to /data/socorro.13-01-2012_16_20_57 Upgrading socorro on sp-web02.phx1.mozilla.com Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/75/artifact/socorro.tar.gz Checksum: 6e53a7647b1926aa781395a4fc2144db Backed up original to /data/socorro.13-01-2012_16_21_00 Upgrading socorro on sp-web03.phx1.mozilla.com Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/75/artifact/socorro.tar.gz Checksum: 6e53a7647b1926aa781395a4fc2144db Backed up original to /data/socorro.13-01-2012_16_21_04 Upgrading socorro on sp-web04.phx1.mozilla.com Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/75/artifact/socorro.tar.gz Checksum: 6e53a7647b1926aa781395a4fc2144db Backed up original to /data/socorro.13-01-2012_16_21_17 Upgrading socorro on sp-web05.phx1.mozilla.com Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/75/artifact/socorro.tar.gz Checksum: 6e53a7647b1926aa781395a4fc2144db Backed up original to /data/socorro.13-01-2012_16_21_20 Upgrading socorro on sp-admin01.phx1.mozilla.com Stopping socorro-monitor: [FAILED] Starting socorro-monitor: [ OK ] Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/75/artifact/socorro.tar.gz Checksum: 6e53a7647b1926aa781395a4fc2144db Backed up original to /data/socorro.13-01-2012_16_21_35 Upgrading socorro on tp-socorro01-master01.phx1.mozilla.com Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/75/artifact/socorro.tar.gz Checksum: 6e53a7647b1926aa781395a4fc2144db Backed up original to /data/socorro.13-01-2012_16_21_37 Upgrading socorro on tp-socorro01-master02.phx1.mozilla.com Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/75/artifact/socorro.tar.gz Checksum: 6e53a7647b1926aa781395a4fc2144db Backed up original to /data/socorro.13-01-2012_16_21_39 [sp-web01.phx1.mozilla.com] running: service memcached restart [sp-web02.phx1.mozilla.com] running: service memcached restart [sp-web03.phx1.mozilla.com] running: service memcached restart [sp-web04.phx1.mozilla.com] running: service memcached restart [sp-web05.phx1.mozilla.com] running: service memcached restart [sp-web02.phx1.mozilla.com] finished: service memcached restart (0.926s) [sp-web02.phx1.mozilla.com] out: Stopping memcached: [ OK ] [sp-web02.phx1.mozilla.com] out: Starting memcached: [ OK ] [sp-web04.phx1.mozilla.com] finished: service memcached restart (1.019s) [sp-web04.phx1.mozilla.com] out: Stopping memcached: [ OK ] [sp-web04.phx1.mozilla.com] out: Starting memcached: [ OK ] [sp-web03.phx1.mozilla.com] finished: service memcached restart (1.021s) [sp-web03.phx1.mozilla.com] out: Stopping memcached: [ OK ] [sp-web03.phx1.mozilla.com] out: Starting memcached: [ OK ] [sp-web05.phx1.mozilla.com] finished: service memcached restart (1.022s) [sp-web05.phx1.mozilla.com] out: Stopping memcached: [ OK ] [sp-web05.phx1.mozilla.com] out: Starting memcached: [ OK ] [sp-web01.phx1.mozilla.com] finished: service memcached restart (1.113s) [sp-web01.phx1.mozilla.com] out: Stopping memcached: [ OK ] [sp-web01.phx1.mozilla.com] out: Starting memcached: [ OK ] [root@sp-admin01.phx1 bin]# update-socorro-processors.sh https://ci.mozilla.org/job/socorro-release/75/artifact/socorro.tar.gz Upgrading socorro to https://ci.mozilla.org/job/socorro-release/75/artifact/socorro.tar.gz Upgrading socorro on sp-processor01.phx1.mozilla.com Stopping socorro-processor: [FAILED] Starting socorro-processor: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/75/artifact/socorro.tar.gz Checksum: 6e53a7647b1926aa781395a4fc2144db Backed up original to /data/socorro.13-01-2012_17_23_38 Upgrading socorro on sp-processor02.phx1.mozilla.com Stopping socorro-processor: [FAILED] Starting socorro-processor: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/75/artifact/socorro.tar.gz Checksum: 6e53a7647b1926aa781395a4fc2144db Backed up original to /data/socorro.13-01-2012_17_23_41 Upgrading socorro on sp-processor03.phx1.mozilla.com Stopping socorro-processor: [FAILED] Starting socorro-processor: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/75/artifact/socorro.tar.gz Checksum: 6e53a7647b1926aa781395a4fc2144db Backed up original to /data/socorro.13-01-2012_17_23_44 Upgrading socorro on sp-processor04.phx1.mozilla.com Stopping socorro-processor: [FAILED] Starting socorro-processor: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/75/artifact/socorro.tar.gz Checksum: 6e53a7647b1926aa781395a4fc2144db Backed up original to /data/socorro.13-01-2012_17_23_48 Upgrading socorro on sp-processor05.phx1.mozilla.com Stopping socorro-processor: [FAILED] Starting socorro-processor: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/75/artifact/socorro.tar.gz Checksum: 6e53a7647b1926aa781395a4fc2144db Backed up original to /data/socorro.13-01-2012_17_23_51 Upgrading socorro on sp-processor06.phx1.mozilla.com Stopping socorro-processor: [FAILED] Starting socorro-processor: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/75/artifact/socorro.tar.gz Checksum: 6e53a7647b1926aa781395a4fc2144db Backed up original to /data/socorro.13-01-2012_17_23_54 Upgrading socorro on sp-processor07.phx1.mozilla.com Stopping socorro-processor: [FAILED] Starting socorro-processor: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/75/artifact/socorro.tar.gz Checksum: 6e53a7647b1926aa781395a4fc2144db Backed up original to /data/socorro.13-01-2012_17_23_57 Upgrading socorro on sp-processor08.phx1.mozilla.com Stopping socorro-processor: [FAILED] Starting socorro-processor: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/75/artifact/socorro.tar.gz Checksum: 6e53a7647b1926aa781395a4fc2144db Backed up original to /data/socorro.13-01-2012_17_24_00 Upgrading socorro on sp-processor09.phx1.mozilla.com Stopping socorro-processor: [FAILED] Starting socorro-processor: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/75/artifact/socorro.tar.gz Checksum: 6e53a7647b1926aa781395a4fc2144db Backed up original to /data/socorro.13-01-2012_17_24_03 Upgrading socorro on sp-processor10.phx1.mozilla.com Stopping socorro-processor: [FAILED] Starting socorro-processor: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/75/artifact/socorro.tar.gz Checksum: 6e53a7647b1926aa781395a4fc2144db Backed up original to /data/socorro.13-01-2012_17_24_06
Status: NEW → RESOLVED
Closed: 14 years ago
Resolution: --- → FIXED
QA verified ... the release looks great. UTC bliss is upon us. Manual bfts, server status, and ganglia all look great. Automation is going to need some love but is passing after the work done on bug 718218. Automation is currently green.
Status: RESOLVED → VERIFIED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.