Closed Bug 853163 Opened 12 years ago Closed 12 years ago

push Socorro 39 release

Categories

(Infrastructure & Operations Graveyard :: WebOps: Other, task, P3)

Product:
Infrastructure & Operations Graveyard
Component:
WebOps: Other
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: lonnen, Assigned: bburton)

Details

(Whiteboard: [push interrupt])

Build: https://ci.mozilla.org/job/socorro-release/351/artifact/socorro.tar.gz This release contains code for a new collector. It is preffed off in the config by default. Following the release, we want to disable puppet on one collector and pref on the new collector. Then we'll observe it overnight. If it runs without error for a while, we'll enable puppet and pref on the new collector in the global config.
Assignee: server-ops-webops → bburton
Status: NEW → ASSIGNED
Whiteboard: [push interrupt]
Push complete [root@sp-admin01.phx1 ~]# /root/bin/update-socorro-all.sh https://ci.mozilla.org/job/socorro-release/351/artifact/socorro.tar.gz Upgrading socorro to https://ci.mozilla.org/job/socorro-release/351/artifact/socorro.tar.gz Upgrading socorro on sp-collector01.phx1.mozilla.com Stopping socorro-crashmover: [ OK ] Starting socorro-crashmover: [ OK ] Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/351/artifact/socorro.tar.gz Checksum: cd4fadf4c6adc1cddbada3f8f5b20e40 Backed up original to /data/socorro.20-03-2013_14_28_46 Upgrading socorro on sp-collector02.phx1.mozilla.com Stopping socorro-crashmover: [ OK ] Starting socorro-crashmover: [ OK ] Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/351/artifact/socorro.tar.gz Checksum: cd4fadf4c6adc1cddbada3f8f5b20e40 Backed up original to /data/socorro.20-03-2013_21_28_50 Upgrading socorro on sp-collector03.phx1.mozilla.com reverse mapping checking getaddrinfo for socorro-collector3.phx1.mozilla.com [10.8.81.24] failed - POSSIBLE BREAK-IN ATTEMPT! Stopping socorro-crashmover: [ OK ] Starting socorro-crashmover: [ OK ] Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/351/artifact/socorro.tar.gz Checksum: cd4fadf4c6adc1cddbada3f8f5b20e40 Backed up original to /data/socorro.20-03-2013_21_28_54 Upgrading socorro on sp-collector04.phx1.mozilla.com reverse mapping checking getaddrinfo for socorro-collector4.phx1.mozilla.com [10.8.81.25] failed - POSSIBLE BREAK-IN ATTEMPT! Stopping socorro-crashmover: [ OK ] Starting socorro-crashmover: [ OK ] Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/351/artifact/socorro.tar.gz Checksum: cd4fadf4c6adc1cddbada3f8f5b20e40 Backed up original to /data/socorro.20-03-2013_21_28_58 Upgrading socorro on sp-collector05.phx1.mozilla.com reverse mapping checking getaddrinfo for socorro-collector5.phx1.mozilla.com [10.8.81.26] failed - POSSIBLE BREAK-IN ATTEMPT! Stopping socorro-crashmover: [ OK ] Starting socorro-crashmover: [ OK ] Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/351/artifact/socorro.tar.gz Checksum: cd4fadf4c6adc1cddbada3f8f5b20e40 Backed up original to /data/socorro.20-03-2013_21_29_02 Upgrading socorro on sp-collector06.phx1.mozilla.com reverse mapping checking getaddrinfo for socorro-collector6.phx1.mozilla.com [10.8.81.27] failed - POSSIBLE BREAK-IN ATTEMPT! Stopping socorro-crashmover: [ OK ] Starting socorro-crashmover: [ OK ] Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/351/artifact/socorro.tar.gz Checksum: cd4fadf4c6adc1cddbada3f8f5b20e40 Backed up original to /data/socorro.20-03-2013_21_29_06 Upgrading socorro on sp-mware01.phx1.mozilla.com Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/351/artifact/socorro.tar.gz Checksum: cd4fadf4c6adc1cddbada3f8f5b20e40 Backed up original to /data/socorro.20-03-2013_14_29_10 Upgrading socorro on sp-mware02.phx1.mozilla.com Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/351/artifact/socorro.tar.gz Checksum: cd4fadf4c6adc1cddbada3f8f5b20e40 Backed up original to /data/socorro.20-03-2013_14_29_12 Upgrading socorro on sp-mware03.phx1.mozilla.com Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/351/artifact/socorro.tar.gz Checksum: cd4fadf4c6adc1cddbada3f8f5b20e40 Backed up original to /data/socorro.20-03-2013_14_29_15 Upgrading socorro on sp-processor01.phx1.mozilla.com Stopping socorro-processor: [ OK ] Starting socorro-processor: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/351/artifact/socorro.tar.gz Checksum: cd4fadf4c6adc1cddbada3f8f5b20e40 Backed up original to /data/socorro.20-03-2013_14_29_16 Upgrading socorro on sp-processor02.phx1.mozilla.com Stopping socorro-processor: [ OK ] Starting socorro-processor: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/351/artifact/socorro.tar.gz Checksum: cd4fadf4c6adc1cddbada3f8f5b20e40 Backed up original to /data/socorro.20-03-2013_14_29_21 Upgrading socorro on sp-processor03.phx1.mozilla.com Stopping socorro-processor: [ OK ] Starting socorro-processor: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/351/artifact/socorro.tar.gz Checksum: cd4fadf4c6adc1cddbada3f8f5b20e40 Backed up original to /data/socorro.20-03-2013_14_29_26 Upgrading socorro on sp-processor04.phx1.mozilla.com Stopping socorro-processor: [ OK ] Starting socorro-processor: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/351/artifact/socorro.tar.gz Checksum: cd4fadf4c6adc1cddbada3f8f5b20e40 Backed up original to /data/socorro.20-03-2013_14_29_30 Upgrading socorro on sp-processor05.phx1.mozilla.com Stopping socorro-processor: [ OK ] Starting socorro-processor: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/351/artifact/socorro.tar.gz Checksum: cd4fadf4c6adc1cddbada3f8f5b20e40 Backed up original to /data/socorro.20-03-2013_14_29_34 Upgrading socorro on sp-processor06.phx1.mozilla.com Stopping socorro-processor: [ OK ] Starting socorro-processor: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/351/artifact/socorro.tar.gz Checksum: cd4fadf4c6adc1cddbada3f8f5b20e40 Backed up original to /data/socorro.20-03-2013_14_29_40 Upgrading socorro on sp-processor07.phx1.mozilla.com Stopping socorro-processor: [ OK ] Starting socorro-processor: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/351/artifact/socorro.tar.gz Checksum: cd4fadf4c6adc1cddbada3f8f5b20e40 Backed up original to /data/socorro.20-03-2013_14_29_45 Upgrading socorro on sp-processor08.phx1.mozilla.com Stopping socorro-processor: [ OK ] Starting socorro-processor: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/351/artifact/socorro.tar.gz Checksum: cd4fadf4c6adc1cddbada3f8f5b20e40 Backed up original to /data/socorro.20-03-2013_14_29_49 Upgrading socorro on sp-processor09.phx1.mozilla.com Stopping socorro-processor: [ OK ] Starting socorro-processor: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/351/artifact/socorro.tar.gz Checksum: cd4fadf4c6adc1cddbada3f8f5b20e40 Backed up original to /data/socorro.20-03-2013_14_29_54 Upgrading socorro on sp-processor10.phx1.mozilla.com Stopping socorro-processor: [ OK ] Starting socorro-processor: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/351/artifact/socorro.tar.gz Checksum: cd4fadf4c6adc1cddbada3f8f5b20e40 Backed up original to /data/socorro.20-03-2013_14_29_59 Upgrading socorro on sp-web01.phx1.mozilla.com Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/351/artifact/socorro.tar.gz Checksum: cd4fadf4c6adc1cddbada3f8f5b20e40 Backed up original to /data/socorro.20-03-2013_14_30_03 Upgrading socorro on sp-web02.phx1.mozilla.com Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/351/artifact/socorro.tar.gz Checksum: cd4fadf4c6adc1cddbada3f8f5b20e40 Backed up original to /data/socorro.20-03-2013_14_30_07 Upgrading socorro on sp-web03.phx1.mozilla.com Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/351/artifact/socorro.tar.gz Checksum: cd4fadf4c6adc1cddbada3f8f5b20e40 Backed up original to /data/socorro.20-03-2013_14_30_09 Upgrading socorro on sp-web04.phx1.mozilla.com Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/351/artifact/socorro.tar.gz Checksum: cd4fadf4c6adc1cddbada3f8f5b20e40 Backed up original to /data/socorro.20-03-2013_14_30_12 Upgrading socorro on sp-web05.phx1.mozilla.com Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/351/artifact/socorro.tar.gz Checksum: cd4fadf4c6adc1cddbada3f8f5b20e40 Backed up original to /data/socorro.20-03-2013_14_30_16 Upgrading socorro on sp-admin01.phx1.mozilla.com Stopping socorro-monitor: [ OK ] Starting socorro-monitor: [ OK ] Stopping httpd: [ OK ] Starting httpd: [ OK ] Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/351/artifact/socorro.tar.gz Checksum: cd4fadf4c6adc1cddbada3f8f5b20e40 Backed up original to /data/socorro.20-03-2013_14_30_19 Upgrading socorro on tp-socorro01-master01.phx1.mozilla.com Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/351/artifact/socorro.tar.gz Checksum: cd4fadf4c6adc1cddbada3f8f5b20e40 Backed up original to /data/socorro.20-03-2013_14_30_48 Upgrading socorro on tp-socorro01-master02.phx1.mozilla.com Socorro build installed successfully! Downloaded from https://ci.mozilla.org/job/socorro-release/351/artifact/socorro.tar.gz Checksum: cd4fadf4c6adc1cddbada3f8f5b20e40 Backed up original to /data/socorro.20-03-2013_14_30_55 [sp-web01.phx1.mozilla.com] running: service memcached restart [sp-web02.phx1.mozilla.com] running: service memcached restart [sp-web03.phx1.mozilla.com] running: service memcached restart [sp-web04.phx1.mozilla.com] running: service memcached restart [sp-web05.phx1.mozilla.com] running: service memcached restart [sp-web01.phx1.mozilla.com] finished: service memcached restart (0.665s) [sp-web01.phx1.mozilla.com] out: Stopping memcached: [ OK ] [sp-web01.phx1.mozilla.com] out: Starting memcached: [ OK ] [sp-web02.phx1.mozilla.com] finished: service memcached restart (1.359s) [sp-web02.phx1.mozilla.com] out: Stopping memcached: [ OK ] [sp-web02.phx1.mozilla.com] out: Starting memcached: [ OK ] [sp-web03.phx1.mozilla.com] finished: service memcached restart (1.418s) [sp-web03.phx1.mozilla.com] out: Stopping memcached: [ OK ] [sp-web03.phx1.mozilla.com] out: Starting memcached: [ OK ] [sp-web05.phx1.mozilla.com] finished: service memcached restart (1.773s) [sp-web05.phx1.mozilla.com] out: Stopping memcached: [ OK ] [sp-web05.phx1.mozilla.com] out: Starting memcached: [ OK ] [sp-web04.phx1.mozilla.com] finished: service memcached restart (2.082s) [sp-web04.phx1.mozilla.com] out: Stopping memcached: [ OK ] [sp-web04.phx1.mozilla.com] out: Starting memcached: [ OK ]
Status: ASSIGNED → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Working on enabling the new code on sp-collector05 [root@socorro-collector5.webapp.phx1 ~]# puppetctl disable --time 'now + 24 hours' -f -m "testing new code on this node only, see https://bugzilla.mozilla.org/show_bug.cgi?id=853163" Puppet has been disabled by bburton at 2013-03-20 21:36 until 2013-03-21 21:36 with the following message: testing new code on this node only, see https://bugzilla.mozilla.org/show_bug.cgi?id=853163
Re-upped the puppet disable [root@socorro-collector5.webapp.phx1 init.d]# puppetctl disable --time 'now + 24 hours' -f -m "testing new code on this node only, see https://bugzilla.mozilla.org/show_bug.cgi?id=853163" Puppet has been disabled by bburton at 2013-03-20 23:12 until 2013-03-21 23:12 with the following message: testing new code on this node only, see https://bugzilla.mozilla.org/show_bug.cgi?id=853163 Downtimed in Nagios 19:13:45 nagios-phx1 | solarce: Downtime for socorro-collector5.webapp.phx1.mozilla.com scheduled for 1 day, 0:00:00 19:13:46 nagios-scl3 | Wed 16:13:07 PDT [568] nsmserver1.private.scl3.mozilla.com:Disk - All is OK: DISK OK (http://m.allizom.org/Disk+-+All)
sp-collector5 has been re-enabled in zeus and is running the new crashmover as of 4:15pm pdt
If the #breakpad folks need collector5 disabled, disable it in https://pp-zlb08.phx.mozilla.net:9090/apps/zxtm/?name=crash-reports%3A80&section=Pools%3AEdit#anchor https://pp-zlb08.phx.mozilla.net:9090/apps/zxtm/?name=crash-reports%3A81&section=Pools%3AEdit#anchor By selecting the dropdown next to 10.8.81.26:80, selecting Disable, and clicking Update below that, do it for both pools
Component: Server Operations: Web Operations → WebOps: Other
Product: mozilla.org → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.