Closed Bug 711103 Opened 13 years ago Closed 13 years ago

Push Socorro 2.3.4.1

Categories

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Product:
Infrastructure & Operations Graveyard
Component:
WebOps: Other
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
VERIFIED FIXED

People

(Reporter: laura, Assigned: jason)

Details

Build
https://ci.mozilla.org/job/socorro-release/58/artifact/socorro.tar.gz

Revision
c920975555e7f0ca5e6aca020037052340f7a36a

Awaiting QA signoff, mbrandt please comment in the bug.

This release fixes bug 710757.
Assignee: server-ops → jthomas
Pushed to prod.

[root@sp-admin01.phx1 ~]# update-socorro-all.sh https://ci.mozilla.org/job/socorro-release/58/artifact/socorro.tar.gz
Upgrading socorro to https://ci.mozilla.org/job/socorro-release/58/artifact/socorro.tar.gz
Upgrading socorro on sp-collector01.phx1.mozilla.com
Stopping socorro-crashmover: [  OK  ]
Starting socorro-crashmover: [  OK  ]
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/58/artifact/socorro.tar.gz
Checksum: c04ea6030793957744cb292c2a935813
Backed up original to /data/socorro.15-12-2011_09_26_21
Upgrading socorro on sp-collector02.phx1.mozilla.com
Stopping socorro-crashmover: [  OK  ]
Starting socorro-crashmover: [  OK  ]
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/58/artifact/socorro.tar.gz
Checksum: c04ea6030793957744cb292c2a935813
Backed up original to /data/socorro.15-12-2011_09_26_30
Upgrading socorro on sp-collector03.phx1.mozilla.com
Stopping socorro-crashmover: [  OK  ]
Starting socorro-crashmover: [  OK  ]
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/58/artifact/socorro.tar.gz
Checksum: c04ea6030793957744cb292c2a935813
Backed up original to /data/socorro.15-12-2011_09_26_35
Upgrading socorro on sp-collector04.phx1.mozilla.com
Stopping socorro-crashmover: [  OK  ]
Starting socorro-crashmover: [  OK  ]
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/58/artifact/socorro.tar.gz
Checksum: c04ea6030793957744cb292c2a935813
Backed up original to /data/socorro.15-12-2011_09_26_50
Upgrading socorro on sp-collector05.phx1.mozilla.com
Stopping socorro-crashmover: [  OK  ]
Starting socorro-crashmover: [  OK  ]
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/58/artifact/socorro.tar.gz
Checksum: c04ea6030793957744cb292c2a935813
Backed up original to /data/socorro.15-12-2011_09_27_09
Upgrading socorro on sp-collector06.phx1.mozilla.com
Stopping socorro-crashmover: [  OK  ]
Starting socorro-crashmover: [  OK  ]
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/58/artifact/socorro.tar.gz
Checksum: c04ea6030793957744cb292c2a935813
Backed up original to /data/socorro.15-12-2011_09_27_15
Upgrading socorro on sp-mware01.phx1.mozilla.com
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/58/artifact/socorro.tar.gz
Checksum: c04ea6030793957744cb292c2a935813
Backed up original to /data/socorro.15-12-2011_09_27_33
Upgrading socorro on sp-mware02.phx1.mozilla.com
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/58/artifact/socorro.tar.gz
Checksum: c04ea6030793957744cb292c2a935813
Backed up original to /data/socorro.15-12-2011_09_27_35
Upgrading socorro on sp-mware03.phx1.mozilla.com
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/58/artifact/socorro.tar.gz
Checksum: c04ea6030793957744cb292c2a935813
Backed up original to /data/socorro.15-12-2011_09_27_38
Upgrading socorro on sp-processor01.phx1.mozilla.com
Stopping socorro-processor: [  OK  ]
Starting socorro-processor: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/58/artifact/socorro.tar.gz
Checksum: c04ea6030793957744cb292c2a935813
Backed up original to /data/socorro.15-12-2011_09_27_40
Upgrading socorro on sp-processor02.phx1.mozilla.com
Stopping socorro-processor: [  OK  ]
Starting socorro-processor: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/58/artifact/socorro.tar.gz
Checksum: c04ea6030793957744cb292c2a935813
Backed up original to /data/socorro.15-12-2011_09_27_44
Upgrading socorro on sp-processor03.phx1.mozilla.com
Stopping socorro-processor: [  OK  ]
Starting socorro-processor: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/58/artifact/socorro.tar.gz
Checksum: c04ea6030793957744cb292c2a935813
Backed up original to /data/socorro.15-12-2011_09_27_49
Upgrading socorro on sp-processor04.phx1.mozilla.com
Stopping socorro-processor: [  OK  ]
Starting socorro-processor: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/58/artifact/socorro.tar.gz
Checksum: c04ea6030793957744cb292c2a935813
Backed up original to /data/socorro.15-12-2011_09_27_55
Upgrading socorro on sp-processor05.phx1.mozilla.com
Stopping socorro-processor: [  OK  ]
Starting socorro-processor: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/58/artifact/socorro.tar.gz
Checksum: c04ea6030793957744cb292c2a935813
Backed up original to /data/socorro.15-12-2011_09_28_02
Upgrading socorro on sp-processor06.phx1.mozilla.com
Stopping socorro-processor: [  OK  ]
Starting socorro-processor: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/58/artifact/socorro.tar.gz
Checksum: c04ea6030793957744cb292c2a935813
Backed up original to /data/socorro.15-12-2011_09_28_09
Upgrading socorro on sp-processor07.phx1.mozilla.com
Stopping socorro-processor: [  OK  ]
Starting socorro-processor: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/58/artifact/socorro.tar.gz
Checksum: c04ea6030793957744cb292c2a935813
Backed up original to /data/socorro.15-12-2011_09_28_14
Upgrading socorro on sp-processor08.phx1.mozilla.com
Stopping socorro-processor: [  OK  ]
Starting socorro-processor: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/58/artifact/socorro.tar.gz
Checksum: c04ea6030793957744cb292c2a935813
Backed up original to /data/socorro.15-12-2011_09_28_17
Upgrading socorro on sp-processor09.phx1.mozilla.com
Stopping socorro-processor: [  OK  ]
Starting socorro-processor: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/58/artifact/socorro.tar.gz
Checksum: c04ea6030793957744cb292c2a935813
Backed up original to /data/socorro.15-12-2011_09_28_24
Upgrading socorro on sp-processor10.phx1.mozilla.com
Stopping socorro-processor: [  OK  ]
Starting socorro-processor: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/58/artifact/socorro.tar.gz
Checksum: c04ea6030793957744cb292c2a935813
Backed up original to /data/socorro.15-12-2011_09_28_29
Upgrading socorro on sp-web01.phx1.mozilla.com
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/58/artifact/socorro.tar.gz
Checksum: c04ea6030793957744cb292c2a935813
Backed up original to /data/socorro.15-12-2011_09_28_36
Upgrading socorro on sp-web02.phx1.mozilla.com
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/58/artifact/socorro.tar.gz
Checksum: c04ea6030793957744cb292c2a935813
Backed up original to /data/socorro.15-12-2011_09_28_54
Upgrading socorro on sp-web03.phx1.mozilla.com
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/58/artifact/socorro.tar.gz
Checksum: c04ea6030793957744cb292c2a935813
Backed up original to /data/socorro.15-12-2011_09_29_01
Upgrading socorro on sp-web04.phx1.mozilla.com
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/58/artifact/socorro.tar.gz
Checksum: c04ea6030793957744cb292c2a935813
Backed up original to /data/socorro.15-12-2011_09_29_06
Upgrading socorro on sp-web05.phx1.mozilla.com
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Checksum: c04ea6030793957744cb292c2a935813
Backed up original to /data/socorro.15-12-2011_09_29_24
update-socorro-all.sh https://ci.mozilla.org/job/socorro-release/58/artifact/socorro.tar.gz
Upgrading socorro to https://ci.mozilla.org/job/socorro-release/58/artifact/socorro.tar.gz
Upgrading socorro on tp-socorro01-master01.phx1.mozilla.com
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/58/artifact/socorro.tar.gz
Checksum: c04ea6030793957744cb292c2a935813
Backed up original to /data/socorro.15-12-2011_09_56_28
Upgrading socorro on tp-socorro01-master02.phx1.mozilla.com
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/58/artifact/socorro.tar.gz
Checksum: c04ea6030793957744cb292c2a935813
Backed up original to /data/socorro.15-12-2011_09_56_30
[root@sp-admin01.phx1 ~]# /data/bin/update-socorro.sh
Stopping socorro-monitor:                                  [FAILED]
Starting socorro-monitor:                                  [  OK  ]
Stopping httpd:                                            [  OK  ]
Starting httpd:                                            [  OK  ]
Socorro build installed successfully!
Downloaded from 
Checksum: c04ea6030793957744cb292c2a935813
Backed up original to /data/socorro.15-12-2011_09_59_47
[sp-web01.phx1.mozilla.com] running: service memcached restart
[sp-web02.phx1.mozilla.com] running: service memcached restart
[sp-web03.phx1.mozilla.com] running: service memcached restart
[sp-web04.phx1.mozilla.com] running: service memcached restart
[sp-web05.phx1.mozilla.com] running: service memcached restart
[sp-web05.phx1.mozilla.com] finished: service memcached restart (0.884s)
[sp-web05.phx1.mozilla.com] out: Stopping memcached: [  OK  ]
[sp-web05.phx1.mozilla.com] out: Starting memcached: [  OK  ]
[sp-web04.phx1.mozilla.com] finished: service memcached restart (1.200s)
[sp-web04.phx1.mozilla.com] out: Stopping memcached: [  OK  ]
[sp-web04.phx1.mozilla.com] out: Starting memcached: [  OK  ]
[sp-web02.phx1.mozilla.com] finished: service memcached restart (1.202s)
[sp-web02.phx1.mozilla.com] out: Stopping memcached: [  OK  ]
[sp-web02.phx1.mozilla.com] out: Starting memcached: [  OK  ]
[sp-web01.phx1.mozilla.com] finished: service memcached restart (1.203s)
[sp-web01.phx1.mozilla.com] out: Stopping memcached: [  OK  ]
[sp-web01.phx1.mozilla.com] out: Starting memcached: [  OK  ]
[sp-web03.phx1.mozilla.com] finished: service memcached restart (1.202s)
[sp-web03.phx1.mozilla.com] out: Stopping memcached: [  OK  ]
[sp-web03.phx1.mozilla.com] out: Starting memcached: [  OK  ]
Status: NEW → RESOLVED
Closed: 13 years ago
Resolution: --- → FIXED
QA verified. Automation passed expect for a test that looks for crashstats data for 9.0b6 which is expected. The cronjob won't run until 2am tomorrow to populate that data. ffts pass. 

The app does return intermittent 500's but this is due to an outage at PHX:
[10:30:29.324] GET https://crash-stats.mozilla.com/query?advanced=1 [HTTP/1.1 500 Internal Server Error 2012ms]


2011-12-15 10:30:30 -08:00 --- error: [5xx Error] File: application/libraries/drivers/Database/Pgsql.php; Line: 48; Message: pg_connect(): Unable to connect to PostgreSQL server: could not connect to server: Connection refused

nagios-phx1: [161] sp-admin01.phx1:File Age - /var/log/socorro/cron_submitdumpstostaging.log is CRITICAL: Connection refused or timed out
nagios-phx1: [176] sp-admin01.phx1:hplog is CRITICAL: CRITICAL 0002: Uncorrectable Machine Check Exception (Board 0, Processor 1, APIC ID 0x00000011, Bank 0x00000005, Status 0xBE00000000800400, Address 0x00003FFFA02872E0, Misc 0x0000000000007FFF)
Status: RESOLVED → VERIFIED
Component: Server Operations: Web Operations → WebOps: Other
Product: mozilla.org → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.