Closed Bug 717055 Opened 8 years ago Closed 8 years ago

Push Socorro 2.4 release

Categories

(mozilla.org Graveyard :: Server Operations, task)

task
Not set

Tracking

(Not tracked)

VERIFIED FIXED

People

(Reporter: rhelmer, Assigned: jason)

References

()

Details

(Whiteboard: ETA: Friday, January 13 2011 8AM Pacific)

We need to push Socorro Friday (the 13th) morning (starting at 8 AM).

This release is a bit more involved than most, since we need a 2 hour database downtime and will need to have crontab and monitor shut down during this time, and push the code after the DB is back up.

Can someone from IT be on-deck to help us do this from 8 AM to around 10 AM this Friday?

We're working through the process in https://etherpad.mozilla.org/socorro-2-4-release-process and still testing the code so no final URL yet, but I wanted to file this now to give heads-up.
Assignee: server-ops → jthomas
Sure, I will reach out to you at 8AM PST.
Assignee: jthomas → server-ops
(In reply to Jason Thomas [:jason] from comment #1)
> Sure, I will reach out to you at 8AM PST.

Oops, mid-aired with you sorry.

Thanks!
Assignee: server-ops → jthomas
Is it possible to send traffic bound for https://crash-stats.mozilla.com to a static downtime page for the duration (maybe using zeus)?

Since the DB is down, Socorro will be entirely inaccessible for the downtime.
(In reply to Robert Helmer [:rhelmer] from comment #3)
> Is it possible to send traffic bound for https://crash-stats.mozilla.com to
> a static downtime page for the duration (maybe using zeus)?
> 
> Since the DB is down, Socorro will be entirely inaccessible for the downtime.

stephend filed bug 717079 for this, also adding new RPM dep in bug 716247 as blocking this
Depends on: 716247, 717079
Pushed code except to processors.

[root@sp-admin01.phx1 ~]# update-socorro-not-processors.sh https://ci.mozilla.org/job/socorro-release/72/artifact/socorro.tar.gz
Upgrading socorro to https://ci.mozilla.org/job/socorro-release/72/artifact/socorro.tar.gz
Upgrading socorro on sp-collector01.phx1.mozilla.com
Stopping socorro-crashmover: [  OK  ]
Starting socorro-crashmover: [  OK  ]
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/72/artifact/socorro.tar.gz
Checksum: aec430615b2df8814f85fdc168c88769
Backed up original to /data/socorro.13-01-2012_09_14_42
Upgrading socorro on sp-collector02.phx1.mozilla.com
Stopping socorro-crashmover: [  OK  ]
Starting socorro-crashmover: [  OK  ]
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/72/artifact/socorro.tar.gz
Checksum: aec430615b2df8814f85fdc168c88769
Backed up original to /data/socorro.13-01-2012_09_14_51
Upgrading socorro on sp-collector03.phx1.mozilla.com
Stopping socorro-crashmover: [  OK  ]
Starting socorro-crashmover: [  OK  ]
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/72/artifact/socorro.tar.gz
Checksum: aec430615b2df8814f85fdc168c88769
Backed up original to /data/socorro.13-01-2012_09_15_10
Upgrading socorro on sp-collector04.phx1.mozilla.com
Stopping socorro-crashmover: [  OK  ]
Starting socorro-crashmover: [  OK  ]
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/72/artifact/socorro.tar.gz
Checksum: aec430615b2df8814f85fdc168c88769
Backed up original to /data/socorro.13-01-2012_09_15_18
Upgrading socorro on sp-collector05.phx1.mozilla.com
Stopping socorro-crashmover: [  OK  ]
Starting socorro-crashmover: [  OK  ]
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/72/artifact/socorro.tar.gz
Checksum: aec430615b2df8814f85fdc168c88769
Backed up original to /data/socorro.13-01-2012_09_15_24
Upgrading socorro on sp-collector06.phx1.mozilla.com
Stopping socorro-crashmover: [  OK  ]
Starting socorro-crashmover: [  OK  ]
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/72/artifact/socorro.tar.gz
Checksum: aec430615b2df8814f85fdc168c88769
Backed up original to /data/socorro.13-01-2012_09_15_44
Upgrading socorro on sp-mware01.phx1.mozilla.com
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/72/artifact/socorro.tar.gz
Checksum: aec430615b2df8814f85fdc168c88769
Backed up original to /data/socorro.13-01-2012_09_16_06
Upgrading socorro on sp-mware02.phx1.mozilla.com
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/72/artifact/socorro.tar.gz
Checksum: aec430615b2df8814f85fdc168c88769
Backed up original to /data/socorro.13-01-2012_09_16_11
Upgrading socorro on sp-mware03.phx1.mozilla.com
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/72/artifact/socorro.tar.gz
Checksum: aec430615b2df8814f85fdc168c88769
Backed up original to /data/socorro.13-01-2012_09_16_14
Upgrading socorro on sp-web01.phx1.mozilla.com
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/72/artifact/socorro.tar.gz
Checksum: aec430615b2df8814f85fdc168c88769
Backed up original to /data/socorro.13-01-2012_09_16_21
Upgrading socorro on sp-web02.phx1.mozilla.com
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/72/artifact/socorro.tar.gz
Checksum: aec430615b2df8814f85fdc168c88769
Backed up original to /data/socorro.13-01-2012_09_16_26
Upgrading socorro on sp-web03.phx1.mozilla.com
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/72/artifact/socorro.tar.gz
Checksum: aec430615b2df8814f85fdc168c88769
Backed up original to /data/socorro.13-01-2012_09_16_31
Upgrading socorro on sp-web04.phx1.mozilla.com
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/72/artifact/socorro.tar.gz
Checksum: aec430615b2df8814f85fdc168c88769
Backed up original to /data/socorro.13-01-2012_09_16_38
Upgrading socorro on sp-web05.phx1.mozilla.com
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/72/artifact/socorro.tar.gz
Checksum: aec430615b2df8814f85fdc168c88769
Backed up original to /data/socorro.13-01-2012_09_16_48
Upgrading socorro on sp-admin01.phx1.mozilla.com
Stopping socorro-monitor: [FAILED]
Starting socorro-monitor: [  OK  ]
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/72/artifact/socorro.tar.gz
Checksum: aec430615b2df8814f85fdc168c88769
Backed up original to /data/socorro.13-01-2012_09_16_52
Upgrading socorro on tp-socorro01-master01.phx1.mozilla.com
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/72/artifact/socorro.tar.gz
Checksum: aec430615b2df8814f85fdc168c88769
Backed up original to /data/socorro.13-01-2012_09_17_08
Upgrading socorro on tp-socorro01-master02.phx1.mozilla.com
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/72/artifact/socorro.tar.gz
Checksum: aec430615b2df8814f85fdc168c88769
Backed up original to /data/socorro.13-01-2012_09_17_11
[sp-web01.phx1.mozilla.com] running: service memcached restart
[sp-web02.phx1.mozilla.com] running: service memcached restart
[sp-web03.phx1.mozilla.com] running: service memcached restart
[sp-web04.phx1.mozilla.com] running: service memcached restart
[sp-web05.phx1.mozilla.com] running: service memcached restart
[sp-web05.phx1.mozilla.com] finished: service memcached restart (1.000s)
[sp-web05.phx1.mozilla.com] out: Stopping memcached: [  OK  ]
[sp-web05.phx1.mozilla.com] out: Starting memcached: [  OK  ]
[sp-web04.phx1.mozilla.com] finished: service memcached restart (1.656s)
[sp-web04.phx1.mozilla.com] out: Stopping memcached: [  OK  ]
[sp-web04.phx1.mozilla.com] out: Starting memcached: [  OK  ]
[sp-web03.phx1.mozilla.com] finished: service memcached restart (2.058s)
[sp-web03.phx1.mozilla.com] out: Stopping memcached: [  OK  ]
[sp-web03.phx1.mozilla.com] out: Starting memcached: [  OK  ]
[sp-web02.phx1.mozilla.com] finished: service memcached restart (2.058s)
[sp-web02.phx1.mozilla.com] out: Stopping memcached: [  OK  ]
[sp-web02.phx1.mozilla.com] out: Starting memcached: [  OK  ]
[sp-web01.phx1.mozilla.com] finished: service memcached restart (2.059s)
[sp-web01.phx1.mozilla.com] out: Stopping memcached: [  OK  ]
[sp-web01.phx1.mozilla.com] out: Starting memcached: [  OK  ]
Pushed https://ci.mozilla.org/job/socorro-release/75/artifact/socorro.tar.gz to prod

[root@sp-admin01.phx1 ~]# update-socorro-not-processors.sh https://ci.mozilla.org/job/socorro-release/75/artifact/socorro.tar.gz
Upgrading socorro to https://ci.mozilla.org/job/socorro-release/75/artifact/socorro.tar.gz
Upgrading socorro on sp-collector01.phx1.mozilla.com
Stopping socorro-crashmover: [  OK  ]
Starting socorro-crashmover: [  OK  ]
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/75/artifact/socorro.tar.gz
Checksum: 6e53a7647b1926aa781395a4fc2144db
Backed up original to /data/socorro.13-01-2012_16_19_53
Upgrading socorro on sp-collector02.phx1.mozilla.com
Stopping socorro-crashmover: [  OK  ]
Starting socorro-crashmover: [  OK  ]
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/75/artifact/socorro.tar.gz
Checksum: 6e53a7647b1926aa781395a4fc2144db
Backed up original to /data/socorro.13-01-2012_16_19_58
Upgrading socorro on sp-collector03.phx1.mozilla.com
Stopping socorro-crashmover: [  OK  ]
Starting socorro-crashmover: [  OK  ]
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/75/artifact/socorro.tar.gz
Checksum: 6e53a7647b1926aa781395a4fc2144db
Backed up original to /data/socorro.13-01-2012_16_20_07
Upgrading socorro on sp-collector04.phx1.mozilla.com
Stopping socorro-crashmover: [  OK  ]
Starting socorro-crashmover: [  OK  ]
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/75/artifact/socorro.tar.gz
Checksum: 6e53a7647b1926aa781395a4fc2144db
Backed up original to /data/socorro.13-01-2012_16_20_12
Upgrading socorro on sp-collector05.phx1.mozilla.com
Stopping socorro-crashmover: [  OK  ]
Starting socorro-crashmover: [  OK  ]
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/75/artifact/socorro.tar.gz
Checksum: 6e53a7647b1926aa781395a4fc2144db
Backed up original to /data/socorro.13-01-2012_16_20_18
Upgrading socorro on sp-collector06.phx1.mozilla.com
Stopping socorro-crashmover: [  OK  ]
Starting socorro-crashmover: [  OK  ]
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/75/artifact/socorro.tar.gz
Checksum: 6e53a7647b1926aa781395a4fc2144db
Backed up original to /data/socorro.13-01-2012_16_20_34
Upgrading socorro on sp-mware01.phx1.mozilla.com
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/75/artifact/socorro.tar.gz
Checksum: 6e53a7647b1926aa781395a4fc2144db
Backed up original to /data/socorro.13-01-2012_16_20_49
Upgrading socorro on sp-mware02.phx1.mozilla.com
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/75/artifact/socorro.tar.gz
Checksum: 6e53a7647b1926aa781395a4fc2144db
Backed up original to /data/socorro.13-01-2012_16_20_51
Upgrading socorro on sp-mware03.phx1.mozilla.com
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/75/artifact/socorro.tar.gz
Checksum: 6e53a7647b1926aa781395a4fc2144db
Backed up original to /data/socorro.13-01-2012_16_20_54
Upgrading socorro on sp-web01.phx1.mozilla.com
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/75/artifact/socorro.tar.gz
Checksum: 6e53a7647b1926aa781395a4fc2144db
Backed up original to /data/socorro.13-01-2012_16_20_57
Upgrading socorro on sp-web02.phx1.mozilla.com
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/75/artifact/socorro.tar.gz
Checksum: 6e53a7647b1926aa781395a4fc2144db
Backed up original to /data/socorro.13-01-2012_16_21_00
Upgrading socorro on sp-web03.phx1.mozilla.com
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/75/artifact/socorro.tar.gz
Checksum: 6e53a7647b1926aa781395a4fc2144db
Backed up original to /data/socorro.13-01-2012_16_21_04
Upgrading socorro on sp-web04.phx1.mozilla.com
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/75/artifact/socorro.tar.gz
Checksum: 6e53a7647b1926aa781395a4fc2144db
Backed up original to /data/socorro.13-01-2012_16_21_17
Upgrading socorro on sp-web05.phx1.mozilla.com
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/75/artifact/socorro.tar.gz
Checksum: 6e53a7647b1926aa781395a4fc2144db
Backed up original to /data/socorro.13-01-2012_16_21_20
Upgrading socorro on sp-admin01.phx1.mozilla.com
Stopping socorro-monitor: [FAILED]
Starting socorro-monitor: [  OK  ]
Stopping httpd: [  OK  ]
Starting httpd: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/75/artifact/socorro.tar.gz
Checksum: 6e53a7647b1926aa781395a4fc2144db
Backed up original to /data/socorro.13-01-2012_16_21_35
Upgrading socorro on tp-socorro01-master01.phx1.mozilla.com
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/75/artifact/socorro.tar.gz
Checksum: 6e53a7647b1926aa781395a4fc2144db
Backed up original to /data/socorro.13-01-2012_16_21_37
Upgrading socorro on tp-socorro01-master02.phx1.mozilla.com
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/75/artifact/socorro.tar.gz
Checksum: 6e53a7647b1926aa781395a4fc2144db
Backed up original to /data/socorro.13-01-2012_16_21_39
[sp-web01.phx1.mozilla.com] running: service memcached restart
[sp-web02.phx1.mozilla.com] running: service memcached restart
[sp-web03.phx1.mozilla.com] running: service memcached restart
[sp-web04.phx1.mozilla.com] running: service memcached restart
[sp-web05.phx1.mozilla.com] running: service memcached restart
[sp-web02.phx1.mozilla.com] finished: service memcached restart (0.926s)
[sp-web02.phx1.mozilla.com] out: Stopping memcached: [  OK  ]
[sp-web02.phx1.mozilla.com] out: Starting memcached: [  OK  ]
[sp-web04.phx1.mozilla.com] finished: service memcached restart (1.019s)
[sp-web04.phx1.mozilla.com] out: Stopping memcached: [  OK  ]
[sp-web04.phx1.mozilla.com] out: Starting memcached: [  OK  ]
[sp-web03.phx1.mozilla.com] finished: service memcached restart (1.021s)
[sp-web03.phx1.mozilla.com] out: Stopping memcached: [  OK  ]
[sp-web03.phx1.mozilla.com] out: Starting memcached: [  OK  ]
[sp-web05.phx1.mozilla.com] finished: service memcached restart (1.022s)
[sp-web05.phx1.mozilla.com] out: Stopping memcached: [  OK  ]
[sp-web05.phx1.mozilla.com] out: Starting memcached: [  OK  ]
[sp-web01.phx1.mozilla.com] finished: service memcached restart (1.113s)
[sp-web01.phx1.mozilla.com] out: Stopping memcached: [  OK  ]
[sp-web01.phx1.mozilla.com] out: Starting memcached: [  OK  ]

[root@sp-admin01.phx1 bin]# update-socorro-processors.sh https://ci.mozilla.org/job/socorro-release/75/artifact/socorro.tar.gz
Upgrading socorro to https://ci.mozilla.org/job/socorro-release/75/artifact/socorro.tar.gz
Upgrading socorro on sp-processor01.phx1.mozilla.com
Stopping socorro-processor: [FAILED]
Starting socorro-processor: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/75/artifact/socorro.tar.gz
Checksum: 6e53a7647b1926aa781395a4fc2144db
Backed up original to /data/socorro.13-01-2012_17_23_38
Upgrading socorro on sp-processor02.phx1.mozilla.com
Stopping socorro-processor: [FAILED]
Starting socorro-processor: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/75/artifact/socorro.tar.gz
Checksum: 6e53a7647b1926aa781395a4fc2144db
Backed up original to /data/socorro.13-01-2012_17_23_41
Upgrading socorro on sp-processor03.phx1.mozilla.com
Stopping socorro-processor: [FAILED]
Starting socorro-processor: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/75/artifact/socorro.tar.gz
Checksum: 6e53a7647b1926aa781395a4fc2144db
Backed up original to /data/socorro.13-01-2012_17_23_44
Upgrading socorro on sp-processor04.phx1.mozilla.com
Stopping socorro-processor: [FAILED]
Starting socorro-processor: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/75/artifact/socorro.tar.gz
Checksum: 6e53a7647b1926aa781395a4fc2144db
Backed up original to /data/socorro.13-01-2012_17_23_48
Upgrading socorro on sp-processor05.phx1.mozilla.com
Stopping socorro-processor: [FAILED]
Starting socorro-processor: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/75/artifact/socorro.tar.gz
Checksum: 6e53a7647b1926aa781395a4fc2144db
Backed up original to /data/socorro.13-01-2012_17_23_51
Upgrading socorro on sp-processor06.phx1.mozilla.com
Stopping socorro-processor: [FAILED]
Starting socorro-processor: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/75/artifact/socorro.tar.gz
Checksum: 6e53a7647b1926aa781395a4fc2144db
Backed up original to /data/socorro.13-01-2012_17_23_54
Upgrading socorro on sp-processor07.phx1.mozilla.com
Stopping socorro-processor: [FAILED]
Starting socorro-processor: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/75/artifact/socorro.tar.gz
Checksum: 6e53a7647b1926aa781395a4fc2144db
Backed up original to /data/socorro.13-01-2012_17_23_57
Upgrading socorro on sp-processor08.phx1.mozilla.com
Stopping socorro-processor: [FAILED]
Starting socorro-processor: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/75/artifact/socorro.tar.gz
Checksum: 6e53a7647b1926aa781395a4fc2144db
Backed up original to /data/socorro.13-01-2012_17_24_00
Upgrading socorro on sp-processor09.phx1.mozilla.com
Stopping socorro-processor: [FAILED]
Starting socorro-processor: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/75/artifact/socorro.tar.gz
Checksum: 6e53a7647b1926aa781395a4fc2144db
Backed up original to /data/socorro.13-01-2012_17_24_03
Upgrading socorro on sp-processor10.phx1.mozilla.com
Stopping socorro-processor: [FAILED]
Starting socorro-processor: [  OK  ]
Socorro build installed successfully!
Downloaded from https://ci.mozilla.org/job/socorro-release/75/artifact/socorro.tar.gz
Checksum: 6e53a7647b1926aa781395a4fc2144db
Backed up original to /data/socorro.13-01-2012_17_24_06
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
QA verified ... the release looks great. UTC bliss is upon us. Manual bfts, server status, and ganglia all look great.

Automation is going to need some love but is passing after the work done on bug 718218. Automation is currently green.
Status: RESOLVED → VERIFIED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.