Please purge /pub/mozilla.org/firefox/releases/36.0b1 from the CDNs and origin server cache

RESOLVED FIXED

Status

Infrastructure & Operations Graveyard
WebOps: Product Delivery
--
blocker
RESOLVED FIXED
3 years ago
a year ago

People

(Reporter: nthomas, Assigned: cyliang)

Tracking

Details

(Whiteboard: [kanban:https://webops.kanbanize.com/ctrl_board/2/281] )

(Reporter)

Description

3 years ago
We need to replace the files at /pub/mozilla.org/firefox/releases/36.0b1 with another build. Please clear the cache for the origin server (download.cdn.mozilla.net), and issue purge requests for the CDNs.

WebOps, could you please do this first thing when you get in ?

Updated

3 years ago
Whiteboard: [kanban:https://webops.kanbanize.com/ctrl_board/2/281]
(Reporter)

Updated

3 years ago
Blocks: 1119236
Lowering I've told some webops about it. no need to page moc.
Severity: critical → normal
(Assignee)

Updated

3 years ago
Assignee: server-ops-webops → cliang
(Assignee)

Comment 2

3 years ago
I've purged the load balancer caches that serve as the CDN origin.
I have issued purge requests at Akamai, Edgecast, and Highwinds.
(Assignee)

Comment 3

3 years ago
Edgecast reports as done.
(Assignee)

Comment 4

3 years ago
Akamai reports as done.
It looks like highwinds is purged but edgecast isn't. We've got a test script (http://hg.mozilla.org/build/braindump/file/default/releases-related/check_cdn) that checks things across VPNs. Everything except Edgecast is returning consistent, correct responses:
➜  releases-related  ./check_cdn firefox/releases/36.0b1/update/mac/an/firefox-35.0b3-36.0b1.partial.mar
http://ftp.mozilla.org/pub/firefox/releases/36.0b1/update/mac/an/firefox-35.0b3-36.0b1.partial.mar
< Content-Range: bytes 0-7/30254709
< Last-Modified: Wed, 14 Jan 2015 23:46:08 GMT
< Content-Length: 8
http://wildcard.cdn.mozilla.net.edgesuite.net/pub/firefox/releases/36.0b1/update/mac/an/firefox-35.0b3-36.0b1.partial.mar
< Last-Modified: Wed, 14 Jan 2015 23:46:08 GMT
< Content-Range: bytes 0-7/30254709
< Content-Length: 8
http://download-akamai.cdn.mozilla.net/pub/firefox/releases/36.0b1/update/mac/an/firefox-35.0b3-36.0b1.partial.mar
< Last-Modified: Wed, 14 Jan 2015 23:46:08 GMT
< Content-Range: bytes 0-7/30254709
< Content-Length: 8
http://cds.d6b5y3z2.hwcdn.net/pub/firefox/releases/36.0b1/update/mac/an/firefox-35.0b3-36.0b1.partial.mar
< Last-Modified: Wed, 14 Jan 2015 23:46:08 GMT
< Content-Length: 8
< Content-Range: bytes 0-7/30254709
http://wpc.1237.edgecastcdn.net/pub/firefox/releases/36.0b1/update/mac/an/firefox-35.0b3-36.0b1.partial.mar
< Content-Range: bytes 0-7/30251653
< Last-Modified: Tue, 13 Jan 2015 06:54:35 GMT
< Content-Length: 8
➜  releases-related  ./check_cdn firefox/releases/36.0b1/update/mac/an/firefox-35.0b3-36.0b1.partial.mar
http://ftp.mozilla.org/pub/firefox/releases/36.0b1/update/mac/an/firefox-35.0b3-36.0b1.partial.mar
< Content-Range: bytes 0-7/30254709
< Last-Modified: Wed, 14 Jan 2015 23:46:08 GMT
< Content-Length: 8
http://wildcard.cdn.mozilla.net.edgesuite.net/pub/firefox/releases/36.0b1/update/mac/an/firefox-35.0b3-36.0b1.partial.mar
< Last-Modified: Wed, 14 Jan 2015 23:46:08 GMT
< Content-Range: bytes 0-7/30254709
< Content-Length: 8
http://download-akamai.cdn.mozilla.net/pub/firefox/releases/36.0b1/update/mac/an/firefox-35.0b3-36.0b1.partial.mar
< Last-Modified: Wed, 14 Jan 2015 23:46:08 GMT
< Content-Range: bytes 0-7/30254709
< Content-Length: 8
http://cds.d6b5y3z2.hwcdn.net/pub/firefox/releases/36.0b1/update/mac/an/firefox-35.0b3-36.0b1.partial.mar
< Last-Modified: Wed, 14 Jan 2015 23:46:08 GMT
< Content-Length: 8
< Content-Range: bytes 0-7/30254709
http://wpc.1237.edgecastcdn.net/pub/firefox/releases/36.0b1/update/mac/an/firefox-35.0b3-36.0b1.partial.mar
< Content-Range: bytes 0-7/30254709
< Last-Modified: Wed, 14 Jan 2015 23:46:08 GMT
< Content-Length: 8
I _think_ things are OK now. I've had consistent results from all CDNs for the past 10min.
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
Scratch that - everything looked fine from my laptop, but from scl3 I'm getting consistent incorrect responses from Edgecast:
------------------
http://ftp.mozilla.org/pub/firefox/releases/36.0b1/update/mac/an/firefox-35.0b3-36.0b1.partial.mar
< Content-Range: bytes 0-7/30254709
< Last-Modified: Wed, 14 Jan 2015 23:46:08 GMT
< Content-Length: 8
http://wildcard.cdn.mozilla.net.edgesuite.net/pub/firefox/releases/36.0b1/update/mac/an/firefox-35.0b3-36.0b1.partial.mar
< Last-Modified: Wed, 14 Jan 2015 23:46:08 GMT
< Content-Range: bytes 0-7/30254709
< Content-Length: 8
http://download-akamai.cdn.mozilla.net/pub/firefox/releases/36.0b1/update/mac/an/firefox-35.0b3-36.0b1.partial.mar
< Last-Modified: Wed, 14 Jan 2015 23:46:08 GMT
< Content-Range: bytes 0-7/30254709
< Content-Length: 8
http://cds.d6b5y3z2.hwcdn.net/pub/firefox/releases/36.0b1/update/mac/an/firefox-35.0b3-36.0b1.partial.mar
< Last-Modified: Wed, 14 Jan 2015 23:46:08 GMT
< Content-Length: 8
< Content-Range: bytes 0-7/30254709
http://wpc.1237.edgecastcdn.net/pub/firefox/releases/36.0b1/update/mac/an/firefox-35.0b3-36.0b1.partial.mar
< Content-Range: bytes 0-7/30251653
< Last-Modified: Tue, 13 Jan 2015 06:54:35 GMT
< Content-Length: 8


If there's anything we can do to poke this along that'd be great, but I'm pretty sure we just have to sit and wait :(.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Can we double check that this is still in progress and/or re-issue the request? We're still seeing tons of inconsistency from Edgecast.
Flags: needinfo?(cliang)
Eg:
http://ftp.mozilla.org/pub/firefox/releases/36.0b1/update/mac/de/firefox-35.0b3-36.0b1.partial.mar
< Content-Range: bytes 0-7/30267355
< Last-Modified: Thu, 15 Jan 2015 01:08:39 GMT
< Content-Length: 8
http://wildcard.cdn.mozilla.net.edgesuite.net/pub/firefox/releases/36.0b1/update/mac/de/firefox-35.0b3-36.0b1.partial.mar
< Last-Modified: Thu, 15 Jan 2015 01:08:39 GMT
< Content-Range: bytes 0-7/30267355
< Content-Length: 8
http://download-akamai.cdn.mozilla.net/pub/firefox/releases/36.0b1/update/mac/de/firefox-35.0b3-36.0b1.partial.mar
< Last-Modified: Thu, 15 Jan 2015 01:08:39 GMT
< Content-Range: bytes 0-7/30267355
< Content-Length: 8
http://cds.d6b5y3z2.hwcdn.net/pub/firefox/releases/36.0b1/update/mac/de/firefox-35.0b3-36.0b1.partial.mar
< Last-Modified: Thu, 15 Jan 2015 01:08:39 GMT
< Content-Length: 8
< Content-Range: bytes 0-7/30267355
http://wpc.1237.edgecastcdn.net/pub/firefox/releases/36.0b1/update/mac/de/firefox-35.0b3-36.0b1.partial.mar
< Content-Range: bytes 0-7/30264278
< Last-Modified: Tue, 13 Jan 2015 08:23:39 GMT
< Content-Length: 8
------------------
(Reporter)

Comment 10

3 years ago
For the sake of clarity, we can't release 36.0b1 until the CDN purge is complete. Any response to comment #8 ?
Severity: normal → blocker
(Reporter)

Comment 11

3 years ago
Edgecast are still serving the old content, lets try purging them again ASAP.
(Assignee)

Comment 12

3 years ago
I believe Edgecast should be clear of old content.   CDN purge instructions updated.
Flags: needinfo?(cliang)
(Assignee)

Comment 13

3 years ago
9:54:55 PM nthomas: I'll start running our verification script
9:55:03 PM nthomas: it iterates over all the possibilities
9:55:45 PM cyliang: K
9:56:03 PM nthomas: runtime is < 10min if all is well
10:03:08 PM nthomas: test passes! woot
10:03:23 PM cyliang: \o/
Status: REOPENED → RESOLVED
Last Resolved: 3 years ago3 years ago
Resolution: --- → FIXED
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.