Closed Bug 1121872 Opened 10 years ago Closed 10 years ago

Please purge /pub/mozilla.org/firefox/releases/36.0b1 from the CDNs and origin server cache

Categories

(Infrastructure & Operations Graveyard :: WebOps: Product Delivery, task)

Product:
Infrastructure & Operations Graveyard
Component:
WebOps: Product Delivery
Platform:
x86
All
Type:
task
Priority:
Not set
Severity:
blocker

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: nthomas, Assigned: cliang)

References

Details

(Whiteboard: [kanban:https://webops.kanbanize.com/ctrl_board/2/281] )

We need to replace the files at /pub/mozilla.org/firefox/releases/36.0b1 with another build. Please clear the cache for the origin server (download.cdn.mozilla.net), and issue purge requests for the CDNs. WebOps, could you please do this first thing when you get in ?
Whiteboard: [kanban:https://webops.kanbanize.com/ctrl_board/2/281]
Blocks: 1119236
Lowering I've told some webops about it. no need to page moc.
Severity: critical → normal
Assignee: server-ops-webops → cliang
I've purged the load balancer caches that serve as the CDN origin. I have issued purge requests at Akamai, Edgecast, and Highwinds.
Edgecast reports as done.
Akamai reports as done.
It looks like highwinds is purged but edgecast isn't. We've got a test script (http://hg.mozilla.org/build/braindump/file/default/releases-related/check_cdn) that checks things across VPNs. Everything except Edgecast is returning consistent, correct responses: ➜ releases-related ./check_cdn firefox/releases/36.0b1/update/mac/an/firefox-35.0b3-36.0b1.partial.mar http://ftp.mozilla.org/pub/firefox/releases/36.0b1/update/mac/an/firefox-35.0b3-36.0b1.partial.mar < Content-Range: bytes 0-7/30254709 < Last-Modified: Wed, 14 Jan 2015 23:46:08 GMT < Content-Length: 8 http://wildcard.cdn.mozilla.net.edgesuite.net/pub/firefox/releases/36.0b1/update/mac/an/firefox-35.0b3-36.0b1.partial.mar < Last-Modified: Wed, 14 Jan 2015 23:46:08 GMT < Content-Range: bytes 0-7/30254709 < Content-Length: 8 http://download-akamai.cdn.mozilla.net/pub/firefox/releases/36.0b1/update/mac/an/firefox-35.0b3-36.0b1.partial.mar < Last-Modified: Wed, 14 Jan 2015 23:46:08 GMT < Content-Range: bytes 0-7/30254709 < Content-Length: 8 http://cds.d6b5y3z2.hwcdn.net/pub/firefox/releases/36.0b1/update/mac/an/firefox-35.0b3-36.0b1.partial.mar < Last-Modified: Wed, 14 Jan 2015 23:46:08 GMT < Content-Length: 8 < Content-Range: bytes 0-7/30254709 http://wpc.1237.edgecastcdn.net/pub/firefox/releases/36.0b1/update/mac/an/firefox-35.0b3-36.0b1.partial.mar < Content-Range: bytes 0-7/30251653 < Last-Modified: Tue, 13 Jan 2015 06:54:35 GMT < Content-Length: 8 ➜ releases-related ./check_cdn firefox/releases/36.0b1/update/mac/an/firefox-35.0b3-36.0b1.partial.mar http://ftp.mozilla.org/pub/firefox/releases/36.0b1/update/mac/an/firefox-35.0b3-36.0b1.partial.mar < Content-Range: bytes 0-7/30254709 < Last-Modified: Wed, 14 Jan 2015 23:46:08 GMT < Content-Length: 8 http://wildcard.cdn.mozilla.net.edgesuite.net/pub/firefox/releases/36.0b1/update/mac/an/firefox-35.0b3-36.0b1.partial.mar < Last-Modified: Wed, 14 Jan 2015 23:46:08 GMT < Content-Range: bytes 0-7/30254709 < Content-Length: 8 http://download-akamai.cdn.mozilla.net/pub/firefox/releases/36.0b1/update/mac/an/firefox-35.0b3-36.0b1.partial.mar < Last-Modified: Wed, 14 Jan 2015 23:46:08 GMT < Content-Range: bytes 0-7/30254709 < Content-Length: 8 http://cds.d6b5y3z2.hwcdn.net/pub/firefox/releases/36.0b1/update/mac/an/firefox-35.0b3-36.0b1.partial.mar < Last-Modified: Wed, 14 Jan 2015 23:46:08 GMT < Content-Length: 8 < Content-Range: bytes 0-7/30254709 http://wpc.1237.edgecastcdn.net/pub/firefox/releases/36.0b1/update/mac/an/firefox-35.0b3-36.0b1.partial.mar < Content-Range: bytes 0-7/30254709 < Last-Modified: Wed, 14 Jan 2015 23:46:08 GMT < Content-Length: 8
I _think_ things are OK now. I've had consistent results from all CDNs for the past 10min.
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Scratch that - everything looked fine from my laptop, but from scl3 I'm getting consistent incorrect responses from Edgecast: ------------------ http://ftp.mozilla.org/pub/firefox/releases/36.0b1/update/mac/an/firefox-35.0b3-36.0b1.partial.mar < Content-Range: bytes 0-7/30254709 < Last-Modified: Wed, 14 Jan 2015 23:46:08 GMT < Content-Length: 8 http://wildcard.cdn.mozilla.net.edgesuite.net/pub/firefox/releases/36.0b1/update/mac/an/firefox-35.0b3-36.0b1.partial.mar < Last-Modified: Wed, 14 Jan 2015 23:46:08 GMT < Content-Range: bytes 0-7/30254709 < Content-Length: 8 http://download-akamai.cdn.mozilla.net/pub/firefox/releases/36.0b1/update/mac/an/firefox-35.0b3-36.0b1.partial.mar < Last-Modified: Wed, 14 Jan 2015 23:46:08 GMT < Content-Range: bytes 0-7/30254709 < Content-Length: 8 http://cds.d6b5y3z2.hwcdn.net/pub/firefox/releases/36.0b1/update/mac/an/firefox-35.0b3-36.0b1.partial.mar < Last-Modified: Wed, 14 Jan 2015 23:46:08 GMT < Content-Length: 8 < Content-Range: bytes 0-7/30254709 http://wpc.1237.edgecastcdn.net/pub/firefox/releases/36.0b1/update/mac/an/firefox-35.0b3-36.0b1.partial.mar < Content-Range: bytes 0-7/30251653 < Last-Modified: Tue, 13 Jan 2015 06:54:35 GMT < Content-Length: 8 If there's anything we can do to poke this along that'd be great, but I'm pretty sure we just have to sit and wait :(.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Can we double check that this is still in progress and/or re-issue the request? We're still seeing tons of inconsistency from Edgecast.
Flags: needinfo?(cliang)
Eg: http://ftp.mozilla.org/pub/firefox/releases/36.0b1/update/mac/de/firefox-35.0b3-36.0b1.partial.mar < Content-Range: bytes 0-7/30267355 < Last-Modified: Thu, 15 Jan 2015 01:08:39 GMT < Content-Length: 8 http://wildcard.cdn.mozilla.net.edgesuite.net/pub/firefox/releases/36.0b1/update/mac/de/firefox-35.0b3-36.0b1.partial.mar < Last-Modified: Thu, 15 Jan 2015 01:08:39 GMT < Content-Range: bytes 0-7/30267355 < Content-Length: 8 http://download-akamai.cdn.mozilla.net/pub/firefox/releases/36.0b1/update/mac/de/firefox-35.0b3-36.0b1.partial.mar < Last-Modified: Thu, 15 Jan 2015 01:08:39 GMT < Content-Range: bytes 0-7/30267355 < Content-Length: 8 http://cds.d6b5y3z2.hwcdn.net/pub/firefox/releases/36.0b1/update/mac/de/firefox-35.0b3-36.0b1.partial.mar < Last-Modified: Thu, 15 Jan 2015 01:08:39 GMT < Content-Length: 8 < Content-Range: bytes 0-7/30267355 http://wpc.1237.edgecastcdn.net/pub/firefox/releases/36.0b1/update/mac/de/firefox-35.0b3-36.0b1.partial.mar < Content-Range: bytes 0-7/30264278 < Last-Modified: Tue, 13 Jan 2015 08:23:39 GMT < Content-Length: 8 ------------------
For the sake of clarity, we can't release 36.0b1 until the CDN purge is complete. Any response to comment #8 ?
Severity: normal → blocker
Edgecast are still serving the old content, lets try purging them again ASAP.
I believe Edgecast should be clear of old content. CDN purge instructions updated.
Flags: needinfo?(cliang)
9:54:55 PM nthomas: I'll start running our verification script 9:55:03 PM nthomas: it iterates over all the possibilities 9:55:45 PM cyliang: K 9:56:03 PM nthomas: runtime is < 10min if all is well 10:03:08 PM nthomas: test passes! woot 10:03:23 PM cyliang: \o/
Status: REOPENED → RESOLVED
Closed: 10 years ago → 10 years ago
Resolution: --- → FIXED
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.