1159060 - Please purge /pub/mozilla.org/firefox/releases/38.0b8 from the CDNs and origin server cache

Reporter

Description

•

10 years ago

We need to replace the files at /pub/mozilla.org/firefox/releases/38.0b8 with another build. Please clear the cache for the origin server (download-origin.cdn.mozilla.net), and issue purge requests for the CDNs.

:kanban

Updated

•

10 years ago

Whiteboard: [kanban:https://webops.kanbanize.com/ctrl_board/2/1062]

Ashish Vijayaram [:ashish]

Updated

•

10 years ago

Assignee: server-ops-webops → ashish

Status: NEW → ASSIGNED

Ashish Vijayaram [:ashish]

Comment 1

•

10 years ago

Zeus cleared

Rail Aliiev [:rail]

Reporter

Comment 2

•

10 years ago

Thanks for taking this so fast! The push is planned for tomorrow European time, this is why I filed this a blocker.

Ashish Vijayaram [:ashish]

Comment 3

•

10 years ago

Edgecast requests in purge queue

Ashish Vijayaram [:ashish]

Comment 4

•

10 years ago

Akamai purge request has been submitted, this would take upto 40 mins to complete

Ashish Vijayaram [:ashish]

Comment 5

•

10 years ago

:rail NP, we go this! If this needs to be reopened for any reason, please unassign. Thanks!

Status: ASSIGNED → RESOLVED

Closed: 10 years ago

Resolution: --- → FIXED

Rail Aliiev [:rail]

Reporter

Comment 6

•

10 years ago

Thanks!

Rail Aliiev [:rail]

Reporter

Comment 7

•

10 years ago

We see some issues with final verification. stage.mozilla.org doesn't agree with download.cdn.mozilla.net. As an example, see: curl -sI "http://download.cdn.mozilla.net/pub/firefox/releases/38.0b8/update/linux-x86_64/cy/firefox-38.0b8.complete.mar" | grep Content-Length: Content-Length: 45970006 curl -sI http://stage.mozilla.org/pub/mozilla.org/firefox/releases/38.0b8/update/linux-x86_64/cy/firefox-38.0b8.complete.mar | grep Content-Length: Content-Length: 45972761

Assignee: ashish → server-ops-webops

Status: RESOLVED → REOPENED

Resolution: FIXED → ---

:kanban

Updated

•

10 years ago

Assignee: server-ops-webops → ashish

:Atoll

Comment 8

•

10 years ago

Verified as CDN issue rather than ftp server mismatches: $ curl -I "http://download.cdn.mozilla.net/pub/firefox/releases/38.0b8/update/linux-x86_64/cy/firefox-38.0b8.complete.mar" HTTP/1.1 200 OK Accept-Ranges: bytes Access-Control-Allow-Origin: * Cache-Control: max-age=604800 Content-Type: application/octet-stream Date: Tue, 28 Apr 2015 09:25:50 GMT Etag: "480bf70-2bd7256-514ac48e5690a" Expires: Tue, 05 May 2015 09:25:50 GMT Last-Modified: Mon, 27 Apr 2015 03:24:33 GMT Server: ECAcc (rhv/8115) X-Backend-Server: ftp1.dmz.scl3.mozilla.com X-Cache: HIT X-Cache-Info: caching Content-Length: 45970006 $ curl -I http://stage.mozilla.org/pub/mozilla.org/firefox/releases/38.0b8/update/linux-x86_64/cy/firefox-38.0b8.complete.mar HTTP/1.1 200 OK Date: Tue, 28 Apr 2015 09:26:10 GMT Server: Apache X-Backend-Server: ftp8.dmz.scl3.mozilla.com Last-Modified: Mon, 27 Apr 2015 18:44:22 GMT ETag: "4bb03e5-2bd7d19-514b9226cc5ef" Accept-Ranges: bytes Content-Length: 45972761 Cache-Control: max-age=3600 Expires: Tue, 28 Apr 2015 10:26:10 GMT Access-Control-Allow-Origin: * Content-Type: application/octet-stream $ for i in ftp{1..12}.dmz.scl3.mozilla.com; do curl -I -H 'Host: stage.mozilla.org' "http://$i/pub/mozilla.org/firefox/releases/38.0b8/update/linux-x86_64/cy/firefox-38.0b8.complete.mar"; done 2>&1 | grep ETag ETag: "4bb03e5-2bd7d19-514b9226cc5ef" ETag: "4bb03e5-2bd7d19-514b9226cc5ef" ETag: "4bb03e5-2bd7d19-514b9226cc5ef" ETag: "4bb03e5-2bd7d19-514b9226cc5ef" ETag: "4bb03e5-2bd7d19-514b9226cc5ef" ETag: "4bb03e5-2bd7d19-514b9226cc5ef" ETag: "4bb03e5-2bd7d19-514b9226cc5ef"

:Atoll

Comment 9

•

10 years ago

(In reply to Rail Aliiev [:rail] from comment #7) > We see some issues with final verification. stage.mozilla.org doesn't agree > with download.cdn.mozilla.net. As an example, see: > > curl -sI > "http://download.cdn.mozilla.net/pub/firefox/releases/38.0b8/update/linux- > x86_64/cy/firefox-38.0b8.complete.mar" | grep Content-Length: > Content-Length: 45970006 This URL does not match the purge request in comment 0: (In reply to Rail Aliiev [:rail] from comment #0) > We need to replace the files at /pub/mozilla.org/firefox/releases/38.0b8 > with another build. Please clear the cache for the origin server > (download-origin.cdn.mozilla.net), and issue purge requests for the CDNs. So I've added two new purge requests to the Edgecast queue: http://download.cdn.mozilla.net/pub/firefox/releases/38.0b8/* http://wpc.1237.taucdn.net/801237/download.cdn.mozilla.net/pub/firefox/releases/38.0b8/* I do not have access to the other providers, hopefully this is sufficient.

:Atoll

Comment 10

•

10 years ago

Flush completed at 02:47 for both.

Ludovic Hirlimann [:Usul]

Updated

•

10 years ago

Severity: blocker → normal

Ludovic Hirlimann [:Usul]

Comment 11

•

10 years ago

Change priority so it wouldn't ping oncall.

Sylvestre Ledru [:Sylvestre]

Comment 12

•

10 years ago

Well, Ludovic, this is just blocking the release work on Firefox...

Rail Aliiev [:rail]

Reporter

Comment 13

•

10 years ago

(In reply to Richard Soderberg [:atoll] from comment #9) > (In reply to Rail Aliiev [:rail] from comment #7) > > We see some issues with final verification. stage.mozilla.org doesn't agree > > with download.cdn.mozilla.net. As an example, see: > > > > curl -sI > > "http://download.cdn.mozilla.net/pub/firefox/releases/38.0b8/update/linux- > > x86_64/cy/firefox-38.0b8.complete.mar" | grep Content-Length: > > Content-Length: 45970006 > > This URL does not match the purge request in comment 0: Ah, this is what happened... stage.m.o has mozilla.org in the url while others don't. Would it be better to request purging both URL prefixes? I'll update our template accordingly.

Rail Aliiev [:rail]

Reporter

Comment 14

•

10 years ago

Still getting size mismatches: curl -sI http://download.cdn.mozilla.net/pub/firefox/releases/38.0b8/update/linux-x86_64/es-CL/firefox-38.0b8.complete.mar | grep -i content-le Content-Length: 45914590 curl -sI http://stage.mozilla.org/pub/mozilla.org/firefox/releases/38.0b8/update/linux-x86_64/es-CL/firefox-38.0b8.complete.mar | grep -i content-le Content-Length: 45915767 curl -sI http://ftp.mozilla.org/pub/mozilla.org/firefox/releases/38.0b8/update/linux-x86_64/es-CL/firefox-38.0b8.complete.mar | grep -i content-le Content-Length: 45915767

Massimo Gervasini [:massimo]

Updated

•

10 years ago

Comment 15

•

10 years ago

resetting the state, maybe someone from Europe/East Coast can look at this.

Assignee: ashish → server-ops-webops

Severity: normal → blocker

:kanban

Updated

•

10 years ago

Assignee: server-ops-webops → ashish

Rail Aliiev [:rail]

Reporter

Comment 16

•

10 years ago

FTR, you can find the full log at http://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/38.0b8-candidates/build2/logs/release-mozilla-beta-beta_final_verification-bm94-build1-build2.txt.gz

Ashish Vijayaram [:ashish]

Comment 17

•

10 years ago

> 08:26:39 < rail> ashish: http://download.cdn.mozilla.net/pub/firefox/releases/38.0b8/update/mac/fr/firefox-38.0b6-38.0b8.partial.mar vs http://stage.mozilla.org/pub/mozilla.org/firefox/releases/38.0b8/update/mac/fr/firefox-38.0b6-38.0b8.partial.mar disagree :/ This is surely an Edgecast problem. The origin servers are returning the right file. Edgecast isn't purging the cache. I've requested a list of files that are failing from :rail to put in a bulk purge request.

Rail Aliiev [:rail]

Reporter

Comment 18

•

10 years ago

Ashish Vijayaram [:ashish]

Comment 19

•

10 years ago

I've put in a bulk purge request for all the URLs in Comment 18

Hal Wine [:hwine] use NI!

Comment 20

•

10 years ago

releng tests passed, assuming QE tests will as well, closing.

Status: REOPENED → RESOLVED

Closed: 10 years ago → 10 years ago

Resolution: --- → FIXED

Rail Aliiev [:rail]

Reporter

Comment 21

•

10 years ago

CDNs strike back: curl -sI http://stage.mozilla.org/pub/mozilla.org/firefox/releases/38.0b8/update/linux-x86_64/es-CL/firefox-38.0b8.complete.mar | grep Content-Length: Content-Length: 45915767 curl -sI http://download.cdn.mozilla.net/pub/firefox/releases/38.0b8/update/linux-x86_64/es-CL/firefox-38.0b8.complete.mar | grep Content-Length: Content-Length: 45914590

Status: RESOLVED → REOPENED

Resolution: FIXED → ---

Hal Wine [:hwine] use NI!

Comment 22

•

10 years ago

Results for all CDNs on url in comment 21: http://ftp.mozilla.org/pub/firefox/releases/38.0b8/update/linux-x86_64/es-CL/firefox-38.0b8.complete.mar < Content-Range: bytes 0-7/45915767 < Last-Modified: Mon, 27 Apr 2015 18:53:43 GMT < Content-Length: 8 http://wildcard.cdn.mozilla.net.edgesuite.net/pub/firefox/releases/38.0b8/update/linux-x86_64/es-CL/firefox-38.0b8.complete.mar < Last-Modified: Mon, 27 Apr 2015 03:30:33 GMT < Content-Range: bytes 0-7/45914590 < Content-Length: 8 http://download-akamai.cdn.mozilla.net/pub/firefox/releases/38.0b8/update/linux-x86_64/es-CL/firefox-38.0b8.complete.mar < Last-Modified: Mon, 27 Apr 2015 03:30:33 GMT < Content-Range: bytes 0-7/45914590 < Content-Length: 8 http://cds.d6b5y3z2.hwcdn.net/pub/firefox/releases/38.0b8/update/linux-x86_64/es-CL/firefox-38.0b8.complete.mar < Last-Modified: Mon, 27 Apr 2015 18:53:43 GMT < Content-Length: 8 < Content-Range: bytes 0-7/45915767 http://wpc.1237.edgecastcdn.net/pub/firefox/releases/38.0b8/update/linux-x86_64/es-CL/firefox-38.0b8.complete.mar < Content-Range: bytes 0-7/45915767 < Last-Modified: Mon, 27 Apr 2015 18:53:43 GMT < Content-Length: 8

:Atoll

Comment 23

•

10 years ago

:rail, we need the full headers of the failure, not just the Content-Length - otherwise we can't tell which CDN provided the incorrect hit. Testing from my house, Edgecast bay area is returning 45915767, but I don't know where the above curl was run from or which CDN it hit.

:Atoll

Comment 24

•

10 years ago

Thanks Hal

:Atoll

Comment 25

•

10 years ago

http://download-akamai.cdn.mozilla.net/pub/firefox/releases/38.0b8/update/linux-x86_64/es-CL/firefox-38.0b8.complete.mar works from my house, 45915767

:Atoll

Comment 26

•

10 years ago

I'm going to hand this over to CloudOps at this point, because we don't really have any other avenues left to explore here and I'm not sure what to do. Jason is talking on IRC about this and the bugtrail here has most if not all of hat we've tried, so hopefully that helps somehow.

Assignee: ashish → nobody

Component: WebOps: Product Delivery → Operations

Product: Infrastructure & Operations → Mozilla Services

QA Contact: nmaul

:kanban

Updated

•

10 years ago

Assignee: nobody → ashish

:Atoll

Updated

•

10 years ago

Assignee: ashish → nobody

Whiteboard: [kanban:https://webops.kanbanize.com/ctrl_board/2/1062]

Ashish Vijayaram [:ashish]

Comment 27

•

10 years ago

As one last try, I've submitted "Load" requests in Edgecast for all the URLs in Comment 18. They are visible from the Purge/Load page.

Hal Wine [:hwine] use NI!

Comment 28

•

10 years ago

Let's back off on this -- we're deploying a work around for 38.0b8.

Jason Thomas [:jason]

Assignee

Comment 29

•

10 years ago

(In reply to Hal Wine [:hwine] (use needinfo) from comment #28) > Let's back off on this -- we're deploying a work around for 38.0b8. OK. Let me know how I can assist.

Assignee: nobody → jthomas

Severity: blocker → normal

Hal Wine [:hwine] use NI!

Comment 30

•

10 years ago

The work around for 38.0b8 appears to have worked. We'll have final confirmation tomorrow. That said, we've had similar problems every time we have to purge. It is likely that we will work around the issues with CDN's inability to purge in a timely fashion. Jason - as I understand it, Cloud Services now "owns" the relationships with our CDN vendors. Can you answer the following questions about the CDN SLA's? a) are we paying extra to have the "purge" interface available? b) do the vendors claim to offer any SLA w.r.t purge time? Note that the list of CDN vendors we're currently aware of is at https://nagios.mozilla.org/sentry/.

Flags: needinfo?(jthomas)

Nick Thomas [:nthomas] (UTC+12)

Comment 31

•

10 years ago

FTR, the 'source of truth' for determining what download.cdn.mozilla.net resolves to is Cydexis. You can use https://nagios.mozilla.org/sentry/ as an approximation, if only because it's visible outside of Cloud Services/WebOps, but it's not completely reliable.

Jason Thomas [:jason]

Assignee

Comment 32

•

10 years ago

(In reply to Hal Wine [:hwine] (use needinfo) from comment #30) > That said, we've had similar problems every time we have to purge. It is > likely that we will work around the issues with CDN's inability to purge in > a timely fashion. What is the current process for replacing files with another build? Are we updating the origin (ftp hosts) with the new build before we submit purge requests? > Jason - as I understand it, Cloud Services now "owns" the relationships with > our CDN vendors. Can you answer the following questions about the CDN SLA's? > > a) are we paying extra to have the "purge" interface available? No. This is usually offered by CDN providers as part of the service. > b) do the vendors claim to offer any SLA w.r.t purge time? I don't believe they provide a SLA w.r.t purge time. On a purge request Akamai will provide a estimated time of purge completion, Edgecast will update the purge request state in their control panel. Akamai support documentation states to file a support request if a purge takes too long or if there are inconsistencies. In the future we should definitely reach out the CDN vendors as soon as we are seeing issues.

Flags: needinfo?(jthomas)

Hal Wine [:hwine] use NI!

Comment 33

•

10 years ago

(In reply to Jason Thomas [:jason] from comment #32) > (In reply to Hal Wine [:hwine] (use needinfo) from comment #30) > > That said, we've had similar problems every time we have to purge. It is > > likely that we will work around the issues with CDN's inability to purge in > > a timely fashion. > > What is the current process for replacing files with another build? Are we > updating the origin (ftp hosts) with the new build before we submit purge > requests? We wait to re-publish until we "believe" the purge has taken effect. Given the non-determinism of the purge process below (and generally we're in a time crunch when we have to purge), we'll investigate a "no purge" process. > > > Jason - as I understand it, Cloud Services now "owns" the relationships with > > our CDN vendors. Can you answer the following questions about the CDN SLA's? > > > > a) are we paying extra to have the "purge" interface available? > > No. This is usually offered by CDN providers as part of the service. > > > b) do the vendors claim to offer any SLA w.r.t purge time? > I don't believe they provide a SLA w.r.t purge time. On a purge request > Akamai will provide a estimated time of purge completion, Edgecast will > update the purge request state in their control panel. Akamai support > documentation states to file a support request if a purge takes too long or > if there are inconsistencies. > > In the future we should definitely reach out the CDN vendors as soon as we > are seeing issues. I'll close this bug, and open a new one on avoiding purges.

Status: REOPENED → RESOLVED

Closed: 10 years ago → 10 years ago

Resolution: --- → FIXED

Jason Thomas [:jason]

Assignee

Comment 34

•

10 years ago

Unless there is a reason not to, we should re-publish and then request a purge of ZLB + CDN cache. If a request is made via the CDN after the purge completes and old content is still on the origin then old content will be cached and served.

Hal Wine [:hwine] use NI!

Updated

•

10 years ago

Blocks: 1159804