1023222 - please purge the CDN caches for http://download.cdn.mozilla.net/pub/mozilla.org/firefox/releases/24.6.0esr

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Product Delivery, task)

Description

[[e-mail] (:bhearsum)]

We had to push new files for this release. Can we please purge the CDN caches for it ASAP?

Comment 1

[Daniel Maher [:phrawzty]]

I only have logins for Akamai and Edgecast; if other CDNs are serving this resource, they have not been purged by me at this time.


Submitted purge request to Akamai via web interface:

Your purge request has been submitted to the Akamai system as of Tuesday, June 10, 2014 4:08:38 PM CEST. Your request id code is: b7e3ada4-f0a8-11e3-a797-13cac4dcc908. Please record this number for future reference.

The estimated maximum completion time for this request is: 7 minutes 0 seconds. 



Submitted purge request to Edgecast via the web interface:

Your file was successfully placed into the purge queue.

Comment 2

[[e-mail] (:bhearsum)]

Does Highwinds still exist, too? I'm just guessing based on an older bug: https://bugzilla.mozilla.org/show_bug.cgi?id=966637#c19

Comment 3

[Jake Maul [:jakem]]

Should be good now. Purged highwinds, and re-purged Edgecast - last edgecast purge was non-recursive.

Comment 4

[[e-mail] (:bhearsum)]

For posterity, I've updated the RelEng docs about CDN purges for releases to hopefully make this easier for IT in the future. They now specify to purge download-origin and the CDNs, and should make it easier for us to verify:
https://wiki.mozilla.org/Release:Release_Automation_on_Mercurial:Troubleshooting#Overwriting_files_that_have_been_pushed_to_releases.2F

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

Did we do the origin server too ? There's evidence at bug 996139 comment #1 and 2 that this has not worked or regressed.

Comment 6

[Jake Maul [:jakem]]

Need more info... can you paste the full response headers (and the IP they were fetched from) for a failed query? I can't replicate it myself...

This is important because it will tell us which CDN is misbehaving.

Comment 7

[[e-mail] (:bhearsum)]

(In reply to Jake Maul [:jakem] from comment #6)
> Need more info... can you paste the full response headers (and the IP they
> were fetched from) for a failed query? I can't replicate it myself...
> 
> This is important because it will tell us which CDN is misbehaving.

✗ curl -L -s -v -D- -o ~/tmp/esr https://download.mozilla.org/\?product\=firefox-24.6.0esr\&os\=win\&lang\=en-US          
* Hostname was NOT found in DNS cache
*   Trying 63.245.217.36...
* Connected to download.mozilla.org (63.245.217.36) port 443 (#0)
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Server hello (2):
{ [data not shown]
* SSLv3, TLS handshake, CERT (11):
{ [data not shown]
* SSLv3, TLS handshake, Server key exchange (12):
{ [data not shown]
* SSLv3, TLS handshake, Server finished (14):
{ [data not shown]
* SSLv3, TLS handshake, Client key exchange (16):
} [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Finished (20):
} [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
{ [data not shown]
* SSLv3, TLS handshake, Finished (20):
{ [data not shown]
* SSL connection using DHE-RSA-AES128-SHA
* Server certificate:
* 	 subject: C=US; ST=CA; L=Mountain View; O=Mozilla Foundation; CN=download.mozilla.org
* 	 start date: 2013-11-05 00:00:00 GMT
* 	 expire date: 2015-05-05 12:00:00 GMT
* 	 subjectAltName: download.mozilla.org matched
* 	 issuer: C=US; O=DigiCert Inc; CN=DigiCert Secure Server CA
* 	 SSL certificate verify ok.
> GET /?product=firefox-24.6.0esr&os=win&lang=en-US HTTP/1.1
> User-Agent: curl/7.35.0
> Host: download.mozilla.org
> Accept: */*
> 
< HTTP/1.1 302 Found
HTTP/1.1 302 Found
* Server Apache is not blacklisted
< Server: Apache
Server: Apache
< X-Backend-Server: bouncer2.webapp.phx1.mozilla.com
X-Backend-Server: bouncer2.webapp.phx1.mozilla.com
< Cache-Control: max-age=15
Cache-Control: max-age=15
< Content-Type: text/html; charset=UTF-8
Content-Type: text/html; charset=UTF-8
< Date: Wed, 11 Jun 2014 12:50:27 GMT
Date: Wed, 11 Jun 2014 12:50:27 GMT
< Location: https://download-installer.cdn.mozilla.net/pub/firefox/releases/24.6.0esr/win32/en-US/Firefox%20Setup%2024.6.0esr.exe
Location: https://download-installer.cdn.mozilla.net/pub/firefox/releases/24.6.0esr/win32/en-US/Firefox%20Setup%2024.6.0esr.exe
< Set-Cookie: dmo=10.8.81.218.1402491027849159; path=/; expires=Thu, 11-Jun-15 12:50:27 GMT
Set-Cookie: dmo=10.8.81.218.1402491027849159; path=/; expires=Thu, 11-Jun-15 12:50:27 GMT
< X-Cache-Info: caching
X-Cache-Info: caching
< Content-Length: 0
Content-Length: 0

< 
* Connection #0 to host download.mozilla.org left intact
* Issue another request to this URL: 'https://download-installer.cdn.mozilla.net/pub/firefox/releases/24.6.0esr/win32/en-US/Firefox%20Setup%2024.6.0esr.exe'
* Hostname was NOT found in DNS cache
*   Trying 184.29.116.61...
* Connected to download-installer.cdn.mozilla.net (184.29.116.61) port 443 (#1)
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Server hello (2):
{ [data not shown]
* SSLv3, TLS handshake, CERT (11):
{ [data not shown]
* SSLv3, TLS handshake, Server finished (14):
{ [data not shown]
* SSLv3, TLS handshake, Client key exchange (16):
} [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Finished (20):
} [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
{ [data not shown]
* SSLv3, TLS handshake, Finished (20):
{ [data not shown]
* SSL connection using AES128-SHA
* Server certificate:
* 	 subject: C=US; ST=CALIFORNIA; L=Mountain View; O=Mozilla; OU=IT; CN=*.cdn.mozilla.net
* 	 start date: 2014-04-17 00:06:37 GMT
* 	 expire date: 2015-04-17 00:06:37 GMT
* 	 common name: *.cdn.mozilla.net (matched)
* 	 issuer: O=Cybertrust Inc; CN=Cybertrust Public SureServer SV CA
* 	 SSL certificate verify ok.
> GET /pub/firefox/releases/24.6.0esr/win32/en-US/Firefox%20Setup%2024.6.0esr.exe HTTP/1.1
> User-Agent: curl/7.35.0
> Host: download-installer.cdn.mozilla.net
> Accept: */*
> 
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
* Server Apache is not blacklisted
< Server: Apache
Server: Apache
< X-Backend-Server: ftp6.dmz.scl3.mozilla.com
X-Backend-Server: ftp6.dmz.scl3.mozilla.com
< Content-Type: application/octet-stream
Content-Type: application/octet-stream
< Accept-Ranges: bytes
Accept-Ranges: bytes
< Access-Control-Allow-Origin: *
Access-Control-Allow-Origin: *
< ETag: "4b1a24a-15b0cd0-4faf5328ee640"
ETag: "4b1a24a-15b0cd0-4faf5328ee640"
< Last-Modified: Tue, 03 Jun 2014 21:28:49 GMT
Last-Modified: Tue, 03 Jun 2014 21:28:49 GMT
< X-Cache-Info: caching
X-Cache-Info: caching
< Content-Length: 22744272
Content-Length: 22744272
< Cache-Control: max-age=517230
Cache-Control: max-age=517230
< Expires: Tue, 17 Jun 2014 12:30:58 GMT
Expires: Tue, 17 Jun 2014 12:30:58 GMT
< Date: Wed, 11 Jun 2014 12:50:28 GMT
Date: Wed, 11 Jun 2014 12:50:28 GMT
< Connection: keep-alive
Connection: keep-alive



...which gives me a sha1 of 83e174b48bfc752197345ff433a13d03d09f06e5, which is build1

Comment 8

[Lawrence Mandel [:lmandel] (use needinfo)]

Does this mean that some instances will have been updated to ESR 24.6.0 build 1?

Comment 9

[[e-mail] (:bhearsum)]

(In reply to Lawrence Mandel [:lmandel] from comment #8)
> Does this mean that some instances will have been updated to ESR 24.6.0
> build 1?

Nope, this means that any users that got the wrong bits will have been unable to update because the bits they received didn't match what AUS told them they should get.

Comment 10

[[e-mail] (:bhearsum)]

(In reply to Ben Hearsum [:bhearsum] from comment #9)
> (In reply to Lawrence Mandel [:lmandel] from comment #8)
> > Does this mean that some instances will have been updated to ESR 24.6.0
> > build 1?
> 
> Nope, this means that any users that got the wrong bits will have been
> unable to update because the bits they received didn't match what AUS told
> them they should get.

And just to avoid confusion: the clients will continue to retry on whatever cycle it does (every 24h I think?). Their next attempt after we resolve this will end up with them updating to the correct build.

Comment 11

[[e-mail] (:bhearsum)]

This is actively breaking updates for real world ESR users. Can somebody please have a look ASAP?

Comment 12

[Jake Maul [:jakem]]

That curl is for Akamai (184.29.116.61)... I was told Akamai was purged already and in our testing it was returning proper results so I didn't bother to re-purge. Of course it's a distributed CDN so full testing is virtually impossible.

My guess is the same as Edgecast, where it wasn't purged recursively... don't know for sure though. I've issued a recursive purge on it. Akamai says 30-40min for full effectiveness. No way to expedite.

Comment 13

[[e-mail] (:bhearsum)]

I think this is fixed now. I've been making requests every minute and the last ~25 have all gotten the correct hash. I've asked Lawrence to get someone from the ESR list to verify that they're able to update now, too.

Comment 14

[Lawrence Mandel [:lmandel] (use needinfo)]

I received confirmation from one of the reporters via the enterprise list. Resolving as fixed.