RESOLVED FIXED

Status

Infrastructure & Operations Graveyard
WebOps: Product Delivery
--
blocker
RESOLVED FIXED
3 years ago
a year ago

People

(Reporter: bhearsum, Assigned: jakem)

Tracking

Details

(Reporter)

Description

3 years ago
We had to push new files for this release. Can we please purge the CDN caches for it ASAP?

Updated

3 years ago
Assignee: server-ops-webops → rwatson
I only have logins for Akamai and Edgecast; if other CDNs are serving this resource, they have not been purged by me at this time.


Submitted purge request to Akamai via web interface:

Your purge request has been submitted to the Akamai system as of Tuesday, June 10, 2014 4:08:38 PM CEST. Your request id code is: b7e3ada4-f0a8-11e3-a797-13cac4dcc908. Please record this number for future reference.

The estimated maximum completion time for this request is: 7 minutes 0 seconds. 



Submitted purge request to Edgecast via the web interface:

Your file was successfully placed into the purge queue.
(Reporter)

Comment 2

3 years ago
Does Highwinds still exist, too? I'm just guessing based on an older bug: https://bugzilla.mozilla.org/show_bug.cgi?id=966637#c19
(Assignee)

Comment 3

3 years ago
Should be good now. Purged highwinds, and re-purged Edgecast - last edgecast purge was non-recursive.
Assignee: rwatson → nmaul
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
(Reporter)

Comment 4

3 years ago
For posterity, I've updated the RelEng docs about CDN purges for releases to hopefully make this easier for IT in the future. They now specify to purge download-origin and the CDNs, and should make it easier for us to verify:
https://wiki.mozilla.org/Release:Release_Automation_on_Mercurial:Troubleshooting#Overwriting_files_that_have_been_pushed_to_releases.2F
Status: RESOLVED → VERIFIED
Did we do the origin server too ? There's evidence at bug 996139 comment #1 and 2 that this has not worked or regressed.
Status: VERIFIED → REOPENED
Resolution: FIXED → ---
(Assignee)

Comment 6

3 years ago
Need more info... can you paste the full response headers (and the IP they were fetched from) for a failed query? I can't replicate it myself...

This is important because it will tell us which CDN is misbehaving.
Flags: needinfo?(nthomas)
(Reporter)

Comment 7

3 years ago
(In reply to Jake Maul [:jakem] from comment #6)
> Need more info... can you paste the full response headers (and the IP they
> were fetched from) for a failed query? I can't replicate it myself...
> 
> This is important because it will tell us which CDN is misbehaving.

✗ curl -L -s -v -D- -o ~/tmp/esr https://download.mozilla.org/\?product\=firefox-24.6.0esr\&os\=win\&lang\=en-US          
* Hostname was NOT found in DNS cache
*   Trying 63.245.217.36...
* Connected to download.mozilla.org (63.245.217.36) port 443 (#0)
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Server hello (2):
{ [data not shown]
* SSLv3, TLS handshake, CERT (11):
{ [data not shown]
* SSLv3, TLS handshake, Server key exchange (12):
{ [data not shown]
* SSLv3, TLS handshake, Server finished (14):
{ [data not shown]
* SSLv3, TLS handshake, Client key exchange (16):
} [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Finished (20):
} [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
{ [data not shown]
* SSLv3, TLS handshake, Finished (20):
{ [data not shown]
* SSL connection using DHE-RSA-AES128-SHA
* Server certificate:
* 	 subject: C=US; ST=CA; L=Mountain View; O=Mozilla Foundation; CN=download.mozilla.org
* 	 start date: 2013-11-05 00:00:00 GMT
* 	 expire date: 2015-05-05 12:00:00 GMT
* 	 subjectAltName: download.mozilla.org matched
* 	 issuer: C=US; O=DigiCert Inc; CN=DigiCert Secure Server CA
* 	 SSL certificate verify ok.
> GET /?product=firefox-24.6.0esr&os=win&lang=en-US HTTP/1.1
> User-Agent: curl/7.35.0
> Host: download.mozilla.org
> Accept: */*
> 
< HTTP/1.1 302 Found
HTTP/1.1 302 Found
* Server Apache is not blacklisted
< Server: Apache
Server: Apache
< X-Backend-Server: bouncer2.webapp.phx1.mozilla.com
X-Backend-Server: bouncer2.webapp.phx1.mozilla.com
< Cache-Control: max-age=15
Cache-Control: max-age=15
< Content-Type: text/html; charset=UTF-8
Content-Type: text/html; charset=UTF-8
< Date: Wed, 11 Jun 2014 12:50:27 GMT
Date: Wed, 11 Jun 2014 12:50:27 GMT
< Location: https://download-installer.cdn.mozilla.net/pub/firefox/releases/24.6.0esr/win32/en-US/Firefox%20Setup%2024.6.0esr.exe
Location: https://download-installer.cdn.mozilla.net/pub/firefox/releases/24.6.0esr/win32/en-US/Firefox%20Setup%2024.6.0esr.exe
< Set-Cookie: dmo=10.8.81.218.1402491027849159; path=/; expires=Thu, 11-Jun-15 12:50:27 GMT
Set-Cookie: dmo=10.8.81.218.1402491027849159; path=/; expires=Thu, 11-Jun-15 12:50:27 GMT
< X-Cache-Info: caching
X-Cache-Info: caching
< Content-Length: 0
Content-Length: 0

< 
* Connection #0 to host download.mozilla.org left intact
* Issue another request to this URL: 'https://download-installer.cdn.mozilla.net/pub/firefox/releases/24.6.0esr/win32/en-US/Firefox%20Setup%2024.6.0esr.exe'
* Hostname was NOT found in DNS cache
*   Trying 184.29.116.61...
* Connected to download-installer.cdn.mozilla.net (184.29.116.61) port 443 (#1)
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Server hello (2):
{ [data not shown]
* SSLv3, TLS handshake, CERT (11):
{ [data not shown]
* SSLv3, TLS handshake, Server finished (14):
{ [data not shown]
* SSLv3, TLS handshake, Client key exchange (16):
} [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Finished (20):
} [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
{ [data not shown]
* SSLv3, TLS handshake, Finished (20):
{ [data not shown]
* SSL connection using AES128-SHA
* Server certificate:
* 	 subject: C=US; ST=CALIFORNIA; L=Mountain View; O=Mozilla; OU=IT; CN=*.cdn.mozilla.net
* 	 start date: 2014-04-17 00:06:37 GMT
* 	 expire date: 2015-04-17 00:06:37 GMT
* 	 common name: *.cdn.mozilla.net (matched)
* 	 issuer: O=Cybertrust Inc; CN=Cybertrust Public SureServer SV CA
* 	 SSL certificate verify ok.
> GET /pub/firefox/releases/24.6.0esr/win32/en-US/Firefox%20Setup%2024.6.0esr.exe HTTP/1.1
> User-Agent: curl/7.35.0
> Host: download-installer.cdn.mozilla.net
> Accept: */*
> 
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
* Server Apache is not blacklisted
< Server: Apache
Server: Apache
< X-Backend-Server: ftp6.dmz.scl3.mozilla.com
X-Backend-Server: ftp6.dmz.scl3.mozilla.com
< Content-Type: application/octet-stream
Content-Type: application/octet-stream
< Accept-Ranges: bytes
Accept-Ranges: bytes
< Access-Control-Allow-Origin: *
Access-Control-Allow-Origin: *
< ETag: "4b1a24a-15b0cd0-4faf5328ee640"
ETag: "4b1a24a-15b0cd0-4faf5328ee640"
< Last-Modified: Tue, 03 Jun 2014 21:28:49 GMT
Last-Modified: Tue, 03 Jun 2014 21:28:49 GMT
< X-Cache-Info: caching
X-Cache-Info: caching
< Content-Length: 22744272
Content-Length: 22744272
< Cache-Control: max-age=517230
Cache-Control: max-age=517230
< Expires: Tue, 17 Jun 2014 12:30:58 GMT
Expires: Tue, 17 Jun 2014 12:30:58 GMT
< Date: Wed, 11 Jun 2014 12:50:28 GMT
Date: Wed, 11 Jun 2014 12:50:28 GMT
< Connection: keep-alive
Connection: keep-alive



...which gives me a sha1 of 83e174b48bfc752197345ff433a13d03d09f06e5, which is build1
Flags: needinfo?(nthomas) → needinfo?(nmaul)
Does this mean that some instances will have been updated to ESR 24.6.0 build 1?
(Reporter)

Comment 9

3 years ago
(In reply to Lawrence Mandel [:lmandel] from comment #8)
> Does this mean that some instances will have been updated to ESR 24.6.0
> build 1?

Nope, this means that any users that got the wrong bits will have been unable to update because the bits they received didn't match what AUS told them they should get.
(Reporter)

Comment 10

3 years ago
(In reply to Ben Hearsum [:bhearsum] from comment #9)
> (In reply to Lawrence Mandel [:lmandel] from comment #8)
> > Does this mean that some instances will have been updated to ESR 24.6.0
> > build 1?
> 
> Nope, this means that any users that got the wrong bits will have been
> unable to update because the bits they received didn't match what AUS told
> them they should get.

And just to avoid confusion: the clients will continue to retry on whatever cycle it does (every 24h I think?). Their next attempt after we resolve this will end up with them updating to the correct build.
(Reporter)

Comment 11

3 years ago
This is actively breaking updates for real world ESR users. Can somebody please have a look ASAP?
(Assignee)

Comment 12

3 years ago
That curl is for Akamai (184.29.116.61)... I was told Akamai was purged already and in our testing it was returning proper results so I didn't bother to re-purge. Of course it's a distributed CDN so full testing is virtually impossible.

My guess is the same as Edgecast, where it wasn't purged recursively... don't know for sure though. I've issued a recursive purge on it. Akamai says 30-40min for full effectiveness. No way to expedite.
Flags: needinfo?(nmaul)
(Reporter)

Comment 13

3 years ago
I think this is fixed now. I've been making requests every minute and the last ~25 have all gotten the correct hash. I've asked Lawrence to get someone from the ESR list to verify that they're able to update now, too.
I received confirmation from one of the reporters via the enterprise list. Resolving as fixed.
Status: REOPENED → RESOLVED
Last Resolved: 3 years ago3 years ago
Resolution: --- → FIXED
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.