Closed
Bug 801017
Opened 12 years ago
Closed 12 years ago
cdn returning 403 errors for installers
Categories
(Infrastructure & Operations Graveyard :: WebOps: Other, task)
Infrastructure & Operations Graveyard
WebOps: Other
Tracking
(Not tracked)
VERIFIED
FIXED
People
(Reporter: bhearsum, Assigned: cshields)
Details
(Whiteboard: [stub+][push-interrupt])
Might be related to changes in bug 798486. Blocking stub installer, which, apparently we're shipping OMGRIGHTNOW. ➜ ~ curl -IL http://download.cdn.mozilla.net/pub/mozilla.org/firefox/nightly/latest-mozilla-central/firefox-19.0a1.en-US.win32.installer.exe HTTP/1.1 403 Forbidden Server: Apache X-Backend-Server: ftp2.dmz.scl3.mozilla.com Content-Type: text/html; charset=iso-8859-1 X-Cache-Info: cached Content-Length: 1 Date: Fri, 12 Oct 2012 17:49:12 GMT Connection: keep-alive Permissions on the filesystem look like this: /pub/mozilla.org/firefox/nightly/latest-mozilla-central -bash-4.1$ ls -l firefox-19.0a1.en-US.win32.installer.exe -rw-r--r-- 3 ffxbld firefox 20218648 Oct 12 06:08 firefox-19.0a1.en-US.win32.installer.exe
Reporter | ||
Comment 1•12 years ago
|
||
https://nagios.mozilla.org/sentry/ shows the CDN being pulled out of the pool just now, not sure if that's because of this 403 error or just the normal reasons.
Comment 2•12 years ago
|
||
Blocker cause this causes the stub to fail intermittently to download the bits for a channel. Reproduced this 3 times so far.
Whiteboard: [stub+]
Comment 3•12 years ago
|
||
Was just told by on-call (Aj) that WebOps is aware and is looking into this.
Updated•12 years ago
|
Assignee: server-ops → server-ops-webops
Component: Server Operations → Server Operations: Web Operations
Assignee | ||
Comment 4•12 years ago
|
||
(In reply to Ben Hearsum [:bhearsum] from comment #1) > https://nagios.mozilla.org/sentry/ shows the CDN being pulled out of the > pool just now, not sure if that's because of this 403 error or just the > normal reasons. trying to decipher this bug and make sense with all of the changes from our plan yesterday, the sentry quote scares me but everything seems normal there.
Assignee | ||
Updated•12 years ago
|
Assignee: server-ops-webops → cshields
Comment 5•12 years ago
|
||
This is expected. The CDN is not supposed to serve anything for nightly and aurora, due to bug 798486. Instead, bouncer should be sending you to the FTP cluster for these requests. If it's not, something is wrong. If you're querying download.cdn.mozilla.net directly (without bouncer sending you there) *you're* wrong. :) To be clear: 403 is intended behavior, and these files are not intended to be accessed via CDN anymore. Use the FTP cluster, or use bouncer. I'm getting on a plane and cannot troubleshoot further at this time.
Reporter | ||
Comment 6•12 years ago
|
||
(In reply to Jake Maul [:jakem] from comment #5) > This is expected. The CDN is not supposed to serve anything for nightly and > aurora, due to bug 798486. Instead, bouncer should be sending you to the FTP > cluster for these requests. If it's not, something is wrong. If you're > querying download.cdn.mozilla.net directly (without bouncer sending you > there) *you're* wrong. :) Bouncer sent us there. Should we be setting SSL-only on these products to restrict it to ftp only, or what?
Comment 7•12 years ago
|
||
SSL-only should not be set for *-latest products... those are full installers, and are expected to be accessible with or without SSL. Most likely this is the psuedo-"race condition" I alluded to in bug 798486. A cache flush on the CDN mirror should fully eliminate it from usage. I have already issued this... takes 30-60min to take full effect.
Assignee | ||
Comment 8•12 years ago
|
||
issue now is that akamai works, but edgecast is 403.. investigating
Comment 9•12 years ago
|
||
I committed the following to the productdelivery cluster + # disabled by solarce for troubleshooting 801017 + #RewriteRule ^/pub/(mozilla.org)*/firefox/nightly/.*$ - [forbidden] In trunk ❯ svn ci -m "temporary for fixing 801017" Sending trunk/modules/webapp/files/ftp/etc-httpd/domains/download.cdn.mozilla.net.conf Transmitting file data . Committed revision 49766. I've confirmed both Akamai and Edgecast IPs are returning a nightly download Akamai: curl -v -H "Host: download.cdn.mozilla.net" http://65.171.167.131/pub/mozilla.org/firefox/nightly/latest-mozilla-central/firefox-19.0a1.en-US.win32.installer.exe -o /tmp/foo Edgecast: curl -v -H "Host: download.cdn.mozilla.net" http://72.21.81.253/pub/mozilla.org/firefox/nightly/latest-mozilla-central/firefox-19.0a1.en-US.win32.installer.exe Both return the install for me
Comment 10•12 years ago
|
||
Also, I initiated a Edgecast purge of download.cdn.mozilla.net earlier 2012-10-12 12:13 PM http://download.cdn.mozilla.net/* 2012-10-12 12:15 PM
Comment 11•12 years ago
|
||
curl -IL http://download.cdn.mozilla.net/pub/mozilla.org/firefox/nightly/latest-mozilla-central/firefox-19.0a1.en-US.win32.installer.exe HTTP/1.1 200 OK Server: Apache X-Backend-Server: ftp1.dmz.scl3.mozilla.com Content-Type: application/octet-stream Accept-Ranges: bytes Access-Control-Allow-Origin: * ETag: "1905de1-1345508-4cba004ac2777" Last-Modified: Tue, 09 Oct 2012 13:07:20 GMT Content-Length: 20206856 X-Cache-Info: cached Cache-Control: max-age=72986 Expires: Sat, 13 Oct 2012 16:30:52 GMT Date: Fri, 12 Oct 2012 20:14:26 GMT Connection: keep-alive Works for me now
Comment 12•12 years ago
|
||
Going to RF this, final apache config will be decided in bug 798486
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Updated•12 years ago
|
Whiteboard: [stub+] → [stub+][push-interrupt]
Verified FIXED; this is good again (automation is happy).
Status: RESOLVED → VERIFIED
Updated•11 years ago
|
Component: Server Operations: Web Operations → WebOps: Other
Product: mozilla.org → Infrastructure & Operations
Updated•5 years ago
|
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in
before you can comment on or make changes to this bug.
Description
•