Might be related to changes in bug 798486. Blocking stub installer, which, apparently we're shipping OMGRIGHTNOW. ➜ ~ curl -IL http://download.cdn.mozilla.net/pub/mozilla.org/firefox/nightly/latest-mozilla-central/firefox-19.0a1.en-US.win32.installer.exe HTTP/1.1 403 Forbidden Server: Apache X-Backend-Server: ftp2.dmz.scl3.mozilla.com Content-Type: text/html; charset=iso-8859-1 X-Cache-Info: cached Content-Length: 1 Date: Fri, 12 Oct 2012 17:49:12 GMT Connection: keep-alive Permissions on the filesystem look like this: /pub/mozilla.org/firefox/nightly/latest-mozilla-central -bash-4.1$ ls -l firefox-19.0a1.en-US.win32.installer.exe -rw-r--r-- 3 ffxbld firefox 20218648 Oct 12 06:08 firefox-19.0a1.en-US.win32.installer.exe
https://nagios.mozilla.org/sentry/ shows the CDN being pulled out of the pool just now, not sure if that's because of this 403 error or just the normal reasons.
Blocker cause this causes the stub to fail intermittently to download the bits for a channel. Reproduced this 3 times so far.
Was just told by on-call (Aj) that WebOps is aware and is looking into this.
Assignee: server-ops → server-ops-webops
Component: Server Operations → Server Operations: Web Operations
(In reply to Ben Hearsum [:bhearsum] from comment #1) > https://nagios.mozilla.org/sentry/ shows the CDN being pulled out of the > pool just now, not sure if that's because of this 403 error or just the > normal reasons. trying to decipher this bug and make sense with all of the changes from our plan yesterday, the sentry quote scares me but everything seems normal there.
This is expected. The CDN is not supposed to serve anything for nightly and aurora, due to bug 798486. Instead, bouncer should be sending you to the FTP cluster for these requests. If it's not, something is wrong. If you're querying download.cdn.mozilla.net directly (without bouncer sending you there) *you're* wrong. :) To be clear: 403 is intended behavior, and these files are not intended to be accessed via CDN anymore. Use the FTP cluster, or use bouncer. I'm getting on a plane and cannot troubleshoot further at this time.
(In reply to Jake Maul [:jakem] from comment #5) > This is expected. The CDN is not supposed to serve anything for nightly and > aurora, due to bug 798486. Instead, bouncer should be sending you to the FTP > cluster for these requests. If it's not, something is wrong. If you're > querying download.cdn.mozilla.net directly (without bouncer sending you > there) *you're* wrong. :) Bouncer sent us there. Should we be setting SSL-only on these products to restrict it to ftp only, or what?
SSL-only should not be set for *-latest products... those are full installers, and are expected to be accessible with or without SSL. Most likely this is the psuedo-"race condition" I alluded to in bug 798486. A cache flush on the CDN mirror should fully eliminate it from usage. I have already issued this... takes 30-60min to take full effect.
issue now is that akamai works, but edgecast is 403.. investigating
I committed the following to the productdelivery cluster + # disabled by solarce for troubleshooting 801017 + #RewriteRule ^/pub/(mozilla.org)*/firefox/nightly/.*$ - [forbidden] In trunk ❯ svn ci -m "temporary for fixing 801017" Sending trunk/modules/webapp/files/ftp/etc-httpd/domains/download.cdn.mozilla.net.conf Transmitting file data . Committed revision 49766. I've confirmed both Akamai and Edgecast IPs are returning a nightly download Akamai: curl -v -H "Host: download.cdn.mozilla.net" http://220.127.116.11/pub/mozilla.org/firefox/nightly/latest-mozilla-central/firefox-19.0a1.en-US.win32.installer.exe -o /tmp/foo Edgecast: curl -v -H "Host: download.cdn.mozilla.net" http://18.104.22.168/pub/mozilla.org/firefox/nightly/latest-mozilla-central/firefox-19.0a1.en-US.win32.installer.exe Both return the install for me
Also, I initiated a Edgecast purge of download.cdn.mozilla.net earlier 2012-10-12 12:13 PM http://download.cdn.mozilla.net/* 2012-10-12 12:15 PM
curl -IL http://download.cdn.mozilla.net/pub/mozilla.org/firefox/nightly/latest-mozilla-central/firefox-19.0a1.en-US.win32.installer.exe HTTP/1.1 200 OK Server: Apache X-Backend-Server: ftp1.dmz.scl3.mozilla.com Content-Type: application/octet-stream Accept-Ranges: bytes Access-Control-Allow-Origin: * ETag: "1905de1-1345508-4cba004ac2777" Last-Modified: Tue, 09 Oct 2012 13:07:20 GMT Content-Length: 20206856 X-Cache-Info: cached Cache-Control: max-age=72986 Expires: Sat, 13 Oct 2012 16:30:52 GMT Date: Fri, 12 Oct 2012 20:14:26 GMT Connection: keep-alive Works for me now
Going to RF this, final apache config will be decided in bug 798486
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
Verified FIXED; this is good again (automation is happy).
Status: RESOLVED → VERIFIED
Component: Server Operations: Web Operations → WebOps: Other
Product: mozilla.org → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.