cdn returning 403 errors for installers

VERIFIED FIXED

Status

Infrastructure & Operations
WebOps: Other
--
blocker
VERIFIED FIXED
5 years ago
4 years ago

People

(Reporter: bhearsum, Assigned: cshields)

Tracking

Details

(Whiteboard: [stub+][push-interrupt])

(Reporter)

Description

5 years ago
Might be related to changes in bug 798486. Blocking stub installer, which, apparently we're  shipping OMGRIGHTNOW.

➜  ~  curl -IL http://download.cdn.mozilla.net/pub/mozilla.org/firefox/nightly/latest-mozilla-central/firefox-19.0a1.en-US.win32.installer.exe
HTTP/1.1 403 Forbidden
Server: Apache
X-Backend-Server: ftp2.dmz.scl3.mozilla.com
Content-Type: text/html; charset=iso-8859-1
X-Cache-Info: cached
Content-Length: 1
Date: Fri, 12 Oct 2012 17:49:12 GMT
Connection: keep-alive


Permissions on the filesystem look like this:
/pub/mozilla.org/firefox/nightly/latest-mozilla-central
-bash-4.1$ ls -l firefox-19.0a1.en-US.win32.installer.exe
-rw-r--r-- 3 ffxbld firefox 20218648 Oct 12 06:08 firefox-19.0a1.en-US.win32.installer.exe
(Reporter)

Comment 1

5 years ago
https://nagios.mozilla.org/sentry/ shows the CDN being pulled out of the pool just now, not sure if that's because of this 403 error or just the normal reasons.
Blocker cause this causes the stub to fail intermittently to download the bits for a channel. Reproduced this 3 times so far.
Whiteboard: [stub+]
Was just told by on-call (Aj) that WebOps is aware and is looking into this.
Assignee: server-ops → server-ops-webops
Component: Server Operations → Server Operations: Web Operations
(Assignee)

Comment 4

5 years ago
(In reply to Ben Hearsum [:bhearsum] from comment #1)
> https://nagios.mozilla.org/sentry/ shows the CDN being pulled out of the
> pool just now, not sure if that's because of this 403 error or just the
> normal reasons.

trying to decipher this bug and make sense with all of the changes from our plan yesterday, the sentry quote scares me but everything seems normal there.
(Assignee)

Updated

5 years ago
Assignee: server-ops-webops → cshields

Comment 5

5 years ago
This is expected. The CDN is not supposed to serve anything for nightly and aurora, due to bug 798486. Instead, bouncer should be sending you to the FTP cluster for these requests. If it's not, something is wrong. If you're querying download.cdn.mozilla.net directly (without bouncer sending you there) *you're* wrong. :)

To be clear: 403 is intended behavior, and these files are not intended to be accessed via CDN anymore. Use the FTP cluster, or use bouncer.


I'm getting on a plane and cannot troubleshoot further at this time.
(Reporter)

Comment 6

5 years ago
(In reply to Jake Maul [:jakem] from comment #5)
> This is expected. The CDN is not supposed to serve anything for nightly and
> aurora, due to bug 798486. Instead, bouncer should be sending you to the FTP
> cluster for these requests. If it's not, something is wrong. If you're
> querying download.cdn.mozilla.net directly (without bouncer sending you
> there) *you're* wrong. :)

Bouncer sent us there. Should we be setting SSL-only on these products to restrict it to ftp only, or what?

Comment 7

5 years ago
SSL-only should not be set for *-latest products... those are full installers, and are expected to be accessible with or without SSL.

Most likely this is the psuedo-"race condition" I alluded to in bug 798486. A cache flush on the CDN mirror should fully eliminate it from usage. I have already issued this... takes 30-60min to take full effect.
(Assignee)

Comment 8

5 years ago
issue now is that akamai works, but edgecast is 403..  investigating
I committed the following to the productdelivery cluster

+    # disabled by solarce for troubleshooting 801017
+    #RewriteRule ^/pub/(mozilla.org)*/firefox/nightly/.*$ - [forbidden]

In 

trunk ❯ svn ci -m "temporary for fixing 801017"
Sending        trunk/modules/webapp/files/ftp/etc-httpd/domains/download.cdn.mozilla.net.conf
Transmitting file data .
Committed revision 49766.

I've confirmed both Akamai and Edgecast IPs are returning a nightly download

Akamai: curl -v -H "Host: download.cdn.mozilla.net" http://65.171.167.131/pub/mozilla.org/firefox/nightly/latest-mozilla-central/firefox-19.0a1.en-US.win32.installer.exe -o /tmp/foo

Edgecast: curl -v -H "Host: download.cdn.mozilla.net" http://72.21.81.253/pub/mozilla.org/firefox/nightly/latest-mozilla-central/firefox-19.0a1.en-US.win32.installer.exe

Both return the install for me
Also, I initiated a Edgecast purge of download.cdn.mozilla.net earlier

2012-10-12 12:13 PM	http://download.cdn.mozilla.net/*	2012-10-12 12:15 PM
curl -IL http://download.cdn.mozilla.net/pub/mozilla.org/firefox/nightly/latest-mozilla-central/firefox-19.0a1.en-US.win32.installer.exe
HTTP/1.1 200 OK
Server: Apache
X-Backend-Server: ftp1.dmz.scl3.mozilla.com
Content-Type: application/octet-stream
Accept-Ranges: bytes
Access-Control-Allow-Origin: *
ETag: "1905de1-1345508-4cba004ac2777"
Last-Modified: Tue, 09 Oct 2012 13:07:20 GMT
Content-Length: 20206856
X-Cache-Info: cached
Cache-Control: max-age=72986
Expires: Sat, 13 Oct 2012 16:30:52 GMT
Date: Fri, 12 Oct 2012 20:14:26 GMT
Connection: keep-alive


Works for me now
Going to RF this, final apache config will be decided in bug 798486
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
Whiteboard: [stub+] → [stub+][push-interrupt]
Verified FIXED; this is good again (automation is happy).
Status: RESOLVED → VERIFIED
Component: Server Operations: Web Operations → WebOps: Other
Product: mozilla.org → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.