801017 - cdn returning 403 errors for installers

Status: VERIFIED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[[e-mail] (:bhearsum)]

Might be related to changes in bug 798486. Blocking stub installer, which, apparently we're  shipping OMGRIGHTNOW.

➜  ~  curl -IL http://download.cdn.mozilla.net/pub/mozilla.org/firefox/nightly/latest-mozilla-central/firefox-19.0a1.en-US.win32.installer.exe
HTTP/1.1 403 Forbidden
Server: Apache
X-Backend-Server: ftp2.dmz.scl3.mozilla.com
Content-Type: text/html; charset=iso-8859-1
X-Cache-Info: cached
Content-Length: 1
Date: Fri, 12 Oct 2012 17:49:12 GMT
Connection: keep-alive


Permissions on the filesystem look like this:
/pub/mozilla.org/firefox/nightly/latest-mozilla-central
-bash-4.1$ ls -l firefox-19.0a1.en-US.win32.installer.exe
-rw-r--r-- 3 ffxbld firefox 20218648 Oct 12 06:08 firefox-19.0a1.en-US.win32.installer.exe

Comment 1

[[e-mail] (:bhearsum)]

https://nagios.mozilla.org/sentry/ shows the CDN being pulled out of the pool just now, not sure if that's because of this 403 error or just the normal reasons.

Comment 2

[Jason Smith [:jsmith]]

Blocker cause this causes the stub to fail intermittently to download the bits for a channel. Reproduced this 3 times so far.

Comment 3

[Stephen Donner [:stephend] Not actively reading bugmail]

Was just told by on-call (Aj) that WebOps is aware and is looking into this.

Comment 4

[Corey Shields [:cshields]]

(In reply to Ben Hearsum [:bhearsum] from comment #1)
> https://nagios.mozilla.org/sentry/ shows the CDN being pulled out of the
> pool just now, not sure if that's because of this 403 error or just the
> normal reasons.

trying to decipher this bug and make sense with all of the changes from our plan yesterday, the sentry quote scares me but everything seems normal there.

Comment 5

[Jake Maul [:jakem]]

This is expected. The CDN is not supposed to serve anything for nightly and aurora, due to bug 798486. Instead, bouncer should be sending you to the FTP cluster for these requests. If it's not, something is wrong. If you're querying download.cdn.mozilla.net directly (without bouncer sending you there) *you're* wrong. :)

To be clear: 403 is intended behavior, and these files are not intended to be accessed via CDN anymore. Use the FTP cluster, or use bouncer.


I'm getting on a plane and cannot troubleshoot further at this time.

Comment 6

[[e-mail] (:bhearsum)]

(In reply to Jake Maul [:jakem] from comment #5)
> This is expected. The CDN is not supposed to serve anything for nightly and
> aurora, due to bug 798486. Instead, bouncer should be sending you to the FTP
> cluster for these requests. If it's not, something is wrong. If you're
> querying download.cdn.mozilla.net directly (without bouncer sending you
> there) *you're* wrong. :)

Bouncer sent us there. Should we be setting SSL-only on these products to restrict it to ftp only, or what?

Comment 7

[Jake Maul [:jakem]]

SSL-only should not be set for *-latest products... those are full installers, and are expected to be accessible with or without SSL.

Most likely this is the psuedo-"race condition" I alluded to in bug 798486. A cache flush on the CDN mirror should fully eliminate it from usage. I have already issued this... takes 30-60min to take full effect.

Comment 8

[Corey Shields [:cshields]]

issue now is that akamai works, but edgecast is 403..  investigating

Comment 9

[Brandon Burton [:solarce]]

I committed the following to the productdelivery cluster

+    # disabled by solarce for troubleshooting 801017
+    #RewriteRule ^/pub/(mozilla.org)*/firefox/nightly/.*$ - [forbidden]

In 

trunk ❯ svn ci -m "temporary for fixing 801017"
Sending        trunk/modules/webapp/files/ftp/etc-httpd/domains/download.cdn.mozilla.net.conf
Transmitting file data .
Committed revision 49766.

I've confirmed both Akamai and Edgecast IPs are returning a nightly download

Akamai: curl -v -H "Host: download.cdn.mozilla.net" http://65.171.167.131/pub/mozilla.org/firefox/nightly/latest-mozilla-central/firefox-19.0a1.en-US.win32.installer.exe -o /tmp/foo

Edgecast: curl -v -H "Host: download.cdn.mozilla.net" http://72.21.81.253/pub/mozilla.org/firefox/nightly/latest-mozilla-central/firefox-19.0a1.en-US.win32.installer.exe

Both return the install for me

Comment 10

[Brandon Burton [:solarce]]

Also, I initiated a Edgecast purge of download.cdn.mozilla.net earlier

2012-10-12 12:13 PM	http://download.cdn.mozilla.net/*	2012-10-12 12:15 PM

Comment 11

[raymond [:retornam] (inactive)]

curl -IL http://download.cdn.mozilla.net/pub/mozilla.org/firefox/nightly/latest-mozilla-central/firefox-19.0a1.en-US.win32.installer.exe
HTTP/1.1 200 OK
Server: Apache
X-Backend-Server: ftp1.dmz.scl3.mozilla.com
Content-Type: application/octet-stream
Accept-Ranges: bytes
Access-Control-Allow-Origin: *
ETag: "1905de1-1345508-4cba004ac2777"
Last-Modified: Tue, 09 Oct 2012 13:07:20 GMT
Content-Length: 20206856
X-Cache-Info: cached
Cache-Control: max-age=72986
Expires: Sat, 13 Oct 2012 16:30:52 GMT
Date: Fri, 12 Oct 2012 20:14:26 GMT
Connection: keep-alive


Works for me now

Comment 12

[Brandon Burton [:solarce]]

Going to RF this, final apache config will be decided in bug 798486

Comment 13

[Stephen Donner [:stephend] Not actively reading bugmail]

Verified FIXED; this is good again (automation is happy).