Closed Bug 654678 Opened 15 years ago Closed 15 years ago

addons.allizom.org has ssl_error_bad_cert_domain

Categories

(mozilla.org Graveyard :: Server Operations, task)

Product:
mozilla.org Graveyard
Component:
Server Operations
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
VERIFIED FIXED

People

(Reporter: linostar, Assigned: nmaul)

References

Details

which is hindering the proper usage of the site.
Severity: normal → critical
Anas: we're testing a new CDN. IT: A lot of people aside from the devs use the staging site. What's the long term plan here?
Assignee: nobody → server-ops
Severity: critical → normal
Component: Public Pages → Server Operations
Product: addons.mozilla.org → mozilla.org
QA Contact: web-ui → mrz
Version: unspecified → other
Assignee: server-ops → jeremy.orem+bugs
Assignee: jeremy.orem+bugs → nmaul
(In reply to comment #1) > IT: A lot of people aside from the devs use the staging site. What's the long > term plan here? Can you explain this a bit more, I'm probably missing the big picture here. Why is stage used by non-devs instead of prod?
The long term plan is whatever we want it to be (that is to say, there isn't one yet). I see a few options: 1) Do nothing. It works, but is annoying. It mirrors production fairly well- same basic layout, same CDN provider / product. 2) Pay Edgecast to add the staging name to the cert. Not sure this is really on the table, because I don't know what the cost is. 3) Switch addons.allizom.org over to use the Edgecast DNS name instead of a mozilla name. No error, but presents a non-mozilla property to visitors- they won't see it in the URL bar, of course, but they will in the page source, or NoScript. This is just a STATIC_URL change (and then the elimination of the now-unused DNS records on our end), and thus can be done very quickly. 4) Change staging to a different, cheaper SSL CDN... Edgecast does offer one that Security was not comfortable with for production, but maybe would be okay with for staging. 5) Stop using CDN for staging, and go back to a simple STATIC_URL that ultimately points back to us directly. This is what we had before. Doesn't mirror production in some respects, but as long as it goes through Zeus it will (more or less) mirror most of the *effects* of a CDN, from a coding perspective. Thoughts?
(In reply to comment #2) > (In reply to comment #1) > > > IT: A lot of people aside from the devs use the staging site. What's the long > > term plan here? > > Can you explain this a bit more, I'm probably missing the big picture here. > Why is stage used by non-devs instead of prod? - localizers need it to see new strings - QA volunteers (QA has test days) - users looking for new features - Firefox devs testing new features (currently, blocklist) - security researchers testing new features - anyone who reports a bug who wants to see/verify the fix
> 1) Do nothing. It works, but is annoying. It mirrors production fairly well- > same basic layout, same CDN provider / product. > > 2) Pay Edgecast to add the staging name to the cert. Not sure this is really on > the table, because I don't know what the cost is. > > 3) Switch addons.allizom.org over to use the Edgecast DNS name instead of a > mozilla name. No error, but presents a non-mozilla property to visitors- they > won't see it in the URL bar, of course, but they will in the page source, or > NoScript. This is just a STATIC_URL change (and then the elimination of the > now-unused DNS records on our end), and thus can be done very quickly. > > 4) Change staging to a different, cheaper SSL CDN... Edgecast does offer one > that Security was not comfortable with for production, but maybe would be okay > with for staging. > > 5) Stop using CDN for staging, and go back to a simple STATIC_URL that > ultimately points back to us directly. This is what we had before. Doesn't > mirror production in some respects, but as long as it goes through Zeus it will > (more or less) mirror most of the *effects* of a CDN, from a coding > perspective. > > > Thoughts? #1 isn't good (see this bug). #3 sounds reasonable on the surface - I'm not sure what it means technically. #4 and #5 I don't like because it's different than production. #5 _sounds_ fine, but we found a few bugs when using the new CDN with the site which prevented us from launching last week. It makes me nervous to not have a way to test that stuff going forward in the future.
#3 just means that instead of: https://addons-cdn.allizom.org/css/files/styles.css they will see https://<some-edgecast-name>.com/path/to/our/stuff/css/files/style.css This would apply to all static content, including JS files... so NoScript users would have to whitelist it (but that goes with the territory if you're using NoScript- I don't see that as a major issue). I'm fine with this, and it'd be my personal choice as well. Stephen / mrz, do you have any concerns about this?
(In reply to comment #6) > #3 just means that instead of: > > https://addons-cdn.allizom.org/css/files/styles.css > > they will see > > https://<some-edgecast-name>.com/path/to/our/stuff/css/files/style.css > > This would apply to all static content, including JS files... so NoScript users > would have to whitelist it (but that goes with the territory if you're using > NoScript- I don't see that as a major issue). > > I'm fine with this, and it'd be my personal choice as well. > > Stephen / mrz, do you have any concerns about this? Just for staging, right? If so, it's not ideal, but it works. Krupa, would you be fine with that? Seems cosmetic, but I'm not sure.
Sounds worth trying #3 at least. Nothing to lose?
I am happy to try #3. Having the bad cert affects automation testing against staging, having to _hack_ the profile just to have coverage means if we move machines it impacts the ramp up.
mrz and I don't know of any reason this should be a problem, so I will make the change to settings_local.py on staging. Here is the change: #STATIC_URL = 'https://addons-cdn.allizom.org' STATIC_URL = 'https://gs1.adn.edgecastcdn.net/801237/addons-cdn.allizom.org' This should be pushed automatically in a few minutes...
Status: NEW → ASSIGNED
By the way, this requires zero changes on Edgecast's side... it's the same property, addons-cdn.allizom.org is just an alias for it. Any/all Edgecast settings for addons-cdn.allizom.org are already in place for this other name.
This change has propagated, and a lot of things are now requesting from the Edgecast property. However, for some reason, many things are still hitting addons-cdn.allizom.org. I suspect Zeus cache, or possibly Edgecast's own cache, may be caching old data. Okay to flush them?
With Wil's permission, I have flushed both the Zeus and Edgecast caches. However, I'm still seeing addons-cdn.allizom.org being used in some places. If you go to https://addons.allizom.org/, you should see it. Here's one such item: https://addons-cdn.allizom.org/media/img/zamboni/global/bg-header.png Are there any places other tan STATIC_URL that this needs to be adjusted? .htaccess files or something, perhaps?
looks like the vhost mentions addons-cdn quite a few times.
I believe what's happening is that Apache is redirecting traffic to addons-cdn.allizom.org, if it believes the traffic is of a certain type, or is in a certain location. Basically, there are RewriteRules that attempt to determine if a file should be served via the CDN... and RewriteRules that 1) deny anything coming to addons-cdn.allizom.org that shouldn't be, and 2) redirect everything that *doesn't* come to addons-cdn.allizom.org to there. I want to remove these rules, as well as the ServerAlias that defines addons-cdn.allizom.org. Here's the diff: =================================================================== --- preview.addons.mozilla.org.conf (revision 14352) +++ preview.addons.mozilla.org.conf (working copy) @@ -12,7 +12,7 @@ <VirtualHost *:80 *:81> ServerName addons.allizom.org - ServerAlias addons-preview-cdn.mozilla.net addons-cdn.allizom.org + ServerAlias addons-preview-cdn.mozilla.net SetEnv HTTPS on DocumentRoot /data/www/addons.mozilla.org-preview/site/app/webroot @@ -64,14 +64,6 @@ RewriteCond %{ENV:REQ_URI} ^/media/js/workers/.*$ RewriteRule . - [E=IS_CDN_REQ:false] - RewriteCond %{HTTP_HOST} ^addons-cdn.allizom.org$ - RewriteCond %{ENV:IS_CDN_REQ} !^true$ - RewriteRule . - [F] - - RewriteCond %{HTTP_HOST} !^addons-cdn.allizom.org$ - RewriteCond %{ENV:IS_CDN_REQ} ^true$ - RewriteRule /(.*) https://addons-cdn.allizom.org/$1 [R=301,L] - #Bug 580427 RewriteRule ^(?:/\w{2,3}(?:-\w{2})?)?/firefoxcup.*$ http://blog.mozilla.com/blog/2010/07/15/thanks-for-an-exciting-firefox-cup/ [R=301,L] =================================================================== Okay to commit and reload Apache?
Where will the rules live then, or what are you going to replace them with?
Actually I wasn't planning on replacing them with anything. The Edgecast CDN sends its requests to addons.allizom.org- there's no good way to differentiate which requests are from the CDN and which are normal requests not from the CDN. We could create an origin specifically for CDN traffic if we want: addons-origin.allizom.org, and have the CDN send its requests there. This would give you a way to differentiate which requests came from the CDN. With that, we could change the conditions to look for addons-origin as the HTTP_HOST, and change the RewriteRule to direct to 'https://gs1.adn.edgecastcdn.net/801237/addons-cdn.allizom.org/$1'. This would effectively duplicate the old behavior (pre-real-cdn). However, my real concern is why these rules would exist in the first place. Why would we be receiving requests for content to the wrong hosts? This is why I hadn't planned on replacing them- it seems like they should be unnecessary.
All of this stuff is in the same repository, so it's not really a right vs. wrong host, it's just that we want all static and/or user data coming from one of the hosts. The rewrites are there to guarantee that. Eg. This is my picture: https://static.addons.mozilla.net/img/uploads/userpics/0/10/10482.png If the rewrite wasn't there, it would also show up on: https://addons.mozilla.org/img/uploads/userpics/0/10/10482.png The picture is pretty innocent (since we sanitize them now) but consider other user content like add-ons and files (we have a file viewer built in to the site) that would have JS access to the mozilla.org domain as well as be on the add-on whitelist within all of Firefox. That's what we're avoiding by having the rewrites in place.
Our best bet then will be to more closely mirror prod. Namely, prod has: addons.mozilla.org - main site static.addons.mozilla.net - CDN origin site. Requests sent to static-cdn are ultimately served from here and cached on the CDN. static-cdn.addons.mozilla.net - CDN property. This is in STATIC_URL, and it's where all static traffic should be sent to. Based on your description, prod has a slight problem- those things you are referring to are not being redirected to the CDN, but rather to the "origin". The ideal would probably be for us to change prod's rewrites to be something like this: RewriteCond %{HTTP_HOST} !^static.addons.mozilla.net$ RewriteCond %{ENV:IS_CDN_REQ} ^true$ RewriteRule /(.*) https://static-cdn.addons.mozilla.net/$1 [R=301,L] (note the static vs static-cdn in the condition vs action) As for preview, this means we need to create 'addons-origin.allizom.org', make it an alias of the main site (like addons-cdn currently is), and change the rules to look something like this: RewriteCond %{HTTP_HOST} !^addons-origin.allizom.org$ RewriteCond %{ENV:IS_CDN_REQ} ^true$ RewriteRule /(.*) https://gs1.adn.edgecastcdn.net/801237/addons-cdn.allizom.org/$1 [R=301,L] Does that make sense?
It sounds right, but is something I'd want to try before we turned it on live. :) oremj wrote all the rewrite rules here - he might have more input.
See Also: → 655032
Talked this over with Jeremy, we're on the same page. I'll check in with you guys in a bit in IRC and make this change when you/QA is ready- this is on preview, of course. We'll be replacing the existing rules with similar ones that make sense for the Edgecast CDN. This will also mean the addition of the CNAME addons-origin.allizom.org, as a ServerAlias to this one, and then modifying the CDN to use that as the origin host. The DNS addition will be first, then the CDN change, then the Rewrite changes. Then once we've confirmed no more 'addons-cdn' requests, we can remove that from the ServerAlias line and from DNS. We may want to wait on this for a day or so, since DNS is involved.
(In reply to comment #22) This is Krupa's call from the QA side -- I don't have a strong opinion either way, but as long as the site is functionally the same and still uses the CDN, I don't see a problem with your proposal.
I'm okay with the proposal in comment 22. Is there a chance that this will break preview?
There is a chance it will, yes. However, the parts that might break preview are easily reversible, and would not require propagation delays to fix.
I'm okay with doing this tomorrow after 3.30pm. Does that work for everyone?
(In reply to comment #26) > I'm okay with doing this tomorrow after 3.30pm. Does that work for everyone? wfm
DNS for addons-origin.allizom.org added, and propagated ServerAlias for addons-origin.allizom.org added We're ready to update Edgecast and change over the mod_rewrite rules tomorrow. I was mistaken in my earlier statement- there is 1 setting that requires propagation, and it's the Edgecast origin host. We cannot change that now as I'd intended, as doing so would cause a loop, due to the second rewrite below. So, we will want to do this in 3 steps: 1) Update the first rule and disable the second. 2) Update Edgecast's origin... 1 hour or less propagation delay. 3) Update the second rule and re-enable it. The rollback is simply the reverse- revert/disable the second rule, change Edgecast back, revert the first rule, and enable the second rule. It sounds kinda complicated, but it's actually very simple. If we care enough, we can work up the syntax to make rule 2 work in both cases and not need the enable/disable steps. The new rules will be: - RewriteCond %{HTTP_HOST} ^addons-cdn.allizom.org$ + RewriteCond %{HTTP_HOST} ^addons-origin.allizom.org$ RewriteCond %{ENV:IS_CDN_REQ} !^true$ RewriteRule . - [F] - RewriteCond %{HTTP_HOST} !^addons-cdn.allizom.org$ + RewriteCond %{HTTP_HOST} !^addons-origin.allizom.org$ RewriteCond %{ENV:IS_CDN_REQ} ^true$ - RewriteRule /(.*) https://addons-cdn.allizom.org/$1 [R=301,L] + RewriteRule /(.*) https://gs1.adn.edgecastcdn.net/801237/addons-cdn.allizom.org/$1 [R=301,L] Explanation: 1) If the request was made to addons-origin.allizom.org, but it's not supposed to be a CDN request, return 403 Forbidden. Only the CDN should be hitting that name. 2) If the request was *not* made to addons-origin.allizom.org, but it should have come from the CDN, redirect to the CDN.
1 and 2 above are completed. Additionally, we discovered that the MEDIA_URL setting was still referencing 'addons-cdn.allizom.org'... this was responsible for a lot of the traffic to that name. It's been fixed also,. However, the Edgecast CDN is not propagating the origin Host change. It's still sending requests to our servers with a Host header of 'addons-cdn.allizom.org'. This should have completed propagation over a half hour ago, so I have opened a ticket with Edgecast about it. I will keep this bug open until that gets solved, but at this point the focus of this bug is solved- the cert error should be gone.
Edgecast replied and said that the propagation was delayed, but did not elaborate as to why. In any case, this is resolved. I have completed step 3 above. Based on traffic to the server, it is no longer being used (as expected). I have thus removed the 'addons-cdn.allizom.org' "Edge Cname" in Edgecast's config, and removed the 'addons-cdn.allizom.org' DNS record entirely, and removed the ServerAlias from the Apache config. It might be nice to rename this property in Edgecast, as the directory is still named 'addons-cdn.allizom.org'. This is a cosmetic change only though, and when looking at the Edgecast control panel it's pretty obvious what's really going on... especially since 'addons-cdn.allizom.org' no longer exists anywhere else. I'm going to close this bug and possibly open a new one for that at a later date.
Status: ASSIGNED → RESOLVED
Closed: 15 years ago
Resolution: --- → FIXED
Our automation hasn't caught any CDN/static-resources regressions/issues, so this can be verified if Krupa hasn't noticed any fallout. I'll note that I've asked around, and tested myself, and the CDN is working just fine on staging for mobile phones, too, where before we were seeing the cert issues.
You do not see cert issues with new profiles or installs anymore. Marking this verified.
Status: RESOLVED → VERIFIED
Thanks for the fix! Our todays Mozmill testrun now passes without leaving running instances behind due to the unexpected modal cert dialog.
OS: Linux → All
Hardware: x86 → All
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.