1457306 - Put a CDN in front of more of normandy

Status: RESOLVED FIXED

(Cloud Services :: Operations: Normandy, task)

Description

[Brian Pitts]

Current confusing state of the world is summarized at https://docs.google.com/document/d/1FpHvB-Y_ZorRbb4O8cEIfaNoV9dylmZ3XfILEEUwrOU/edit#

Work to do is

1) Set settings in stage to its equivalent of normandy.services.mozilla.com then verify /api/v1/ now returns this domain for classify_client endpoint. Best not to go through cdn, even uncached, because of ip geolocation. Once tested good, change in prod.

2) Change cloudfront origin to normandy.services.mozilla.com

3) Figure out how to stop stripping cache-control in nginx and strip in cloudfront instead, so that cloudfront caches appropriately but no other middleboxes try to cache . lambda@edge might be a way. Set this up.

4) Change normandy.cdn.mozilla.net to point to the cloudfront distribution with a short TTL

5) Once traffic is gone, kill unused ELB

6) Change other settings to use normandy.cdn.mozilla.net Once rquests to that domian are gone, kill unused domain

Comment 1

[Brian Pitts]

#1 was done last week.

For #3 we're going to change the approach. I've filed bug 1471286 for it.

Comment 2

[Brian Pitts]

We did #3 last week. I just did #2 now.

We should be ready to do #4 and start sending more traffic through the CDN.

Comment 3

[Brian Pitts]

mythmon, I noticed that normandy is only returning cache-control for

action-signed
action-list
recipe-signed
recipe-list

It is not returning any cache-related headers for 

approvalrequest-list
reciperevision-list
filters

Without them, cloudfront will cache those results for its default tl, which is currently configured for 24 hours. Is that okay?

Comment 4

[Brian Pitts]

Attachment: normandy_url_test.py

Hits normandy bypassing clodufront once, then cloudfront three times with a 31 second sleep before the final request. Intent is to show how headers different with and without cloudfront and to verify cloudfront caching behavior.

Comment 5

[Michael Cooper [:mythmon]]

Thanks for checking the cache headers there.

"filters" isn't likely to change fast enough to matter, but "approvalrequest-list" and "reciperevision-list" probably shouldn't be cached for 24 hours. However, I don't think we can go ahead with the plan, assuming the admin-bearer host isn't going through the CDN.

Nothing that reads approval requests and revision lists (currently) reads from the CDN endpoint. Delivery console might soon read from the CDN urls, but it doesn't yet, and we can fix the cache headers before then. I'll file an issue to make sure we fix this on Normandy.

Comment 6

[Michael Cooper [:mythmon]]

I mis-edited my last email. I meant to say I think we *can* go ahead with the plan.

Comment 7

[Brian Pitts]

We plan to do this on Monday.

Right now

normandy-cloudfront.cdn.mozilla.net is an alias for d6wjo2hisqfy2.cloudfront.net

and

normandy.cdn.mozilla.net is an alias for normandy-cdn.services.mozilla.com.
normandy-cdn.services.mozilla.com is an alias for normandy-cdn-elb-prod-787321160.us-west-2.elb.amazonaws.com.

I will make normandy-cdn.services.mozilla.com also be an for d6wjo2hisqfy2.cloudfront.net.

Comment 8

[Brian Pitts]

We're now sending 10% or traffic through cloudfront (via weighted record sets in route 53) and it's looking good so far. Plan is to ramp up to 100% tomorrow.

Comment 9

[Brian Pitts]

We ramped up to 50% but then back down to 0% when we realized there were some issues. Chiefly, cloudfront doesn't pass the Host header though, and normandy.cdn.mozilla.net/api/v1 endpoint started returning urls with the domain normandy.services.mozilla.com instead of normandy.cdn.mozilla.net. Which kinda defeats the whole purpose of this.

Also, since Accept header wasn't being passed through we lost the pretty django rest framework view.

I realized I can't just pass the Host header through with the current config, because in some testing simulating that nginx started returning 404s when the host was normandy-cloudfront.cdn.mozilla.net which is used for actions which go through cloudfront now.

I'll get staging in better shape to test all this and update the course of action in the gdoc.

Comment 10

[Brian Pitts]

Attachment: test_cache.py

pytest for desired cache behavior

Comment 11

[Brian Pitts]

gdoc is up to data with the current config in stage and prod and next steps.

I've written a new set of tests to verify the behavior we want. They pass in stage.

Comment 12

[Brian Pitts]

All the work is done to start trying to shift traffic (via weighted dns) to cloudfront again. We'll target that for 9/13.

Comment 13

[Brian Pitts]

100% of traffic for normandy.cdn.mozilla.net is now routed through Cloudfront.

Comment 14

[Brian Pitts]

This has been good for a couple weeks so I'm going to close.