Closed Bug 946845 Opened 12 years ago Closed 11 years ago

Front-end cache URLs on -dev/-altdev for homepage/category/app detail API endpoints

Categories

(Cloud Services :: Operations: Marketplace, task, P2)

task

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: cvan, Assigned: oremj)

Details

(Whiteboard: waiting on akamai)

Let's start (on Marketplace -dev/-altdev) front-end caching the GET responses for these following API URLs: /api/v1/fireplace/search/featured/ /api/v1/apps/category/ /api/v1/apps/search/ It should be easily to disable in cache we have problems (pun intended). From my nginx configuration, the rule went something like this: location ~ '^/api/v\d+/(fireplace/search/featured/|apps/category/|apps/search/)' { proxy_pass https://proxy; break; } Zeus, varnish, nginx, whatever works. We could start with a TTL of a few minutes, but we ought to try something like an asynchronous cache refresh before rolling this out to production. Thanks, and let me know if you have any questions about this. - I built a cache-proxy prototype in node: https://github.com/cvan/rapidorapido#how-it-works Any cache hits - the responses for any subsequent requests (until 60 seconds have elapsed) - are retrieved from the cache (redis) and delivered immediately; asynchronously the cache is refreshed (if the content has changed) with that URL's response and set to expire again in 60 seconds. This way the content is never stale because every request is actually refreshing the cache for that given URL. There's also a TTL for every cached response (I've set my default locally to 5 minutes; you can remove the TTL setting or lower it as you please).
Assignee: server-ops-amo → oremj
Added caching on dev in https://github.com/mozilla-services/svcops-puppet/commit/a61b4edbada807b262405aab14314c296e880c77 Are all of these varies actually needed? Vary: Accept, API-Filter, Accept-Language, Cookie, X-Mobile, User-Agent
Accept-Language and X-Mobile no
Thanks!! :] https://marketplace-dev.allizom.org/api/v1/fireplace/search/featured/ Before Response time: 2.43 secs After Response time: 1.97 secs https://marketplace-dev.allizom.org/api/v1/apps/category/ Before Response time: 1.57 secs After Response time: 0.18 secs https://marketplace-dev.allizom.org/api/v1/apps/search/ Before Response time: 2.23 secs After Response time: 1.33 secs I'm trying to understand how it's actually getting cached in nginx. Is this all stored in memcache or what?
When the load balancer sees HTTP cache headers, it should honor them.
(In reply to Christopher Van Wiemeersch [:cvan] from comment #3) > Thanks!! :] > > https://marketplace-dev.allizom.org/api/v1/fireplace/search/featured/ > Before Response time: 2.43 secs > After Response time: 1.97 secs > > https://marketplace-dev.allizom.org/api/v1/apps/category/ > Before Response time: 1.57 secs > After Response time: 0.18 secs > > > https://marketplace-dev.allizom.org/api/v1/apps/search/ > Before Response time: 2.23 secs > After Response time: 1.33 secs > > > I'm trying to understand how it's actually getting cached in nginx. Is this > all stored in memcache or what? Jeremy, why are these speeds still so high? Do you have ideas?
Flags: needinfo?(oremj)
Can you send expires headers from the app?
Flags: needinfo?(oremj)
Flags: needinfo?(cvan)
(In reply to Jeremy Orem [:oremj] from comment #6) > Can you send expires headers from the app? Why exactly? Is there any reason why it's still slower from nginx on our boxes vs. from a node proxy on basta's cheap linode server? BEFORE: URL: https://marketplace.firefox.com (production with appcached assets - CSS, JS, favicons, and images defined in CSS) Logs: http://dpaste.com/1493082/ https://marketplace-dev.allizom.org/api/v1/fireplace/search/featured/ Response time: 2.43 secs https://marketplace-dev.allizom.org/api/v1/apps/category/ Response time: 1.57 secs https://marketplace-dev.allizom.org/api/v1/apps/search/ Response time: 2.23 secs AFTER: URL: http://mmmmarketplace.firefox.com (my local Marketplace, also appcached, using this nginx configuration https://gist.github.com/cvan/7794203) Logs: http://dpaste.com/1493112/ http://serverboy.net/api/v1/fireplace/search/featured/ Response time: 0.36 secs http://serverboy.net/api/v1/apps/category/ Response time: 0.10 secs http://serverboy.net/api/v1/apps/search/ Response time: 0.28 secs
Flags: needinfo?(cvan)
The expires header will tell our frontend cache that the URL is cacheable. The response time differences could be a number of things. Let's do a few tests and discuss your set up tomorrow, if you are available, on IRC.
Priority: -- → P2
Whiteboard: waiting on akamai
We are caching these on the CDN now.
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Component: Server Operations: AMO Operations → Operations: Marketplace
Product: mozilla.org → Mozilla Services
You need to log in before you can comment on or make changes to this bug.