Closed
Bug 946845
Opened 12 years ago
Closed 11 years ago
Front-end cache URLs on -dev/-altdev for homepage/category/app detail API endpoints
Categories
(Cloud Services :: Operations: Marketplace, task, P2)
Cloud Services
Operations: Marketplace
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: cvan, Assigned: oremj)
Details
(Whiteboard: waiting on akamai)
Let's start (on Marketplace -dev/-altdev) front-end caching the GET responses for these following API URLs:
/api/v1/fireplace/search/featured/
/api/v1/apps/category/
/api/v1/apps/search/
It should be easily to disable in cache we have problems (pun intended).
From my nginx configuration, the rule went something like this:
location ~ '^/api/v\d+/(fireplace/search/featured/|apps/category/|apps/search/)' {
proxy_pass https://proxy;
break;
}
Zeus, varnish, nginx, whatever works.
We could start with a TTL of a few minutes, but we ought to try something like an asynchronous cache refresh before rolling this out to production.
Thanks, and let me know if you have any questions about this.
-
I built a cache-proxy prototype in node: https://github.com/cvan/rapidorapido#how-it-works
Any cache hits - the responses for any subsequent requests (until 60 seconds have elapsed) - are retrieved from the cache (redis) and delivered immediately; asynchronously the cache is refreshed (if the content has changed) with that URL's response and set to expire again in 60 seconds. This way the content is never stale because every request is actually refreshing the cache for that given URL.
There's also a TTL for every cached response (I've set my default locally to 5 minutes; you can remove the TTL setting or lower it as you please).
Assignee | ||
Updated•12 years ago
|
Assignee: server-ops-amo → oremj
Assignee | ||
Comment 1•12 years ago
|
||
Added caching on dev in https://github.com/mozilla-services/svcops-puppet/commit/a61b4edbada807b262405aab14314c296e880c77
Are all of these varies actually needed?
Vary: Accept, API-Filter, Accept-Language, Cookie, X-Mobile, User-Agent
Reporter | ||
Comment 2•12 years ago
|
||
Accept-Language and X-Mobile no
Reporter | ||
Comment 3•12 years ago
|
||
Thanks!! :]
https://marketplace-dev.allizom.org/api/v1/fireplace/search/featured/
Before Response time: 2.43 secs
After Response time: 1.97 secs
https://marketplace-dev.allizom.org/api/v1/apps/category/
Before Response time: 1.57 secs
After Response time: 0.18 secs
https://marketplace-dev.allizom.org/api/v1/apps/search/
Before Response time: 2.23 secs
After Response time: 1.33 secs
I'm trying to understand how it's actually getting cached in nginx. Is this all stored in memcache or what?
Assignee | ||
Comment 4•12 years ago
|
||
When the load balancer sees HTTP cache headers, it should honor them.
Reporter | ||
Comment 5•12 years ago
|
||
(In reply to Christopher Van Wiemeersch [:cvan] from comment #3)
> Thanks!! :]
>
> https://marketplace-dev.allizom.org/api/v1/fireplace/search/featured/
> Before Response time: 2.43 secs
> After Response time: 1.97 secs
>
> https://marketplace-dev.allizom.org/api/v1/apps/category/
> Before Response time: 1.57 secs
> After Response time: 0.18 secs
>
>
> https://marketplace-dev.allizom.org/api/v1/apps/search/
> Before Response time: 2.23 secs
> After Response time: 1.33 secs
>
>
> I'm trying to understand how it's actually getting cached in nginx. Is this
> all stored in memcache or what?
Jeremy, why are these speeds still so high? Do you have ideas?
Reporter | ||
Updated•12 years ago
|
Flags: needinfo?(oremj)
Assignee | ||
Updated•12 years ago
|
Flags: needinfo?(cvan)
Reporter | ||
Comment 7•12 years ago
|
||
(In reply to Jeremy Orem [:oremj] from comment #6)
> Can you send expires headers from the app?
Why exactly? Is there any reason why it's still slower from nginx on our boxes vs. from a node proxy on basta's cheap linode server?
BEFORE:
URL: https://marketplace.firefox.com (production with appcached assets - CSS, JS, favicons, and images defined in CSS)
Logs: http://dpaste.com/1493082/
https://marketplace-dev.allizom.org/api/v1/fireplace/search/featured/
Response time: 2.43 secs
https://marketplace-dev.allizom.org/api/v1/apps/category/
Response time: 1.57 secs
https://marketplace-dev.allizom.org/api/v1/apps/search/
Response time: 2.23 secs
AFTER:
URL: http://mmmmarketplace.firefox.com (my local Marketplace, also appcached, using this nginx configuration https://gist.github.com/cvan/7794203)
Logs: http://dpaste.com/1493112/
http://serverboy.net/api/v1/fireplace/search/featured/
Response time: 0.36 secs
http://serverboy.net/api/v1/apps/category/
Response time: 0.10 secs
http://serverboy.net/api/v1/apps/search/
Response time: 0.28 secs
Flags: needinfo?(cvan)
Assignee | ||
Comment 8•12 years ago
|
||
The expires header will tell our frontend cache that the URL is cacheable. The response time differences could be a number of things. Let's do a few tests and discuss your set up tomorrow, if you are available, on IRC.
Assignee | ||
Updated•11 years ago
|
Priority: -- → P2
Assignee | ||
Updated•11 years ago
|
Whiteboard: waiting on akamai
Assignee | ||
Comment 9•11 years ago
|
||
We are caching these on the CDN now.
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Updated•11 years ago
|
Component: Server Operations: AMO Operations → Operations: Marketplace
Product: mozilla.org → Mozilla Services
You need to log in
before you can comment on or make changes to this bug.
Description
•