Tune down memcache on addons.mozilla.org

RESOLVED FIXED

Status

RESOLVED FIXED
11 years ago
4 years ago

People

(Reporter: alex, Assigned: oremj)

Tracking

Details

(Reporter)

Description

11 years ago
Hey guys. Lots of add-ons updated on March 20th are having corrupt hash issues upon download. See bug 424339.

We've been trying to diagnose the issue in bug 422562. However, this is nasty enough now that we need to expire existing caches and turn down the overall cache time period. I think it is at 2hrs now, and it should only be 1 min. 

It was originally turned up to 2hrs when services and addons were the same host. Now they have been split up -- so services can have the long cache -- and addons can have the short one.
(Assignee)

Comment 1

11 years ago
Has the cache namespace patch been landed?  Without that tweaking one config and not the other will probably just cause some items to be cached for 1 min then possibly recached for 2 hours. 

Also, even before services the addons master was being pushed to its limits.  We need to make sure almost 100% of selects are going to the slave databases ( where we have a little room for horizontal scalability ).  I remember work being done on that when we were having load problems, but am not sure what the status is now.

Comment 2

11 years ago
This will send load through the roof - can we have oremj to manual purges of the cache to help with testing?  I really would rather not basically turn off caching (1 min is almost as bad as off) for amo...other ideas?
(In reply to comment #1)
> Has the cache namespace patch been landed?  Without that tweaking one config
> and not the other will probably just cause some items to be cached for 1 min
> then possibly recached for 2 hours.
SAMO has it, but not 3.1.  However since at least one of them does you should be able to flip it.

> Also, even before services the addons master was being pushed to its limits. 
> We need to make sure almost 100% of selects are going to the slave databases (
> where we have a little room for horizontal scalability ).  I remember work
> being done on that when we were having load problems, but am not sure what the
> status is now.
 Large majority of search and addon controller traffic gets sent to read-only datasource.  So we should have both of these bases covered.
 

(Assignee)

Updated

11 years ago
Severity: critical → normal
Per Justin there's nothing we can do from the server end on this without absolutely killing performance on the servers (because it's tuned as high as it is now specifically to avoid killing the servers because it was killing them before).  This needs to be an application-side fix to flush the appropriate items from the cache when they get updated.
Status: NEW → RESOLVED
Last Resolved: 11 years ago
Resolution: --- → WONTFIX
There is a middle ground - don't know why it has to be 7200 seconds.  If possible we should lower this.  We have a bug filed to improve AMO's caching policy but until that is fixed (big-ish change) if we can cut the cache time it'd improve user experience.

60-7200 is a big range, is there a reason we can't do 1800 or 3600?
Status: RESOLVED → REOPENED
Resolution: WONTFIX → ---
(Assignee)

Updated

11 years ago
Assignee: server-ops → oremj
Status: REOPENED → NEW
(Assignee)

Comment 6

11 years ago
define('CACHE_PAGES_FOR', 3600);
Status: NEW → RESOLVED
Last Resolved: 11 years ago11 years ago
Resolution: --- → FIXED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.