1223384 - Stop running clear_cache to clear memcached on stage/prod deploy

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Infrastructure, defect, P1)

Description

[Ed Morley [:emorley]]

On stage/prod deploy (but not Heroku), we run `./manage.py clear_cache`, which clears memcached.

This means that the builds-{4hr,running,pending} as well as pushlog ingestion tasks are all more demanding the next time they are run, since they can't use the cached values.

The reason mentioned for doing this in the past is in case we change the meaning of one of the memcached keys in-code. However I would argue that we should *never* do that and keep the cache key name the same.

I think we should stop clearing the cache on deploy - I think it exacerbates the problems seen in bug 1223335

Mauro/Cameron: thoughts?

Comment 1

[Ed Morley [:emorley]]

(In reply to Ed Morley [:emorley] from comment #0)
> However I would argue that we should *never* do that and keep the cache key name the same.

To be clearer:
If we ever do that, we should always rename the key at the same time. eg when we switched from revision SHA to push ID we renamed the key rather than trying to use the prior name.

Comment 2

[Ed Morley [:emorley]]

Attachment: Stop running clear_cache on stage/prod deploy

Comment 3

[Cameron Dawson [:camd]]

Comment on attachment 8686007 [details] [review]
Stop running clear_cache on stage/prod deploy

Yeah, I agree.  I would imagine at the time that was implemented things were more in flux than they are today.  Like you say, we just need to be smarter about this rather than paying a penalty on EVERY deploy.

Comment 4

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/26946f754ca8b641807c98daebbed88c9a894859
Bug 1223384 - Stop running clear_cache on stage/prod deploy

Since it should be unnecessary, and causes additional load until the
cache is re-populated, since:
* the builds-{4hr,running,pending} tasks have to ingest all jobs,
  including those previously seen
* the pushlog task defaults back to "the last 10 pushes", which in the
  case of inactive repos, means extra busy work, and for active repos
  means we might actually miss commits (particularly if the prod push
  was to fix an issue that has say broken ingestion for the last X mins)

Comment 5

[Ed Morley [:emorley]]

Thank you for the reviews :-)