1191581 - Use Django Cache Machine to cache frequent queries (e.g. refdata lookups)

Status: RESOLVED WONTFIX

(Tree Management :: Treeherder, defect)

Description

[Ed Morley [:emorley]]

We currently manually handle caching for Datasources:
https://github.com/mozilla/treeherder/blob/3b3f3fabd6edadd3d813b5896314195073d8f8ed/treeherder/model/models.py#L166-L175

However this is slightly broken at the moment (bug 1187304).

In addition, I wonder if we want to start using caching for more models (eg some of the reference data).

There are packages out there that could do this for us in a more automatic manner.

eg:
http://tech.yipit.com/2011/12/07/django-queryset-cachinge280a6or-how-we-slayed-the-beast-known-as-mysql-joins/

http://jbalogh.me/2010/02/09/cache-machine/
https://cache-machine.readthedocs.org/en/latest/

https://pythonhosted.org/johnny-cache/

https://github.com/dziegler/django-cachebot


Cache Machine looks promising - the only thing is that it apparently doesn't currently support QuerySet.values or QuerySet.values_list:
https://cache-machine.readthedocs.org/en/latest/#cache-manager

...however we could rework some of the uses of these - or else even add support for them to Cache Manager ourselves.

Mauro - what are your thoughts? :-)

Comment 1

[Mauro Doglio [:mdoglio]]

There's a comparison grid on https://www.djangopackages.com/grids/g/caching/. I would go with django-cache-machine because it seems to do what we need and other mozilla web properties (e.g. zamboni) use it.

Comment 3

[William Lachance (:wlach)]

I've been experimenting with Django cache machine with excellent results.

Comment 4

[William Lachance (:wlach)]

Attachment: Screen Shot 2016-08-26 at 2.04.18 PM.png

On further investigation I'm not seeing much benefit from using memcache to cache refdata lookups, at least. I think MySQL's built-in cache is already pretty good with this type of information, and thus there's not a huge benefit to taking it out of the loop. 

Attached is a screenshot of the top database transactions on stage (where I was testing this work). The retrieval patterns are pretty much the same before & after.

Comment 5

[GitHub Autolander Bot]

Attachment: [treeherder] mozilla:stage > mozilla:master

Comment 6

[William Lachance (:wlach)]

Going to close this as wontfix for now, we can always reopen if we have a workload which benefits from db caching in the future.