1277574 - Decide whether the Django filesystem-based log cache makes sense on Heroku

Status: RESOLVED DUPLICATE of bug 1328469

(Tree Management :: Treeherder, defect, P3)

Description

[Ed Morley [:emorley]]

For the log slice API, we currently cache the requested log file on the filesystem of that webhead, to speed up subsequent seeks within that log file:
https://github.com/mozilla/treeherder/blob/3402f00fbc99d336118f5cde0fe3c36ec8f11b76/treeherder/webapp/api/logslice.py#L70
https://github.com/mozilla/treeherder/blob/3402f00fbc99d336118f5cde0fe3c36ec8f11b76/treeherder/config/settings.py#L504-L512

For this to be effective:
1) Subsequent requests must be served by the same webhead (or at least all webheads will have to have their caches warmed)
2) The cache must exist for sufficient time relative to when requests are made (note also that currently MAX_ENTRIES is set to 1000 in settings.py)

Re #1: For current stage/prod, I'm not sure whether there is any persistence of requests across webheads. For Heroku there definitely isn't.

re #2: For current stage/prod, I don't believe there is any manual purging. For Heroku any time we do a deploy or after the 24hr automatic dyno restart, the cache is purged (since the filesystem isn't persistent).

Also, given that most of the logs are being hosted in AWS, it may be that the download times to Heroku are actually much less of an issue than they were to SCl3.

Given the above, it may be worth evaluating whether we:
1) Stop caching entirely
2) Switch to another cache type
3) Stick with the disk cache, but tweak MAX_ENTRIES and or TIMEOUT to more suitable values (ie: what's the disk size of a dyno? we don't want to fill it up accidentally)

Comment 1

[Ed Morley [:emorley]]

The logslice API is being removed in bug 1328469 :-)