975573 - Provide HTTP caching layer for outbound requests

Status: RESOLVED FIXED

(Cloud Services :: Operations: Marketplace, task)

Description

[Austin King [:ozten]]

The original proposal for the APK Factory service had suggested a HTTP reverse caching layer using Squid.

Why do we need this?

The Factory makes two kinds of outbound requests:
1) manifest urls
2) packages (zip files)

The APK Factory provides an "App Update" API which takes a list of App manifest urls and version numbers. It then checks to see if any of the applications on the public internet have been updated or changed since the last build.

If a user installs 6 packaged apps, we have to make 12 calls to find out that their app has not changed. These calls are getting the manifest url and getting the packages.

Squid would provide a way to do the following:
1) share HTTP request caching across N servers
2) Respect HTTP Cache headers robustly
3) Give items a default TTL

If we don't want to go the Squid route, any suggestions?

Gene Wood is familiar with a similar deployment for Persona.

Thanks!

Comment 1

[Austin King [:ozten]]

This would improve the performance of Bug#958329.

Comment 2

[Austin King [:ozten]]

oremj had a genius idea of not using the package for cache invalidation logic. Only using the manifest which is much lighter weight.

clouserw said this is fine and it is how the Marketplace works today.

Comment 3

[Austin King [:ozten]]

I will add an LRU cache with a 1 minute timeout to where we download manifests as well as when we query the database for metadata.

Comment 4

[Austin King [:ozten]]

Fixed in 5dbec97003 of https://github.com/mozilla/apk-factory-service/pull/53

New Config value - manifestCacheTTL
New Schema migration

Details will be given with a deployment bug.

Holding off on merging to master and deploying until after MWC.