Closed Bug 1512346 Opened 4 years ago Closed 4 years ago

Cache raw files command responses

Categories

(Developer Services :: Mercurial: hg.mozilla.org, enhancement)

enhancement
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: gps, Assigned: gps)

References

(Blocks 1 open bug)

Details

Attachments

(1 file)

The Mercurial server doesn't cache wire protocol responses for the rawfiledata command. Try pushes reveal this to be a bottleneck. Caching the responses should make things substantially faster.

IIRC the reason I didn't implement caching in upstream Mercurial was I wasn't sure how to efficiently compute a cache key. Thinking about it some more, I think we should hash the DAG heads in the changelog and manifestlog. That should be sufficient to derive a suitable cache key.
Partial clones make a request to this API for the changelog + manifestlog
data. This API will generate a few hundred megabytes for Firefox repos.
Having the hg.mo servers in the critical path of serving that much data
would introduce a lot of data egress from the servers.

This commit teaches that wire protocol command to cache responses.
Combined with our S3 caching, this should result in much fewer bytes
being transmitted over the wire by hg.mo.

The added test extension is copied from the Mercurial repository.
Pushed by gszorc@mozilla.com:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/c66c9a8c33dc
hgmo: cache rawstorefiledata response ; r=sheehan
Status: ASSIGNED → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
Flags: needinfo?(Vip256123)
Flags: needinfo?(Vip256123)
You need to log in before you can comment on or make changes to this bug.