1476669 - Discuss: High volume endpoint for hg.mo?

Status: RESOLVED INACTIVE

(Developer Services :: Mercurial: hg.mozilla.org, enhancement)

Description

[Kyle Lahnakoski [:ekyle]]

ActiveData-ETL consists of many machines transforming data and requiring hg.mo to  help markup that data with repository information. The machines use Elasticsearch as a cache; reducing query load on hg.mo by sharing results from previous requests made by other machines. The Elasticsearch sometimes fails causing the ETL machines to ask hg.mo for the information; which causes high load.
https://bugzilla.mozilla.org/show_bug.cgi?id=1472251
https://bugzilla.mozilla.org/show_bug.cgi?id=1417720

This problem is currently mitigated by adding decentralized heuristics; like send-alert-and-wait-an-hour-if-es-is-down-before-trying-again. But these heuristics are messy; they make the code less readable as lines are added to catch-and-respond to various failure modes. These heuristics are prone to miss-optimization; Waiting an hour may be too long if there is one machine, but too short if there are 250 machines. Finally, the decentralized nature makes answering "how many requests are you sending" a very difficult task; requiring summing log data across many spot (read ephemeral) machines. 

The TUID project now has the same problem, although not as distributed:
https://github.com/mozilla/TUID/issues/60

What are the possible solutions to this problem?
* Maybe we should centralize all requests so they can be rate-limited, cached, and measured:  https://github.com/mozilla/TUID/issues/65
* Maybe hg.mo can setup a dedicated mirror to handle the high load from specific applications
* Maybe a third party (me) can setup our own hg.mo mirrors
* Maybe there are better solutions?

gps, fubar: Thoughts?

Thank you

Comment 1

[Kendall Libby [:fubar] (he/him)]

I'm going to keep my input limited - once hgmo is stable in MDC1 I intend to start the process of transferring operational ownership to another team (TBD). 

I think I'd suggest looking for a way to have hgmo send its information to ES, rather than polling for data, or finding a better way to reduce your polling.

We have a reasonably strong system already, and it's easy enough for ActiveData to degrade the service, that trying to create a separate mirror will take considerable resources to handle the random load increase. That is, if things are normal, existing hgmo is fine and you don't need a mirror, but if things go wonky you're likely to crush whatever mirror you/we set up.

Comment 2

[Gregory Szorc [:gps]]

Mercurial is a distributed version control system. I have a strong preference for any downstream consumer that needs to bulk spider data to do so against a local clone. That way, there is a near 0% chance that they can impact the operational availability of https://hg.mozilla.org/.

I understand there are things that can make this difficult:

1) The repositories are large and require local storage.
2) There are N repositories on hg.mo that people usually care about.
3) We've customized hg.mozilla.org and sometimes consumers rely on those customizations.
4) Sometimes downstream consumers need to act on new data and a replication / latency problem arises.

That being said, you can make an argument that https://hg.mozilla.org/ should support all legitimate consumers and ActiveData, TUID, and others are legitimate consumers and we should adapt the service to support those consumers.

FWIW our plan of record is to establish read-only hg.mozilla.org mirrors for the private use of Firefox CI. At that point, traffic to the public https://hg.mozilla.org/ servers should not cause a CI outage and we'll be a bit more forgiving about high-traffic consumers. Of course, the same system to support private mirrors could also support private infrastructure for other consumers, such as ActiveData and TUID. I can't commit to doing that though: there are costs and complexities associated with it that would need to be thought through / justified.

Comment 3

[Kyle Lahnakoski [:ekyle]]

:fubar I like your idea of streaming the data to where it is needed, rather than polling. I have no idea where to begin, or even if it is possible, but it's the type of idea I am looking for.

gps: Thank you for mentioning "read-only hg.mozilla.org mirrors for the private use". Just knowing  something like that is possible is a big help.  

FYI: For this bug, I am expecting no commitment from either of you. I only want to explore what is possible, what is impossible, and ensure I am not overlooking existing solutions.

Thank you both!

Comment 4

[Kyle Lahnakoski [:ekyle]]

tomprince mentioned: Change Notifications — Mozilla Version Control Tools 0 documentation (https://mozilla-version-control-tools.readthedocs.io/en/latest/hgmo/notifications.html)

Comment 5

[Julien Cristau [:jcristau]]

Closing this since ActiveData is gone.