498641 - Create a cross-repo pushlog to see all locales's checkins

Status: NEW

(Developer Services :: Mercurial: hg.mozilla.org, defect, P3)

Description

[Armen [:armenzg]]

Having a push-log that shows the checkins of all locales will help us with polling for changes on the different locales.

We currently poll more than 70 different repos for 2 branches. This actually hits the CPU for a little less than a minute every 5 mins (see bug 498468 for considerations on increasing that time).

The long term solution would be to just check in one place.

Comment 1

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

mrz, ted: is this something usually done by IT or RelEng or Ted-the-solver-of-all-problems? :-)

Comment 2

[Axel Hecht [:Pike]]

The code side is mostly ted, with some patches from folks like me right now. Deployment is IT.

I expect that this system will be something related to what I have running on the l10n server, based on my chat with ted at the last onsite.

The princples of the schema are not totally figured out, in particular if we want the parent info in the database scheme, which may or may not happen at the webdev chat tomorrow.

Comment 3

[(not currently active) Ted Mielczarek]

Makes sense to use what Axel already has working. I don't think it matters if this lives in RelEng or Hg: Customizations as long as someone is signed up to do the work.

Comment 4

[Armen [:armenzg]]

Would this be a good option to consider? http://bm-l10n-dashboard01/stage/pushes/

Comment 5

[Axel Hecht [:Pike]]

I actually am as far as having an hg push hook at http://hg.mozilla.org/l10n/django-site/file/default/pushes/hghooks.py, but aravind doesn't like installing django code on the hg server.

stage/pushes is currently working on a poller-based pushlog clone db, which isn't all too stable in the long run.

There are some lessons I learned recently about how to model stuff in django, and it might be worth looking at those, the copying the schema and trying to find someone to own writing to that schema without the db model at hand.

Comment 6

[Chris Cooper [:coop] (he/him)]

Mass move of bugs from Release Engineering:Future -> Release Engineering. See
http://coop.deadsquid.com/2010/02/kiss-the-future-goodbye/ for more details.

Comment 7

[Chris Cooper [:coop] (he/him)]

This landed on my plate during the recent old bug triage. 

Axel: hghooks.py looks recently updated in hg...is it still the way we want to go here? Has anything happened to make it more acceptable to IT in the intervening 10 months? What can I do to help drive this to completion?

Comment 8

[Axel Hecht [:Pike]]

Let's put it this way: hghooks.py is the code I can offer, and having it used upstream would obviously make a flock of infra decisions easier for the l10n dashboard.

I didn't invest into making it depend less on django, because I'm not the right person to do so. Replacing the django abstraction with raw SQL would require me to learn that, bad investment. If we'd want that, someone that's willing to maintain that code would need to take it forward, and I'd be happy to see it re-using the schema for the reason above.

Comment 9

[Chris Cooper [:coop] (he/him)]

(In reply to comment #5)
> I actually am as far as having an hg push hook at
> http://hg.mozilla.org/l10n/django-site/file/default/pushes/hghooks.py, but
> aravind doesn't like installing django code on the hg server.

aravind/mrz: is this still the case, re: no django code allowed on the hg server? Trying to figure out what our options are here.

Comment 10

[Aravind Gottipati [:aravind]]

(In reply to comment #9)
> 
> aravind/mrz: is this still the case, re: no django code allowed on the hg
> server? Trying to figure out what our options are here.

I don't mind having some stripped down django libraries for using in other stuff (like hooks).  However, I don't think that running an entire django instance on these servers is a good idea.  I really would like to keep the services on these servers limited to whats required for svn/hg to function and thats about it.  Also, these servers are python 2.4, if you need django libraries that depend on 2.6 etc.. that will be hard.

I am hesitant to do this is because it adds another software layer (that isn't maintained by rhel), that we have to keep track of, for security patches, etc.  And I am also against running a heavy app on these servers.  Also, if the hooks you are talking about, will add additional dependencies to the system (like dumping stuff into a db server etc), I would think twice before allowing them.

If we can run this django app elsewhere, that had read-only access to these repositories, that's fine by me.

Comment 11

[Chris Cooper [:coop] (he/him)]

From discussions during releng triage, I'm going to pass this over to IT, although it could equally go over to webdev, depending on who has the relevant skillset to take Axel's hghooks.py and convert it from django to SQLAlchemy (or similar).

From the releng side, we have a workaround in place to poll less frequently. This workaround is fine for us, but less good for l10n because they need to wait longer for changes to get noticed.

Comment 12

[Corey Shields [:cshields]]

(In reply to comment #11)
> From discussions during releng triage, I'm going to pass this over to IT,
> although it could equally go over to webdev, depending on who has the
> relevant skillset to take Axel's hghooks.py and convert it from django to
> SQLAlchemy (or similar).

We do not have the resources to do this.  Can you find someone in webdev to pass this on to, or should we WONTFIX this?

Comment 13

[Axel Hecht [:Pike]]

There are two new items since we made a decision last time:

- aravind left, so we could reconsider using the django sql abstraction on the mercurial server again
- we landed (and sadly bounced?) the mercurial pulse hooks, which, if working, could allow us to do the conversion into a queriable db somewhere else.
- and, still, look for someone to convert the code to something that doesn't sound so scary (I really think django just sounds scary)

Comment 14

[Phong Tran [:phong]]

Working on an upgrade and rolling into that.

Comment 15

[Ben Kero [:bkero]]

This seems like it would be trivial to implement once we migrate pushlog to an actual database. Marking this dependent on that bug.

Comment 16

[Gregory Szorc [:gps]]

It's been 5+ years. Could you please restate the use case and impact for this?

Comment 17

[Armen [:armenzg]]

Releng has been able to survive without for all these many years.
I don't know what value would actually give anymore.

Unless there are any objections I would ask to close this bug.

coop, are you fine if we close this?

Comment 18

[Axel Hecht [:Pike]]

Here'd be my take:

Both releng and I are constantly pounding our webheads to find new pushes, there's basically not a second that at least my infrastructure doesn't have an http request to a pushlog open.

Having a single database or upstream API that crosses repositories would be one way to reduce infrastructure load, and improve latency in the downstream automations. My cycle through the repos takes some 500 seconds, and that's just time that passes between pushes and the automation doing something with them.

Comment 19

[Chris Cooper [:coop] (he/him)]

gps is in the cc list for this bug. Now that he's part of dev services and can affect change here, maybe he can chime in with whether this is possible and how hard it might be to implement.

Comment 20

[Gregory Szorc [:gps]]

I think having a unified timeline of all changes to hg.mozilla.org repos since date X or push event Y is achievable and may naturally fall out of some of the Q1 work I'm doing to make replication more robust. But don't expect it to have the same rich query API we have today with json-pushes: it would likely only consist of a list of {date, repo, push ID, author, [changesets]} and the ability to query said list from an offset.

Also, hg.mozilla.org is grossly over-provisioned these days. You should not be concerned about hammering the servers and creating load. What you should be concerned about is the efficiency of being able to discover pushes. We can't do that optimally today because you have to resort to brute force over all repos.

Here is another hypothetical: if you could get a "ping" when a repo is pushed to, would that help? push-based protocols aren't reliable, so you'd still need to have a system periodically polling. But it may help with some latency, which is a Q1 goal for parts of RelEng, no?