443600 - Optimize HgLocalePoller and pushlog communication

Status: RESOLVED WORKSFORME

(Developer Services :: Mercurial: hg.mozilla.org, defect, P3)

Description

[Axel Hecht [:Pike]]

The HgLocalePoller in http://hg.mozilla.org/users/bsmedberg_mozilla.com/index.cgi/l10n-buildbot/file/7926829fa055/mozillal10n.py#l318 currently hits each l10n repository per se.

So we're talking about opening 50-100+ connections between the buildmaster and the hg server. Doing so in serial isn't great, and doing so in parallel isn't really either?

If we want to kick off nightly builds as triggered builds from en-US nightlies, which we do, then we'll likely even open these connections from our most precious master. Frank. N. Furter comes to my mind right now, excuse the distraction.

I would imagine that we could optimize this by extending the pushlog hook on http://hg.mozilla.org/users/bsmedberg_mozilla.com/index.cgi/hghooks/file/6209a348c992/mozhghooks/pushlog.py and to add an additional commit hook to all l10n repositories, adding themselves to a single db with an additional column for the locale.

Ted, Benjamin, one thing I don't know is if we can actually extend hgweb at a parent dir of repositories. Something like http://hg.mozilla.org/l10n-central/index.cgi/l10n-pushlog, picking up the style stuff from #developers.

I'll need you guys to tell me if that makes any sense at all, too.

And probably joduinn to say if it has to make sense in one way or another.

Comment 1

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Unclear what we are supposed to do here. Can we try getting l10n builds started first and then if this is really a problem, we can reopen this bug?

Comment 2

[Axel Hecht [:Pike]]

I have a buildbot running pounding those feeds and it responds quickly, so it doesn't seem to care too much.

Resolving WORKSFORME, in the right component though.