665118 - further decouple hg to pulse hook

Status: RESOLVED WONTFIX

(Developer Services :: Mercurial: hg.mozilla.org, defect, P2)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

We had some failures to push to hg toady (e.g., bug 665077) that were due to problems with pulse due to a mis-ordering of timeouts.  Fixing the timeouts is the short-term solution, but the long-term solution is to make the hg hook non-blocking, rather than using timeouts, so that it always completes regardless of the speed, state, upness, etc. of the pulse system.

In releng, catlee's working on a system to dump pulse messages from the master into a maildir, with a distinct process picking up those files as they appear and sending them to pulse.  We'll then monitor the size of that queue and the upness of the send-to-pulse daemon, and take appropriate action if either of those alerts, confident that any failures will not affect the buildmasters.  Perhaps this makes a good model for the hg hook?

Comment 1

[LegNeato]

Either that, or we can stick the logic in another script and just exec it from the hook with appropriate options.

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

I assume you mean exec in the background?  The worry then is that a processing backlog results in a full process table (impacting other operations) and no good way to unwedge things without losing data.

Comment 3

[Mark Côté [:mcote]]

Talked about this with dustin and catlee; we should go with the maildir approach (called "QueueDir").

It's implemented here: http://hg.mozilla.org/build/tools/file/default/lib/python/mozilla_buildtools/queuedir.py

And the BuildBot publisher uses it here: http://hg.mozilla.org/build/tools/file/67cca6c691a6/buildbot-helpers/pulse_publisher.py

We just need a publishing shim that hooks into hg's QueueDir.

catlee, what do the hg QueueDir messages look like (or do you have a link to a class that reads from hg's QueueDir)?

Comment 4

[Chris AtLee [:catlee]]

I don't think there is a queuedir for hg at this point, is there? Maybe you mean pulse's queuedir?

Comment 5

[Mark Côté [:mcote]]

I think I may be confused here.  My understanding was that something in releng is using QueueDir, or a similar approach, with hg; in other words, there's already an hg server hook that can write to a QueueDir.  If not, I imagine we'll have to write that in addition to a daemon that slurps them up and publishes them.  Probably neither is terribly difficult, but I don't know for sure.

Comment 6

[Gregory Szorc [:gps]]

My reading of this bug is that it is related to a hook installed on the Mercurial server. Said hook is no longer deployed. So closing.