We had some failures to push to hg toady (e.g., bug 665077) that were due to problems with pulse due to a mis-ordering of timeouts. Fixing the timeouts is the short-term solution, but the long-term solution is to make the hg hook non-blocking, rather than using timeouts, so that it always completes regardless of the speed, state, upness, etc. of the pulse system. In releng, catlee's working on a system to dump pulse messages from the master into a maildir, with a distinct process picking up those files as they appear and sending them to pulse. We'll then monitor the size of that queue and the upness of the send-to-pulse daemon, and take appropriate action if either of those alerts, confident that any failures will not affect the buildmasters. Perhaps this makes a good model for the hg hook?
Either that, or we can stick the logic in another script and just exec it from the hook with appropriate options.
I assume you mean exec in the background? The worry then is that a processing backlog results in a full process table (impacting other operations) and no good way to unwedge things without losing data.
Talked about this with dustin and catlee; we should go with the maildir approach (called "QueueDir"). It's implemented here: http://hg.mozilla.org/build/tools/file/default/lib/python/mozilla_buildtools/queuedir.py And the BuildBot publisher uses it here: http://hg.mozilla.org/build/tools/file/67cca6c691a6/buildbot-helpers/pulse_publisher.py We just need a publishing shim that hooks into hg's QueueDir. catlee, what do the hg QueueDir messages look like (or do you have a link to a class that reads from hg's QueueDir)?
OS: Mac OS X → All
Priority: -- → P2
Hardware: x86 → All
I don't think there is a queuedir for hg at this point, is there? Maybe you mean pulse's queuedir?
I think I may be confused here. My understanding was that something in releng is using QueueDir, or a similar approach, with hg; in other words, there's already an hg server hook that can write to a QueueDir. If not, I imagine we'll have to write that in addition to a daemon that slurps them up and publishes them. Probably neither is terribly difficult, but I don't know for sure.
My reading of this bug is that it is related to a hook installed on the Mercurial server. Said hook is no longer deployed. So closing.
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Component: Pulse → Mercurial: hg.mozilla.org
Product: Webtools → Developer Services
QA Contact: hwine
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.