further decouple hg to pulse hook

RESOLVED WONTFIX

Status

Developer Services
Mercurial: hg.mozilla.org
P2
normal
RESOLVED WONTFIX
7 years ago
3 years ago

People

(Reporter: dustin, Unassigned)

Tracking

Details

We had some failures to push to hg toady (e.g., bug 665077) that were due to problems with pulse due to a mis-ordering of timeouts.  Fixing the timeouts is the short-term solution, but the long-term solution is to make the hg hook non-blocking, rather than using timeouts, so that it always completes regardless of the speed, state, upness, etc. of the pulse system.

In releng, catlee's working on a system to dump pulse messages from the master into a maildir, with a distinct process picking up those files as they appear and sending them to pulse.  We'll then monitor the size of that queue and the upness of the send-to-pulse daemon, and take appropriate action if either of those alerts, confident that any failures will not affect the buildmasters.  Perhaps this makes a good model for the hg hook?

Comment 1

7 years ago
Either that, or we can stick the logic in another script and just exec it from the hook with appropriate options.
I assume you mean exec in the background?  The worry then is that a processing backlog results in a full process table (impacting other operations) and no good way to unwedge things without losing data.

Updated

4 years ago
Blocks: 1022701

Updated

4 years ago
Assignee: christian → nobody

Comment 3

4 years ago
Talked about this with dustin and catlee; we should go with the maildir approach (called "QueueDir").

It's implemented here: http://hg.mozilla.org/build/tools/file/default/lib/python/mozilla_buildtools/queuedir.py

And the BuildBot publisher uses it here: http://hg.mozilla.org/build/tools/file/67cca6c691a6/buildbot-helpers/pulse_publisher.py

We just need a publishing shim that hooks into hg's QueueDir.

catlee, what do the hg QueueDir messages look like (or do you have a link to a class that reads from hg's QueueDir)?
Flags: needinfo?(catlee)
OS: Mac OS X → All
Priority: -- → P2
Hardware: x86 → All
I don't think there is a queuedir for hg at this point, is there? Maybe you mean pulse's queuedir?
Flags: needinfo?(catlee)

Comment 5

4 years ago
I think I may be confused here.  My understanding was that something in releng is using QueueDir, or a similar approach, with hg; in other words, there's already an hg server hook that can write to a QueueDir.  If not, I imagine we'll have to write that in addition to a daemon that slurps them up and publishes them.  Probably neither is terribly difficult, but I don't know for sure.

Comment 6

3 years ago
My reading of this bug is that it is related to a hook installed on the Mercurial server. Said hook is no longer deployed. So closing.
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Component: Pulse → Mercurial: hg.mozilla.org
Product: Webtools → Developer Services
QA Contact: hwine
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.