603029 - Roll out hg hook to publish messages from hg.mozilla.org to pulse.mozilla.org

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[LegNeato]

This is a tracking bug for rolling out http://hg.mozilla.org/users/clegnitto_mozilla.com/hg-broker/ on hg.mozilla.org.

The hook will make it so pushes are published into pulse.mozilla.org and I can turn off my hgweb scrapers.

The hook is not 100% ready yet and hasn't gone through security review, but I wanted a bug to track progress.

Comment 1

[Axel Hecht [:Pike]]

Christian, how far along is the code here?

Comment 2

[Dirkjan Ochtman (:djc)]

Looks good from the hg side. Maybe it's slightly more robust to use mercurial.util.datestr() instead of wiring up your own date conversion routine.

Comment 3

[LegNeato]

(In reply to comment #1)
> Christian, how far along is the code here?

I actually just picked this up again yesterday. I have a couple of polish things I need to do and have some tests to write. ETA is today.

(In reply to comment #2)
> Looks good from the hg side. Maybe it's slightly more robust to use
> mercurial.util.datestr() instead of wiring up your own date conversion routine.

I'll likely do that, thanks for the pointer!

Comment 4

[LegNeato]

Attachment: HG hook to send messages to pulse.mozilla.org

I am going to keep hg-broker general and do a Pulse-specific script for Mozilla. In the future once I get hg-broker solid we can switch. We'll at least get the benefit in the meantime though.

Attached is the hook I want to put in place on hg.mozilla.org (checked into mozhghooks I guess).

Outstanding questions:
#0 - Can we install carrot on the machine with `easy_install`? Where do I codify that requirement? There doesn't look to be any dependencies in mozhghooks's setup.py even though the current hooks use stuff like sqlite

#1 - Where should I read login/password information from? Can we just stick them in the file and be done with it? (they aren't really meant to be secure yet, note that the connection isn't made over SSL)

#2 - Is there any way to get the repo root w/o setting it manually? I don't want to include the whole FS path, merely the part that makes up the repo (mozilla-central, releases/mozilla-1.9.2, etc)

#3 - Are tests required for landing? I can get some tests working but they will take time. I've tested it locally by modifying run_tests.py in the mozhghooks repo, but I don't have infrastructure to switch between test settings and production in the hook yet

I am also assuming for shadow/user repos we just wouldn't enable the hook, so no repo filtering is involved.

I'd like to enable it for these repos:
  mozilla-central
  comm-central
  mobile-browser
  try-comm-central
  mozilla-1.9.2
  mozilla-1.9.1
  tracemonkey
  try
  try-1.9.2

If we want to try to roll it out slowly, let's just enable it for my user repo and I can test it:

http://hg.mozilla.org/users/clegnitto_mozilla.com/

Comment 5

[LegNeato]

Attachment: HG hook to send messages to pulse.mozilla.org , now with less mispellings

Comment 6

[LegNeato]

Another outstanding question:

#4 - Do the firewall routes exist so hg.mozilla.org can talk to pulse.mozilla.org?

Comment 7

[Axel Hecht [:Pike]]

I'm at least interested in 

dom-inspector
releases/comm-1.9.1
releases/comm-1.9.2
releases/mobile-1.1

and all of

l10n-central/*
releases/l10n-mozilla-1.9.1/*
releases/l10n-mozilla-1.9.2/*
testpilot-l10n/*
weave-l10n/*

though I'd think that having all hg repos including user repos and whatnot would be the right thing to do, aka, make it a global hook.

We could also include the top-level path in the global config, so that you can get the repo name by cutting that off.

Comment 8

[LegNeato]

(In reply to comment #7)
> though I'd think that having all hg repos including user repos and whatnot
> would be the right thing to do, aka, make it a global hook.

That's probably the correct way to do it in the long run. We'd need to maintain a blacklist though (for shadow repos) or have some metadata in the repos themselves to signal no messages be sent.

Comment 9

[Dirkjan Ochtman (:djc)]

We probably want Aravind for at least some of these questions.

Comment 10

[Aravind Gottipati [:aravind]]

What happens when/if pulse is down?  Are we willing to hang hg pushes if the hook fails (like we do with pushlog).

Also, answers to some of the questions here..

1) Yes, I can install a lone package (carrot) on the box, if the hook needs it.

2) I don't think I can trim the path of these repos, but since its going to be the same for all of them, it's probably pretty easy to trim in the hook itself?

3) That's for Ted to answer.  I merely deploy these hooks as folks request them.

4) Don't know, but it can be opened up if it needs to.

Also, none of the hooks we have had so far send stuff to external services.  Do you think this hook should go through an infra-sec audit?

Comment 11

[LegNeato]

(In reply to comment #10)
> What happens when/if pulse is down?  Are we willing to hang hg pushes if the
> hook fails (like we do with pushlog).

There is an option in the hook to fail the push or to warn and continue with the push. For hg.mozilla.org we'll want to have it ignore the failure and continue for now. In the future once Pulse has proved reliable we'll make the push fail if the message send fails.

> 
> 1) Yes, I can install a lone package (carrot) on the box, if the hook needs it.

I think carrot will pull in other dependencies FYI.

> 
> 2) I don't think I can trim the path of these repos, but since its going to be
> the same for all of them, it's probably pretty easy to trim in the hook itself?

You can set it in the hook and it will trim it off currently. I just wondered if there was a way w/o setting this manually. Guess not, as repos can be located anywhere so Mercurial can't expose a "base path"

> Also, none of the hooks we have had so far send stuff to external services.  Do
> you think this hook should go through an infra-sec audit?

I've casually talked to the infra-sec folks about it and have asked for audits of other system instrumentation (bmo, etc). I don't think this needs a formal review due to:

1) Only dealing with public info
2) Only people pushing are trusted by Mozilla
3) It isn't transforming any data, merely packaging it up and shipping it to another system
4) It doesn't have any permissions-type logic that would need to be vetted

CCing Michael just in case he disagrees though.

Comment 12

[Axel Hecht [:Pike]]

We could add a 

[mozilla]
repo_base=/path/to/where/we/store/repos

to the system-wide config on hg, and make the pulse hook get that setting.

Comment 13

[(not currently active) Ted Mielczarek]

(In reply to comment #4)
> #3 - Are tests required for landing? I can get some tests working but they will
> take time. I've tested it locally by modifying run_tests.py in the mozhghooks
> repo, but I don't have infrastructure to switch between test settings and
> production in the hook yet

They haven't been historically. I added the tests because I was tired of breaking stuff when I made code changes. Your confidence level may vary. :)

Comment 14

[Michael Coates [:mcoates] (acct no longer active)]

(In reply to comment #11)
> CCing Michael just in case he disagrees though.
All good here. Christian and I chatted.  This change is just providing a more effective access method to the public data.

Comment 15

[LegNeato]

Changed direction a bit again. Based on Axel's input in comment 12 I have generalized it. There are no more hard-coded values to configure in the script. It should also now easily support specifying different configurations for running tests.

The code is @ https://github.com/LegNeato/hg-broker 

(seems odd to put something that deals with mercurial into git but I like GitHub ha)

Comment 16

[Axel Hecht [:Pike]]

I've been looking at https://github.com/LegNeato/hg-broker/blob/master/broker.py a bit. Is there a benefit from putting the config stuff into the module instead of the hook? I would think this is fired up from scratch for each push anyway, so doing it inside the hook would have the benefit that the ui object has them config files parsed in, too. Sounds like you'd be iterating over them twice now.

Comment 17

[LegNeato]

Yeah, I noticed the UI config stuff the other day, just haven't got a chance to update to it yet. The API I am using isn't really public I think

Comment 18

[LegNeato]

Ok, the UI stuff is fixed on GitHub. I think this is ready. Over to Aravind for rollout.

I'll need:

1 - Confirmation that hg.mozilla.org can contact pulse.mozilla.org via port 5672

2 - carrot (and its dependencies) installed on hg.mozilla.org)

3 - The hook checked out of git and stuck in the PYTHONPATH somewhere (probably into mozhghooks where the other hooks are?)

4 - The following added to the (global?) hg config file in the "hooks" section:
    pretxnchangegroup.z_broker = python:mozhghooks.broker.send_message
(assuming the broker hook is stored where mozhghooks are)

5 - The following added to the (global?) hg config file:

[broker]
BROKER_HOST = pulse.mozilla.org
BROKER_USER = (ask me)
BROKER_PASS = (ask me)
MSG_EXCHANGE = org.mozilla.exchange.code
REPO_ROOT = /path/to/where/the/repos/are/stored

Let me know if you want me to modify the hg-broker script to have a setup.py that can be installed with easy_install or pip. I figured it was easy to just drop it in somewhere, but if that's painful / not maintainable I can have it be its own module.

Comment 19

[LegNeato]

Any ETA on this so I can plan to be around if things go awry?

Comment 20

[Aravind Gottipati [:aravind]]

staging this is a significant amount of work for us, since we usually don't have these python packages handy.  I will work on this next week.

Comment 21

[LegNeato]

Ok, thanks for the info.

Comment 22

[LegNeato]

Aravind told me in person he was going to roll it out and only enable it for my user repo. Once it looks good we'll expand it to others, followed by every repo.

Did this get bumped for other higher-priority work?

Comment 23

[Aravind Gottipati [:aravind]]

Hook deployed from a systems perspective into Legneato's repo.  Changes to note.

1) Fix funtion name in the hook call
2) Change REPO_ROOT to something more general, like COMMON_PREFIX_PATH or something.
3) Add uuid as a dependency for carrot.
4) Provide IT a real user / password for pulse.

Comment 24

[LegNeato]

1) Fixed in https://github.com/LegNeato/hg-broker/commit/e5b1c647b28b092aad240eab3ba0a839fc6b823e

2) Fixed in https://github.com/LegNeato/hg-broker/commit/885e7989be1d90f14deb009d23fadd9abefcc9c5

3) Fixed in https://github.com/LegNeato/hg-broker/commit/f1b818f2e157a112b6d9d46efea6598b9d6f6f0b

4) Will do so in person

Comment 25

[LegNeato]

Reopening as I want this rolled out everywhere, looked good in my repo! Should take less than 5 mins. When this gets assigned to someone doing the work please ping me and we'll talk credentials (#4 in comment 23 & comment 24).

Comment 26

[LegNeato]

Ping? Can this get assigned to someone that I can bug? :-)

Comment 27

[Dirkjan Ochtman (:djc)]

I believe noahm might be able to help?

Comment 28

[Corey Shields [:cshields]]

(In reply to comment #27)
> I believe noahm might be able to help?

yes, but not with that address. we should probably get that one out of bugzilla  :)

Comment 29

[Noah Meyerhans [:noahm]]

(In reply to comment #28)
> (In reply to comment #27)
> > I believe noahm might be able to help?
> 
> yes, but not with that address. we should probably get that one out of
> bugzilla  :)

*sigh*  I've tried that.  Bug 645941

Anyway, I'll take a look at what it'll take to roll this out globally...

Comment 30

[LegNeato]

Yay, thanks guys!

Comment 31

[Shyam Mani [:fox2mike]]

(In reply to comment #28)
> yes, but not with that address. we should probably get that one out of
> bugzilla  :)

Or merge his accounts!

Comment 32

[Noah Meyerhans [:noahm]]

So, to be clear, the required change is to add the following to the side-wide hgrc:

[broker]
BROKER_HOST = pulse.mozilla.org
BROKER_USER = public
BROKER_PASS = public
MSG_EXCHANGE = org.mozilla.exchange.code
REPO_ROOT = /repo/hg/mozilla

[hooks]
pretxnchangegroup.z_broker = python:mozhghooks.broker.send_messages

If that's the correct change, I'll go ahead and enable it as soon as I get confirmation.  Otherwise, please send corrections.