665021 - Provide mechanism for receiving hg notifications from mirrors

Status: RESOLVED WONTFIX

(Release Engineering :: General, defect, P5)

Description

[Chris AtLee [:catlee]]

Once we figure out what data we need about a push, we need to provide a way for the hg mirrors to let us know about a push.

Some options here are:
* sendchange port
* have mirror ssh in, run a script
* message queues

Comment 1

[Chris Cooper [:coop] (he/him)]

(In reply to Chris AtLee [:catlee] from comment #0)
> Some options here are:
> * sendchange port
> * have mirror ssh in, run a script
> * message queues

To recap what I just heard from catlee, arr/dustin ruled out using ssh, and our message queues are currently unreliable.

If using the sendchange port is not viable, we'll need to come up with some new options or revisit the old ones. Let's be sure to discuss this next week at the all-hands.

Comment 2

[Mike Taylor [:bear]]

I don't think we need to pull message queues from the short list - their are a number of low-infra required message queues that could be used.

Going to meet with RelOps folks at the allhands to discuss the options

Comment 3

[Chris Cooper [:coop] (he/him)]

(In reply to Mike Taylor [:bear] from comment #2) 
> Going to meet with RelOps folks at the allhands to discuss the options

Did you get a chance to discuss this with relops?

Comment 4

[Mike Taylor [:bear]]

(In reply to Chris Cooper [:coop] from comment #3)
> (In reply to Mike Taylor [:bear] from comment #2) 
> > Going to meet with RelOps folks at the allhands to discuss the options
> 
> Did you get a chance to discuss this with relops?

sadly no, I think I saw amy once and dustin a handful of times - all of them being "oh hai" moments as they were travelling to/from colo :/

Comment 5

[Chris AtLee [:catlee]]

I chatted with Dustin and Corey about this. Dustin and I think the best short term solution is to re-use the QueueDir code we have for pulse/log uploading. In this scenario, the hg master would create an entry in the QueueDir once the mirrors have the new changes. A queue processor on the hg master would pick up the items, and run a command on them (probably ssh/rsync to one of the build machines).

Longer term, we should be using message queues. Corey had some ideas about machines we could use for rabbitmq hosts, but I've forgotten the details.

Comment 6

[Chris AtLee [:catlee]]

How's this:

We set up durable topic exchanges for the different branches:
'hg.pushes.mozilla-central'

hg.m.o will publish a message to this exchange using the format described in bug 665020 after successfully pushing a new commit to the mirror(s)

set up a durable queue bound to 'hg.pushes.#' called 'hg.pushes'

releng sets up a consumer that connects to the 'hg.pushes' queue. When messages are received, we insert the data into our databases and ack the message.

I'm still a bit fuzzy on the rmq details...does this make sense?

Comment 7

[Mike Taylor [:bear]]

(In reply to Chris AtLee [:catlee] from comment #6)
> How's this:
> 
> We set up durable topic exchanges for the different branches:
> 'hg.pushes.mozilla-central'
> 
> hg.m.o will publish a message to this exchange using the format described in
> bug 665020 after successfully pushing a new commit to the mirror(s)
> 
> set up a durable queue bound to 'hg.pushes.#' called 'hg.pushes'
> 
> releng sets up a consumer that connects to the 'hg.pushes' queue. When
> messages are received, we insert the data into our databases and ack the
> message.
> 
> I'm still a bit fuzzy on the rmq details...does this make sense?

no need to make it durable IMO - but yes, a topic exchange would work just fine for this.

Comment 8

[Chris AtLee [:catlee]]

not using hg mirrors any more