1104911 - pulsetranslator test app

Status: RESOLVED INVALID

(Webtools :: Pulse, defect)

Description

[Mark Côté [:mcote]]

There have been many problems with pulsetranslator lately, and we don't have good insight into what's going on.  Given several services rely on the normalized-build stream, we need to monitor it until we're sure it's running well.

A basic test app would listen to both the "raw" and the normalized exchanges and record basic information from messages in order to correlate them.  The goal is to (a) ensure each raw message is translated in a timely manner and (b) there are no duplicate messages (both of which were mentioned in bug 1094272).

This is a bit complicated given that not every message is normalized and that there's no reference to the original message.  The latter is easy to fix; we should preserve the original name in a new property.  I'll file a bug for that. The former will be a bit trickier and require going to the pulsetranslator source to figure out what should not be tracked.

If a certain time elapses without seeing the associated normalized message, or if a duplicate is seen, we should log a message and ideally send out an email.

Comment 1

[Ricardo Castro]

I'll start looking into this one.

Comment 2

[Ricardo Castro]

Some questions asked, by me to Mark outside Bugzilla, that are relevant to the bug:

> The basic idea is that we would create an app/service that would monitor pulse,
> decide which messages should be translated and check if pulsetranslator is doing
> the translation correctly and in a timely manner.
Yeah, a consumer for each stream, "raw" and normalized, and some logic to compare them.

> Should this be more of a background service or some sort of app
> with possibly some web interface of some kind?
I think a background service, with logs, is all we need here.

> Should the code we develop for this be included in the
> pulsetranslator code base or should this be a separate project?
Separate, for now at least. I don't foresee this as something that will
be run forever; we can just use it on demand to debug problems.

Comment 3

[Ricardo Castro]

I believe we can take 2 approaches to this:

1) Create a script that is run by a cron job. 

2) Create a script that will run a loop, and that script will be run and monitored by supervisord.

Comment 4

[Mark Côté [:mcote]]

It will have to run continuously, since it launches two consumers that remain connected and consume messages in order to compare them.  So option 2), but it'll have to be multithreaded or multiprocess, probably, and the looping will be implicit in the consumer listen() calls.

Comment 5

[Ricardo Castro]

Maybe Celery (http://www.celeryproject.org/) can lend us a hand on this.

Comment 6

[Mark Côté [:mcote]]

You could try that, although personally I would probably use stuff from the standard library, e.g. multiprocessing.  Check out mozillapulse/test/runtests.py where we have a publisher and a consumer running in separate processes and communicating via a queue.

Comment 7

[Ricardo Castro]

Precisely! I was looking through that code you mentioned, this morning before coming into work. I believe that to be a better option

Comment 8

[Ricardo Castro]

I'll use this repository: https://github.com/mccricardo/pulsetranslator_monitor

Comment 9

[Ricardo Castro]

In Pulse, there are several consumers for different exchanges. I believe we should monitor all those exchanges.

Comment 10

[Mark Côté [:mcote]]

The only consumers that are relevant for this bug are the BuildConsumer and the NormalizedBuildConsumer.

Comment 11

[Dustin J. Mitchell [:dustin] (he/him)]

Pulsetranslator translated buildbot messages, and buildbot is no more.