Status

ASSIGNED
4 years ago
4 years ago

People

(Reporter: mcote, Assigned: mcc.ricardo)

Tracking

Trunk
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

4 years ago
There have been many problems with pulsetranslator lately, and we don't have good insight into what's going on.  Given several services rely on the normalized-build stream, we need to monitor it until we're sure it's running well.

A basic test app would listen to both the "raw" and the normalized exchanges and record basic information from messages in order to correlate them.  The goal is to (a) ensure each raw message is translated in a timely manner and (b) there are no duplicate messages (both of which were mentioned in bug 1094272).

This is a bit complicated given that not every message is normalized and that there's no reference to the original message.  The latter is easy to fix; we should preserve the original name in a new property.  I'll file a bug for that. The former will be a bit trickier and require going to the pulsetranslator source to figure out what should not be tracked.

If a certain time elapses without seeing the associated normalized message, or if a duplicate is seen, we should log a message and ideally send out an email.
(Reporter)

Updated

4 years ago
Depends on: 1104912
(Reporter)

Updated

4 years ago
Blocks: 1094272
(Assignee)

Comment 1

4 years ago
I'll start looking into this one.
Assignee: nobody → mcc.ricardo
(Assignee)

Updated

4 years ago
Status: NEW → ASSIGNED
(Assignee)

Comment 2

4 years ago
Some questions asked, by me to Mark outside Bugzilla, that are relevant to the bug:

> The basic idea is that we would create an app/service that would monitor pulse,
> decide which messages should be translated and check if pulsetranslator is doing
> the translation correctly and in a timely manner.
Yeah, a consumer for each stream, "raw" and normalized, and some logic to compare them.

> Should this be more of a background service or some sort of app
> with possibly some web interface of some kind?
I think a background service, with logs, is all we need here.

> Should the code we develop for this be included in the
> pulsetranslator code base or should this be a separate project?
Separate, for now at least. I don't foresee this as something that will
be run forever; we can just use it on demand to debug problems.
(Assignee)

Comment 3

4 years ago
I believe we can take 2 approaches to this:

1) Create a script that is run by a cron job. 

2) Create a script that will run a loop, and that script will be run and monitored by supervisord.
Flags: needinfo?(mcote)
(Reporter)

Comment 4

4 years ago
It will have to run continuously, since it launches two consumers that remain connected and consume messages in order to compare them.  So option 2), but it'll have to be multithreaded or multiprocess, probably, and the looping will be implicit in the consumer listen() calls.
Flags: needinfo?(mcote)
(Assignee)

Comment 5

4 years ago
Maybe Celery (http://www.celeryproject.org/) can lend us a hand on this.
(Reporter)

Comment 6

4 years ago
You could try that, although personally I would probably use stuff from the standard library, e.g. multiprocessing.  Check out mozillapulse/test/runtests.py where we have a publisher and a consumer running in separate processes and communicating via a queue.
(Assignee)

Comment 7

4 years ago
Precisely! I was looking through that code you mentioned, this morning before coming into work. I believe that to be a better option
(Assignee)

Comment 8

4 years ago
I'll use this repository: https://github.com/mccricardo/pulsetranslator_monitor
(Assignee)

Comment 9

4 years ago
In Pulse, there are several consumers for different exchanges. I believe we should monitor all those exchanges.
Flags: needinfo?(mcote)
(Reporter)

Comment 10

4 years ago
The only consumers that are relevant for this bug are the BuildConsumer and the NormalizedBuildConsumer.
Flags: needinfo?(mcote)
You need to log in before you can comment on or make changes to this bug.