937743 - Change Notification System ZeroMQ support

Status: RESOLVED WONTFIX

(bugzilla.mozilla.org Graveyard :: Bugzilla Change Notification System, defect)

Description

[Mark Côté [:mcote]]

The Bugzilla Change Notification System currently depends on Pulse, Mozilla's RabbitMQ-based messaging system.  In order to make the Notification System more generally consumable, it would be nice to support a more generic system such as ZeroMQ, which is also simpler to deploy than RabbitMQ.

Comment 1

[Nitin Garg [:musingmario]]

Hi Mark, does this require a deep knowledge of ZeroMQ? I know the basics of RabbitMQ and heard about ZeroMQ but never played around with it.

Comment 2

[Mark Côté [:mcote]]

To be honest, I'm not sure. :) It *should* be pretty straightforward, but I haven't done anything more than read the docs for ZeroMQ, so I'm not sure myself!

Comment 3

[Nitin Garg [:musingmario]]

Haha, okay. May I ask, based on your initial reading, why do you think ZeroMQ is 'more generic'? Also, what exactly is meant by 'more generally consumable'?

Comment 4

[Mark Côté [:mcote]]

It's just because Mozilla's Pulse service is a layer around RabbitMQ--it expects Rabbit to be set up in a particular way, and it relies on a package (mozillapulse) for the consumer and producer (or at least greatly benefits from it).  Since the change notification system uses very simple messages, I think that we could make the shim (which reads entries off the Bugzilla db table) and the WebSocket server also support zeromq really simply. This would ease deployments for Bugzilla installations other than BMO.

Comment 5

[Nitin Garg [:musingmario]]

Btw, I'm totally new here. So..
1. I have no idea where these components are located in the code, how they interact with each other and how the whole thing even looks like in a production environment.
2. What do you think I should do to at least use and get acquainted with this notification system?
3. Is there any documentation for the notification system?

Comment 6

[Mark Côté [:mcote]]

The whole system is documented at https://wiki.mozilla.org/BMO/ChangeNotificationSystem.  There are links to the code there.  bugzfeed, the WebSocket server, has a HACKING.md file with more info on getting started. I just added more instructions to that file for getting started without having to set up a Bugzilla instance first, which should help you get going faster.  Feel free to ask questions here or in #bteam on irc.mozilla.org.

Comment 7

[Nitin Garg [:musingmario]]

Attachment: BMO - Change Notification System.PNG

I came up with a diagrammatic representation of the Change Notification System. It's based on my reading of the wiki link you gave. Could you tell me how accurate it is?

Comment 8

[Nitin Garg [:musingmario]]

Okay, I think I know my first mistake. The shim should be in the Change Notifications Server, right?

Comment 9

[Nitin Garg [:musingmario]]

No wait.. I'm confused now.

The wiki says: "We will implement a separate server that polls the database directly for changes to bugs.." and "The changes are picked up and delivered to Pulse by the Bugzilla Simple Shim". Those indicate that the shim runs on this separate server.

Whereas HACKING.md says: "Grab the Bugzilla Simple Shim and run it on your Bugzilla box."

Comment 10

[Mark Côté [:mcote]]

Wow, thanks!  Great diagram.  To answer your question from comment #9 first, they can live together on the same server but don't have to, as long as both can access the database.  It will depend on how much load the shim generates.

Your diagram is pretty much accurate for a basic setup, for local development and for the initial deployment for testing.  At first, the shim will live on the bugzilla-dev server (bugzilla-dev.allizom.org, which we use for testing code changes before deploying them to production), as you described.  However when we move to production, it will probably move to its own server, although it's hard to say before knowing how much load it will generate; it might just be able to live on a webhead (production has multiple servers running Bugzilla to share the load).

The only other inaccuracy of your diagram is that the ZPushNotify extension and the REST API are all part of the Bugzilla application, so they live together.  They are logically separate pieces of code but all run in the same process(es).  Not sure if you want to (or how you would) clarify that in your diagram.

Also, in production (and actually on our bugzilla-dev test system) the database runs on its own machine, but for local testing, and in small installations elsewhere, Bugzilla and the db are generally on the same system.

Comment 11

[Nitin Garg [:musingmario]]

Thanks for the detailed reply Mark.

Is there a name for this process group / daemon that ZPushNotify and REST API are part of?

Who exactly is the client? (I left it as a '?' in the diagram). Is it a piece of software running on your personal computers? When I first looked at this bug, I assumed that the notifications are Facebook style notifications on the bugzilla website. But from my understanding so far, that's not the case.

I also find the subscriptions with a short life span of a session slightly odd. Unless these sessions are going to be indefinitely running websocket connections.

I'm going to setup my dev VMs now as explained in HACKING.md

After your next reply, I'll update the diagram. Perhaps we can put it up on the Wiki? I can refine it further in case it doesn't meet some standards.

Comment 12

[Mark Côté [:mcote]]

(In reply to musingmario from comment #11)
> Is there a name for this process group / daemon that ZPushNotify and REST
> API are part of?

Bugzilla. :)

> Who exactly is the client? (I left it as a '?' in the diagram). Is it a
> piece of software running on your personal computers? When I first looked at
> this bug, I assumed that the notifications are Facebook style notifications
> on the bugzilla website. But from my understanding so far, that's not the
> case.

The intended use would be dashboards that use the Bugzilla REST API, e.g. bzdeck.com.  We might integrate it into Bugzilla proper at some point, but we have no definite plans there at the moment.

> I also find the subscriptions with a short life span of a session slightly
> odd. Unless these sessions are going to be indefinitely running websocket
> connections.

Yeah, that would be the idea.  You would load some sort of dashboard that displays one or more bugs, and then the dashboard's client-side JS would open some WebSocket connections so that it could update the display when bugs change, or at least notify the user that there was a change.  We didn't want to bother with having durable queues that clients could reconnect to, at least for a first version, since it would be significantly more complicated.

It is also possible to subscribe to Pulse to get notifications for use in other applications.  The WebSocket server is just one possible way to serve notifications. 

> I'm going to setup my dev VMs now as explained in HACKING.md

Cool, as I mentioned in that file, you may not want to bother with a whole Bugzilla setup right now, since the developer box is old.  The code that's packaged in the box doesn't have the ZPushNotify extension, which was only committed a couple weeks ago, and merging in recent changes from the Bazaar repo is problematic because of a major schema change that happened a month ago or so (the tracking-flags migration).  It's probably possible to grab just that extension and throw it into the box and run checksetup.pl, or of course you could set up a BMO box yourself from scratch, but to get up and running more quickly you'll probably want to use my minimal instructions which just involves simulating changes by writing rows directly to a database.

You'll still need the Pulse test VM, though, which should work fine as it is.

> After your next reply, I'll update the diagram. Perhaps we can put it up on
> the Wiki? I can refine it further in case it doesn't meet some standards.

Yeah, I was going to ask if I could use your diagram. :)  That would be great!

Comment 13

[Nitin Garg [:musingmario]]

Attachment: BMO - Change Notification System v0.2.PNG

Hi, here's the updated version.

Comment 14

[Nitin Garg [:musingmario]]

Attachment: BMO - Change Notification System v0.3.PNG

Updated yet again. Yes, WebSockets are built on top of TCP. And yes, I'm spying on your IRC chats. :D

Comment 15

[Mark Côté [:mcote]]

Cool!  I just have a few little tweaks if you're up for it:

* It should be just "Bugzilla" instead of "Bugzilla daemon", since it's not necessarily a daemon (it can be run as CGI, for example).
* Looks like there is an extra space in "ZPushNotify  Extension".
* Missing a closing parenthesis in the API description.
* Just "Bugzilla API", since there are multiple ways of accessing it, not just via REST (though that is the most common).
* Just "WebSocket Server" (WebSockets are always TCP)
* Maybe change "E.g. bzdeck.com" to just "Web Browser", since that is what is opening the WebSocket and API connections.

Comment 16

[Nitin Garg [:musingmario]]

Attachment: BMO - Change Notification System v0.4.PNG

Made the changes accordingly.

Comment 17

[Mark Côté [:mcote]]

Thanks, I've uploaded the most recent diagram to wiki.mozilla.org and linked to it from the Change Notification System page.  I also marked the older versions as obsolete.

Comment 18

[Mark Côté [:mcote]]

How's it going with your setup? Need any help?

Comment 19

[Mark Côté [:mcote]]

Also just to let you know, there is a new version of the Bugzilla DeveloperBox up; it's much smaller and contains recent code & schema changes.

https://groups.google.com/forum/#!topic/mozilla.tools.bmo/mqaqqq5Fd-g

Comment 20

[Dejan Strbad]

Hi, is this still active? If so, where can I get more info regarding project details? Thanks.

Comment 21

[Mark Côté [:mcote]]

It is; I haven't heard from the previous commenter in a while.  The above comments should have lots of background; let me know if you have any specific questions (or ask in #bteam on irc.mozilla.org).

Comment 22

[Dejan Strbad]

Thanks for prompt response. I'll check the comments, setup test environment and get back to you on IRC.

Comment 23

[Mark Côté [:mcote]]

I am going to un-mentor this bug; it seems hard for people to make much traction, and there's no use case yet.

Comment 24

[Mark Côté [:mcote]]

bugzfeed as it is currently implemented has been decommissioned.