655935 - firebot send buildbot failures to pushers

Status: RESOLVED WONTFIX

(Webtools Graveyard :: Mozbot, enhancement)

Description

[Steve Fink [:sfink] [:s:]]

It'd be handy if firebot could maintain a registry of pusher email -> irc nicks, then watch buildbot results and send IRC messages to pushers when builds/tests break. (You opt-in by giving firebot your pushing email address, and perhaps configure which results you want to be notified of.)

Comment 1

[Wolf [:wolf]]

This has come up before (in a slightly different form - i.e. not an IRC notification).. However, mozbot does not have good integration with buildbot, which is something I hope to improve on - this would have to wait for that.

Comment 2

[LegNeato]

All the buildbot events/messages are being sent through pulse.mozilla.org via AMQP. It should be fairly easy to listen to finished events. If firebot is in python I have an easy AMQP helper lib @ http://hg.mozilla.org/users/clegnitto_mozilla.com/mozillapulse/.

You can see the kind of messages flowing through the system at http://pulse.mozilla.org/live

Comment 3

[Mark Banner (:standard8)]

Something I was actually thinking of was to uplift tinderstatus a bit more - tinderstatus can already notify you when particular builders (or the tree) changes state, but I suspect we need more resolution as to what changed with what changeset (e.g. register your push email with tinderstatus and it'll give you big warnings when something you pushed failed).

Of course, pulse might make it easier, if there's a way we can interface to it via js.

Comment 4

[Steve Fink [:sfink] [:s:]]

(In reply to comment #2)
> All the buildbot events/messages are being sent through pulse.mozilla.org
> via AMQP. It should be fairly easy to listen to finished events. If firebot
> is in python I have an easy AMQP helper lib @
> http://hg.mozilla.org/users/clegnitto_mozilla.com/mozillapulse/.

Nope, firebot is in Perl. I looked into this some. Perl has a (as in one, a single) module for handling AMQP. I managed to struggle through getting it to talk to pulse.mozilla.org, and can see the build messages now. Just two problems: first, the current CPAN version of the AMQP module doesn't read the messages properly, and will fail after half a dozen or so build messages come in. I patched that and forked the project on github (after talking to the author), in the process discovering that it's been fixed in two other ways already. But at any rate, you can get versions of the relevant modules that work.

Second, I'm not sure if having AMQP support in Perl really helps. Monitoring a live network queue doesn't really fit naturally into mozbot's extension mechanism, afaict. So you'd almost want a separate server that firebot would poll with its existing scheduling mechanism, and that server would watch the Pulse feed. (If mozbot mozbot were written using POE, it might be different.)

But if you're using a separate server, it doesn't matter much what language it's written in. I guess it's probably better to keep it all one language, but the AMQP stuff pulls in a bunch of dependencies.

Wolf, which path would you rather take? Use Legnitto's python stuff, try to get a POE-based thing working on your Windows server, or do your own buildbot integration? (Note that using Pulse could give useful integration with other tools for little extra effort -- hg, bugzilla, etc.)

Comment 5

[Steve Fink [:sfink] [:s:]]

Wow, I had totally forgotten about this bug, even though I just finished implementing a very limited version of it in mrgiggles (my irc bot). It lets you ask for notifications of failures for a particular revision. I don't think we can really do much better right now, given our intermittent orange rate. I suppose it could just notify for build failures, not test failures, but even then there are quite a few intermittents.

Comment 6

[:glob ✱]

this is a reasonable amount of work, and not something that i'm keen to put any time upon - mostly due to firebot's archaic code.

glandium has an irc bot that is already hooked up to pulse (called "pulsebot") which may be a better place to build this notification system should it still be desired.

Comment 7

[Mike Hommey [:glandium]]

That's on the pulsebot roadmap.

Comment 8

[Steve Fink [:sfink] [:s:]]

As of yesterday, you can

  /msg mrgiggles tell me when 988779675ea6 jobs fail

and it will privmsg you when any job for that revision returns a status code other than 0 and 5. The request will auto-expire after 24 hours. (I think. Can't remember how long I set it for.) I implemented the functionality a while back, but the syntax was pretty cryptic (!watch bad 988779675ea6, I think, which will still work) until yesterday.

pulsebot would be a better place for it. I only have pulse-monitoring capability in mrgiggles because it needs to pull down the latest results from hazard builds. But now that I have it, I've wedged in a couple of other things like this. And I can't directly share code with pulsebot, because mrgiggles is based on Twisted.

https://bitbucket.org/sfink/mrgiggles