16226 - [HAVE REVIEW] Channel Logging module

Status: VERIFIED FIXED

(Webtools Graveyard :: Mozbot, enhancement, P4)

Description

[Ben Bucksch (:BenB)]

Since some important discussions/decision are made in #mozilla, I'd appreciate a
logging facility for mozbot.

Stampede log their channel, too.
http://www.stampede.org/news.php3
http://www.stampede.org/irc.php3

Comment 1

[Hixie (not reading bugmail)]

Sounds like a very simple module that could be added to mozbot2.

How would the logs be made available? Daily files on a website? By mail?
How much backlog would we keep? What are the implications of this as far as
making people reconsider how freely they speak on #mozilla? Should there be
a 'mozbot this is off the record' mode? Should individual people be able to 
enable/disable the logging of themselves? Should people be able to prefix their
text with 'Off the record: ...' or 'OTR: ...'? How much disk space would this
take? Who is volunteering the space? Should the output format be plain text,
raw (easily machine-readable) text, HTML, XML?

Comment 2

[David Krause]

Ian, first it should be noted that most clients have logging capabilities.
Personally I feel that anything said on irc is public knowledge and anyone could
be logging it.  This may be a feature that would be off by default.  Then when
someone wants a log of the converation, they (a blessed person) would tell
mozbot to start logging.  When the subject being discussed was finished they
would tell mozbot to stop.  Otherwise irc logs can reach huge sizes in just a
few days.  But I do think this would be a useful feature and probably the
responsiblity of the person running the bot to keep the logs.

Comment 3

[Hixie (not reading bugmail)]

Agreed. Taking this bug. Looks like a simple enough module to write.

Comment 4

[Ben Bucksch (:BenB)]

Ian, cool.

I agree with David that open IRC channels are public. Many technical discussions
are held in #mozilla, and not everybody can or wants to be logged on all the
time. Enabling on request would propably not work, as the technical discussions
are often very short, and it is often not clear, if they will get important.
OHTOH, I can see arguments for not publishing logs, too.

As for size, I have logging in #mozilla on all the time, and the complete log
from > 1 year is ~11 MB in size. OK, I wasn't even nearly logged on all the
time, but *IMO*, some 100 MB should be worth it.

Comment 5

[Hixie (not reading bugmail)]

I need some sort of data as to where/how to store these logs and and how to make
them available. At the moment, its looking like the bot will just dump dump dump
(when logging is turned on), ignoring any lines prefixed with 'otr:' or 'off the
record:', and will then mail the logs to people upon request.

This is open to abuse, however. Better solutions are warmly welcomed.

Comment 6

[Hixie (not reading bugmail)]

still looking for ideas...

Comment 7

[Ben Bucksch (:BenB)]

publish on <http://www.mozilla.org>.

Comment 8

[Hixie (not reading bugmail)]

I mean how to do it generically. Don't forget mozbot is a mozilla.org product,
like Bugzilla -- many other people use it. So, how should it publish stuff?
I can't really require the module to use CVS, let alone automatically...

Comment 9

[Matthew Tuck [:CodeMachine]]

New QA.

Comment 10

[timeless]

publishing huge logs to cvs is not a good idea.

for one, logs should be compressed daily or weekly.

i think it's better to rely on a future mozbot http interface.

Comment 11

[Matt Jones]

I've developed a simple logging facility for mozbot.  It uses XML as its log
file format.  It has a simple "rotatelogs" command to start a new set of logs
for the channels.  You can use patterns ala MiniLogger to selectively log.  We
use it on our IRC server to log everything, and one day's log for one channel is
about 100K, with about 15 people generally logged in through the day.

Not sure if this will help as a start or not.  Seems like it does some of what
you want as described in this bug. Maybe it'll be useful to you.

I plan on writing an XSLT stylesheet so that I can easily publish these as HTML.
 I'll post that here too if there is interest.

I will attach some sample log output and the bot module code.

Comment 12

[Matt Jones]

Attachment: sample XML log file

This is an example XML log file as produced by the XMLLogger.bm module to
mozbot.

Comment 13

[Matt Jones]

Attachment: XMLLogger module

Module to log IRC channels in XML format.

Comment 14

[Hixie (not reading bugmail)]

wow

thanks!!

I'll look at it closer later. I think we should add logging functionality for
mode changes, joining/parting, and the like.

Also, for Mozilla at least, a CSS stylesheet is all that is needed (rather than
an XSLT one).

Comment 15

[Ben Bucksch (:BenB)]

> Also, for Mozilla at least, a CSS stylesheet is all that is needed

One file includes all channels, so a server-side XSLT stylesheet would help,
assuming mozbot logs more than just #mozilla.

Comment 16

[Hixie (not reading bugmail)]

Alternate stylesheets would help with the alternate views problem.

Comment 17

[Matt Jones]

Actually, the way it currently works each channel is logged in a different file.
 I put the "channel" attribute in anticipating that some people would want
everything to go into one file.  That should probably be an option (one file or
one per channel), but for now it is one per channel.

CSS is fine for some browsers, but at least for our site I'll need a server-side
transform to HTML + CSS to support older browsers.  Thus, XSLT is probably still
needed for some situations.

Comment 18

[Mika Morell]

I have a working module that logs public irc chatter to a Mysql database.
Currently it reads the Heard event and just inserts nick,userhost,channel and
what the person said into the database and gives the entry a timestamp. This way
you get a really finegrained timestamp. This module is still in beta so there is
no fancy output or reporting yet and all the database login information is
hardcoded into the module. This might be the best since you probably don't want
to change that once.   Logging to a database might seem to be a bit 'overkill'
but I think it is as god as any :). I will post the module here in a little
while when I have added a readme.

Comment 19

[Mika Morell]

Attachment: Mysql logging module

Comment 20

[Hixie (not reading bugmail)]

Attachment: XMLLogger module (Version 0.2)

This is a work in progress based on the previous XMLLogger module. I have a few
more modifications to make before it is ready for review.

Comment 21

[Hixie (not reading bugmail)]

Attachment: XMLLogger module (Version 1)

Ok, comments and reviews please!

Comment 22

[Hixie (not reading bugmail)]

Attachment: sample XML log file

Comment 23

[Hixie (not reading bugmail)]

(note: the code changes for bug 133509 are also in this bug's patch)

Comment 24

[Hixie (not reading bugmail)]

kerz said r=kerz on IRC assuming I change the if ($channel) { to check for
defined(), not just truth.

Comment 25

[Hixie (not reading bugmail)]

Checked in.

Comment 26

[Tanner M. Young [:tmyoung]]

Verified Fixed.