799965 - Implement syslog-to-database insertion for imaging service

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task)

Description

[(not currently active) Ted Mielczarek]

This is a pretty standalone component. We need to monitor the syslog output from the devices and feed it into the logs table in the database.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

So, rsyslogd can insert directly into mysql, and that seems like the best option here.  I can implement that when the DB flow is in place (bug 799624).

The separate question of how to *react* to those log entries will need to be handled elsewhere.  I suspect that will be a daemon polling the DB, and consulting the logs table when necessary.

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

I also discovered a bug in the rsyslogd config:

*.info;mail.none;authpriv.none;cron.none;local.none     /var/log/messages

should be

*.info;mail.none;authpriv.none;cron.none                /var/log/messages

and the line is rejected (silently, yay!) otherwise.

I'll fix that up here.

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: insert-fn.patch

rsyslogd on each imaging server will invoke this stored procedure to do the inserts.  It encapsulates the find-id-then-insert pattern nicely in the DB.

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: insert-fn.patch

this adds ltrim() - I hadn't noticed the spurious space before

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: bug799965.patch

* slurp logs from rsyslogd into mysql using a stored procedure
* do so with an in-memory queue to ride out short downtimes (10m), but discard over longer times
* also log to /var/log/boards
* install logrotate and use it to rotate /var/log/boards
* replace bb_db_url secret (python-specific) with a host/user/pass/db tuple

Comment 6

[Jake Watkins [:dividehex]]

Comment on attachment 670874 [details] [diff] [review]
bug799965.patch

>+class packages::rsyslog_mysql {
>+    case $operatingsystem {
>+        CentOS: {
>+            package {
>+                "rsyslog-mysql":
>+                    ensure => latest;
>+            }
>+        }

Probably want to require rsyslog here.  Other than that, looks good.

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 670874 [details] [diff] [review]
bug799965.patch

committed with the require added

Comment 8

[(not currently active) Ted Mielczarek]

Since the tftpd logs to syslog, could we hook up some magic that could feed the syslog entry from when a board fetches its PXE config into the logs table? Presumably we'd have to look up the board by MAC address.

Comment 9

[Rail Aliiev [:rail]]

Comment on attachment 670874 [details] [diff] [review]
bug799965.patch

I pushed a bustage fix for this: http://hg.mozilla.org/build/puppet/rev/711679a2cb4b

Comment 10

[Dustin J. Mitchell [:dustin] (he/him)]

comment #8 spawned bug 801643