Implement syslog-to-database insertion for imaging service

RESOLVED FIXED

Status

Infrastructure & Operations
RelOps
RESOLVED FIXED
5 years ago
5 years ago

People

(Reporter: ted, Assigned: dustin)

Tracking

Details

Attachments

(2 attachments, 1 obsolete attachment)

(Reporter)

Description

5 years ago
This is a pretty standalone component. We need to monitor the syslog output from the devices and feed it into the logs table in the database.
Assignee: server-ops-releng → dustin
So, rsyslogd can insert directly into mysql, and that seems like the best option here.  I can implement that when the DB flow is in place (bug 799624).

The separate question of how to *react* to those log entries will need to be handled elsewhere.  I suspect that will be a daemon polling the DB, and consulting the logs table when necessary.
Depends on: 799624
I also discovered a bug in the rsyslogd config:

*.info;mail.none;authpriv.none;cron.none;local.none     /var/log/messages

should be

*.info;mail.none;authpriv.none;cron.none                /var/log/messages

and the line is rejected (silently, yay!) otherwise.

I'll fix that up here.
Created attachment 670861 [details] [diff] [review]
insert-fn.patch

rsyslogd on each imaging server will invoke this stored procedure to do the inserts.  It encapsulates the find-id-then-insert pattern nicely in the DB.
Attachment #670861 - Flags: review?(ted.mielczarek)
Created attachment 670870 [details] [diff] [review]
insert-fn.patch

this adds ltrim() - I hadn't noticed the spurious space before
Attachment #670861 - Attachment is obsolete: true
Attachment #670861 - Flags: review?(ted.mielczarek)
Created attachment 670874 [details] [diff] [review]
bug799965.patch

* slurp logs from rsyslogd into mysql using a stored procedure
* do so with an in-memory queue to ride out short downtimes (10m), but discard over longer times
* also log to /var/log/boards
* install logrotate and use it to rotate /var/log/boards
* replace bb_db_url secret (python-specific) with a host/user/pass/db tuple
Attachment #670874 - Flags: review?(jwatkins)
Comment on attachment 670874 [details] [diff] [review]
bug799965.patch

>+class packages::rsyslog_mysql {
>+    case $operatingsystem {
>+        CentOS: {
>+            package {
>+                "rsyslog-mysql":
>+                    ensure => latest;
>+            }
>+        }

Probably want to require rsyslog here.  Other than that, looks good.
Attachment #670874 - Flags: review?(jwatkins) → review+
Comment on attachment 670874 [details] [diff] [review]
bug799965.patch

committed with the require added
Attachment #670874 - Flags: checked-in+
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
(Reporter)

Comment 8

5 years ago
Since the tftpd logs to syslog, could we hook up some magic that could feed the syslog entry from when a board fetches its PXE config into the logs table? Presumably we'd have to look up the board by MAC address.
Component: Server Operations: RelEng → RelOps
Product: mozilla.org → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.