Please sync kublad's log to cm-vpn01

RESOLVED FIXED

Status

mozilla.org
Webdev
RESOLVED FIXED
10 years ago
9 years ago

People

(Reporter: clouserw, Assigned: clouserw)

Tracking

Details

Attachments

(1 attachment)

(Assignee)

Description

10 years ago
I just checked kublad's heartbeat and it says it's tried to update 12 files and failed over the past weekend.  The last time this happened there was a permissions problem.  I'd like to sync /data/kublad.log to cm-vpn01:/var/log/clusterlogs so I can see what went wrong (and have it synchronized so I can see it in the future).

Updated

10 years ago
Assignee: server-ops → oremj

Comment 1

10 years ago
Can kublad log to '/usr/bin/logger -p local7.crit -t http_kublad'.  That's all it will take to start sending the logs to /var/log/clusterlogs.
(Assignee)

Comment 2

10 years ago
kublad just opens a log file with open($name, 'a').  The log file name & path is set in the config.  I don't know anything about logger - is it expecting log messages on stdin or what?  Do you have some example python code for using it?

Comment 3

10 years ago
Logger is expecting messages via stdin. I don't have an example of a python program, but on the command line 'echo foo | /usr/bin/logger -p local7.crit -t http_kublad' is what we want.  Popen would probably do the trick.  Let me know if you need more information or examples.

Comment 4

10 years ago
It might work to just run tail -f | logger in the background...

Comment 5

10 years ago
Please reassign to me when something is in place to write to logger.
Assignee: oremj → clouserw

Comment 6

10 years ago
Wil - can kublad just call syslog directly instead of exec'ing logger?  Just put in:

syslog("local7", $message);

?

Updated

10 years ago
Component: Server Operations → Webdev
QA Contact: justin → webdev
(Assignee)

Comment 7

10 years ago
Created attachment 331794 [details] [diff] [review]
syslog patch

Will this work?

It runs for me and logs to local7.  It also seems to spam any terminal logged in to sm-cms01 but that's probably a syslog config issue.
Attachment #331794 - Flags: review?(mrz)
(Assignee)

Comment 8

10 years ago
Comment on attachment 331794 [details] [diff] [review]
syslog patch

Asking for review from oremj too - either wfm.  The heartbeat says that kubla has failed to update something 75 times - i'd really like to see what the problem is...
Attachment #331794 - Flags: review?(oremj)

Comment 9

10 years ago
Comment on attachment 331794 [details] [diff] [review]
syslog patch

Would like to change a couple things.

1) The first arg to the syslog method only sets the priority and not the facility.
2) I'd like to tag the syslogs as http_kubla.

Working example script:

import syslog

msg = "foobar"
syslog.openlog('http_kubla', 0, syslog.LOG_LOCAL7)
syslog.syslog(msg)
syslog.closelog()
Attachment #331794 - Flags: review?(oremj) → review-
(Assignee)

Comment 10

10 years ago
Thanks, that works better than mine.  New change is on trunk and production.
(Assignee)

Comment 11

10 years ago
Woo, this is working now.  Right now it is splitting log files by the hour - is it possible to split by the day or even month?  kublad doesn't generate enough output to justify splitting them up and it makes tailing more of a pain.
All of the error logs are now split by day.
(Assignee)

Updated

9 years ago
Status: NEW → RESOLVED
Last Resolved: 9 years ago
Resolution: --- → FIXED

Updated

9 years ago
Attachment #331794 - Flags: review?(mrz) → review-
You need to log in before you can comment on or make changes to this bug.