Releng celery servers are not reporting celery logs to papertrail

RESOLVED FIXED

Status

RESOLVED FIXED
4 years ago
3 years ago

People

(Reporter: Callek, Assigned: dustin)

Tracking

Details

(Reporter)

Description

4 years ago
https://papertrailapp.com/systems/93137064/events  Has system level events just not actual app events (ala badpenny stuff)
Let me verify whether those are going to syslog at all.
Assignee: relops → dustin
No -- and that makes sense, since the logging setup is in relengapi.wsgi, which isn't used in celery.

But worse, application logging doesn't appear in celeryd.log, either.
I found https://pypi.python.org/pypi/logging_tree while debugging this -- very cool!

However, I was wrong in comment 2 (just forgot to deploy my tweaks).  The log messages *are* appearing in celeryd.log.  So we just need to find a way to get those to syslog.
A partial solution is to forward logs via supervisord, which I've just implemented.  It gets the data there, but since celeryd is adding its own log format, it ends up duplicating a lot of info on each line.
Oh, we have /data/www/relengapi/celery which is a Python script just like relengapi.wsgi.  So I can just add the logging there.  Duh.
Actually I didn't land the partial solution in comment 4.  I went with https://github.com/mozilla/build-relengapi/pull/286 instead.  Among other things, this should be easily made compatible with structured logging.
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
(Reporter)

Updated

3 years ago
Blocks: 1183247
It looks like I never deployed this to production.  Deployed now.
You need to log in before you can comment on or make changes to this bug.