1175411 - Releng celery servers are not reporting celery logs to papertrail

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task)

Description

[Justin Wood (:Callek)]

https://papertrailapp.com/systems/93137064/events  Has system level events just not actual app events (ala badpenny stuff)

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

Let me verify whether those are going to syslog at all.

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

No -- and that makes sense, since the logging setup is in relengapi.wsgi, which isn't used in celery.

But worse, application logging doesn't appear in celeryd.log, either.

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

I found https://pypi.python.org/pypi/logging_tree while debugging this -- very cool!

However, I was wrong in comment 2 (just forgot to deploy my tweaks).  The log messages *are* appearing in celeryd.log.  So we just need to find a way to get those to syslog.

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

A partial solution is to forward logs via supervisord, which I've just implemented.  It gets the data there, but since celeryd is adding its own log format, it ends up duplicating a lot of info on each line.

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

Oh, we have /data/www/relengapi/celery which is a Python script just like relengapi.wsgi.  So I can just add the logging there.  Duh.

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

Actually I didn't land the partial solution in comment 4.  I went with https://github.com/mozilla/build-relengapi/pull/286 instead.  Among other things, this should be easily made compatible with structured logging.

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

It looks like I never deployed this to production.  Deployed now.