https://papertrailapp.com/systems/93137064/events Has system level events just not actual app events (ala badpenny stuff)
Let me verify whether those are going to syslog at all.
Assignee: relops → dustin
No -- and that makes sense, since the logging setup is in relengapi.wsgi, which isn't used in celery. But worse, application logging doesn't appear in celeryd.log, either.
I found https://pypi.python.org/pypi/logging_tree while debugging this -- very cool! However, I was wrong in comment 2 (just forgot to deploy my tweaks). The log messages *are* appearing in celeryd.log. So we just need to find a way to get those to syslog.
A partial solution is to forward logs via supervisord, which I've just implemented. It gets the data there, but since celeryd is adding its own log format, it ends up duplicating a lot of info on each line.
Oh, we have /data/www/relengapi/celery which is a Python script just like relengapi.wsgi. So I can just add the logging there. Duh.
Actually I didn't land the partial solution in comment 4. I went with https://github.com/mozilla/build-relengapi/pull/286 instead. Among other things, this should be easily made compatible with structured logging.
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
It looks like I never deployed this to production. Deployed now.
You need to log in before you can comment on or make changes to this bug.