Add internal logging to treeherder



Tree Management
3 years ago
3 years ago


(Reporter: wlach, Assigned: wlach)




(1 attachment)

We should add internal logging to treeherder, to make it easier to diagnose problems (like e.g. the recent issue where we were constantly downloading and reprocessing a full set of revision data due to a broken memcached).

I'm inclined to follow the practices of Mozilla webdev here. It looks like they use a module that they created called "commonware" for this, which enables some nifty features like per component logging:

I've been playing with this stuff a bit and have things mostly going for the development deployment. We'll probably want to consult with fubar or someone in ops on what the production deployment should look like.
Assignee: nobody → wlachance
Created attachment 8544253 [details] [review]
Add basic logging

This adds basic logging to treeherder. On dev (vagrant) instances, log messages will be written to /var/log/treeherder/treeherder.log (rotated with /var/log/treeherder/treeherder.log.1 after a 5M cap is reached). 

On production we add syslog-level logging, modeled after the setup for kitsune ( Note that the production logging is not yet tested, though I see no reason it shouldn't work.

After this lands I recommend we put in some basic info-level logging in various parts of treeherder so we have a better idea of what's being ingested and what decisions it's making (i.e. deciding we need to re-ingest the full pushlog).
Attachment #8544253 - Flags: review?(mdoglio)
Attachment #8544253 - Flags: feedback?(klibby)
Attachment #8544253 - Flags: review?(mdoglio) → review+


3 years ago
Priority: -- → P2
Comment on attachment 8544253 [details] [review]
Add basic logging

Awesome! Rather than going with a log file on production, I think it should just go straight to syslog. Then I can see what tag it's using by logging locally, get syslog1 set up to receive, and then forward the log onward.
Attachment #8544253 - Flags: feedback?(klibby) → feedback+


3 years ago
Priority: P2 → P3
I believe this bug is fixed. On production, we should be logging both to /var/log/treeherder/treeherder.log and /var/log/syslog

There is more work to be done on production to make logging more transparent, but we can leave that to later.

Feel free to reopen if I'm missing something.
Last Resolved: 3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.