We should add internal logging to treeherder, to make it easier to diagnose problems (like e.g. the recent issue where we were constantly downloading and reprocessing a full set of revision data due to a broken memcached). I'm inclined to follow the practices of Mozilla webdev here. It looks like they use a module that they created called "commonware" for this, which enables some nifty features like per component logging: http://playdoh.readthedocs.org/en/latest/userguide/logging.html I've been playing with this stuff a bit and have things mostly going for the development deployment. We'll probably want to consult with fubar or someone in ops on what the production deployment should look like.
Assignee: nobody → wlachance
Created attachment 8544253 [details] [review] Add basic logging This adds basic logging to treeherder. On dev (vagrant) instances, log messages will be written to /var/log/treeherder/treeherder.log (rotated with /var/log/treeherder/treeherder.log.1 after a 5M cap is reached). On production we add syslog-level logging, modeled after the setup for kitsune (https://github.com/mozilla/kitsune/blob/master/kitsune/log_settings.py#L21). Note that the production logging is not yet tested, though I see no reason it shouldn't work. After this lands I recommend we put in some basic info-level logging in various parts of treeherder so we have a better idea of what's being ingested and what decisions it's making (i.e. deciding we need to re-ingest the full pushlog).
Attachment #8544253 - Flags: review?(mdoglio) → review+
Commits pushed to master at https://github.com/mozilla/treeherder-service https://github.com/mozilla/treeherder-service/commit/8919d4f98c298955bc47db42c1cf3ff65f14ddaa Bug 1117856 - Add basic internal logging to treeherder https://github.com/mozilla/treeherder-service/commit/a10d257c5f83a4b4e1779b99c5f50ee8522df8c4 Merge pull request #314 from wlach/logging Bug 1117856 - Add basic internal logging to treeherder
Comment on attachment 8544253 [details] [review] Add basic logging Awesome! Rather than going with a log file on production, I think it should just go straight to syslog. Then I can see what tag it's using by logging locally, get syslog1 set up to receive, and then forward the log onward.
Attachment #8544253 - Flags: feedback?(klibby) → feedback+
Commits pushed to master at https://github.com/mozilla/treeherder-service https://github.com/mozilla/treeherder-service/commit/6a2fdd347256fa1ffc45bb3549ff50f5a028ff57 Bug 1117856 - Fix error in logging configuration https://github.com/mozilla/treeherder-service/commit/856cbf3b9e5be9390d7f3e6d36f9e47c47ae283c Merge pull request #352 from wlach/1117856 Bug 1117856 - Fix error in logging configuration
I believe this bug is fixed. On production, we should be logging both to /var/log/treeherder/treeherder.log and /var/log/syslog There is more work to be done on production to make logging more transparent, but we can leave that to later. Feel free to reopen if I'm missing something.
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.