Papertail is currently so slow it's killing us. I'm open to giving loggly a try again - just don't switch over exclusively. (I papertrail is awesome for text logs and so simple) I was reasonably happy with papertrail before we had 15k systems. I've reached out to them to see if it can be fixed on their end. Note, My only regret with papertrail is that it's so easy to end up search across all systems when you click on say web.1, this is still doable by searching with "system:taskcluster-queue program:web.1" Anyways, right now I can't add new systems to an existing group because we have so many systems that you can open the page with list of systems... Also opening up papertrail overview is really slow. We can probably scale down some logging, but we should figure out how to do logging right.
Note, we might also try with https://logentries.com/
Update, Papertail said they: - want to support our obscure case :) - agree it's slow, and - will get back to us with more info. So let's give them some time. I love it when problems solves themselves.
:kang, suggests that we consider mozdef as a potential service for logging. We should look more into that. Obviously debuggability features, speed, availability and scalability is important to us. Anyways, we might end up piping logs here for sec grepping anyways, as far as I understand.
Component: TaskCluster → General
Product: Testing → Taskcluster
Papertrail got a bit, faster... could be better... but it's a start... Yes, we should log less and possibly find better solutions. But for now this works.
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.