Fix papertrail -- make it not slow again :)



3 years ago
3 years ago


(Reporter: jonasfj, Unassigned)





3 years ago
Papertail is currently so slow it's killing us.
I'm open to giving loggly a try again - just don't switch over exclusively.
(I papertrail is awesome for text logs and so simple)

I was reasonably happy with papertrail before we had 15k systems.
I've reached out to them to see if it can be fixed on their end.

My only regret with papertrail is that it's so easy to end up search across all
systems when you click on say web.1, this is still doable by
searching with "system:taskcluster-queue program:web.1"

Anyways, right now I can't add new systems to an existing group because we have
so many systems that you can open the page with list of systems...
Also opening up papertrail overview is really slow.

We can probably scale down some logging, but we should figure out how to do logging right.

Comment 1

3 years ago
Note, we might also try with

Comment 2

3 years ago
Update, Papertail said they:
 - want to support our obscure case :)
 - agree it's slow, and
 - will get back to us with more info.
So let's give them some time. I love it when problems solves themselves.

Comment 3

3 years ago
:kang, suggests that we consider mozdef as a potential service for logging.
We should look more into that.

Obviously debuggability features, speed, availability and scalability is important to us. 
Anyways, we might end up piping logs here for sec grepping anyways, as far as I understand.
Component: TaskCluster → General
Product: Testing → Taskcluster

Comment 4

3 years ago
Papertrail got a bit, faster... could be better... but it's a start...

Yes, we should log less and possibly find better solutions.
But for now this works.
Last Resolved: 3 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.