1143253 - Fix papertrail -- make it not slow again :)

Status: RESOLVED WONTFIX

(Taskcluster :: General, defect)

Description

[Jonas Finnemann Jensen (:jonasfj)]

Papertail is currently so slow it's killing us.
I'm open to giving loggly a try again - just don't switch over exclusively.
(I papertrail is awesome for text logs and so simple)

I was reasonably happy with papertrail before we had 15k systems.
I've reached out to them to see if it can be fixed on their end.

Note,
My only regret with papertrail is that it's so easy to end up search across all
systems when you click on say web.1, this is still doable by
searching with "system:taskcluster-queue program:web.1"

Anyways, right now I can't add new systems to an existing group because we have
so many systems that you can open the page with list of systems...
Also opening up papertrail overview is really slow.

We can probably scale down some logging, but we should figure out how to do logging right.

Comment 1

[Jonas Finnemann Jensen (:jonasfj)]

Note, we might also try with https://logentries.com/

Comment 2

[Jonas Finnemann Jensen (:jonasfj)]

Update, Papertail said they:
 - want to support our obscure case :)
 - agree it's slow, and
 - will get back to us with more info.
So let's give them some time. I love it when problems solves themselves.

Comment 3

[Jonas Finnemann Jensen (:jonasfj)]

:kang, suggests that we consider mozdef as a potential service for logging.
We should look more into that.

Obviously debuggability features, speed, availability and scalability is important to us. 
Anyways, we might end up piping logs here for sec grepping anyways, as far as I understand.

Comment 4

[Jonas Finnemann Jensen (:jonasfj)]

Papertrail got a bit, faster... could be better... but it's a start...

Yes, we should log less and possibly find better solutions.
But for now this works.