1621938 - decide how to handle huge logs from isolation tasks

Status: NEW

(Tree Management :: Treeherder: Data Ingestion, task, P3)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Today, some isolation tasks had logs above the threshold for getting parsed (to prevent and ingestion backlog):

https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&group_state=expanded&selectedJob=292692175&resultStatus=testfailed%2Cbusted%2Cexception%2Csuperseded%2Cretry%2Cusercancel&revision=6b1d5e72a65b5d97025a566196fa5cb63b7e4549

https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&group_state=expanded&selectedJob=292692177&resultStatus=testfailed%2Cbusted%2Cexception%2Csuperseded%2Cretry%2Cusercancel&revision=6b1d5e72a65b5d97025a566196fa5cb63b7e4549

https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&group_state=expanded&selectedJob=292692178&resultStatus=testfailed%2Cbusted%2Cexception%2Csuperseded%2Cretry%2Cusercancel&revision=6b1d5e72a65b5d97025a566196fa5cb63b7e4549

We need to know if that will become more common and how to handle these tasks (ignore them like now, increase the threshold, assign their own ingestion queue).

Bob will investigate and answer.

Comment 1

[Bob Clary [:bc] (inactive)]

This was introduced in Bug 1295997 where the max download size of the compressed file is set to be 5 Mb with idea that the spammy logs have compression rates of 20-50x. But these examples belie that assumption

filename	size
live_backing.log	78803763
live_backing.log.gz	5457372

camd: Thoughts?

Comment 2

[Cameron Dawson [:camd]]

I think the best solution here is, as Sebastion suggested, to create a separate ingestion queue. To be honest, I have not done a deep-dive into this to see if this would have other implications. But I think it would work to not jam up the works with normal log parsing. However, I must wonder why Ed didn't take this approach in the first place. It seems like that would have occurred to him. But perhaps the thought was that there was no value in parsing these huge logs, so why bother.

In this case, we'd just create a separate celery queue. But we may need a separate worker to handle the parsing? So that may feel like a waste of money if we don't already have a worker that does low-priority work. Still, the cost may be worth it.

Sarah, is this something that would fit into your back-end Treeherder workload?

Comment 3

[Sarah Clements [:sclements]]

Sure, I can add it to my list. Not sure when I'll get to it though. If you think this is high priority, I can bump it to the front of the queue.

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

that seems like a lot of work for something that is more experimental- we have found what looks like value from isolation tasks, but it still needs more investigation/work before we can narrow down a specific protocol and say we have a final plan - there are no plans in the next 2 months to specifically finish that work.

can we just skip these?

Comment 5

[Sarah Clements [:sclements]]

I'll be starting a new project in a few weeks and I already have enough on my plate. If this becomes a high priority, ping someone from the team and we'll figure out who can work on it.