1652931 - Investigated increased limit of failure lines stored per task (210 instead of 100 failure lines)

Status: NEW

(Tree Management :: Treeherder: Data Ingestion, task)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Since omitting the text_log_step table, tasks can have 210 failure lines compared to 100 before. Please check if this is intentional.

Examples:
https://treeherder.mozilla.org/logviewer.html#?job_id=309822869&repo=autoland
https://treeherder.mozilla.org/logviewer.html#?job_id=309822132&repo=autoland

Comment 1

[Sarah Clements [:sclements]]

(In reply to Sebastian Hengst [:aryx] (needinfo on intermittent or backout) from comment #0)

Since omitting the text_log_step table, tasks can have 210 failure lines compared to 100 before. Please check if this is intentional.

Interesting. Did you compare it locally to a previous commit and see less failure lines previously for this same job? Looking at those links, I don't see duplicate line numbers... Is there a reason why we wouldn't want it to find all the failure lines?

Btw, I haven't removed the TextLogStep table yet - just the StepParser and stopped storing "steps" in that table.

Comment 2

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

This query counts the failure line for successful jobs and compares the count to one month ago.

Afaik the limit is in place to prevent people spamming the database.

Comment 3

[Sarah Clements [:sclements]]

Ah, yeah, it was flagged during review: https://github.com/mozilla/treeherder/pull/6546#discussion_r445223394

We can keep an eye on it and if it's a problem add it back in.

Comment 4

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Current maximum is 11,467 failure lines for a task which also causes the browser to hang when such tasks get selected.

Comment 5

[Armen [:armenzg]]

Let's add the limit back or back out the current log parsing code.
See bug 1653250.

Comment 7

[Armen [:armenzg]]

I'm backing out for now and let you follow up with time.
The spikes can also be seen on New Relic for treeherder-stage.

Perhaps modifying this management command and picking a log that takes long to parse will help find why certain logs are failing:

./manage.py test_parse_log https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/AG7pAidnSnSJjfewPLrV5w/runs/0/artifacts/public/logs/live_backing.log

There's also some spamming print statement in Papertrail:

Jul 16 13:24:29 treeherder-prod app/worker_log_parser.3 NoneType: None

Comment 8

[Sarah Clements [:sclements]]

I'll add that limit back in.

There's also some spamming print statement in Papertrail:
Jul 16 13:24:29 treeherder-prod app/worker_log_parser.3 NoneType: None

That's been happening for a while and is unrelated. I'll see if I can pinpoint what's causing it.

Comment 9

[Sarah Clements [:sclements]]

(In reply to Sebastian Hengst [:aryx] (needinfo on intermittent or backout) from comment #4)

Current maximum is 11,467 failure lines for a task which also causes the browser to hang when such tasks get selected.

Wow, good to know. Thanks for flagging this Sebastian.

Comment 10

[Sarah Clements [:sclements]]

I have a pr for this btw: https://github.com/mozilla/treeherder/pull/6634