1473565 - Bug suggestions not shown after log format changes in bug 1471717

Status: RESOLVED FIXED

(Tree Management :: Treeherder, defect, P1)

Description

[Andreea Pavel [:apavel]]

There is the following failure: 
https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-classifiedState=unclassified&selectedJob=186601855

This failure has a bug already created, with an exact summary match, but it's not being shown in the failure summary: https://bugzilla.mozilla.org/show_bug.cgi?id=1414495

This is the most recent example, there are others.

Comment 1

[Ed Morley [:emorley]]

Hi!

That's because the error line shown in the failure summary, is:

ERROR - TEST-UNEXPECTED-TIMEOUT | automation.py | application timed out after 370 seconds with no output


...rather than:

TEST-UNEXPECTED-TIMEOUT | automation.py | application timed out after 370 seconds with no output

Looking at the raw log, I see:

[task 2018-07-05T12:55:50.089Z]    ERROR - TEST-UNEXPECTED-TIMEOUT | automation.py | application timed out after 370 seconds with no output

Whereas logs previously used to have the format:

[task 2018-07-02T21:03:31.262Z] 21:03:31    ERROR - TEST-UNEXPECTED-FAIL | ...

(see this mozilla-beta log for example: https://treeherder.mozilla.org/logviewer.html#?job_id=186038852&repo=mozilla-beta&lineNumber=3401)

It looks like the log format has been changed (to remove the duplicated timestamp; which is a reasonable change) but without a corresponding Treeherder bug to confirm it's still compatible / add support. 

Searching Bugzilla I see bug 1471717.

Geoff, I don't suppose you'd mind updating the regex here:
https://github.com/mozilla/treeherder/blob/efcff8e586c02e9d94d76ca7c175eed2c6eb8dd7/treeherder/log_parser/parsers.py#L394-L409

...and adding a test here?
https://github.com/mozilla/treeherder/blob/master/tests/model/test_error_summary.py

It may also be worth adding a test for the old format, to ensure we don't regress bug suggestions on beta/... when updating the regexes.

For some extra context: The reason bug suggestions don't work in this specific case but do in others, is that "automation.py" is on `is_helpful_search_term()`'s blacklist, so this case uses the "full line search term" mode (rather than "just search for the test name from between the pipe symbols" default mode), which requires that the prefxing trimming be accurate.

Comment 2

[Geoff Brown [:gbrown]]

Sigh. Sorry about that.

I can try to fix this, but it might be faster if a regular treeherder contributor took it on. I'll need to ask dumb questions like:

 - Why doesn't this just search for '^.* +(ERROR|...'?
 - How do I run those tests?

Comment 3

[Ed Morley [:emorley]]

(In reply to Geoff Brown [:gbrown] from comment #2)
>  - Why doesn't this just search for '^.* +(ERROR|...'?

Possibly to avoid mid-line matches (eg: `TEST-UNEXPECTED-FAIL | some-test.py | ERROR: Blah`), though I don't entirely know (has been a while since those regexes were last updated / I've had to think about the log parser).

>  - How do I run those tests?

https://treeherder.readthedocs.io/installation.html
https://treeherder.readthedocs.io/common_tasks.html#running-the-tests

(or open a PR and Travis will run them)

The Treeherder team is pretty resource-constrained at the moment - perhaps a backout of bug 1471717 would be easiest for now?

Comment 4

[Geoff Brown [:gbrown]]

I took a run at this, but was defeated by vagrant installation. I'll leave a real solution to real treeherder devs.

Meanwhile, bug 1471717 is being backed out.

Comment 5

[Andreea Pavel [:apavel]]

Hi guys, i keep having issues and filing new bugs because there are no suggestions, even thought there is an exact specific path. Who needs to do the backout of bug 1471717 until a better solution is available? 

Thank you.

Comment 6

[Geoff Brown [:gbrown]]

Bug 1471717 was backed out as of 2018-07-06. If you are still having trouble, let's look at some examples...maybe it's a different issue?

Comment 7

[Andreea Pavel [:apavel]]

(In reply to Geoff Brown [:gbrown] from comment #6)
> Bug 1471717 was backed out as of 2018-07-06. If you are still having
> trouble, let's look at some examples...maybe it's a different issue?

Sure, i've had plenty tonight :D

https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&group_state=expanded&selectedJob=187024945

https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&group_state=expanded&selectedJob=187028621

https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&group_state=expanded&selectedJob=187026978

 https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&fromchange=cc3401e78e8bbae22e6dbc854e525ceae4923bcf&filter-classifiedState=unclassified&group_state=expanded&selectedJob=187023649 
^ the same failure, here has a bug suggestion:

 https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&fromchange=cc3401e78e8bbae22e6dbc854e525ceae4923bcf&filter-classifiedState=unclassified&group_state=expanded&selectedJob=187023649

 https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&fromchange=cc3401e78e8bbae22e6dbc854e525ceae4923bcf&filter-classifiedState=unclassified&group_state=expanded&selectedJob=187023649


 https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&fromchange=cc3401e78e8bbae22e6dbc854e525ceae4923bcf&filter-classifiedState=unclassified&group_state=expanded&selectedJob=187023649
^ the same failure here has a suggestion:

 https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&fromchange=cc3401e78e8bbae22e6dbc854e525ceae4923bcf&filter-classifiedState=unclassified&group_state=expanded&selectedJob=187023649


Hope these suffice, there were others. What am i seeing wrong?

Comment 8

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

(In reply to Andreea Pavel [:apavel] from comment #7)
>  https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-
> resultStatus=testfailed&filter-resultStatus=busted&filter-
> resultStatus=exception&fromchange=cc3401e78e8bbae22e6dbc854e525ceae4923bcf&fi
> lter-classifiedState=unclassified&group_state=expanded&selectedJob=187023649 
> ^ the same failure, here has a bug suggestion:
That is bug 1407118.

The following URLs are all the same: 
>  https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-
> resultStatus=testfailed&filter-resultStatus=busted&filter-
> resultStatus=exception&fromchange=cc3401e78e8bbae22e6dbc854e525ceae4923bcf&fi
> lter-classifiedState=unclassified&group_state=expanded&selectedJob=187023649
> 
>  https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-
> resultStatus=testfailed&filter-resultStatus=busted&filter-
> resultStatus=exception&fromchange=cc3401e78e8bbae22e6dbc854e525ceae4923bcf&fi
> lter-classifiedState=unclassified&group_state=expanded&selectedJob=187023649
> 
> 
>  https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-
> resultStatus=testfailed&filter-resultStatus=busted&filter-
> resultStatus=exception&fromchange=cc3401e78e8bbae22e6dbc854e525ceae4923bcf&fi
> lter-classifiedState=unclassified&group_state=expanded&selectedJob=187023649
> ^ the same failure here has a suggestion:
> 
>  https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-
> resultStatus=testfailed&filter-resultStatus=busted&filter-
> resultStatus=exception&fromchange=cc3401e78e8bbae22e6dbc854e525ceae4923bcf&fi
> lter-classifiedState=unclassified&group_state=expanded&selectedJob=187023649

Comment 9

[Ed Morley [:emorley]]

The bug suggestions appear fine for me, for the jobs linked in comment 7.

Comment 10

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Saw some devtools jobs from last European night without suggestions.

E.g. https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=72882160c090ad132256ef7b8557f82133311f80&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=runnable&filter-searchStr=devtools&selectedJob=187091145 both jobs have the same failure and one has the suggestion the other one not.

The dt8s at https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=ae04f1ff217f85049f1d157517241274d15ab5bb&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=runnable&selectedJob=187091987 should have bug 1472657 as suggestion.

Comment 11

[Ed Morley [:emorley]]

The bug that caused the issue in this bug has since been backed out, so any other bug suggestion problems are very unlikely to be unrelated. Please can new bugs be filed?

Comment 12

[Andreea Pavel [:apavel]]

:Aryx well, the jobs i selected were different ones, so i don't really know why that did not happen. I found the bugs for all those mentioned links, the problem is there were not suggested in treeherder. 

emorley: when i posted the links, the bug was not backed out.

Comment 13

[Ed Morley [:emorley]]

(In reply to Andreea Pavel [:apavel] from comment #12)
> emorley: when i posted the links, the bug was not backed out.

It was - Geoff's message about the backout is quoted in comment 7. If you mean the jobs were old jobs from before the backout was merged around - then in those situations it's best to check the jobs before pasting, otherwise multiple people spend time investigating them :-)

Comment 14

[Andreea Pavel [:apavel]]

:emorley The backout was made after i pointed out that the issue still occurs and i cannot find the backout in mozilla-central, please see Comment 7 in bug 1473565 and comment 8 in bug 1471717. It was not my intention to make people spend time on issues that are solved, but 2 days ago, this was not solved.

Comment 15

[Ed Morley [:emorley]]

Fixed by backout.

If we still decide to go ahead with bug 1471717 let's open a new bug for any log parser changes required, for easier to follow bug history.

Comment 16

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

(In reply to Sebastian Hengst [:aryx] (needinfo on intermittent or backout) from comment #8)
> (In reply to Andreea Pavel [:apavel] from comment #7)
> >  https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-
> > resultStatus=testfailed&filter-resultStatus=busted&filter-
> > resultStatus=exception&fromchange=cc3401e78e8bbae22e6dbc854e525ceae4923bcf&fi
> > lter-classifiedState=unclassified&group_state=expanded&selectedJob=187023649 
> > ^ the same failure, here has a bug suggestion:
> That is bug 1407118.
Quoted wrong link here, was for https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=84262b86904dd58d544c486b95ca0625cd34860a&selectedJob=187028621