1422920 - Support and document comments in bug summary of intermittent failure which will be ignored for matching

Status: RESOLVED INCOMPLETE

(Tree Management :: Treeherder, enhancement)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

• The bug summary is used to match the test file and the failure message.
    • Adding information after the first word “Intermittent” sometimes works.
    • For information added at the end (like if other lines must be in the log earlier), it seems to depend on the length of that comment if they will be shown because the matcher uses them for matching the failure.
    • There should be support for comments which the matcher ignores, e.g. “Intermittent <Win7 debug> folder/test_example.js | Something went wrong <after Assertion failure: false at GC.cpp>” which should be a perfact match for the failure line “TEST-UNEXPECTED-FAIL  folder/test_example.js | Something went wrong”

Comment 1

[Ed Morley [:emorley]]

Hi!

(In reply to Sebastian Hengst [:aryx][:archaeopteryx] (needinfo on intermittent or backout) from comment #0)
> • The bug summary is used to match the test file and the failure message.
>     • Adding information after the first word “Intermittent” sometimes works.
>     • For information added at the end (like if other lines must be in the
> log earlier), it seems to depend on the length of that comment if they will
> be shown because the matcher uses them for matching the failure.

I'm struggling to work out what's being requested for these two?

>     • There should be support for comments which the matcher ignores, e.g.

It seems like this is a solution to a problem that isn't mentioned? (http://xyproblem.info/) Could you describe the original problem with comments in bug summaries, since it's possible that there is a simpler solution to resolve it?

Comment 2

[Phil Ringnalda (:philor)]

FWIW, the problem not mentioned is that if you give autoclassify the two bug summaries

Intermittent filename.html | foo
Intermittent filename.html | bar (when baz quux with foopy goats)

it will select the first one rather than the second for a "filename.html | bar" failure, because the first only has one wrong word and the second has six wrong words. Odds are "after the first word “Intermittent” sometimes works" means that for cases where there's a recognizable filename, words before it are discarded, but for full-line searches they are not (and knowing whether or not a failure is matched by a full-line search requires quite a bit of knowledge of unexposed treeherder internal behavior, so it would seem like randomness that "Intermittent foo bar baz tp6_google | foopy" would be a perfect match for "tp6_google | foopy" while "Intermittent foo bar baz tp6 | foopy" would not be a match for "tp6 | foopy").

Comment 3

[Ed Morley [:emorley]]

Ah - in which case a dupe of bug 1196875.