Closed Bug 1016251 Opened 7 years ago Closed 7 years ago

Reftest analyzer shows failures twice because of logcat

Categories

(Testing :: Reftest, defect)

x86
macOS
defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED
mozilla32

People

(Reporter: kats, Assigned: kats)

References

Details

Attachments

(1 file)

Using the try push at https://tbpl.mozilla.org/?tree=Try&rev=866e11df3666 as an example. Click on R1, and open the reftest analyzer. The http://10.0.2.2:8888/tests/layout/reftests/reftest-sanity/async-scroll-1b.html test is listed twice.

This happens because the log excerpt provided by tbpl is this: https://tbpl.mozilla.org/php/getLogExcerpt.php?id=40432940&type=reftest

Note here that the REFTEST IMAGE line shows up twice - once printed by the reftest harness itself, and once because logcat output is echoed back into the log file. Also important is that the logcat version is truncated because of logcat line length limits.

Ideally the analyzer should ignore the logcat output and just use the harness output.
Not sure whether best to fix in the reftest viewer, or the TBPL parser that generates the excerpt that gets passed to the viewer (https://hg.mozilla.org/webtools/tbpl/file/03d9ff82b54a/php/inc/ReftestFailureFilter.php#l19) - or both?
I thought it would be better to fix the analyzer for when people are running it locally. When I ran b2g-emulator reftests locally the logcat was separate from the reftest output so the mixing wasn't a problem, but using the logcat version would still result in the truncation due to line length limits. So I think it's better for the analyzer to detect and ignore stuff from logcat, to remove this footgun.
Comment on attachment 8429127 [details] [diff] [review]
Ignore logcat output

Could you add an "// ignore duplicated output in logcat" or something similar?

r=dbaron
Attachment #8429127 - Flags: review?(dbaron) → review+
https://hg.mozilla.org/mozilla-central/rev/1da9076a74ed
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla32
Ugh. Looks like some devices generate logcat without the slash between I and Gecko. For example https://tbpl.mozilla.org/php/getLogExcerpt.php?id=41621436&type=reftest has a line of the form

06-12 19:16:41.395   691   691 I Gecko   : REFTEST   IMAGE 1 (TEST) [snip]

and that doesn't get caught by my regex.
(In reply to Kartikaya Gupta (email:kats@mozilla.com) from comment #7)
> Ugh. Looks like some devices generate logcat without the slash between I and
> Gecko. For example
> https://tbpl.mozilla.org/php/getLogExcerpt.php?id=41621436&type=reftest has
> a line of the form
> 
> 06-12 19:16:41.395   691   691 I Gecko   : REFTEST   IMAGE 1 (TEST) [snip]
> 
> and that doesn't get caught by my regex.

See bug 1016371 comment 122 onwards - I'm planning on seeing if I can fix the other instances to use the same format, which will fix this for free :-)
Depends on: 1027574
Duplicate of this bug: 1027116
Reopening this pending the state of bug 1027574.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Assignee: bugmail.mozilla → emorley
Given that bug 1016371 has fixed this on TBPL for both the normal case and the B2G 'threadtime' format logcat (and landed today) and that I hope to switch that format to 'time' like everywhere else in bug 1027574 (now that I've tracked down where it's coming from), I think actually that the patch as landed here is absolutely fine (if we adjust the regex to support threadtime format, we'll only have to undo it later).

As such, closing and restoring assignee; sorry for the churn.
Assignee: emorley → bugmail.mozilla
Status: REOPENED → RESOLVED
Closed: 7 years ago7 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.