807707 - If a filename can't be found in a log failure line, search for bug suggestions using the whole line instead of giving up

Status: RESOLVED FIXED

(Tree Management Graveyard :: TBPL, defect)

Description

[Ed Morley [:emorley]]

ie we currently do:

>   protected function processLine(&$annotatedLines, $line) {
>     $annotatedLines[] = htmlspecialchars($line);
> 
>     $tokens = preg_split("/\s\\|\s/", $line);
>     if (count($tokens) < 3)
>       return;

Instead of returning, let's just use the whole line, so we can star talosError and more.

#107 of the "why didn't we do this sooner" list (albeit I guess until recently we didn't have annotated summary support for talosError, but still).

Comment 1

[Ed Morley [:emorley]]

Attachment: Part 1: s/filename/searchTerm/g

...since we'll be searching for more than just filenames now.

Comment 2

[Ed Morley [:emorley]]

Attachment: Part 2: Fallback to whole line search

Diff looks noisier than it actually is.

Only real change is that we replace the return with '$searchTerm = trim($line);'

Comment 3

[Ed Morley [:emorley]]

Attachment: Part 3: $searchTerm tweaks

* Trim $searchTerm for both filename & whole line codepaths
* Bail early if post-trim the search term is empty
* Cap the search term at 200 characters, because:
Bugzilla's max summary length is 256 characters, so once "Intermittent " and platform/suite information is prefixed, it often only leaves ~200 characters for the failure, so anything more will be truncated in the summary. Capping the length (a) prevents crazy length bzapi requests, and (b) might mean we still match the truncated failure in the summary.

Comment 4

[Ed Morley [:emorley]]

Attachment: Screenshot

huzzah!

Comment 5

[Ed Morley [:emorley]]

Attachment: bzapi calls

A quick glance at the bzapi requests being made with this patch shows that we now search for a few extra things that are false positives. However many of them will require switching to a regex-based blacklist, so I think we should defer to another bug, given we already search for stupid things (a la bug 788518).

eg:
{
PGOCVT : fatal error PG0001: An unexpected internal error was detected in source file 'f:\dd\vctools\compiler\utc\src\tools\pogo\cvtpgd\cvtpgd.cpp', line 800.
Thread 12 (crashed)
The web-page 'stop build' button was pressed by '[e-mail]': Cancelled via self-serve
abort: HTTP Error 404: Not Found
buildbot.slave.commands.TimeoutError: command timed out: 1200 seconds without output, attempting to kill
buildbot.slave.commands.TimeoutError: command timed out: 2400 seconds without output, killing pid 34848
make[5]: *** [BaselineFrameInfo.o] Error 1
}

We'll also need to deal with mozharness' prefixing:

{
09:15:44     INFO -  TEST-UNEXPECTED-FAIL : TimeoutException: socket.timeout
09:16:03     INFO -  TEST-UNEXPECTED-FAIL : TimeoutException: socket.timeout
}

On the plus side, we now catch all of the following (and this is just from the last hour or two of running TBPL locally):

{
talosError: "Could not find report in browser output: [('tsformat', ('__start_report', '__end_report')), ('tpformat', ('__start_tp_report', '__end_tp_report'))] [browser_output.txt]"
talosError: 'browser non-zero return code (1) [browser_output.txt]'
talosError: 'crash during run (stack found)'
talosError: 'failed to initialize browser'
talosError: 'initialization timed out'
talosError: 'stack found after process termination (org.mozilla.fennec: terminated by testAgent.killProcess, plugin-container: terminated by testAgent.killProcess, crashreporter: terminated by testAge
talosError: Unable to proceed with missing counter 'tp5n_main_startup_fileio'
TEST-UNEXPECTED-FAIL | Disconnect Error: Application unexpectedly closed
TEST-UNEXPECTED-FAIL | | exception while running reftests
}

Comment 6

[Ed Morley [:emorley]]

Thank you once again :-)

https://hg.mozilla.org/users/mstange_themasta.com/tinderboxpushlog/rev/72c65e5f7f04
https://hg.mozilla.org/users/mstange_themasta.com/tinderboxpushlog/rev/1b245dbb7151
https://hg.mozilla.org/users/mstange_themasta.com/tinderboxpushlog/rev/535535d1bb0c

Comment 7

[Ed Morley [:emorley]]

With this patch we can now star all of:
https://bugzil.la/803413,732325,796914,690176,739089,686085,686245,802475,781107,675297,806123,798219,781106,739089,802289,690176,782189
(and that's just open bugs & a very rough search).

:-D

Comment 8

[Ed Morley [:emorley]]

In production.

Comment 9

[Ed Morley [:emorley]]

(In reply to Ed Morley [:edmorley UTC+0] from comment #5)
> 09:15:44     INFO -  TEST-UNEXPECTED-FAIL : TimeoutException: socket.timeout
> 09:16:03     INFO -  TEST-UNEXPECTED-FAIL : TimeoutException: socket.timeout

This will soon be fixed in mozharness afaict (bug 804366).