1057359 - [Meta] Incorrect/missing bug suggestions for failures

Status: RESOLVED FIXED

(Tree Management :: Treeherder, defect, P1)

Description

[Ed Morley [:emorley]]

Breaking this out of bug 1032643.

This bug is for collecting examples of incorrect/missing bug suggestions for failures, compared to TBPL. If the error summary is wrong/missing failure lines, that should be filed separately, since this is just about suggestion generation.

Comment 1

[Ed Morley [:emorley]]

https://treeherder.mozilla.org/ui/logviewer.html#?job_id=282513&repo=mozilla-central

 command timed out: 3600 seconds without output running ['/tools/buildbot/bin/python', 'scripts/scripts/android_panda_talos.py', '--talos-suite', 'remote-tspaint', '--cfg', 'android/android_panda_talos_releng.py', '--branch-name', 'mobile', '--blob-upload-branch', 'mobile'], attempting to kill 

Treeherder doesn't suggest anything, TBPL suggests bug 1054456.

Note:
https://hg.mozilla.org/webtools/tbpl/file/06af9dad7e20/php/inc/AnnotatedSummaryGenerator.php#l167

Comment 2

[Ed Morley [:emorley]]

https://treeherder.mozilla.org/ui/logviewer.html#?job_id=282913&repo=mozilla-central

 TEST-UNEXPECTED-FAIL | /builds/slave/test/gaia/build/test/integration/build.test.js | Build Integration tests make with GAIA_OPTIMIZE=1 BUILD_DEBUG=1 

Treeherder suggests:
1022192 Intermittent build.test.js | Build GAIA from differece app list GAIA_DEVICE_TYPE=tablet make
1024047 Intermittent build.test.js | Node modules tests make node_modules from github or git mirror
1025111 Intermittent build.test.js | Build Integration tests make with HAIDA=1 (and more) 

TBPL:
Bug 1022192 - Intermittent build.test.js | Build GAIA from differece app list GAIA_DEVICE_TYPE=tablet make
Bug 1024047 - Intermittent build.test.js | Node modules tests make node_modules from github or git mirror
Bug 1025111 - Intermittent build.test.js | Build Integration tests make with HAIDA=1 (and more)
Bug 1055754 - Intermittent build.test.js | Build Integration tests make with GAIA_OPTIMIZE=1 BUILD_DEBUG=1 | Build Integration tests "after each" hook after "Component returned failure code: 0x80520006 (NS_ERROR_FILE_TARGET_DOES_NOT_EXIST)" error
[closed] Bug 1022196 - Intermittent build.test.js | Build Integration tests make with DEBUG=1
[closed] Bug 1047244 - gaia-build jobs failing with "build.test.js | Build GAIA from differece app list GAIA_DEVICE_TYPE=tablet" or "multilocale.test.js | multilocale Integration tests "before all" hook"

The bug we're after here is bug 1055754 - is treeherder's bug cache not up to date?

Comment 3

[Ed Morley [:emorley]]

https://treeherder.mozilla.org/ui/logviewer.html#?job_id=1683386&repo=mozilla-inbound

Treeherder:
{
TEST-UNEXPECTED-FAIL | test_execute_isolate.py TestExecuteIsolationChrome.test_execute_async_isolate | AssertionError: ScriptTimeoutException not raised
TEST-UNEXPECTED-ERROR | test_shared_all.py TestSharedUnits.test_units | NoSuchElementException: NoSuchElementException: Unable to locate element: complete 
   1054793 Intermittent test_shared_all.py TestSharedUnits.test_units | NoSuchElementException: NoSuchElementException: Unable to locate element: complete
Return code: 10 
   1018943 Intermittent Image Loader Test Suite > imgsLoader resuming > resuming calls new Image() | expected Image to be called once but was called twice Return code: 10
}

TBPL:
{
TEST-UNEXPECTED-FAIL | test_execute_isolate.py TestExecuteIsolationChrome.test_execute_async_isolate | AssertionError: ScriptTimeoutException not raised
    Bug 1001296 - Intermittent test_execute_isolate.py test_execute_isolate.TestExecuteIsolationChrome.test_execute_async_isolate | AssertionError: ScriptTimeoutException not raised TEST-UNEXPECTED-ERROR | test_shared_all.py TestSharedUnits.test_units | NoSuchElementException: NoSuchElementException: Unable to locate element: complete
    Bug 1054793 - Intermittent test_shared_all.py TestSharedUnits.test_units | NoSuchElementException: NoSuchElementException: Unable to locate element: complete
Return code: 10
}

1) Bug 1001296 was missing from treeherder's suggestion
2) Treeherder should blacklist 'Return code: 10' (note: https://hg.mozilla.org/webtools/tbpl/file/tip/php/inc/AnnotatedSummaryGenerator.php#l142)

Comment 4

[Ed Morley [:emorley]]

(In reply to Ed Morley [:edmorley] from comment #3)
> 2) Treeherder should blacklist 'Return code: 10' (note:
> https://hg.mozilla.org/webtools/tbpl/file/tip/php/inc/
> AnnotatedSummaryGenerator.php#l142)

Seems like we're for some reason not using the blacklist?
https://github.com/mozilla/treeherder-service/blob/master/treeherder/log_parser/utils.py#L12

Comment 5

[Mauro Doglio [:mdoglio]]

I think the bug is here
https://github.com/mozilla/treeherder-service/blob/master/treeherder/log_parser/utils.py#L77
We should get rid of the 'or error_line' I guess. If the error line is not an helpful search term it should return None

Comment 6

[Ed Morley [:emorley]]

(In reply to Mauro Doglio [:mdoglio] from comment #5)
> I think the bug is here
> https://github.com/mozilla/treeherder-service/blob/master/treeherder/
> log_parser/utils.py#L77
> We should get rid of the 'or error_line' I guess. If the error line is not
> an helpful search term it should return None

Broken this out to bug 1057377 :-)

Comment 7

[Ed Morley [:emorley]]

Unlike some of the other meta bugs, the issues under this one are quite discrete, rather than a fuzzy cumulative-UX thing. As such, I thing the dependants of this bug should block the blocker bug directly, rather than this one. I'll keep this bug around for now to link together the various bugs tracking improvements and close it out once they're all done.

Comment 8

[Ryan VanderMeulen [:RyanVM]]

Other common examples:
* Bug 1078237
* Bug 1083952

Comment 9

[Ed Morley [:emorley]]

(In reply to Ryan VanderMeulen [:RyanVM UTC-4] from comment #8)
> Other common examples:
> * Bug 1078237

eg:
https://treeherder.mozilla.org/ui/logviewer.html#?repo=mozilla-aurora&job_id=353503

Which leads to:
https://treeherder.mozilla.org/api/project/mozilla-aurora/artifact/?job_id=353503&name=Bug+suggestions&type=json

And thus:

"search": "TEST-UNEXPECTED-FAIL | test_switch_frame.py TestSwitchFrame.test_should_be_able_to_carry_on_working_if_the_frame_is_deleted_from_under_us | AssertionError: 0 != 1",

("search" is actually not a great descriptor, I've filed bug 1091060 for renaming it)

Querying the bugscache for what we _should_ be using as a search term:
https://treeherder.mozilla.org/api/bugscache/?search=test_switch_frame.py%20TestSwitchFrame.test_should_be_able_to_carry_on_working_if_the_frame_is_deleted_from_under_us

Successfully finds bugs.

So this is a problem at some point prior to querying the bugscache (eg search term extraction). Going to clean up the tests and add one for this case, to track it down.

Comment 10

[Ed Morley [:emorley]]

(In reply to Ed Morley [:edmorley] from comment #9)
> Querying the bugscache for what we _should_ be using as a search term:
> https://treeherder.mozilla.org/api/bugscache/?search=test_switch_frame.
> py%20TestSwitchFrame.
> test_should_be_able_to_carry_on_working_if_the_frame_is_deleted_from_under_us
> 
> Successfully finds bugs.
> 
> So this is a problem at some point prior to querying the bugscache (eg
> search term extraction). Going to clean up the tests and add one for this
> case, to track it down.

Ha, as is always the way, actually writing the tests out in bug 1059686 (https://github.com/mozilla/treeherder-service/pull/271) made me realise that the above isn't true. The search term above isn't actually what we're using, since it's being truncated at 100 characters (and by design) - in this case the test name is so long we hit the 100 char cap (whereas normally we'd only expect to do so for the long non-pipe delimited error lines).

As such, we're searching for:
https://treeherder.mozilla.org/api/bugscache/?search=test_switch_frame.py%20TestSwitchFrame.test_should_be_able_to_carry_on_working_if_the_frame_is_deleted

Which returns 0 results, and so means it's the bugscache search at fault.

I would imagine this is due to our current use of boolean mode full-text searches:
https://github.com/mozilla/treeherder-service/blob/master/treeherder/model/sql/reference.json#L324
http://dev.mysql.com/doc/refman/5.5/en/fulltext-boolean.html

I'll carry on looking into this tomorrow.

Comment 11

[Ed Morley [:emorley]]

Seems like we're all good here now :-)
Let's file any other issues as new bugs.