1384015 - some failures are missing in Test-Centric UI

Status: RESOLVED DUPLICATE of bug 1414949

(Tree Management Graveyard :: Treeherder: Test-based View, defect)

Description

[Yoshi Cheng-Hao Huang [:allstars.chh][:allstarschh][:yoshi]]

Try https://treeherder.mozilla.org/#/jobs?repo=try&revision=06bdaafcfde78695af02cd0f242eb9402bd42ed5

You can see mochitest-chrome 1 is failing
https://treeherder.mozilla.org/#/jobs?repo=try&revision=06bdaafcfde78695af02cd0f242eb9402bd42ed5&selectedJob=116786116

FAIL | dom/base/test/chrome/test_title.xul | XUL <window> title attribute
got "Problem loading page", expected "Test"
[log…]


However in Test-Centric UI
https://treeherder-manifest.herokuapp.com/?repo=try&revision=06bdaafcfde78695af02cd0f242eb9402bd42ed5

The test dom/base/test/chrome/test_title.xul is missing

Comment 1

[Cameron Dawson [:camd]]

Thanks for reporting this.  I'm very keen to flesh out any discrepancies.

Though, in this case, I'm actually seeing the test there now.  There is a lot of noise (failing tests) on that page, so it was hard for me to spot it without filtering.  If you type "dom/base/test" in the filter field, are you able to see it now?

Comment 2

[Yoshi Cheng-Hao Huang [:allstars.chh][:allstarschh][:yoshi]]

I am really sorry I found that I pasted a wrong try link 

The correct try should be 
https://treeherder.mozilla.org/#/jobs?repo=try&revision=8457500b8f53732247e552b9a51c504e5eb8261e&selectedJob=116728829


As you can see there are many failures in c1,

like 

TEST-UNEXPECTED-FAIL | devtools/server/tests/mochitest/test_Debugger.Source.prototype.introductionType.html | Test timed out. [log…]

 
TEST-UNEXPECTED-FAIL | devtools/server/tests/mochitest/test_makeGlobalObjectReference.html | uncaught exception - TypeError: g2iDO.getOwnPropertyDescriptor(...) is undefined at iframe2OnLoad@chrome://mochitests/content/chrome/devtools/server/tests/mochitest/test_makeGlobalObjectReference.html:80:21 [log…]

 
TEST-UNEXPECTED-FAIL | devtools/shared/webconsole/test/test_commands_other.html | Test timed out. [log…]

 
TEST-UNEXPECTED-FAIL | devtools/shared/webconsole/test/test_commands_registration.html | top.foo should equal to 'bar' - got undefined, expected "bar" [log…]

 
TEST-UNEXPECTED-FAIL | devtools/shared/webconsole/test/test_commands_registration.html | Test timed out. [log…]

 
TEST-UNEXPECTED-FAIL | devtools/shared/webconsole/test/test_console_serviceworker.html | received correct number of console calls - got +0, expected 6 [log…]

 
TEST-UNEXPECTED-FAIL | devtools/shared/webconsole/test/test_console_serviceworker.html | unexpected error: TypeError: call is undefined [log…]

 
TEST-UNEXPECTED-FAIL | devtools/shared/webconsole/test/test_console_serviceworker_cached.html | 'from' is undefined [log…]

     

    1232949 - Intermittent test_console_serviceworker_cached.html | received correct number of console calls - got +0, expected 1, Test timed out [ 0.10 ]

 
TEST-UNEXPECTED-FAIL | devtools/shared/webconsole/test/test_console_serviceworker_cached.html | 'input' is undefined [log…]

     

    1232949 - Intermittent test_console_serviceworker_cached.html | received correct number of console calls - got +0, expected 1, Test timed out [ 0.10 ]

 
TEST-UNEXPECTED-FAIL | devtools/shared/webconsole/test/test_console_serviceworker_cached.html | 'result' is undefined [log…]

     

    1232949 - Intermittent test_console_serviceworker_cached.html | received correct number of console calls - got +0, expected 1, Test timed out [ 0.10 ]

 
TEST-UNEXPECTED-FAIL | devtools/shared/webconsole/test/test_console_serviceworker_cached.html | #foo textContent should equal to ">o_/" - got "", expected ">o_/" [log…]

     

    1232949 - Intermittent test_console_serviceworker_cached.html | received correct number of console calls - got +0, expected 1, Test timed out [ 0.27 ]

 
TEST-UNEXPECTED-FAIL | devtools/shared/webconsole/test/test_console_serviceworker_cached.html | 'result' is undefined [log…]

     

    1232949 - Intermittent test_console_serviceworker_cached.html | received correct number of console calls - got +0, expected 1, Test timed out [ 0.10 ]

 
TEST-UNEXPECTED-FAIL | devtools/shared/webconsole/test/test_console_serviceworker_cached.html | 'result' is undefined [log…]

     

    1232949 - Intermittent test_console_serviceworker_cached.html | received correct number of console calls - got +0, expected 1, Test timed out [ 0.10 ]

 
TEST-UNEXPECTED-FAIL | devtools/shared/webconsole/test/test_console_serviceworker_cached.html | 'result' is undefined [log…]

     

    1232949 - Intermittent test_console_serviceworker_cached.html | received correct number of console calls - got +0, expected 1, Test timed out [ 0.10 ]

 
TEST-UNEXPECTED-FAIL | devtools/shared/webconsole/test/test_console_styling.html | 'result' is undefined [log…]

 
TEST-UNEXPECTED-FAIL | devtools/shared/webconsole/test/test_jsterm_queryselector.html | Test timed out. [log…]

 
TEST-UNEXPECTED-FAIL | devtools/shared/webconsole/test/test_network_get.html | Test timed out. [log…]

     

    1248111 - Intermittent test_network_get.html | Test timed out. [ 0.89 ]

 
TEST-UNEXPECTED-FAIL | (SimpleTest/TestRunner.js) | 4 test timeouts, giving up. [log…]

 
TEST-UNEXPECTED-FAIL | (SimpleTest/TestRunner.js) | Skipping 12 remaining tests. [log…]

 
TEST-UNEXPECTED-FAIL | (SimpleTest/TestRunner.js) | chrome://mochitests/content/chrome/devtools/shared/webconsole/test/test_network_get.html - finished in a non-clean fashion, probably because it didn't call SimpleTest.finish() [log…]

 
TEST-UNEXPECTED-ERROR | (SimpleTest/TestRunner.js) | Finished in 326904ms [log…]

 
TEST-UNEXPECTED-FAIL | docshell/test/chrome/test_principalInherit.xul | load in type=content iframe inherited principal of same type parent - got undefined, expected true [log…]

 
TEST-UNEXPECTED-FAIL | docshell/test/chrome/test_principalInherit.xul | Test timed out. [log…]

 
TEST-UNEXPECTED-FAIL | dom/base/test/chrome/test_title.xul | XUL <window> title attribute - got "Problem loading page", expected "Test" [log…]

 
TEST-UNEXPECTED-FAIL | dom/base/test/chrome/test_title.xul | Setting XUL <window> title attribute: DOMTitleChange not fired [log…]

 
TEST-UNEXPECTED-FAIL | dom/base/test/chrome/test_title.xul | DOM: Setting 'title' in XUL - got null, expected "Hello" [log…]

 
TEST-UNEXPECTED-FAIL | dom/base/test/chrome/test_title.xul | DOM: Setting 'title' in XUL - got [object Text], expected null [log…] 


But from its test-Centric UI
https://treeherder.mozilla.org/#/jobs?repo=try&revision=8457500b8f53732247e552b9a51c504e5eb8261e

I cannot find those failures.

Comment 3

[Cameron Dawson [:camd]]

Yoshi--  Ahh, ok.  Yes, I can reproduce this.  Thank you so much for reporting.  I'll investigate and get to the bottom of this.

Comment 4

[Cameron Dawson [:camd]]

James--  It looks like Yoshi is seeing a test that only gets the error lines from a non-structured log.  Does this look right to you?

https://treeherder.mozilla.org/#/jobs?repo=try&revision=8457500b8f53732247e552b9a51c504e5eb8261e&selectedJob=116728829&filter-searchStr=6583ca4e0b17fde38f8860c24a197ad41679e26f

The TestGroup UI only fetches FailureLine entries.  But I know the Autoclassify UI gets TextLogError entries that have been cross-referenced with what our log-parser has extracted.  So there will always be some tests that your UI will show that mine won't.  Unless of course I start showing test lines from TextLogError, but that'd be a big expansion.  :)

Comment 5

[James Graham [:jgraham]]

It looks like that job doesn't have an *_errorsummary.log file uploaded at all. So there will only be text_log_error entires, no failure_line rows at all. I don't know what's causing that, but at a guess the problem is that the task times out, so we force kill the container, which means we only end up with the text logs that taskcluster deals with directly and miss all the structured logging that's happening inside mozharness. So I think a proper solution here would need input from the taskcluster people to ensure that we extricate the structured logging up to the point at which the container is killed, e.g. by writing it to some shared area and having taskcluster itself take responsibility for uploading any files written there, rather than assuming that mozharness will run to completion.

Comment 6

[Cameron Dawson [:camd]]

This looks like a symptom of hitting a harness level error and not completing so we don't get that log to parse.  So duping to bug 1414949