579184 - mobile unit test parser flags tests as green even though they have failed 100% of tests

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P3)

Description

[John Ford [:jhford] CET/CEST Berlin Time]

Basically, log parsing works as expected unless 100% of tests fail because the browser hasn't launched.  If every single test is failing, the log parser reports the build as green which is definitely not good.

We should make sure that the tests show as failing if 100% of the tests fail, or the summary generated would be 'T-FAIL'

http://staging-mobile-master.build.mozilla.org:8020/builders/n810%20mozilla-central%20unit%20xpcshell/builds/2/steps/xpcshell/logs/summary is an example where there were 0 failures reported so the test is green even though the summary message is T-FAIL and there were 0 successful passes.  In this case, I think a status of Fail(red) is appropriate.

Comment 1

[Aki Sasaki (not active)]

tempted to dup to bug 511174.

Comment 2

[Aki Sasaki (not active)]

Attachment: now tested

may or may not make things better.

Comment 3

[Aki Sasaki (not active)]

Comment on attachment 458471 [details] [diff] [review]
now tested

Tested on smm2.
Haven't been able to find any unit test runs after the reconfig that have 0 successful tests, but are green.

Comment 4

[John Ford [:jhford] CET/CEST Berlin Time]

if the the GLIBCXX_3.4.9 issue has not been fixed yet, every n810 unit test run on mozilla-central should pass 0 tests, did they start going orange?

Comment 5

[John Ford [:jhford] CET/CEST Berlin Time]

Comment on attachment 458471 [details] [diff] [review]
now tested

looks good assuming that none of our test harnesses use zero padded test counts or octal representation.

Comment 6

[Aki Sasaki (not active)]

Comment on attachment 458471 [details] [diff] [review]
now tested

http://hg.mozilla.org/build/buildbotcustom/rev/c3edcb488bad