1156070 - Rename test_errors.py's Exception tests so they don't confuse mozharness and thus Treeherder into thinking there are six failures in every single Mn run

Status: RESOLVED FIXED

(Remote Protocol :: Marionette, defect)

Description

[Phil Ringnalda (:philor)]

At least I *think* my second guess about the cause is correct.

If you click on any Mn job in treeherder (green or orange), the panel at the bottom which lists failures will show

TEST-START | test_errors.py TestAllExceptions.test_properties
TEST-START | test_errors.py TestExceptionType.test_cause_string
TEST-START | test_errors.py TestExceptionType.test_cause_tuple
TEST-START | test_errors.py TestExceptionType.test_construction
TEST-START | test_errors.py TestExceptionType.test_defaults
TEST-START | test_errors.py TestExceptionType.test_str 

and every Treeherder Robot bug comment on every Mn intermittent failure includes those six lines too. My first thought was that the filename was the problem, but if you look at the raw log,

07:21:41    ERROR -  TEST-START | test_errors.py TestAllExceptions.test_properties
07:21:41    ERROR -  TEST-PASS | test_errors.py TestAllExceptions.test_properties | took 704ms
07:21:41    ERROR -  TEST-START | test_errors.py TestExceptionType.test_cause_string
07:21:42    ERROR -  TEST-PASS | test_errors.py TestExceptionType.test_cause_string | took 699ms
07:21:42    ERROR -  TEST-START | test_errors.py TestExceptionType.test_cause_tuple
07:21:43    ERROR -  TEST-PASS | test_errors.py TestExceptionType.test_cause_tuple | took 845ms
07:21:43    ERROR -  TEST-START | test_errors.py TestExceptionType.test_construction
07:21:44    ERROR -  TEST-PASS | test_errors.py TestExceptionType.test_construction | took 1122ms
07:21:44    ERROR -  TEST-START | test_errors.py TestExceptionType.test_defaults
07:21:45    ERROR -  TEST-PASS | test_errors.py TestExceptionType.test_defaults | took 1099ms
07:21:45    ERROR -  TEST-START | test_errors.py TestExceptionType.test_str
07:21:46    ERROR -  TEST-PASS | test_errors.py TestExceptionType.test_str | took 1081ms
07:21:46     INFO -  TEST-START | test_errors.py TestLookup.test_by_known_number
07:21:47     INFO -  TEST-PASS | test_errors.py TestLookup.test_by_known_number | took 1094ms

I think it's actually mozharness calling them an ERROR because of "Exception", and then Treeherder trusting mozharness, since only the Exception tests get highlighted.

Comment 2

[David Burns :automatedtester]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=e6322d72ca7a

Comment 3

[David Burns :automatedtester]

Attachment: MozReview Request: Bug 1156070: Rename test class to prevent test harnesses thinking their is a failure; r?jgriffin

Bug 1156070: Rename test class to prevent test harnesses thinking their is a failure; r?jgriffin

Mozharness appears to do a check for the word exception and is seeing these
tests as failures when they are not.

Comment 4

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8650058 [details]
MozReview Request: Bug 1156070: Rename test class to prevent test harnesses thinking their is a failure; r?jgriffin

https://reviewboard.mozilla.org/r/16539/#review14805

Thanks!

Comment 5

[David Burns :automatedtester]

url:        https://hg.mozilla.org/integration/mozilla-inbound/rev/38f5e5feea729b41139e487a4f676f0fa2389474
changeset:  38f5e5feea729b41139e487a4f676f0fa2389474
user:       David Burns <[e-mail]>
date:       Wed Aug 19 15:57:06 2015 +0100
description:
Bug 1156070: Rename test class to prevent test harnesses thinking there is a failure; r=jgriffin

Mozharness appears to do a check for the word exception and is seeing these
tests as failures when they are not.

Comment 6

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/38f5e5feea72

Comment 7

[David Burns :automatedtester]

Please sir, can you uplift to aurora a=test-only

Comment 8

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/c468a9aab0d1