Closed Bug 1156070 Opened 6 years ago Closed 6 years ago

Rename test_errors.py's Exception tests so they don't confuse mozharness and thus Treeherder into thinking there are six failures in every single Mn run

Categories

(Testing :: Marionette, defect)

defect
Not set
normal

Tracking

(firefox40 affected, firefox42 fixed, firefox43 fixed)

RESOLVED FIXED
mozilla43
Tracking Status
firefox40 --- affected
firefox42 --- fixed
firefox43 --- fixed

People

(Reporter: philor, Assigned: automatedtester)

References

Details

(Keywords: pi-marionette-task)

Attachments

(1 file)

At least I *think* my second guess about the cause is correct.

If you click on any Mn job in treeherder (green or orange), the panel at the bottom which lists failures will show

TEST-START | test_errors.py TestAllExceptions.test_properties
TEST-START | test_errors.py TestExceptionType.test_cause_string
TEST-START | test_errors.py TestExceptionType.test_cause_tuple
TEST-START | test_errors.py TestExceptionType.test_construction
TEST-START | test_errors.py TestExceptionType.test_defaults
TEST-START | test_errors.py TestExceptionType.test_str 

and every Treeherder Robot bug comment on every Mn intermittent failure includes those six lines too. My first thought was that the filename was the problem, but if you look at the raw log,

07:21:41    ERROR -  TEST-START | test_errors.py TestAllExceptions.test_properties
07:21:41    ERROR -  TEST-PASS | test_errors.py TestAllExceptions.test_properties | took 704ms
07:21:41    ERROR -  TEST-START | test_errors.py TestExceptionType.test_cause_string
07:21:42    ERROR -  TEST-PASS | test_errors.py TestExceptionType.test_cause_string | took 699ms
07:21:42    ERROR -  TEST-START | test_errors.py TestExceptionType.test_cause_tuple
07:21:43    ERROR -  TEST-PASS | test_errors.py TestExceptionType.test_cause_tuple | took 845ms
07:21:43    ERROR -  TEST-START | test_errors.py TestExceptionType.test_construction
07:21:44    ERROR -  TEST-PASS | test_errors.py TestExceptionType.test_construction | took 1122ms
07:21:44    ERROR -  TEST-START | test_errors.py TestExceptionType.test_defaults
07:21:45    ERROR -  TEST-PASS | test_errors.py TestExceptionType.test_defaults | took 1099ms
07:21:45    ERROR -  TEST-START | test_errors.py TestExceptionType.test_str
07:21:46    ERROR -  TEST-PASS | test_errors.py TestExceptionType.test_str | took 1081ms
07:21:46     INFO -  TEST-START | test_errors.py TestLookup.test_by_known_number
07:21:47     INFO -  TEST-PASS | test_errors.py TestLookup.test_by_known_number | took 1094ms

I think it's actually mozharness calling them an ERROR because of "Exception", and then Treeherder trusting mozharness, since only the Exception tests get highlighted.
Bug 1156070: Rename test class to prevent test harnesses thinking their is a failure; r?jgriffin

Mozharness appears to do a check for the word exception and is seeing these
tests as failures when they are not.
Attachment #8650058 - Flags: review?(jgriffin)
Attachment #8650058 - Flags: review?(jgriffin) → review+
Comment on attachment 8650058 [details]
MozReview Request: Bug 1156070: Rename test class to prevent test harnesses thinking their is a failure; r?jgriffin

https://reviewboard.mozilla.org/r/16539/#review14805

Thanks!
url:        https://hg.mozilla.org/integration/mozilla-inbound/rev/38f5e5feea729b41139e487a4f676f0fa2389474
changeset:  38f5e5feea729b41139e487a4f676f0fa2389474
user:       David Burns <dburns@mozilla.com>
date:       Wed Aug 19 15:57:06 2015 +0100
description:
Bug 1156070: Rename test class to prevent test harnesses thinking there is a failure; r=jgriffin

Mozharness appears to do a check for the word exception and is seeing these
tests as failures when they are not.
https://hg.mozilla.org/mozilla-central/rev/38f5e5feea72
Assignee: nobody → dburns
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla43
Please sir, can you uplift to aurora a=test-only
Flags: needinfo?(ryanvm)
Flags: needinfo?(ryanvm)
You need to log in before you can comment on or make changes to this bug.