Rename test_errors.py's Exception tests so they don't confuse mozharness and thus Treeherder into thinking there are six failures in every single Mn run

RESOLVED FIXED in Firefox 42

Status

Testing
Marionette
RESOLVED FIXED
3 years ago
2 years ago

People

(Reporter: philor, Assigned: automatedtester)

Tracking

({ateam-marionette-task})

Trunk
mozilla43
ateam-marionette-task
Points:
---

Firefox Tracking Flags

(firefox40 affected, firefox42 fixed, firefox43 fixed)

Details

MozReview Requests

Submitter Diff Changes Open Issues Last Updated
Loading...
Error loading review requests:

Attachments

(1 attachment)

(Reporter)

Description

3 years ago
At least I *think* my second guess about the cause is correct.

If you click on any Mn job in treeherder (green or orange), the panel at the bottom which lists failures will show

TEST-START | test_errors.py TestAllExceptions.test_properties
TEST-START | test_errors.py TestExceptionType.test_cause_string
TEST-START | test_errors.py TestExceptionType.test_cause_tuple
TEST-START | test_errors.py TestExceptionType.test_construction
TEST-START | test_errors.py TestExceptionType.test_defaults
TEST-START | test_errors.py TestExceptionType.test_str 

and every Treeherder Robot bug comment on every Mn intermittent failure includes those six lines too. My first thought was that the filename was the problem, but if you look at the raw log,

07:21:41    ERROR -  TEST-START | test_errors.py TestAllExceptions.test_properties
07:21:41    ERROR -  TEST-PASS | test_errors.py TestAllExceptions.test_properties | took 704ms
07:21:41    ERROR -  TEST-START | test_errors.py TestExceptionType.test_cause_string
07:21:42    ERROR -  TEST-PASS | test_errors.py TestExceptionType.test_cause_string | took 699ms
07:21:42    ERROR -  TEST-START | test_errors.py TestExceptionType.test_cause_tuple
07:21:43    ERROR -  TEST-PASS | test_errors.py TestExceptionType.test_cause_tuple | took 845ms
07:21:43    ERROR -  TEST-START | test_errors.py TestExceptionType.test_construction
07:21:44    ERROR -  TEST-PASS | test_errors.py TestExceptionType.test_construction | took 1122ms
07:21:44    ERROR -  TEST-START | test_errors.py TestExceptionType.test_defaults
07:21:45    ERROR -  TEST-PASS | test_errors.py TestExceptionType.test_defaults | took 1099ms
07:21:45    ERROR -  TEST-START | test_errors.py TestExceptionType.test_str
07:21:46    ERROR -  TEST-PASS | test_errors.py TestExceptionType.test_str | took 1081ms
07:21:46     INFO -  TEST-START | test_errors.py TestLookup.test_by_known_number
07:21:47     INFO -  TEST-PASS | test_errors.py TestLookup.test_by_known_number | took 1094ms

I think it's actually mozharness calling them an ERROR because of "Exception", and then Treeherder trusting mozharness, since only the Exception tests get highlighted.
Duplicate of this bug: 1191378
(Assignee)

Updated

2 years ago
Keywords: ateam-marionette-task
(Assignee)

Comment 2

2 years ago
https://treeherder.mozilla.org/#/jobs?repo=try&revision=e6322d72ca7a
(Assignee)

Comment 3

2 years ago
Created attachment 8650058 [details]
MozReview Request: Bug 1156070: Rename test class to prevent test harnesses thinking their is a failure; r?jgriffin

Bug 1156070: Rename test class to prevent test harnesses thinking their is a failure; r?jgriffin

Mozharness appears to do a check for the word exception and is seeing these
tests as failures when they are not.
Attachment #8650058 - Flags: review?(jgriffin)
Attachment #8650058 - Flags: review?(jgriffin) → review+
Comment on attachment 8650058 [details]
MozReview Request: Bug 1156070: Rename test class to prevent test harnesses thinking their is a failure; r?jgriffin

https://reviewboard.mozilla.org/r/16539/#review14805

Thanks!
(Assignee)

Comment 5

2 years ago
url:        https://hg.mozilla.org/integration/mozilla-inbound/rev/38f5e5feea729b41139e487a4f676f0fa2389474
changeset:  38f5e5feea729b41139e487a4f676f0fa2389474
user:       David Burns <dburns@mozilla.com>
date:       Wed Aug 19 15:57:06 2015 +0100
description:
Bug 1156070: Rename test class to prevent test harnesses thinking there is a failure; r=jgriffin

Mozharness appears to do a check for the word exception and is seeing these
tests as failures when they are not.
https://hg.mozilla.org/mozilla-central/rev/38f5e5feea72
Assignee: nobody → dburns
Status: NEW → RESOLVED
Last Resolved: 2 years ago
status-firefox43: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla43
(Assignee)

Comment 7

2 years ago
Please sir, can you uplift to aurora a=test-only
Flags: needinfo?(ryanvm)
Flags: needinfo?(ryanvm)
https://hg.mozilla.org/releases/mozilla-aurora/rev/c468a9aab0d1
status-firefox42: --- → fixed
You need to log in before you can comment on or make changes to this bug.