1300892 - Autophone - should S1S2 tests which fail to get a single measurement be treated as failures?

Status: RESOLVED FIXED

(Testing Graveyard :: Autophone, defect)

Description

[Bob Clary [:bc] (inactive)]

Autophone currently treats the failure to obtain a single measurement out of all of the iterations as a test failure which turns the test orange. We have not made any effort or headway in triaging or fixing these issues which results in the Sheriffs spending time classifying failures for an issue which will not be fixed. I'm a firm believer that if a condition isn't something that we will actively work to resolve it shouldn't turn the test orange.

One possibility is to automatically retry the iteration (as long as the failure is not due to a crash). This may eliminate the situation where we have fewer than the requested number of iterations for most situations.

However, if we fail to get the measurement during the retry and were able to obtain at least N-1 of the requested iterations I think we should treat the test as a success and not a failure.

Thoughts?

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

Auto retry glosses over the real issues- although fixing the real issues might be a wasted effort.

I think it is reasonable to do what is proposed- we will  have real data, just a small bit less than ideal, but useful enough.

Comment 2

[James Willcox (:snorp) ([e-mail]) (he/him)]

Sounds fine to me

Comment 3

[Bob Clary [:bc] (inactive)]

Attachment: bug-1300892-v1.patch

Examples:

https://treeherder.allizom.org/#/jobs?repo=autoland&filter-searchStr=autophone&startdate=2016-09-06&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=runnable&filter-resultStatus=success&filter-resultStatus=retry&filter-resultStatus=pending&filter-resultStatus=running&tochange=afb8c6617510e7a8a0132e8412ca1aa8c049933a&selectedJob=3247856

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8789436 [details] [diff] [review]
bug-1300892-v1.patch

Review of attachment 8789436 [details] [diff] [review]:
-----------------------------------------------------------------

great, I am eager to see if this will greatly reduce our oranges.

Comment 5

[Bob Clary [:bc] (inactive)]

https://github.com/mozilla/autophone/commit/086707f07f47440a46569c42729ad098615142f9
deployed 2016-09-08 ~10:15 PDT