Closed
Bug 1301795
Opened 8 years ago
Closed 6 years ago
Autoclassify calls 100% occurring failures on a push intermittent if it has ever seen a push where they were intermittent
Categories
(Tree Management :: Treeherder, defect)
Tree Management
Treeherder
Tracking
(Not tracked)
RESOLVED
INVALID
People
(Reporter: ehsan.akhgari, Unassigned)
References
Details
See this try push for example: <https://treeherder.mozilla.org/#/jobs?repo=try&revision=4dc3cf4f0b9e8c4c9ee1cd8002cd554955e6c4f6>
Here, devtools/client/debugger/test/mochitest/browser_dbg_sources-webext-contentscript.js has been failing on all of the platforms and configurations that it has ran on, however it has been mis-starred on OS X and Windows, but not Linux!
Comment 1•8 years ago
|
||
dt6? It looks to me like it only failed on Linux and MacOS X:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=4dc3cf4f0b9e8c4c9ee1cd8002cd554955e6c4f6&filter-searchStr=dt6
Comment 2•8 years ago
|
||
Wouldn't that be neat, if we ran the same number of chunks on every platform, and had so few disabled-per-platform tests that the chunking was the same between platforms?
dt8 and dt6 on asan, dt6 and dt2 on mac, dt8 and dt2 on Win7 opt, dt2 and dt1 on Win7 debug.
Updated•8 years ago
|
Summary: TreeHerder mis-stars 100% occurring failures → Autoclassify calls 100% occurring failures on a push intermittent if it has ever seen a push where they were intermittent
Comment 3•8 years ago
|
||
Is this due to detect-intermittents? If so, it's just been disabled in bug 1301434, so this is a dupe of that.
Flags: needinfo?(james)
Comment 4•8 years ago
|
||
In principle no, because a sheriff staring a bug that is first intermittent and then later permafail leads to the same outcome. OTOH it may occur less frequently when that task is disabled.
But I don't really want to spend time special-casing 100% permafails because the actual problem seems to be "tests that fail too often should be fixed with urgency, and may lead to tree closures". "Too often" here could be 50% or lower. That seems like a sub-problem of the larger "detect when the frequency of an intermittent increased" problem, which should be solved anyway. So I agree this is a real problem, but I don't think the fix implied by the bug summary is the right one.
Flags: needinfo?(james)
Comment 5•8 years ago
|
||
This feels like a UI problem to me, in that the star next to the jobs gives the user the false impression that a permafail is just a "known intermittent" (certainly this was my first impression looking at Ehsan's push).
A "test centric" view of results would be one possible solution to this problem (see bug 1059770), since it would probably highlight the fact that it's one test in particular that's failing (rather than the usual pattern of orange that we see on every push these days).
See Also: → 1059770
Updated•7 years ago
|
Component: Treeherder → Treeherder: Log Parsing & Classification
Comment 6•6 years ago
|
||
AutoclassifyTab has been removed from the app.
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → INVALID
Assignee | ||
Updated•3 years ago
|
Component: Treeherder: Log Parsing & Classification → TreeHerder
You need to log in
before you can comment on or make changes to this bug.
Description
•