1184736 - Show bar charts for results with medium confidence in comparison view

Status: RESOLVED DUPLICATE of bug 1170305

(Tree Management :: Perfherder, defect)

Description

[William Lachance (:wlach)]

:BenWa pointed out that we don't show the bar charts when we don't have high confidence of regression, even if the magnitude is fairly large. This makes it difficult to see by glancing down the graph column if there aren't potential things which need more retriggers to appear.

See some of the results here (from bug 1184277 comment 5):

https://treeherder.mozilla.org/perf.html#/comparesubtest?originalProject=try&originalRevision=3a36fc31a1a1&newProject=try&newRevision=01ee797fc2c9&originalSignature=bd72d04511c657c5c5040f1633fe73642fcdcb3b&newSignature=bd72d04511c657c5c5040f1633fe73642fcdcb3b

I suspect we should probably show yellow bars or something similar to show what the magnitude of the potential regression/improvement might be.

Comment 1

[Benoit Girard (:BenWa)]

FWIW If we have yellow along side red bars then some people might be lead to believe that yellow means not so bad compared to the red bars. Perhaps that's fine or perhaps that's not quite what you want to express.

Comment 2

[William Lachance (:wlach)]

(In reply to Benoit Girard (:BenWa) from comment #1)
> FWIW If we have yellow along side red bars then some people might be lead to
> believe that yellow means not so bad compared to the red bars. Perhaps
> that's fine or perhaps that's not quite what you want to express.

Good point. I was thinking maybe a lighter shade of red (or green) might be better here.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

how about light grey bars with a button to initiate a retrigger?

Comment 4

[Avi Halachmi (:avih)]

(In reply to Benoit Girard (:BenWa) from comment #1)
> FWIW If we have yellow along side red bars then some people might be lead to
> believe that yellow means not so bad compared to the red bars. Perhaps
> that's fine or perhaps that's not quite what you want to express.

Well, it's half the problem TBH. The thing is that the design has 4 CSS classes:
- sure regression (red)
- sure improvement (green)
- not-sure (orange-ish)
- nothing (plain)

Because there's only one "not sure" class but two cases where it can be applied (either for regression or improvement), I assigned the "not sure" semantically only to "regression but not sure" since we're more interested in the regressions to pop-out.

(In reply to William Lachance (:wlach) from comment #2)
> Good point. I was thinking maybe a lighter shade of red (or green) might be
> better here.

Same of sorts. I think the best approach here would to split not-sure into not-sure-regression and not-sure-improvement, and make them some "lighter" visual indication of the "sure" variants (e.g. maybe lighter shades of green/red, or just red/green border with white content, probably best to experiment with it a bit to understand what feels better).

(In reply to Joel Maher (:jmaher) from comment #3)
> how about light grey bars with a button to initiate a retrigger?

Retrigger from the compare page, while potentially useful right now, might be less useful when it will auto-retrigger talos try pushes, and also, probably not worth the work. IMO better to link back to the commit treeherder view which allows retriggering already. Though this subject is maybe off topic for this specific bug.



So, would this be a good summary of how it should be?
- regression/improvement stays solid red/green (already exists).
- not-sure regression/improvement becomes some lighter shades of red/green (changes from yellow-ish only for not-sure-regression).
- bard are always displayed at the color of the line/status (and appear as plain gray if low confidence or no regression/improvement).

Sounds about right?

Comment 5

[Avi Halachmi (:avih)]

(In reply to Avi Halachmi (:avih) from comment #4)
> - bard are always displayed at the color of the line/status (and appear as
> plain gray if low confidence or no regression/improvement).

s/bard/bars

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

I think this is just fine.  As long as it is easy to retrigger a job with more more than 3 clicks from the data point where you determine you need to retrigger, that is fine with me.

we can maybe mark the colors as softer or checkerboarded/pixelated colors to indicate it is a 'not sure' or 'not severe' change.

Comment 7

[Avi Halachmi (:avih)]

(In reply to Joel Maher (:jmaher) from comment #6)
> I think this is just fine.  As long as it is easy to retrigger a job with
> more more than 3 clicks from the data point where you determine you need to
> retrigger, that is fine with me.

I'm not against it, but I don't think it can currently be less than 3 clicks from the compare view to the treeherder commit view and retriggering. Or maybe 3 clicks, but one still needs to know the job's letter for retrigger, and find the specific platform which requires the retrigger. Maybe those can be arguments of the URL such that the platform+job-letter are already selected and all that's left is click the retrigger button?

> we can maybe mark the colors as softer or checkerboarded/pixelated colors to
> indicate it is a 'not sure' or 'not severe' change.

There's a big distinction between not-sure and not-severe. The former is currently only represented by the yellow-ish thingy for regressions only, and the latter is indicated by the change % value and bar's length (where we arbitrarily assigned 1.5% as the severe threshold, though obviously eventually we judge it as humans and apply contextual weight to the severity level).

IMO it's OK to not address the severity level (% change) with further visual indications beyond those which we have already.

Comment 8

[William Lachance (:wlach)]

I wound up addressing this in bug 1170305. Let's continue the discussion (if needed) there.