1865721 - Show a dot plot with jitter of before and after on hover for easier visual comparison

Status: ASSIGNED

(Testing :: PerfCompare, enhancement)

Description

[Jeff Muizelaar [:jrmuizel]]

This would be a lot more useful than the current plot of dots that shows up when you hover on the before and after individually.

https://github.com/cxli233/FriendsDontLetFriends#1-friends-dont-let-friends-make-bar-plots-for-means-separation has an example of the difference this can make.

https://plotly.com/javascript/box-plots/#box-plot-that-displays-the-underlying-data has an example of how to do this with plotly and might be the quickest way to get something working soon.

Comment 1

[Julien Wajsberg [:julienw] (PTO -> Aug 24)]

We implemented kde (kernel density estimation) in the new perfcompare, hopefully this will be a better visualization for you. I can't provide a link just yet but will do once it's deployed and usable.

Comment 2

[Jeff Muizelaar [:jrmuizel]]

I originally thought I wanted a kde, but now I'm thinking a dot plot might be better. It'd be nice to try both and see what works better. When will the new perfcompare be deployed?

Comment 3

[Julien Wajsberg [:julienw] (PTO -> Aug 24)]

You can access it on https://beta--mozilla-perfcompare.netlify.app/, but it has a lot of buggy behaviors as well as not performant in the result page (we're working on it). I can try to provide a working link with a KDE tomorrow.

Comment 4

[Julien Wajsberg [:julienw] (PTO -> Aug 24)]

(In reply to Jeff Muizelaar [:jrmuizel] from comment #0)

https://github.com/cxli233/FriendsDontLetFriends#1-friends-dont-let-friends-make-bar-plots-for-means-separation has an example of the difference this can make.

I really like the 3rd graph here, in our context.
This could be implemented by https://observablehq.com/plot/transforms/dodge, possibly with chartjs scatter chart.
(or we could also move everything to d3)

Comment 5

[Julien Wajsberg [:julienw] (PTO -> Aug 24)]

Hey Jeff, could you try the currently deployed version? What do you think of the recent improvements?
Thanks!

Comment 6

[Jeff Muizelaar [:jrmuizel]]

Do you have an example report you can link to?

Comment 7

[Julien Wajsberg [:julienw] (PTO -> Aug 24)]

You can see for example here with devtools tests, where some tests are quite flaky but others are good: https://perf.compare/subtests-compare-results?baseRev=b045e71e64271296c9deef554f1e635f69bd82fd&baseRepo=try&newRev=d1813fcf0c0b49f89229f6a22f0cbd88cd4a3dd6&newRepo=try&framework=12&baseParentSignature=4763542&newParentSignature=4763542

Comment 8

[Jeff Muizelaar [:jrmuizel]]

What are the recent improvements?

On that link the scatter plots are on different sets axis and beside each other so you can't compare them visually. They are also not jittered and so it's harder to get a feel for the density of clumps

Comment 9

[Julien Wajsberg [:julienw] (PTO -> Aug 24)]

The recent improvements are:

• all graphs are on the same scale now, previously they were on different scales. I agree that putting them on the same graph and vertically aligned would be a better outcome.
• there was the question in comment 2 whether the density graph would be useful for you
• the plots are not bubbles anymore but triangles, so it's easier to tell them apart when there's a bigger density of them. They're also smaller.

I'm trying to get a feel whether the jittering them is, after you, as needed as it was before.

Comment 10

[Jeff Muizelaar [:jrmuizel]]

Attachment: browser-toolbox.connect.DAMP.html

I think the density graph is somewhat useful but it's hard to judge without being able to compare the raw data better.

One issue with the current density estimate is that the underlying distribution is likely not two tailed and so by using a KDE with a Gaussian kernel the KDE plot suggests that values below the minimum are a lot more likely than they actually are.

e.g. If you compare the browser-toolbox.connect.DAMP results in the link that you provided with the attached visualization of the same data, I think the attached visualization tells a different story than the density graph and current plots. i.e. The clump of data around 1750 is still there and there's more dispersion in the lower values.

Some other comments:

• Standard deviation shows up as 0
• It's not clear to me what mean difference is. Is that the difference of the means or the mean of differences between paired samples?

Comment 11

[Julien Wajsberg [:julienw] (PTO -> Aug 24)]

(In reply to Jeff Muizelaar [:jrmuizel] from comment #10)

Created attachment 9420856 [details]
browser-toolbox.connect.DAMP.html

I think the density graph is somewhat useful but it's hard to judge without being able to compare the raw data better.

One issue with the current density estimate is that the underlying distribution is likely not two tailed and so by using a KDE with a Gaussian kernel the KDE plot suggests that values below the minimum are a lot more likely than they actually are.

The bandwidth I used for the KDE computation is not perfect (I simply used "full range / 15", while normally we should use a root finding of some function algorithm), maybe by doing it better we'd also have a better graph.

e.g. If you compare the browser-toolbox.connect.DAMP results in the link that you provided with the attached visualization of the same data, I think the attached visualization tells a different story than the density graph and current plots. i.e. The clump of data around 1750 is still there and there's more dispersion in the lower values.

This brings you more data in case of a modal distribution, but when it's bimodal (or more) then it will be not so useful.

Some other comments:

• Standard deviation shows up as 0

Ah I know where this comes from: the plots use the replicates data, but the stddev comes from the 1 reported result that's a mean of these replicates. All the other measurements come from the means, not the replicates (eg: confidence, etc). I admit it's not clear to me at the moment why we're not using these replicates for this computation.

• It's not clear to me what mean difference is. Is that the difference of the means or the mean of differences between paired samples?

Good remark, yes this is the difference of the means.

Comment 12

[Benjamin De Kosnik [:bdekoz]]

Quick comment: this came up in the Perf x A11Y x Design working group as well. The suggestions were to use beeswarm or raincloud plot for this.

Comment 13

[Julien Wajsberg [:julienw] (PTO -> Aug 24)]

We just landed some improvements to our graphs (not all of the improvements are deployed to production yet but you can see the main branch deployment in https://main--mozilla-perfcompare.netlify.app/).

Still no jitter, but the KDE graph should be much better now, and all the graphs are now aligned and bigger. Hopefully this will make the issue a bit less annoying.

I started to look at adding a jitter, this might come soon as well. But I'll also be transitioning to my new role soon, with less time to work on this.