Open Bug 2056802 Opened 7 days ago Updated 3 days ago

Mann-Whitney results are hard / impossible to use for devtools DAMP subtests

Categories

(Testing :: PerfCompare, defect, P2)

Product:
Testing
Component:
PerfCompare
Type:
defect

Tracking

(Not tracked)

People

(Reporter: jdescottes, Assigned: kala)

References

(Depends on 1 open bug)

Details

(Whiteboard: [pcf])

Ever since the introduction of Mann-Whitney, the result pages for the DevTools DAMP (talos) tests have been hard to use and interpret.

The main reason is that it flags a lot of subtests as regressions/improvements, many of which are false positives.
It makes it difficult to identify changes in our own try pushes.

For instance a push from me yesterday:

The Mann-Whitney variant flags 19 subtests as regressions with "high significance", whereas Student-T classifies them as "low" confidence. The only actual expected change from my push is the screenshot test, which is flagged as an improvement in both cases. With Mann-Whitney I notice it has CD = 1 and CLES = 100%, while all the other noisy tests had values all over the spectrum.

If we take for example custom.jsdebugger.stepOut.big-bundle.DAMP, it has CD=-0.16 and CLES=42%. Looking at the distribution, it seems that during one run, it had a bimodal distribution, during the other one it didn't. While this might be an interesting information about that subtest itself, the actual change is ~1ms over a 700ms test, and is unrelated to my change.

I am fairly confident that all the other regressions highlighted by the Mann Whitney variant are just noise/test variance wrongly interpreted. I will check what happens when comparing two jobs for the same changeset.

Is this something which affects other performance test suites as well ? Will there be some effort to tune the thresholds for CD / CLES before flagging something as an actual regression / improvement?

Edit: I removed the section about "incorrect values" in the summary, I just mixed up two subtests, sorry!

(In reply to Julian Descottes [:jdescottes] from comment #1)

Here is an example where the Mann-Whitney results found 13 regressions for 2 pushes using the exact same code: https://perf.compare/subtests-compare-results?baseRev=85dd14e115fd25a7fbdafe694e4df1fbabc1e43e&baseRepo=try&newRev=19fb6068048f0e8d6598eb0d651a6e7de88194b9&newRepo=try&framework=12&baseParentSignature=5288973&newParentSignature=5288973&test_version=mann-whitney-u

What strikes me is that we have all these regressions despite this is the same code. Statistically we should have either nothing, or maybe a good balance of regressions and improvements.

So I wonder if there's something fishy going on, maybe with the test machines?

I realize that we don't expose the Cliff's delta interpretation in the summary row, I think it's more interesting than the actual value . Of course the actual value can be used to sort.

Looking at the interpretation, we more clearly see that most of the values end up in either "small" or "moderate", while the screenshot change that Julian was interested in is "Large". That's why exposing this in a better way might be a solution for the devtools folks.

Also if I understood correctly, CLES is just Cliff's Delta shown in a different way, so maybe this could be removed. Cliff's delta interpretation looks more interesting.

Depends on: 2039401

I plan to create a simplified view for PerfCompare so I'll look into showing the Cliff's Delta interpretation in the row while removing the CLES column (which has already been planned). See Bug 2039401

No longer depends on: 2039401
Severity: -- → S2
Priority: -- → P2
Assignee: nobody → csevere
Depends on: 2039401

Hi Julian, here's a link with your comparison showing the simplified view: https://deploy-preview-1069--mozilla-perfcompare.netlify.app/subtests-compare-results?baseRev=85dd14e115fd25a7fbdafe694e4df1fbabc1e43e&baseRepo=try&newRev=19fb6068048f0e8d6598eb0d651a6e7de88194b9&newRepo=try&framework=12&baseParentSignature=5288973&newParentSignature=5288973&test_version=mann-whitney-u&filter_status=improvement%2Cregression

I removed the CD/CLES from the default view (can be added with a checkbox); updated the median change column, and added the difference size or Cliff's Delta interpretation right below the status so you can see how big the difference is between Base and New without having to expand the row.

Thanks for the link Kala. It's nice to be able to see "small" / "moderate" / "large" directly in the table.
Is there any way we could have a filter on those? I would like to have a way to quickly see only moderate and large.

Some insights about how DAMP tests run can be helpful here. For each job (eg damp-other), we will start Firefox 5 times (= 5 cycle). For each cycle we will run all the tests for the corresponding job 5 times. Those tests are devtools UI tests, so they require first navigating to a page, then opening devtools, performing some actions etc... Many scenarios we are testing are very heavy on the memory consumption side , because we want to exercise DevTools performance when the webpage has many nodes, many css variables, many log messages. Because of that our tests are often subject to GC pauses which don't occur at the same time during different runs.

With Student-T we for sure could not pick up small regressions, but I don't think DAMP tests are stable and isolated enough to detect those at the moment. The fact that MW consistently detects "real" regressions on pushes using exactly the same code, to me indicates that the shape of the data generated by our tests is not clean enough for MW.

My hope is that focusing on "large" or "large" + "moderate" can somewhat be equivalent to the results we had with Student T, but that might not be the case. But that would be a start.

I was looking at other test suites, between two central pushes one from July 24th, one from July 23rd).
I spotted the pdfpaint test suite, which interestingly also uses subtests, same as devtools.

And the MW results are flagging hundreds of regressions / improvements, whereas Student-t doesn't flag any:

Granted I don't know if some changes landed which might impact pdfpaint, so I'll do another test pushing twice from the same commit.

Here is a compare for pdfpaint with two pushes for the exact same changeset and similar to devtools, many regressions / improvements are showing up for the MW results

https://perf.compare/subtests-compare-results?baseRev=10028b40a7b02c975e7924eaee0f7e3fe7616e9a&baseRepo=try&newRev=e8d60b129eb737da6d4082805be6db264d69b552&newRepo=try&framework=1&baseParentSignature=291091&newParentSignature=291091&test_version=mann-whitney-u&filter_status=improvement%2Cregression

Could it mean that there is an issue with subtests ?

Overall, checking a compare for central between 2 days, there really seems to be a high number of regressions and improvements spotted for subtests. Other suspicious suites: tart, tp5o etc... https://perf.compare/compare-results?baseRev=db59e73163ec54922cf9f65d125f9c06a477e50e&baseRepo=mozilla-central&newRev=6d21b1fa30fe11dfef209e78ffc4aa762013407b&newRepo=mozilla-central&framework=1

You need to log in before you can comment on or make changes to this bug.