Ever since the introduction of Mann-Whitney, the result pages for the DevTools DAMP (talos) tests have been hard to use and interpret. The main reason is that it flags a lot of subtests as regressions/improvements, many of which are false positives. It makes it difficult to identify changes in our own try pushes. For instance a push from me yesterday: - with Mann-Whitney https://perf.compare/subtests-compare-results?baseRev=3a3daf9ae871ce28b8b8a8819359d86440406635&baseRepo=try&newRev=09ac790cb4d802c23eda10210060200bc72154c5&newRepo=try&framework=12&baseParentSignature=5716596&newParentSignature=5716596&test_version=mann-whitney-u - with Student-T https://perf.compare/subtests-compare-results?baseRev=3a3daf9ae871ce28b8b8a8819359d86440406635&baseRepo=try&newRev=09ac790cb4d802c23eda10210060200bc72154c5&newRepo=try&framework=12&baseParentSignature=5716596&newParentSignature=5716596&test_version=student-t ### False positives The Mann-Whitney variant flags 19 subtests as regressions with "high significance", whereas Student-T classifies them as "low" confidence. The only actual expected change from my push is the screenshot test, which is flagged as an improvement in both cases. With Mann-Whitney I notice it has CD = 1 and CLES = 100%, while all the other noisy tests had values all over the spectrum. If we take for example `custom.jsdebugger.stepOut.big-bundle.DAMP`, it has CD=-0.16 and CLES=42%. Looking at the distribution, it seems that during one run, it had a bimodal distribution, during the other one it didn't. While this might be an interesting information about that subtest itself, the actual change is ~1ms over a 700ms test, and is unrelated to my change. I am fairly confident that all the other regressions highlighted by the Mann Whitney variant are just noise/test variance wrongly interpreted. I will check what happens when comparing two jobs for the same changeset. ### Incorrect values I also don't understand where the values from the perfcompare UI are from. It reports ~700ms of average for custom.jsdebugger.stepOver.big-bundle.DAMP. But Student-T reports around 530ms? Is this something which affects other performance test suites as well ? Will there be some effort to tune the thresholds for CD / CLES before flagging something as an actual regression / improvement?
Bug 2056802 Comment 0 Edit History
Note: The actual edited comment in the bug view page will always show the original commenter’s name and original timestamp.
Ever since the introduction of Mann-Whitney, the result pages for the DevTools DAMP (talos) tests have been hard to use and interpret. The main reason is that it flags a lot of subtests as regressions/improvements, many of which are false positives. It makes it difficult to identify changes in our own try pushes. For instance a push from me yesterday: - with Mann-Whitney https://perf.compare/subtests-compare-results?baseRev=3a3daf9ae871ce28b8b8a8819359d86440406635&baseRepo=try&newRev=09ac790cb4d802c23eda10210060200bc72154c5&newRepo=try&framework=12&baseParentSignature=5716596&newParentSignature=5716596&test_version=mann-whitney-u - with Student-T https://perf.compare/subtests-compare-results?baseRev=3a3daf9ae871ce28b8b8a8819359d86440406635&baseRepo=try&newRev=09ac790cb4d802c23eda10210060200bc72154c5&newRepo=try&framework=12&baseParentSignature=5716596&newParentSignature=5716596&test_version=student-t ### False positives The Mann-Whitney variant flags 19 subtests as regressions with "high significance", whereas Student-T classifies them as "low" confidence. The only actual expected change from my push is the screenshot test, which is flagged as an improvement in both cases. With Mann-Whitney I notice it has CD = 1 and CLES = 100%, while all the other noisy tests had values all over the spectrum. If we take for example `custom.jsdebugger.stepOut.big-bundle.DAMP`, it has CD=-0.16 and CLES=42%. Looking at the distribution, it seems that during one run, it had a bimodal distribution, during the other one it didn't. While this might be an interesting information about that subtest itself, the actual change is ~1ms over a 700ms test, and is unrelated to my change. I am fairly confident that all the other regressions highlighted by the Mann Whitney variant are just noise/test variance wrongly interpreted. I will check what happens when comparing two jobs for the same changeset. ### Incorrect values I also don't understand where the values from the perfcompare UI are from. It reports ~700ms of average for custom.jsdebugger.stepOver.big-bundle.DAMP. But Student-T reports around 530ms? And looking at the logs of the test, I only see values around 500-600ms. Where is the 700ms reported by Mann-Whitney coming from? Is this something which affects other performance test suites as well ? Will there be some effort to tune the thresholds for CD / CLES before flagging something as an actual regression / improvement?
Ever since the introduction of Mann-Whitney, the result pages for the DevTools DAMP (talos) tests have been hard to use and interpret. The main reason is that it flags a lot of subtests as regressions/improvements, many of which are false positives. It makes it difficult to identify changes in our own try pushes. For instance a push from me yesterday: - with Mann-Whitney https://perf.compare/subtests-compare-results?baseRev=3a3daf9ae871ce28b8b8a8819359d86440406635&baseRepo=try&newRev=09ac790cb4d802c23eda10210060200bc72154c5&newRepo=try&framework=12&baseParentSignature=5716596&newParentSignature=5716596&test_version=mann-whitney-u - with Student-T https://perf.compare/subtests-compare-results?baseRev=3a3daf9ae871ce28b8b8a8819359d86440406635&baseRepo=try&newRev=09ac790cb4d802c23eda10210060200bc72154c5&newRepo=try&framework=12&baseParentSignature=5716596&newParentSignature=5716596&test_version=student-t ### False positives The Mann-Whitney variant flags 19 subtests as regressions with "high significance", whereas Student-T classifies them as "low" confidence. The only actual expected change from my push is the screenshot test, which is flagged as an improvement in both cases. With Mann-Whitney I notice it has CD = 1 and CLES = 100%, while all the other noisy tests had values all over the spectrum. If we take for example `custom.jsdebugger.stepOut.big-bundle.DAMP`, it has CD=-0.16 and CLES=42%. Looking at the distribution, it seems that during one run, it had a bimodal distribution, during the other one it didn't. While this might be an interesting information about that subtest itself, the actual change is ~1ms over a 700ms test, and is unrelated to my change. I am fairly confident that all the other regressions highlighted by the Mann Whitney variant are just noise/test variance wrongly interpreted. I will check what happens when comparing two jobs for the same changeset. ### Incorrect values I also don't understand where the values from the Mann Whitney UI are from. It reports ~700ms of average for custom.jsdebugger.stepOver.big-bundle.DAMP. But Student-T reports around 530ms? And looking at the logs of the test, I only see values around 500-600ms. Where is the 700ms reported by Mann-Whitney coming from? Is this something which affects other performance test suites as well ? Will there be some effort to tune the thresholds for CD / CLES before flagging something as an actual regression / improvement?
Ever since the introduction of Mann-Whitney, the result pages for the DevTools DAMP (talos) tests have been hard to use and interpret. The main reason is that it flags a lot of subtests as regressions/improvements, many of which are false positives. It makes it difficult to identify changes in our own try pushes. For instance a push from me yesterday: - with Mann-Whitney https://perf.compare/subtests-compare-results?baseRev=3a3daf9ae871ce28b8b8a8819359d86440406635&baseRepo=try&newRev=09ac790cb4d802c23eda10210060200bc72154c5&newRepo=try&framework=12&baseParentSignature=5716596&newParentSignature=5716596&test_version=mann-whitney-u - with Student-T https://perf.compare/subtests-compare-results?baseRev=3a3daf9ae871ce28b8b8a8819359d86440406635&baseRepo=try&newRev=09ac790cb4d802c23eda10210060200bc72154c5&newRepo=try&framework=12&baseParentSignature=5716596&newParentSignature=5716596&test_version=student-t The Mann-Whitney variant flags 19 subtests as regressions with "high significance", whereas Student-T classifies them as "low" confidence. The only actual expected change from my push is the screenshot test, which is flagged as an improvement in both cases. With Mann-Whitney I notice it has CD = 1 and CLES = 100%, while all the other noisy tests had values all over the spectrum. If we take for example `custom.jsdebugger.stepOut.big-bundle.DAMP`, it has CD=-0.16 and CLES=42%. Looking at the distribution, it seems that during one run, it had a bimodal distribution, during the other one it didn't. While this might be an interesting information about that subtest itself, the actual change is ~1ms over a 700ms test, and is unrelated to my change. I am fairly confident that all the other regressions highlighted by the Mann Whitney variant are just noise/test variance wrongly interpreted. I will check what happens when comparing two jobs for the same changeset. Is this something which affects other performance test suites as well ? Will there be some effort to tune the thresholds for CD / CLES before flagging something as an actual regression / improvement?