Bugzilla

Ran a validation experiment to test whether the VMs can actually detect real improvement despite the variance we've been seeing. (it should, but would good to see the extent...)

I backed out Bug 2030147 (a word cache optimization) as the base

https://perf.compare/compare-results?baseRev=ae5ad2bf98a2821fdd83b67f979da2a2381e3ce3&baseRepo=try&newRev=87e0f02f184337dbd24cbfcfd088c37304ec969e&newRepo=try&framework=13&title=VM+before%2Fafter+with++Bug+2030147+reverted&test_version=mann-whitney-u

and the VM runs were able to pick up that improvement (kind of?)

Looking at the subtest view, Editor-TipTap/total was the highest delta , which is expected if you look at https://bugzilla.mozilla.org/show_bug.cgi?id=2030147#c11 and examining the [alert summary](https://treeherder.mozilla.org/perfherder/alerts?id=49627) tiptap was what was affected.

The perfcompare shows significant changes in all the subtests, but they are lower magnitude compared to TipTap

Ran a validation experiment to test whether the VMs can actually detect real improvement despite the variance we've been seeing. (it should, but would good to see the extent...)

I backed out Bug 2030147 (a word cache optimization) as the base

https://perf.compare/compare-results?baseRev=ae5ad2bf98a2821fdd83b67f979da2a2381e3ce3&baseRepo=try&newRev=87e0f02f184337dbd24cbfcfd088c37304ec969e&newRepo=try&framework=13&title=VM+before%2Fafter+with++Bug+2030147+reverted&test_version=mann-whitney-u

and the VM runs were able to pick up that improvement (kind of?)

Looking at the subtest view, Editor-TipTap/total was the highest delta , which is expected if you look at https://bugzilla.mozilla.org/show_bug.cgi?id=2030147#c11 and examining the [alert summary](https://treeherder.mozilla.org/perfherder/alerts?id=49627) tiptap was what was affected.

The perfcompare shows significant changes in all the subtests, but they are lower magnitude compared to TipTap, and may or may not be real improvements

Ran a validation experiment to test whether the VMs can actually detect real improvement despite the variance we've been seeing. (it should, but would good to see the extent...)

I backed out Bug 2030147 (a word cache optimization) as the base

https://perf.compare/compare-results?baseRev=ae5ad2bf98a2821fdd83b67f979da2a2381e3ce3&baseRepo=try&newRev=87e0f02f184337dbd24cbfcfd088c37304ec969e&newRepo=try&framework=13&title=VM+before%2Fafter+with++Bug+2030147+reverted&test_version=mann-whitney-u

and the VM runs were able to pick up that improvement (kind of?)

Looking at the subtest view, Editor-TipTap/total was the highest delta (to be more comparable with alert summary, which uses t-test, switch to t-test view in the perfcompare and look at subtests),

This is expected if you look at https://bugzilla.mozilla.org/show_bug.cgi?id=2030147#c11 and examine the [alert summary](https://treeherder.mozilla.org/perfherder/alerts?id=49627) you'll see TipTap was what was affected from that patch.

The perfcompare shows significant changes in all the subtests, but they are lower magnitude compared to TipTap, and may or may not be real improvements...

Ran a validation experiment to test whether the VMs can actually detect real improvement despite the variance we've been seeing. (it should, but would good to see the extent...)

I backed out Bug 2030147 (a word cache optimization) as the base

https://perf.compare/compare-results?baseRev=ae5ad2bf98a2821fdd83b67f979da2a2381e3ce3&baseRepo=try&newRev=87e0f02f184337dbd24cbfcfd088c37304ec969e&newRepo=try&framework=13&title=VM+before%2Fafter+with++Bug+2030147+reverted&test_version=mann-whitney-u

and the VM runs were able to pick up that improvement (kind of?)

Looking at the subtest view, Editor-TipTap/total was the highest delta (to be more comparable with alert summary, which uses t-test, switch to t-test view in the perfcompare and look at subtests e.g. [LINK](https://perf.compare/subtests-compare-results?baseRev=ae5ad2bf98a2821fdd83b67f979da2a2381e3ce3&baseRepo=try&newRev=87e0f02f184337dbd24cbfcfd088c37304ec969e&newRepo=try&framework=13&baseParentSignature=5835677&newParentSignature=5835677&test_version=student-t&sort=delta%7Cdesc) ),

This is expected if you look at https://bugzilla.mozilla.org/show_bug.cgi?id=2030147#c11 and examine the [alert summary](https://treeherder.mozilla.org/perfherder/alerts?id=49627) you'll see TipTap was what was affected from that patch.

The perfcompare shows significant changes in all the subtests, but they are lower magnitude compared to TipTap, and may or may not be real improvements...

Ran a validation experiment to test whether the VMs can actually detect real improvement despite the variance we've been seeing. (it should, but would be good to see the extent...)

I backed out Bug 2030147 (a word cache optimization) as the base

https://perf.compare/compare-results?baseRev=ae5ad2bf98a2821fdd83b67f979da2a2381e3ce3&baseRepo=try&newRev=87e0f02f184337dbd24cbfcfd088c37304ec969e&newRepo=try&framework=13&title=VM+before%2Fafter+with++Bug+2030147+reverted&test_version=mann-whitney-u

and the VM runs were able to pick up that improvement (kind of?)

Looking at the subtest view, Editor-TipTap/total was the highest delta (to be more comparable with alert summary, which uses t-test, switch to t-test view in the perfcompare and look at subtests e.g. [LINK](https://perf.compare/subtests-compare-results?baseRev=ae5ad2bf98a2821fdd83b67f979da2a2381e3ce3&baseRepo=try&newRev=87e0f02f184337dbd24cbfcfd088c37304ec969e&newRepo=try&framework=13&baseParentSignature=5835677&newParentSignature=5835677&test_version=student-t&sort=delta%7Cdesc) ),

This is expected if you look at https://bugzilla.mozilla.org/show_bug.cgi?id=2030147#c11 and examine the [alert summary](https://treeherder.mozilla.org/perfherder/alerts?id=49627) you'll see TipTap was what was affected from that patch.

The perfcompare shows significant changes in all the subtests, but they are lower magnitude compared to TipTap, and may or may not be real improvements...

Ran a validation experiment to test whether the VMs can actually detect real improvement despite the variance we've been seeing. (it should, but would be good to see the extent...)

I reverted Bug 2030147 (a word cache optimization) as the base

https://perf.compare/compare-results?baseRev=ae5ad2bf98a2821fdd83b67f979da2a2381e3ce3&baseRepo=try&newRev=87e0f02f184337dbd24cbfcfd088c37304ec969e&newRepo=try&framework=13&title=VM+before%2Fafter+with++Bug+2030147+reverted&test_version=mann-whitney-u

and the VM runs were able to pick up that improvement (kind of?)

Looking at the subtest view, Editor-TipTap/total was the highest delta (to be more comparable with alert summary, which uses t-test, switch to t-test view in the perfcompare and look at subtests e.g. [LINK](https://perf.compare/subtests-compare-results?baseRev=ae5ad2bf98a2821fdd83b67f979da2a2381e3ce3&baseRepo=try&newRev=87e0f02f184337dbd24cbfcfd088c37304ec969e&newRepo=try&framework=13&baseParentSignature=5835677&newParentSignature=5835677&test_version=student-t&sort=delta%7Cdesc) ),

This is expected if you look at https://bugzilla.mozilla.org/show_bug.cgi?id=2030147#c11 and examine the [alert summary](https://treeherder.mozilla.org/perfherder/alerts?id=49627) you'll see TipTap was what was affected from that patch.

The perfcompare shows significant changes in all the subtests, but they are lower magnitude compared to TipTap, and may or may not be real improvements...

Ran a validation experiment to test whether the VMs can actually detect real improvement despite the variance we've been seeing. (it should, but would be good to see the extent...)

I reverted Bug 2030147 (a word cache optimization) as the base

https://perf.compare/compare-results?baseRev=ae5ad2bf98a2821fdd83b67f979da2a2381e3ce3&baseRepo=try&newRev=87e0f02f184337dbd24cbfcfd088c37304ec969e&newRepo=try&framework=13&title=VM+before%2Fafter+with++Bug+2030147+reverted&test_version=mann-whitney-u

and the VM runs were able to pick up that improvement (kind of?)

Looking at the subtest view, Editor-TipTap/total was the highest delta (to be more comparable with alert summary, which uses t-test, switch to t-test view in the perfcompare and look at subtests e.g. [LINK](https://perf.compare/subtests-compare-results?baseRev=ae5ad2bf98a2821fdd83b67f979da2a2381e3ce3&baseRepo=try&newRev=87e0f02f184337dbd24cbfcfd088c37304ec969e&newRepo=try&framework=13&baseParentSignature=5835677&newParentSignature=5835677&test_version=student-t&sort=delta%7Cdesc) ),

This is expected if you look at https://bugzilla.mozilla.org/show_bug.cgi?id=2030147#c11 and examine the [alert summary](https://treeherder.mozilla.org/perfherder/alerts?id=49627) you'll see TipTap was what was affected from that patch.

The perfcompare shows significant changes in all the subtests, but they are lower magnitude compared to TipTap, and may or may not be real improvements...

**edit:** actually quick correction to my comment, it's the MWU view showing tons of improvements in the subtests. switching to T-test view, that amount goes down considerable, bringing us closer to the original patch alert!

Ran a validation experiment to test whether the VMs can actually detect real improvement despite the variance we've been seeing. (it should, but would be good to see the extent...)

I reverted Bug 2030147 (a word cache optimization) as the base (t-test view for alert parity)

https://perf.compare/compare-results?baseRev=ae5ad2bf98a2821fdd83b67f979da2a2381e3ce3&baseRepo=try&newRev=87e0f02f184337dbd24cbfcfd088c37304ec969e&newRepo=try&framework=13&title=VM+before%2Fafter+with++Bug+2030147+reverted&test_version=student-t

and the VM runs were able to pick up that improvement

Looking at the subtest view, Editor-TipTap/total was the highest delta (to be more comparable with alert summary, looking at t-test view for subtests as well: e.g. [LINK](https://perf.compare/subtests-compare-results?baseRev=ae5ad2bf98a2821fdd83b67f979da2a2381e3ce3&baseRepo=try&newRev=87e0f02f184337dbd24cbfcfd088c37304ec969e&newRepo=try&framework=13&baseParentSignature=5835677&newParentSignature=5835677&test_version=student-t&sort=delta%7Cdesc) ),

This is expected if you look at https://bugzilla.mozilla.org/show_bug.cgi?id=2030147#c11 and examine the [alert summary](https://treeherder.mozilla.org/perfherder/alerts?id=49627) you'll see TipTap was what was affected from that patch.

the MWU view shows a lot of improvements which may or may not be real, but for comparison with the original bug we are just looking at the t-test view (since t-test is used currently for generating alerts)