1398068 - DAMP still miss performances regressions on inspector.open

Status: NEW

(DevTools :: General, enhancement, P3)

Description

[Alexandre Poirot [:ochameau]]

Even with bug 1396539 and bug 1394804, we still don't catch significant improvements being made to the inspector.

In the following talos results I ran a first reference build including bug 1396539 and bug 1394804:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=5689dc70e2cc6104b87678e3d609d860cfa531af

Then I pushed another one, also including this two bugs, plus a bunch of recent optimizations made to the inspector: bug 1396600, bug 1397343 and bug 1397330:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=280605daeddeeb64edf33a0bdcb74cd5ee2f493a

This end up reporting this performance numbers:
https://treeherder.mozilla.org/perf.html#/comparesubtest?originalProject=try&originalRevision=5689dc70e2cc&newProject=try&newRevision=280605daeddeeb64edf33a0bdcb74cd5ee2f493a&originalSignature=edaec66500db21d37602c99daa61ac983f21a6ac&newSignature=edaec66500db21d37602c99daa61ac983f21a6ac&filter=inspector.open&framework=1

No alert, confidence is always low.
But the main issue is that it only report regressions instead of improvements!
The timing of the second build are all higher.

I have no clue why the results are wrong. May be we miss something similar to bug 1396539??

Comment 1

[Alexandre Poirot [:ochameau]]

Or could it be related to bug 1394804 comment 30 and the numbers are not as good as before?
If I understand this correctly, we only take 1 run for cold and 5 runs for warm into account.

Comment 2

[Alexandre Poirot [:ochameau]]

Note that if I check performances for even more patches (all my current patch queue related to performances), I see a win:
https://treeherder.mozilla.org/perf.html#/comparesubtest?originalProject=try&originalRevision=5689dc70e2cc&newProject=try&newRevision=27483b916801a063a8a7536fe321715f2f53aa94&originalSignature=edaec66500db21d37602c99daa61ac983f21a6ac&newSignature=edaec66500db21d37602c99daa61ac983f21a6ac&filter=.open&showOnlyImportant=1&framework=1
Between 5 to 17% win. It includes bug 1397341, bug 1396619 and bug 1396783 in addition to the others from comment 0.