1487359 - Add a DAMP test for typing in JsTerm

Status: RESOLVED FIXED

(DevTools :: Console, enhancement, P1)

Description

[Nicolas Chevobbe [:nchevobbe]]

Typing in the console input revealed to be quite slow lately due to slow reflows.
Having a damp test would help us keep track of this metric and make sure we are improving this (and making sure we stay fast in the future).

Comment 1

[Nicolas Chevobbe [:nchevobbe]]

Attachment: Bug 1487359 - Add a DAMP test for typing in the console input; r=ochameau.

Here we are simulating a user typing a few letters in
the console input, when the console output has 500 messages.
We wait between each 'keystroke' in order to have a
realistic measure. Also, we are typing a string that is
triggering the popup  as it also impacts typing performance.

Comment 2

[Nicolas Chevobbe [:nchevobbe]]

I was able to test the test with a couple of changeset : 
- disabling codeMirror: https://treeherder.mozilla.org/perf.html#/comparesubtest?originalProject=try&originalRevision=a1c4b7e0c766b2213cef1fa1b8cbd0f965f61632&newProject=try&newRevision=c300f4079d62165bb4cc91e3bcf3b82abb4dcd06&originalSignature=f79b6a4f1f8f53326d3056f6c8008c0ff4de0a94&newSignature=f79b6a4f1f8f53326d3056f6c8008c0ff4de0a94&filter=typing&framework=12 
- modify some CSS: https://treeherder.mozilla.org/perf.html#/comparesubtest?originalProject=try&originalRevision=c300f4079d62165bb4cc91e3bcf3b82abb4dcd06&newProject=try&newRevision=6aec83d3460acc5ea298f10adb75246b8f673df6&originalSignature=f79b6a4f1f8f53326d3056f6c8008c0ff4de0a94&newSignature=f79b6a4f1f8f53326d3056f6c8008c0ff4de0a94&filter=typing&framework=12

Standard deviation of the test looks quite low (~0.2%)

Comment 3

[Nicolas Chevobbe [:nchevobbe]]

So I tried a few thing with the test: 
 1 - subtracting the time waiting (200ms for each key stroke) at the end of the test. That sounds good and something we could apply right away. Also it would better conveys what the results means (going from 2000ms to 1500ms does not report the same improvement as going from 1000ms (2000ms - delay time) to 500ms ).
 2 - not ending with a "." to not trigger the popup in the end. This seems legit and doesn't seems to affect the test stability.
 3 - setting the flex elements in the console to "block" to basically remove our reflow issue. The test then still reports taking ~100ms (still with 5 chars being typed) , which I think is sensitive enough to report future variations.
 4 - lowering the time we wait for between key strokes: This causes some issues. If we don't wait enough, we are sometimes showing the popup, sometimes not, and the test results tend to vary more because of that. I think it's still fine if we wait 200ms between each key stroke (if we go with what I did in 1. , I think it's fine).

Here's a comparison without and with the test: https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=b93a664dd349676f4d44af739a33a282d1851a77&newProject=try&newRevision=a1c4b7e0c766b2213cef1fa1b8cbd0f965f61632&framework=1

I don't think the other tests suffer from this new addition.

Comment 4

[Nicolas Chevobbe [:nchevobbe]]

okay, this time I managed to get the test to not impact other tests by clearing the output at the end of the test.

Comment 5

[Nicolas Chevobbe [:nchevobbe]]

Latest TRY push https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=93b224af9d87a82a08690425a3f26289dc8dbe68&newProject=try&newRevision=0f2f61d00f9980021f5938e8891161b81e24bd80&framework=12

Comment 6

[Alexandre Poirot [:ochameau]]

Comment on attachment 9005157 [details]
Bug 1487359 - Add a DAMP test for typing in the console input; r=ochameau.

Alexandre Poirot [:ochameau] PTO back on 17th has approved the revision.

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/549c61f38514
Add a DAMP test for typing in the console input; r=ochameau.

Comment 8

[Eliza Balazs [:ebalazs_]]

https://hg.mozilla.org/mozilla-central/rev/549c61f38514