Closed
Bug 946298
Opened 10 years ago
Closed 10 years ago
Measure performance of keyboard after OOP patches land
Categories
(Firefox OS Graveyard :: General, defect, P2)
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: davehunt, Assigned: wlach)
References
Details
(Keywords: perf, Whiteboard: [c=automation p= s=2014.02.14 u=])
Attachments
(1 file)
1.40 KB,
text/plain
|
Details |
We should run some performance tests for the keyboard once the OOP patches land. I have written a simple eideticker script that allows us to record a video of using the keyboard. We need to run this before and after the patches land, ideally on Hamachi for analysis.
Reporter | ||
Updated•10 years ago
|
Status: NEW → ASSIGNED
Reporter | ||
Comment 1•10 years ago
|
||
Will: Would you mind taking this? My camera is now in the London office and I probably won't be traveling in for at least a couple of weeks.
Assignee: dave.hunt → wlachance
Assignee | ||
Comment 2•10 years ago
|
||
I'm seeing this error when running this code on the 1.4 nightly (which I assume is what we want?) (eideticker)wlach@popsicle:~/src/eideticker$ python bin/keyboard-test.py Traceback (most recent call last): File "bin/keyboard-test.py", line 40, in <module> main() File "bin/keyboard-test.py", line 24, in main new_contact = contacts.tap_new_contact() File "/home/wlach/src/eideticker/local/lib/python2.7/site-packages/gaiatest/apps/contacts/app.py", line 72, in tap_new_contact return NewContact(self.marionette) File "/home/wlach/src/eideticker/local/lib/python2.7/site-packages/gaiatest/apps/contacts/regions/contact_form.py", line 169, in __init__ done = self.marionette.find_element(*self._done_button_locator) File "/home/wlach/src/eideticker/local/lib/python2.7/site-packages/marionette/marionette.py", line 1100, in find_element response = self._send_message('findElement', 'value', **kwargs) File "/home/wlach/src/eideticker/local/lib/python2.7/site-packages/marionette/marionette.py", line 577, in _send_message self._handle_error(response) File "/home/wlach/src/eideticker/local/lib/python2.7/site-packages/marionette/marionette.py", line 598, in _handle_error raise NoSuchElementException(message=message, status=status, stacktrace=stacktrace) marionette.errors.NoSuchElementException: Unable to locate element: save-button
Flags: needinfo?(dave.hunt)
Reporter | ||
Comment 3•10 years ago
|
||
This is working for me with gaiatest 0.21.2. What version are you using? What state is the device in when the exception is thrown?
Flags: needinfo?(dave.hunt)
Reporter | ||
Comment 4•10 years ago
|
||
I just updated again with the latest master build (Hamachi) and the script is still passing for me. You mentioned a keyboard issue after resolving the wait?
Flags: needinfo?(wlachance)
Assignee | ||
Comment 5•10 years ago
|
||
Ok, so we needed a patch from Zac. The script wasn't working with my pointgrey camera for some reason so I resorted to just using my nexus 5 to record stuff. The keyboard code changed so I did a capture with an older version as well to make sure that we're not seeing differences that are due to test changes. Uploaded videos here: http://people.mozilla.com/~wlachance/keyboard-b2g12-2013-12-22.mp4 http://people.mozilla.com/~wlachance/keyboard-b2g14-2014-01-06.mp4 The first is with a build from the 22nd (my understanding is this is from before the patches landed). The second is from a build 2 days ago (after it landed). I haven't done any analysis on the difference.
Flags: needinfo?(wlachance)
Reporter | ||
Comment 6•10 years ago
|
||
Thanks Will. Clint, would you be able to run your analysis on these videos, or let me know your process so I can work on it?
Flags: needinfo?(ctalbert)
I just look through the video frame by frame using openshot - nothing magic here. 12-22 video Add contact window first loaded and stable: 1.018 Tap on Name: 2.012 Keyboard finishes rendering (you see the shift key highlight): 4.008 R typed (first time when the 'R' bubble looks fully rendered): 9.025 R shows in name field: 10.001 Suggestions show up: 10.009 E typed: 11.006 e shows up: 11.009 suggestions updated: 11.015 1-6 build Add contact window first loaded and stable: 1.026 Tap on Name: 3.008 Keyboard finishes rendering (you see the shift key highlight): 4.008 R typed (first time when the 'R' bubble looks fully rendered): 9.020 R shows in name field: 9.027 Suggestions show up: 10.008 E typed: 10.026 e shows up: 11.004 suggestions updated: 11.013 So, if you look at the event to event differences for each build. 22 diff 6 diff Time from load to tap on name: 0.994 1.982 Time from name tap to keyboard render: 1.996 1 Keyboard render to r type: 5.017 5.012 R type to R shows in name field: 0.976 0.007 R in name field to suggestions render: 0.008 0.981 Suggestions to e typed: 0.997 0.018 e typed to e in name field: 0.003 0.978 e in name field to suggestions: 0.006 0.009 You see that this code is roughly the same performance wise. We need much better quality videos to really see the elements here to analyze. I'm not sure what branch you're capturing these builds from Will, so I can't say when this code actually landed. Hopefully this shows you how I'm doing the analysis.
Flags: needinfo?(ctalbert)
Assignee | ||
Comment 8•10 years ago
|
||
(In reply to Clint Talbert ( :ctalbert ) from comment #7) > You see that this code is roughly the same performance wise. We need much > better quality videos to really see the elements here to analyze. I'm not > sure what branch you're capturing these builds from Will, so I can't say > when this code actually landed. Hopefully this shows you how I'm doing the > analysis. The exact builds I downloaded/used were: https://pvtbuilds.mozilla.org/pvt/mozilla.org/b2g_ril/hamachi-eng-mozilla-central-20131221040201-ril01.02.00.019.102.zip https://pvtbuilds.mozilla.org/pvt/mozilla.org/b2g_ril/hamachi-eng-mozilla-central-20140106040201-ril01.02.00.019.102.zip
Reporter | ||
Comment 9•10 years ago
|
||
I'm still trying to get remote access to eideticker-london. Once I have this, I should be able to run the script against the inari and provide the videos. Note that the previous results I provided were against inari, and Will's were against the hamachi as requested.
Comment 10•10 years ago
|
||
@Shelly, are you looking for this bug? Please clean the NeedInfo after you see the message. Thanks. :)
Flags: needinfo?(slin)
Updated•10 years ago
|
Flags: needinfo?(slin)
Comment 11•10 years ago
|
||
Thank you William :) I'll see if we can use our TaskTracer tool to provide any useful information.
Comment 12•10 years ago
|
||
@ William L., thanks for all your help. :) I changed bug status. -------------------------------------------------------------------- @ All, I have completed the performance measurement. :) Here are the detailed information. https://docs.google.com/a/mozilla.com/file/d/0B3SBjlVFSBc3bGdaMUExa0lZdFE/edit Comparing with V1.3 build, test results show that 1) the re-launch time of keyboard increases 219.3 centiseconds, and 2) the first switching time of third party keyboard increases 545.2 centiseconds.
Status: ASSIGNED → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Comment 13•10 years ago
|
||
(In reply to William Hsu [:whsu] from comment #12) > @ William L., thanks for all your help. :) > I changed bug status. > > -------------------------------------------------------------------- > @ All, > I have completed the performance measurement. :) > > Here are the detailed information. > https://docs.google.com/a/mozilla.com/file/d/0B3SBjlVFSBc3bGdaMUExa0lZdFE/ > edit > > Comparing with V1.3 build, test results show that > 1) the re-launch time of keyboard increases 219.3 centiseconds, and > 2) the first switching time of third party keyboard increases 545.2 > centiseconds. Mentioned over email - but do you have bugs on file for the perf regressions here?
Comment 14•10 years ago
|
||
Thanks for your reminder! :) Bug 970188 and Bug 970193
Updated•10 years ago
|
Priority: -- → P2
Whiteboard: [c=automation p= s= u=] → [c=automation p= s=2014.02.14 u=]
You need to log in
before you can comment on or make changes to this bug.
Description
•