Measure performance of keyboard after OOP patches land

RESOLVED FIXED

Status

Firefox OS
General
P2
normal
RESOLVED FIXED
4 years ago
4 years ago

People

(Reporter: davehunt, Assigned: wlach)

Tracking

({perf})

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [c=automation p= s=2014.02.14 u=])

Attachments

(1 attachment)

(Reporter)

Description

4 years ago
Created attachment 8342452 [details]
Script for testing the keyboard performance

We should run some performance tests for the keyboard once the OOP patches land. I have written a simple eideticker script that allows us to record a video of using the keyboard. We need to run this before and after the patches land, ideally on Hamachi for analysis.
(Reporter)

Updated

4 years ago
Status: NEW → ASSIGNED

Updated

4 years ago
Keywords: perf
Whiteboard: [c=automation p= s= u=]
(Reporter)

Comment 1

4 years ago
Will: Would you mind taking this? My camera is now in the London office and I probably won't be traveling in for at least a couple of weeks.
Assignee: dave.hunt → wlachance
I'm seeing this error when running this code on the 1.4 nightly (which I assume is what we want?)

(eideticker)wlach@popsicle:~/src/eideticker$ python bin/keyboard-test.py 
Traceback (most recent call last):
  File "bin/keyboard-test.py", line 40, in <module>
    main()
  File "bin/keyboard-test.py", line 24, in main
    new_contact = contacts.tap_new_contact()
  File "/home/wlach/src/eideticker/local/lib/python2.7/site-packages/gaiatest/apps/contacts/app.py", line 72, in tap_new_contact
    return NewContact(self.marionette)
  File "/home/wlach/src/eideticker/local/lib/python2.7/site-packages/gaiatest/apps/contacts/regions/contact_form.py", line 169, in __init__
    done = self.marionette.find_element(*self._done_button_locator)
  File "/home/wlach/src/eideticker/local/lib/python2.7/site-packages/marionette/marionette.py", line 1100, in find_element
    response = self._send_message('findElement', 'value', **kwargs)
  File "/home/wlach/src/eideticker/local/lib/python2.7/site-packages/marionette/marionette.py", line 577, in _send_message
    self._handle_error(response)
  File "/home/wlach/src/eideticker/local/lib/python2.7/site-packages/marionette/marionette.py", line 598, in _handle_error
    raise NoSuchElementException(message=message, status=status, stacktrace=stacktrace)
marionette.errors.NoSuchElementException: Unable to locate element: save-button
Flags: needinfo?(dave.hunt)
(Reporter)

Comment 3

4 years ago
This is working for me with gaiatest 0.21.2. What version are you using? What state is the device in when the exception is thrown?
Flags: needinfo?(dave.hunt)
(Reporter)

Comment 4

4 years ago
I just updated again with the latest master build (Hamachi) and the script is still passing for me. You mentioned a keyboard issue after resolving the wait?
Flags: needinfo?(wlachance)
Ok, so we needed a patch from Zac. The script wasn't working with my pointgrey camera for some reason so I resorted to just using my nexus 5 to record stuff. The keyboard code changed so I did a capture with an older version as well to make sure that we're not seeing differences that are due to test changes. Uploaded videos here:

http://people.mozilla.com/~wlachance/keyboard-b2g12-2013-12-22.mp4
http://people.mozilla.com/~wlachance/keyboard-b2g14-2014-01-06.mp4

The first is with a build from the 22nd (my understanding is this is from before the patches landed). The second is from a build 2 days ago (after it landed). I haven't done any analysis on the difference.
Flags: needinfo?(wlachance)
(Reporter)

Comment 6

4 years ago
Thanks Will. Clint, would you be able to run your analysis on these videos, or let me know your process so I can work on it?
Flags: needinfo?(ctalbert)

Comment 7

4 years ago
I just look through the video frame by frame using openshot - nothing magic here.
12-22 video
Add contact window first loaded and stable: 1.018
Tap on Name: 2.012
Keyboard finishes rendering (you see the shift key highlight): 4.008
R typed (first time when the 'R' bubble looks fully rendered): 9.025
R shows in name field: 10.001
Suggestions show up: 10.009
E typed: 11.006
e shows up: 11.009
suggestions updated: 11.015

1-6 build
Add contact window first loaded and stable: 1.026
Tap on Name: 3.008
Keyboard finishes rendering (you see the shift key highlight): 4.008
R typed (first time when the 'R' bubble looks fully rendered): 9.020
R shows in name field: 9.027
Suggestions show up: 10.008
E typed: 10.026
e shows up: 11.004
suggestions updated: 11.013

So, if you look at the event to event differences for each build.
22 diff	6 diff
	
Time from load to tap on name:         0.994	1.982
Time from name tap to keyboard render: 1.996	1
Keyboard render to r type:             5.017	5.012
R type to R shows in name field:       0.976	0.007
R in name field to suggestions render: 0.008	0.981
Suggestions to e typed:                0.997	0.018
e typed to e in name field:            0.003	0.978
e in name field to suggestions:        0.006	0.009

You see that this code is roughly the same performance wise. We need much better quality videos to really see the elements here to analyze. I'm not sure what branch you're capturing these builds from Will, so I can't say when this code actually landed. Hopefully this shows you how I'm doing the analysis.
Flags: needinfo?(ctalbert)
(In reply to Clint Talbert ( :ctalbert ) from comment #7)

> You see that this code is roughly the same performance wise. We need much
> better quality videos to really see the elements here to analyze. I'm not
> sure what branch you're capturing these builds from Will, so I can't say
> when this code actually landed. Hopefully this shows you how I'm doing the
> analysis.

The exact builds I downloaded/used were:

https://pvtbuilds.mozilla.org/pvt/mozilla.org/b2g_ril/hamachi-eng-mozilla-central-20131221040201-ril01.02.00.019.102.zip
https://pvtbuilds.mozilla.org/pvt/mozilla.org/b2g_ril/hamachi-eng-mozilla-central-20140106040201-ril01.02.00.019.102.zip
(Reporter)

Comment 9

4 years ago
I'm still trying to get remote access to eideticker-london. Once I have this, I should be able to run the script against the inari and provide the videos. Note that the previous results I provided were against inari, and Will's were against the hamachi as requested.

Comment 10

4 years ago
@Shelly, are you looking for this bug?

Please clean the NeedInfo after you see the message.
Thanks. :)
Flags: needinfo?(slin)

Updated

4 years ago
Flags: needinfo?(slin)
Thank you William :) I'll see if we can use our TaskTracer tool to provide any useful information.

Comment 12

4 years ago
@ William L., thanks for all your help. :)
I changed bug status.

--------------------------------------------------------------------
@ All,
I have completed the performance measurement. :)

Here are the detailed information.
https://docs.google.com/a/mozilla.com/file/d/0B3SBjlVFSBc3bGdaMUExa0lZdFE/edit

Comparing with V1.3 build, test results show that
 1) the re-launch time of keyboard increases 219.3 centiseconds, and 
 2) the first switching time of third party keyboard increases 545.2 centiseconds.
Status: ASSIGNED → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED
(In reply to William Hsu [:whsu] from comment #12)
> @ William L., thanks for all your help. :)
> I changed bug status.
> 
> --------------------------------------------------------------------
> @ All,
> I have completed the performance measurement. :)
> 
> Here are the detailed information.
> https://docs.google.com/a/mozilla.com/file/d/0B3SBjlVFSBc3bGdaMUExa0lZdFE/
> edit
> 
> Comparing with V1.3 build, test results show that
>  1) the re-launch time of keyboard increases 219.3 centiseconds, and 
>  2) the first switching time of third party keyboard increases 545.2
> centiseconds.

Mentioned over email - but do you have bugs on file for the perf regressions here?

Comment 14

4 years ago
Thanks for your reminder! :)
Bug 970188 and Bug 970193

Updated

4 years ago
Priority: -- → P2
Whiteboard: [c=automation p= s= u=] → [c=automation p= s=2014.02.14 u=]
You need to log in before you can comment on or make changes to this bug.