1320263 - Feasibility for Fennec performance assessment

Status: RESOLVED FIXED

(Firefox for Android Graveyard :: General, defect, P1)

Description

[Wesley Huang [:wesley_huang] (EPM) (NI me)]

Hasal[1] team had done efforts on measuring the performance of Firefox desktop.
It would be nice if we can integrate Fennec with that as well.  

Some initial questions as below:
a. how to generate screen capture video on android?
b. how to automate the user behavior (launch fennec, initiate a page load, scroll ...etc) on android?

[1] https://wiki.mozilla.org/Hasal

Comment 1

[Max Liu [:maliu]]

Hi Wesley,

a. Screen recording can be performed by below command, but this could also impact overall device performance leads to "Uncertainty Principle"
ref:https://developer.android.com/studio/command-line/adb.html#screenrecord

adb shell screenrecord /sdcard/demo.mp4


b. "adb shell" command can fulfill most requirement(load, tap, scroll, ...etc). Swipe gesture can be simulated by adb command as below. Unfortunately, swipe does not work well on fennec, Google Chrome worked fine. Possible caused by different MotionEvent handling.

adb shell am start -a android.intent.action.VIEW -d "https://www.cnet.com/" org.mozilla.fennec_aurora/org.mozilla.fennec_aurora.App
adb shell "input swipe 100 960 900 960 200"
adb shell "input tap 1020 280"
adb shell "input tap 230 710"

Comment 2

[Wesley Huang [:wesley_huang] (EPM) (NI me)]

Thank you Max for the input.
One open question we got while chatting with the Hasal team: "how do we determine a web page loading DONE?"

Comment 3

[Wesley Huang [:wesley_huang] (EPM) (NI me)]

On the requirement side, we would also need to define:

a. a list of top sites we would like to measure. I believe it's different from the list for desktop. Google doc is very likely not as important as in Fennec.

b. prioritize what else browsers we would like to benchmark.

Comment 4

[Sebastian Kaspari (:sebastian; :pocmo)]

(In reply to Max Liu [:maliu] from comment #1)
> b. "adb shell" command can fulfill most requirement(load, tap, scroll,
> ...etc). Swipe gesture can be simulated by adb command as below.
> Unfortunately, swipe does not work well on fennec, Google Chrome worked
> fine. Possible caused by different MotionEvent handling.

Yep, bug 1312482 should help.

(In reply to Wesley Huang [:wesley_huang] (EPM) (NI me) from comment #2)
> Thank you Max for the input.
> One open question we got while chatting with the Hasal team: "how do we
> determine a web page loading DONE?"

Our s1s2 autophone tests do this by looking at the device log. More about that here:
https://wiki.mozilla.org/EngineeringProductivity/Autophone#S1S2_Test

Comment 5

[Joe Cheng [:jcheng] (please needinfo)]

thanks Wesley. let me create a quick doc to answer your questions.
leaving the NI so I don't forget

Comment 6

[Wesley Huang [:wesley_huang] (EPM) (NI me)]

As per the meeting together with Hasal and Fennec team, we had aligned on the long-term goal and what we can start with. 

The living document[1] has more details. It requires permission so I abstract most info to "user story" of this bug.

For next step, let's target for creating a simpler PoC (proof of concept) that is capable for a pilot run.
With that we should be able to have initial understanding of the page loading, and speed index on Fennec, and other mobile browsers.


[1] https://docs.google.com/document/d/1OJi2ZBKrjvvc0vPF_qsIrTCr4s1NLgw7E6wpMT91p34/edit

Comment 7

[Tim Guan-tin Chien [:timdream] (please needinfo)]

It seems that we are expanding HASAL to measure beyond web content/rendering performance, but also application performance in general. I don't have a strong opinion on that other than we should be clear which part we are measuring and understand what's required to fix a specific performance bottleneck.

It's also worthy to note that we could measure application performance through other means, like telemetry marker etc.

Comment 8

[Tim Guan-tin Chien [:timdream] (please needinfo)]

Updating the summary and remove mention of HASAL, as we are talking about app performance, not content performance. We might be able to share their dashboard infra etc but not now...

Comment 9

[Wesley Huang [:wesley_huang] (EPM) (NI me)]

PoC will be tracked by the follow up Bug 1340965

Comment 10

[Wesley Huang [:wesley_huang] (EPM) (NI me)]

Hi Al,
Since we changed the scope of this bug to be a "preliminary testing", how about attaching the testing result/report here and then we can close it.

Comment 11

[Al Tsai [:atsai]]

Sounds good for me.

https://docs.google.com/spreadsheets/d/1DJfcMf1QAP9iM-4_6uH3viP6iwWp73ppUV8fnAiKDBo/edit#gid=1003657603

Comment 12

[Al Tsai [:atsai]]

Attachment: Fennec Performance Test

Comment 13

[Wesley Huang [:wesley_huang] (EPM) (NI me)]

The preliminary testing was done. 
Follow up will be Bug 1340965 - PoC for Fennec performance assessment