Closed Bug 1320263 Opened 8 years ago Closed 8 years ago

Feasibility for Fennec performance assessment

Categories

(Firefox for Android Graveyard :: General, defect, P1)

Product:
Firefox for Android Graveyard
Component:
General
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: wesley_huang, Assigned: jwu)

References

Details

User Story

We would like to make sure we cover the basics (performance and stability) to make sure users do not leave Firefox to other browsers.

HASAL is a tool that allows us to understand performance of different browsers using video recording and analysis techniques, which will provide us with results in Running Time, Speed Index (SI), Perceptual Speed Index (PSI).


What are we measuring?
App launch time 
Page load time
Perceived page load time 
Scrolling
*reference[1]: We already have such automated but it's only for Fennec https://wiki.mozilla.org/EngineeringProductivity/Autophone#S1S2_Test


What other mobile browsers are being tested against Firefox
Top 3 mobile browsers globally (excluding Safari)
Chrome
UC Browser
Opera

Websites to be tested
TBD

Reference Test Devices
Google Pixel
Sony Z3C
Google Nexus?

“Useful information for users” - (data that user is interested in)
Web page test: https://www.webpagetest.org/
Architecture: https://sites.google.com/a/webpagetest.org/docs/system-design/mobile-testing
Agent to controll device:
https://sites.google.com/a/webpagetest.org/docs/private-instances/node-js-agent/setup
Repo:
https://github.com/WPO-Foundation/webpagetest/tree/master/agent/js

Attachments

(1 file)

Fennec Performance Test
103 bytes, text/x-google-doc
Details 
Hasal[1] team had done efforts on measuring the performance of Firefox desktop.
It would be nice if we can integrate Fennec with that as well.  

Some initial questions as below:
a. how to generate screen capture video on android?
b. how to automate the user behavior (launch fennec, initiate a page load, scroll ...etc) on android?

[1] https://wiki.mozilla.org/Hasal
Hi Wesley,

a. Screen recording can be performed by below command, but this could also impact overall device performance leads to "Uncertainty Principle"
ref:https://developer.android.com/studio/command-line/adb.html#screenrecord

adb shell screenrecord /sdcard/demo.mp4


b. "adb shell" command can fulfill most requirement(load, tap, scroll, ...etc). Swipe gesture can be simulated by adb command as below. Unfortunately, swipe does not work well on fennec, Google Chrome worked fine. Possible caused by different MotionEvent handling.

adb shell am start -a android.intent.action.VIEW -d "https://www.cnet.com/" org.mozilla.fennec_aurora/org.mozilla.fennec_aurora.App
adb shell "input swipe 100 960 900 960 200"
adb shell "input tap 1020 280"
adb shell "input tap 230 710"
Assignee: nobody → max
Flags: needinfo?(whuang)
Thank you Max for the input.
One open question we got while chatting with the Hasal team: "how do we determine a web page loading DONE?"
Flags: needinfo?(whuang)
On the requirement side, we would also need to define:

a. a list of top sites we would like to measure. I believe it's different from the list for desktop. Google doc is very likely not as important as in Fennec.

b. prioritize what else browsers we would like to benchmark.
Flags: needinfo?(jcheng)
(In reply to Max Liu [:maliu] from comment #1)
> b. "adb shell" command can fulfill most requirement(load, tap, scroll,
> ...etc). Swipe gesture can be simulated by adb command as below.
> Unfortunately, swipe does not work well on fennec, Google Chrome worked
> fine. Possible caused by different MotionEvent handling.

Yep, bug 1312482 should help.

(In reply to Wesley Huang [:wesley_huang] (EPM) (NI me) from comment #2)
> Thank you Max for the input.
> One open question we got while chatting with the Hasal team: "how do we
> determine a web page loading DONE?"

Our s1s2 autophone tests do this by looking at the device log. More about that here:
https://wiki.mozilla.org/EngineeringProductivity/Autophone#S1S2_Test
thanks Wesley. let me create a quick doc to answer your questions.
leaving the NI so I don't forget
User Story: (updated)
As per the meeting together with Hasal and Fennec team, we had aligned on the long-term goal and what we can start with. 

The living document[1] has more details. It requires permission so I abstract most info to "user story" of this bug.

For next step, let's target for creating a simpler PoC (proof of concept) that is capable for a pilot run.
With that we should be able to have initial understanding of the page loading, and speed index on Fennec, and other mobile browsers.


[1] https://docs.google.com/document/d/1OJi2ZBKrjvvc0vPF_qsIrTCr4s1NLgw7E6wpMT91p34/edit
Summary: Feasibility of integrating Fennec into Project Hasal (for performance monitoring) → Feasibility/PoC of integrating Fennec into Project Hasal (for performance monitoring)
It seems that we are expanding HASAL to measure beyond web content/rendering performance, but also application performance in general. I don't have a strong opinion on that other than we should be clear which part we are measuring and understand what's required to fix a specific performance bottleneck.

It's also worthy to note that we could measure application performance through other means, like telemetry marker etc.
Updating the summary and remove mention of HASAL, as we are talking about app performance, not content performance. We might be able to share their dashboard infra etc but not now...
Summary: Feasibility/PoC of integrating Fennec into Project Hasal (for performance monitoring) → Feasibility/PoC for Fennec performance assessment
Priority: -- → P1
Assignee: max → topwu.tw
Flags: needinfo?(jcheng)
PoC will be tracked by the follow up Bug 1340965
Summary: Feasibility/PoC for Fennec performance assessment → Feasibility for Fennec performance assessment
Hi Al,
Since we changed the scope of this bug to be a "preliminary testing", how about attaching the testing result/report here and then we can close it.
Flags: needinfo?(atsai)
Sounds good for me.

https://docs.google.com/spreadsheets/d/1DJfcMf1QAP9iM-4_6uH3viP6iwWp73ppUV8fnAiKDBo/edit#gid=1003657603
Flags: needinfo?(atsai)

  

  



    
    

    
  
The preliminary testing was done. 
Follow up will be Bug 1340965 - PoC for Fennec performance assessment

  

  


Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
Product: Firefox for Android → Firefox for Android Graveyard

          You need to log in
          before you can comment on or make changes to this bug.
        






  

    
  







  

    

      
Attachment

      

      
      
      
      
    

    

      

        

          

            
General

            

              Creator: 



            

            
Created: 

            
Updated: 

            
Size: