1365296 - Create reference profiles for heavy users based on Telemetry data

Status: RESOLVED FIXED

(Toolkit :: Startup and Profile System, enhancement)

Description

[:Dominik Strohmeier [:dstrohmeier]]

During the work on improving startup time for Quantum, the question about the impact of profiles on perf came up. In order to run reproducible benchmarks across difference devices like our reference hardware, we suggest to create reference profiles that could be loaded into a browser for benchmarking non-new, i.e. non-empty, users.

Based on Telemetry data about heavy users, the reference profile should have the following characteristics:

* Typical amount of bookmarks
* Typical history size
* Typical auto session-restore
* Typical UI configuration
* Typical amount of add-ons

Comment 1

[brendan c]

in gdoc here, https://docs.google.com/document/d/13kGlF-hKCxsKFrtaDSzZl3ZW1J5hgVaArio-gyU1XSc/edit , there is info on:

> * Typical amount of bookmarks
> * Typical history size



I included some inf from browser_startup_homepage and browser_startup_page -- does that address the question around

> * Typical auto session-restore

I'm not sure what probes address that, but those are the ones bsmedberg mentioned.



> * Typical UI configuration
I think this is only available in UI telemetry, which I think is not available in the data sets I know (longitudinal and main summary), so i won't be delivering this


> * Typical amount of add-ons
I'll try to get to this soon

Comment 2

[Benjamin Smedberg]

I'm putting numbers on the bug so that we can create a reference profile in the user story field.

Brendan, I put the 75% number up for tabs open, bookmarks, and history. Can you confirm that's a reasonable thing to measure?

There's a product question about the browser homepage. The data shows most users, even heavy ones, still use the default home page, but session restore is the next-most-popular option and doesn't currently have a quantum benchmark. We do have some Talos coverage via the `sessionrestore` test although that appears somewhat artificial.

Jeff, can you provide guidance about whether we should compare Firefox session restore with Chrome "continue where you left off" as part of the heavy user cohort in Firefox release criteria, or stick to measuring the default homepage?

Comment 3

[brendan c]

Yes, I think that seems fine-- it's the heavyish end of the heavy users, but it sounds reasonable to me to optimize for a worse case than a more run-of-the-mill case. I certainly don't see anything problematic in that approach.

Comment 4

[Jeff Griffiths (:canuckistani) (:⚡︎)]

(In reply to Benjamin Smedberg [:bsmedberg] from comment #2)
...
> Jeff, can you provide guidance about whether we should compare Firefox
> session restore with Chrome "continue where you left off" as part of the
> heavy user cohort in Firefox release criteria, or stick to measuring the
> default homepage?

I would like to compare session restore in addition to default home page if that's feasible. I would in fact in the future like to explore turning on session restore or test easier access to session restore by default.

I have some ideas that I would like us to research to try to identify "at work 9-5 users" ( if we can ) and what all they do. My unsubstantiated guess is that 9-5 web workers always use session restore.

Comment 5

[Selena Deckelmann :selenamarie :selena]

Benjamin has requested that engineering help create a profile. Details are in the "user story".

Comment 6

[:Dominik Strohmeier [:dstrohmeier]]

jhildebrand asked me to talk to mossop about technical direction for the project. mossop is busy today, but I'll meet with him later this week.

Comment 7

[:Dominik Strohmeier [:dstrohmeier]]

Regarding addons in the reference profiles: How much do we care about this, or maybe better, what are the ways that we will need to do here in order to create profiles that will representative to the only WebExtensions state by FF57?

Comment 8

[Benjamin Smedberg]

We need to measure the impact of addons that these heavy users are actually using, especially relative to Chrome. That means we need to measure the webextensions prototypes of addons that are still porting. (There is no point in measuring the non-webextensions versions.)

Comment 9

[:Dominik Strohmeier [:dstrohmeier]]

I have worked with whimboo today on work on a Marionette test that we can run to (re-)generate the profile. It's still work in progress and needs some tweaks, but it already works to load and bookmarks domains defined in a txt file.

Reporting this progress in case of this bug getting triaged.

Comment 10

[:Dominik Strohmeier [:dstrohmeier]]

Attachment: Marionette script to generate Firefox profiles

This is a Marionette script that we can run to generate profiles according to specific characteristics, such as the heavy user ones mentioned in the user story.

The current version takes a list of urls in heavy_users_urls.txt, visits them and bookmarks the first i of the list.

Calling the script with the --workspace option will make the generated profile permanent and the profile can be copy and pasted into Nightly to launch.

Comment 11

[:Dominik Strohmeier [:dstrohmeier]]

Attachment: test_heavy_users_generator.py

Changed location so that the .txt file holding all the urls to visit can be in the same folder as the script.

Comment 12

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Comment on attachment 8869971 [details]
test_heavy_users_generator.py

>        #with open('./testing/firefox-ui/tests/generate_profiles/heavy_users/heavy_users_urls.txt', 'r') as url_file:
>        with open(os.path.join(here, 'heavy_users_urls.txt'), 'r') as url_file:
>            self.test_urls = url_file.readlines()

If you are getting those files from a URL you could even automate that step in just retrieving the URLs.

>            if i < 3:# Bookmark the current page using the bookmark menu only for the first i items in urls
>                self.browser.menubar.select_by_id('bookmarksMenu', 'menu_bookmarkThisPage')
>
>                # TODO: Replace hard-coded selector with library method when one is available
>                self.bookmark_panel = self.marionette.find_element(By.ID, 'editBookmarkPanel')
>                done_button = self.marionette.find_element(By.ID, 'editBookmarkPanelDoneButton')

Code-wise this is fine. But to meet the criteria as listed in the user story there will have to be made some changes.

Comment 13

[:Dominik Strohmeier [:dstrohmeier]]

Comment on attachment 8869971 [details]
test_heavy_users_generator.py

With regards to generating the profiles, the most time-consuming part in terms of automation execution will definitely be retrieving 34000 history entries, i.e. visiting 34k URLs. Regarding the location where to find these URLs, I don't have a strong opinion. In this case, it's also not important which URLs these are, we just need to fill the history and bookmark list first.

Therefore, my suggestion would be to generate a basic profiles with history entries and bookmarks first with this script, and then from there, generate more specific profiles like session restore on/off, tabs open, either with a second script or manually based on this basic profile.

Does this make sense?

Comment 14

[Benjamin Smedberg]

At the present time I think we should focus on one single "heavy user" profile. Measuring combinations of options will be unwieldy quite quickly. But having the script be able to generate various options for later sounds quite reasonable.

I expect it is in fact important what URLs are visited, though more for pageload than for startup. In particular the history has a large impact on calculcating visited/not-visited status of links.

* Visit the facebook homepage and subgroup pages a lot, and a few other profile pages
* Visit the youtube homepage and trending pages a lot, and a large random sampling of videos
* A bunch of searches and the top result from those searches
* Visit the amazon home page a lot, and a sampling of search result pages and product pages

Comment 15

[Henrik Skupin [:whimboo][⌚️UTC+2]]

How important would it be to actually visit the real pages compared to just using the bookmark service to add the URLs directly?

Comment 16

[Benjamin Smedberg]

We're looking to have realistic data; I'm not sure we care exactly how we get it. But it needs to work equivalently in both Chrome and Firefox.

Comment 17

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Oh, so in such a case it might be worth to check if we can use Selenium + GeckoDriver for running the script. The code will be a bit different, but most of it would directly work cross-browser. So no different scripts would be needed.

https://github.com/SeleniumHQ/selenium/wiki/Python-Bindings
https://github.com/mozilla/geckodriver/

Comment 18

[Naveed Ihsanullah [:naveed]]

This bug does not appear to be a performance issue in the product that is mentioned in the Quantum Release Criteria document. If it is please help correct our understanding.

Comment 19

[Benjamin Smedberg]

Renominating: this directly affects our ability to measure startup performance and pageload for the heavy user cohort, which are release criteria.

Comment 20

[:Dominik Strohmeier [:dstrohmeier]]

There may be some confusion here about the term "profile". Please note that the reference profiles here are user profiles, not Gecko profiles.

Comment 21

[Andrew Overholt [:overholt]]

Is anything still happening here? It doesn't feel like it should remain qf:p1 at this point ...

Comment 22

[:Dominik Strohmeier [:dstrohmeier]]

Andrew, you can p3 this bug. We have a first version of the reference profile, but this isn't going to be an explicit 57 priority. Does this work for you? I will update the bug later this week with the current script.

After updating, I will close the bug and we'll be able to re-open if necessary.

Comment 23

[Jean Gong :jgong]

Andrew, can you get an engineer to record and do a profile analysis on browser startup with the heavy user profile using this current reference profile here https://drive.google.com/open?id=0ByGACjfH7lA-cmdXUUJ5TzJMMzQ. Please note that this has only the 30k page visits, but no bookmarks or addons yet so whoever takes on this effort will have to augment the test script with addons or bookmarks.

Comment 24

[Kan-Ru Chen [:kanru] (UTC+9)]

(In reply to Jean Gong from comment #23)
> Andrew, can you get an engineer to record and do a profile analysis on
> browser startup with the heavy user profile using this current reference
> profile here https://drive.google.com/open?id=0ByGACjfH7lA-cmdXUUJ5TzJMMzQ.
> Please note that this has only the 30k page visits, but no bookmarks or
> addons yet so whoever takes on this effort will have to augment the test
> script with addons or bookmarks.

Will Wang is going to take a look at the profile. Based on the initial analysis we will decide whether to file more bugs.

Comment 25

[:Dominik Strohmeier [:dstrohmeier]]

Kanru, Will,
to use the profile, unzip the folder that Jean linked above and then either copy the whole profile into your profile folder, or which works better for me, create a new profile in your browser and then copy the content of the heavy user profile folder into the folder of the newly created profile.

Comment 26

[Henrik Skupin [:whimboo][⌚️UTC+2]]

FYI there is also a -p command line option which let you specify the profile folder without having to copying the data around.

Comment 27

[Will Wang [:WillWang]]

(In reply to :Dominik Strohmeier [:dstrohmeier] from comment #25)
> Kanru, Will,
> to use the profile, unzip the folder that Jean linked above and then either
> copy the whole profile into your profile folder, or which works better for
> me, create a new profile in your browser and then copy the content of the
> heavy user profile folder into the folder of the newly created profile.

Thanks for your profile and sharing, Dominik and Henrik!

Dominik, could I confirm some questions with you:

1. I found that 2 prefs are manually changed to:
- browser.sessionstore.resume_from_crash = false (default = true)
- browser.startup.page = 0
which means session will not be restored.
I'm not sure the reason behind these pref change since the user story says "show your windows and tabs from last time" (Firefox)"; So I was wondering that is that ok if I change these 2 prefs to make session always restored and do the profiling based on that?

2. I also found that the restored session only contain 1 youtube tab, I'm not sure whether this session is the one you ever opened, did I miss something? Or would it be OK to randomly open other 19 tabs then restart and profile it? Or you have better candidate sites for these 20 tabs?

Thanks a lot! :)

Comment 28

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Will Wang [:WillWang] from comment #27)
> 1. I found that 2 prefs are manually changed to:
> - browser.sessionstore.resume_from_crash = false (default = true)
> - browser.startup.page = 0
> which means session will not be restored.
> I'm not sure the reason behind these pref change since the user story says
> "show your windows and tabs from last time" (Firefox)"; So I was wondering
> that is that ok if I change these 2 prefs to make session always restored
> and do the profiling based on that?

Those are actually preference changes Marionette is doing by default for tests. Feel free to revert that for your profile. In the future it should be done by the script. Good catch btw!

Comment 29

[:Dominik Strohmeier [:dstrohmeier]]

Let me add to this a bit of context:
The purpose for doing this initial profile was to have a baseline profile that can be extended easily. As filling history is the most time-consuming part, this profile only contains history elements.

Anyone using it, can extend this profile and add what's needed, like install addons, enable session restore and open tabs to be restored, etc. as described in the user stories.

Eventually, the idea was to have an incremental heavy user profile that will also help to identify which components add regressions if there are any, e.g. history only -> no regression, history + addons -> regression. So maybe some addons issue.

Does that answer your questions, Will?

Comment 30

[Will Wang [:WillWang]]

(In reply to Henrik Skupin (:whimboo) from comment #28)
> Those are actually preference changes Marionette is doing by default for
> tests. Feel free to revert that for your profile. In the future it should be
> done by the script. Good catch btw!
Thanks for your info for Marionette, Henrik!


(In reply to :Dominik Strohmeier [:dstrohmeier] from comment #29)
> Let me add to this a bit of context:
> The purpose for doing this initial profile was to have a baseline profile
> that can be extended easily. As filling history is the most time-consuming
> part, this profile only contains history elements.
Thank you for the supplement, Dominik :)
To me, it's really not clear how to "make" history on UI, so it's very helpful!

> Eventually, the idea was to have an incremental heavy user profile that will
> also help to identify which components add regressions if there are any,
> e.g. history only -> no regression, history + addons -> regression. So maybe
> some addons issue.
> 
> Does that answer your questions, Will?
Yes, and I love this idea, which can be used to improve our performance test in many circumstances!

Then I utilized the provided profile and extended/profiled it:

* Environment of profiling

1. I unzip the profile and make the profile directory managed by git, thus I can revert or extend it anytime in a clean manner. (This helps me a lot and save much time(unzip again, check dirty state,...etc), I recommend to anyone who would like to do the similar experiment.) (Besides, I found that about:profiles have a bug[1]) 
2. Set Firefox to "show your windows and tabs from last time" as the user story.
3. Open 20 tabs before restart Firefox (Yahoo search x10, and Amazon x10)
4. Build and enable stylo, which should be the default on each platform in the future.
5. Use a machine with hard disk instead of SSD; development is more painful but can expose performance issue more clearly.
6. I made an HTML file(called heavy-link.html) which contains 100,000 different links to try exposing possible history issues (ref :bsmedberg said in comment 14)
7. the startup command(./mach run) I used:
   MOZ_PROFILER_STARTUP_ENTRIES=20000000 MOZ_PROFILER_STARTUP=1 ./mach run -jsconsole -attach-console -n


* Process of profiling

Basically, it's a simple 3 steps:
s1. Start firefox with gecko profiler and wait the session restored. 
s2. Open a tab, go cnn.com, make some scrolling.
s3. Open a tab, open the heavy-link.html, make some scrolling.

The performance profile is here: 
     https://perfht.ml/2wVraOi

We can use a useful marker of the profile called `browser-open-newtab-start` to clearly distinguish each step.
For instance, we can know the start time of each stage:
s1 = 0 sec
s2 = 12.328 sec
s3 = 32.012 sec

So we can:
s1 <--observe startup--> s2 <--observe a normal page--> s3 <--observe influence of history in an extreme way--> end of profile


* Found possible time-consuming functions

Based on the observation above, here are some results. 
Note,
- These functions might actually need that much time, but we can try finding low-hanging fruits from them at this moment.
- These are filtered result; I filtered those are not that related to the heavy user profile, or is about the wait 
  operations(NtWaitForMultipleObjects).
- Please feel free to leave your other findings here :)


[Startup]
    main thread:
        - HistorySidebarInit(), 366ms. (I start up the Firefox with the sidebar opened)

        - mozilla::places::History::RegisterVisitedCallback(), 4 ms. This is for the comparison later.

        - onBrowserReady(), 56 ms. It seems an init for activity stream, but I didn't use it; could we defer this?

        - nsObserverService::NotifyObservers sessionstore-windows-restored, 22 ms. It seems that NotifyObservers is blocked by 
          sync call?

        - mozilla::net::nsHttpHandler::NotifyObservers, 309 ms. It seems that NotifyObservers is blocked by sync call?

    content 2:
        - nsSHistory::ReloadCurrentEntry, 10ms. It might not be a big impact in the content process at that time?


[Normal page(CNN)]
    main thread:
        - mozilla::places::History::RegisterVisitedCallback(), 3 ms. This is for the comparison later.


[Heavy link]
    main thread:
        - mozilla::places::History::RegisterVisitedCallback(), 862 ms. This is hugely notable, but in such an extreme case. It 
          shoud not be a problem if we call it in a normal page(4 or 3 ms).

    content 2:
        - mozilla::places::History::RegisterVisitedCallback(), 1020 ms. same.


Please leave your opinions if any :)



[1] 
In case anyone could run into it, here is the bug description:
I use nightly built from m-c.
about:profiles indicates Firefox is using one of the profiles in a directory where is full of other firefox profiles, but it's NOT!
actually, Firefox uses the (build-generated) profile in the object directory.
Does anyone know this bug?

Comment 31

[Will Wang [:WillWang]]

Attachment: heavy-link.html.gz

FYR, here is the gzip of heavy-link.html which was used during my profiling above.

Comment 32

[:Dominik Strohmeier [:dstrohmeier]]

Attachment: heavy_user_generator.py

Added the current Marionette script for generating the profile. This uses the eager strategy to load pages only until DOM Content Loaded is reached and doesn't wait for page load complete.

Comment 33

[Jean Gong :jgong]

I am moving this bug to P2 since it's not a blocker for 57. We will continue with this effort post 57.

Comment 34

[:Dominik Strohmeier [:dstrohmeier]]

Some updates in this bug, before I close it.

Tarek took this idea and created https://github.com/tarekziade/heavy-profile
This scrpt generates a user profile for Firefox based on defined characteristics like Telemetry data.

The profile gets generated daily on Taskcluster and is available there: https://index.taskcluster.net/v1/task/garbage.heavyprofile/artifacts/public/today-simple.tgz

Further, there is a first version of the heavy user profile integrated into Talos. There, you can use it with the new --profile option e.g.:

$ ./mach talos-test -a tp5o --profile simple

This call will download and use the profile instead of the empty profile Talos usually relies on.

I will close this bug and further changes that are planned on improving the heavy user profile in terms of its characteristics should be tracked in a new bug.

Comment 35

[Sylvestre Ledru [:Sylvestre]]

Clearly ni on Naveed as he no longer works at Mozilla.