1326564 - Nightly top sites screen is slow

Status: RESOLVED FIXED

(Firefox for iOS :: Home screen, defect, P1)

Description

[Richard Newman [:rnewman]]

I have a lot of history. It takes 2-4 seconds to load Fennec on a 6S. I suspect that the Highlights UI isn't cached in the way that our Top Sites results are/used to be — an unchanged cold start is no faster than a changed cold start. 

Either that or we need a focus on perf again.

Comment 1

[Stefan Arentz | :st3fan | ⏰ EST | he/him]

What do you mean with 'to load Fennec'. Where is the delay? What do you have on screen when it hangs?

Comment 2

[Farhan Patel [:farhan]]

If you are talking about initial app launch it could be related to how slow enterprise builds take to load. I find that compared to a build compiled from my desktop, enterprise builds take about 2x longer to launch.

I don't understand why. If you have any thoughts on this would love to hear em.

Comment 3

[Richard Newman [:rnewman]]

This is my own build, so it doesn't suffer from the enterprise slowness (which a year or two ago we hypothesized was due to cert checking).

In any case, I'm measuring from the view being displayed until it being populated, rather than raw startup time.

Stefan: I mean if I set "New Tab" to "Show your Top Sites", close Settings, hit Menu, hit New Tab, I get a new tab with the four pager icons above a big white area, and then 2+ seconds later I get content. If I open reddit.com in that tab, then hit Menu > New Tab, it takes about a second to show the content.

The only time that the grid and Highlights list are shown immediately is when I haven't navigated _anywhere_ since the last time I showed Top Sites.

It's also relatively slow to switch from the History or Bookmarks panes to Top Sites -- feels like about 600-800msec.

If I swipe away Fennec in the app switcher and do a cold launch, it takes about 2 seconds again from showing the four pager icons until Top Sites and Highlights are populated. That's a long time to look at a big white rectangle, and a big motivation for my turning off Top Sites.

Comment 4

[Farhan Patel [:farhan]]

I think a big issue with this is we never show the cached TopSites. We always invalidate and calculate new TopSites every time the AS panel loads.

Comment 5

[Stephan Leroux [:sleroux]]

After doing some testing with my own FxA account which has a relatively large collection of history and small amount of bookmarks, I can definitely see a delay in populating the content as well.

> I think a big issue with this is we never show the cached TopSites. We always invalidate and calculate new TopSites every time the AS panel loads.

Yup - we've essentially gone back to the old way of populating top sites without a cache. On top of that, we also run a costly query to fetch highlights (albeit in parallel) and block the UI from updating until both operations have completed. So it's actually _worse_ than our old top sites problem! I did a quick timing test and found that highlights and top sites are close in duration (175 ms for highlights, 218 for top sites) with my profile.

Sure, we could start using the top site cache table like we were before which would make the top sites section appear instantly but then we're left with the delay with highlights.

Couple of options:

1. Optimize the queries for top sites and highlights more than they have been. I don't think this is feasible since we've tried this approach with the frecency query and it's just complicated in it's nature and hard to optimize fully.

2. Add caching for the highlights query results to avoid blocking on the expensive query.

There are a couple of approach we can take with (2). We could go the same route as we did with top sites and populate a cached_highlights table and use that instead of the query set or store the cached results in-memory.

Comment 6

[Stephan Leroux [:sleroux]]

Attachment: Link to Github pull-request: https://github.com/mozilla-mobile/firefox-ios/pull/2654

Adding :rnewman as f? in case you want to give this a try and see how it performs since you were noticing the slow downs on your large profile.

Comment 7

[Farhan Patel [:farhan]]

Comment on attachment 8860482 [details] [review]
Link to Github pull-request: https://github.com/mozilla-mobile/firefox-ios/pull/2654

I think your PR will also fix this one https://bugzilla.mozilla.org/show_bug.cgi?id=1357737

I'll create a new bug for the improvements I have for the highlights algo.

Comment 8

[Stephan Leroux [:sleroux]]

master b97329bf0008310ec5806e17c84c780c14c3f12d

Comment 9

[Stephan Leroux [:sleroux]]

I'm noticing in the sign off nightly build (#2923) that there is still a delay when loading in the top sites. We might have capped the delay that would be caused by many history items but it still feels sluggish to the point where I'd want to reopen this.

Comment 10

[Stephan Leroux [:sleroux]]

Running the profiler on the latest Firefox Beta, it looks like the delay is being caused by our query to get the highlights out of the cached table. I ran EXPLAIN QUERY PLAN on the query we're using:

explain query plan select * from metadata.highlights left join view_history_id_favicon on view_history_id_favicon.id = historyID left outer join metadata.page_metadata on metadata.page_metadata.cache_key = metadata.highlights.cache_key;
2|0|0|SCAN TABLE favicon_sites USING COVERING INDEX sqlite_autoindex_favicon_sites_1
2|1|1|SEARCH TABLE favicons USING INTEGER PRIMARY KEY (rowid=?)
1|0|0|SCAN TABLE history USING COVERING INDEX sqlite_autoindex_history_2
1|1|1|SEARCH SUBQUERY 2 USING AUTOMATIC COVERING INDEX (siteID=?)
0|0|0|SCAN TABLE highlights
0|1|1|SEARCH SUBQUERY 1 USING AUTOMATIC COVERING INDEX (id=?)
0|2|2|SEARCH TABLE page_metadata USING INDEX idx_page_metadata_cache_key_uniqueindex (cache_key=?)

I think what stands out as the SCAN on the history table when we try to join our limited highlights with the history/favicon view. We probably don't need to do a table SCAN on history for this.

Comment 11

[Stephan Leroux [:sleroux]]

Looks like we can speed up this query significantly by culling down the history/favicon view before doing the join:

left join (select * from view_history_id_favicon where id IN (select historyID from metadata.highlights)) as f1 on f1.id == historyID

Running the timer from sqlite3 shows the progress:

0.017400 old query
0.017292 pure select * from favicon 
0.000133 highlights query w/o favicon join
0.000092 highlights query w/o favicon join, w/o metadata join

0.000497 revised highlights query

Clearly that table scan on history was causing some harm!

Comment 12

[Stephan Leroux [:sleroux]]

Attachment: Link to Github pull-request: https://github.com/mozilla-mobile/firefox-ios/pull/2717#attch-to-bugzilla

Comment 13

[Justin D'Arcangelo [:justindarc]]

Comment on attachment 8866489 [details]
Link to Github pull-request: https://github.com/mozilla-mobile/firefox-ios/pull/2717#attch-to-bugzilla

LGTM

Comment 14

[Stephan Leroux [:sleroux]]

master 699dbceb7761d917844c7db0f1951c12db563b66