1131737 - Expire history and favicons

Status: RESOLVED FIXED

(Firefox for iOS :: Data Storage, defect, P1)

Description

[Wesley Johnston (:wesj)]

Right now we download and store a lot of history and favicon data. We need an expiry for it as well probably.

Comment 1

[johale]

 I want comment for website tsurumi use IOS: http://maybomtsurumi.vn and http://maybomtsurumi.com.vn

Comment 2

[Richard Newman [:rnewman]]

I ran an approximation of our history search implementation on my current iPad data.

Wiping the two favicons tables knocked 25% off our runtime.

Expiring old visits:

   DELETE FROM visits ORDER BY date ASC LIMIT 100000;

didn't make much difference.

Killing 150,000 old history items did:

   DELETE FROM history WHERE NOT EXISTS (SELECT 1 FROM visits WHERE visits.siteID = history.id);

That's because the query plan begins by searching for matching text, which is a table scan, which is linear over the amount of stored history. We need Bug 1173164.

Comment 3

[Richard Newman [:rnewman]]

Requesting tracking: search on my iPad takes up to nine seconds to show results.

Comment 4

[Justin D'Arcangelo [:justindarc]]

Richard, can you let me know if the FTS search in the latest v13 Beta helps with your abundant history?

Comment 5

[Richard Newman [:rnewman]]

Text search performance is much better, yes.

That doesn't entirely obsolete this bug — there are still space and performance reasons to expire old content — but now we can be less destructive, perhaps putting the limit at 100K places or a certain file size.

Comment 6

[Justin D'Arcangelo [:justindarc]]

(In reply to Richard Newman [:rnewman] from comment #5)
> Text search performance is much better, yes.
> 
> That doesn't entirely obsolete this bug — there are still space and
> performance reasons to expire old content — but now we can be less
> destructive, perhaps putting the limit at 100K places or a certain file size.

Awesome! Thanks for checking!

Last question: How often do you think we should run our cleanup process? Surely, we don't want to check on each launch. Every 30 days?

Comment 7

[Richard Newman [:rnewman]]

(In reply to Justin D'Arcangelo [:justindarc] from comment #6)

> Awesome! Thanks for checking!
> 
> Last question: How often do you think we should run our cleanup process?
> Surely, we don't want to check on each launch. Every 30 days?

Consider that it's mostly Sync that will cause an issue here, and consider that what you really want to do is try to clean up when certain conditions are met and performance is suffering.

It's cheap to examine the size of browser.db, very cheap to measure how long a query took to run, and relatively cheap to measure how many rows are in the history table via `SELECT COUNT(rowid) FROM history`.

I would probably hook this into the rebuild of the top sites cache, or trigger a one-time-per-launch run if a history query is particularly slow. You should collect telemetry before you make a decision.

Comment 8

[Justin D'Arcangelo [:justindarc]]

Attachment: GitHub Pull Request

Comment 9

[Justin D'Arcangelo [:justindarc]]

Comment on attachment 8994322 [details] [review]
GitHub Pull Request

Carrying over R+ from GitHub.

Comment 10

[Justin D'Arcangelo [:justindarc]]

Landed on master:

https://github.com/mozilla-mobile/firefox-ios/commit/e613114ec6c625fa791811e4700d782e49951d10