1753729 - Firefox remembers less browsing history than before

Status: RESOLVED FIXED

(Firefox :: Bookmarks & History, defect, P3)

Description

[Botond Ballo [:botond]]

I don't remember exactly when this changed, but I've noticed that Firefox remembers less browsing history than before.

My recollection is it used to remember something like the past 6 months, and these days it's only the past ~2 months or so.

I run into this a lot -- typing things into the awesomebar that I know I have visited within the past 6 months or so but are no longer in my history -- and it would be nice to be able to increase the retention period back to 6 months (or another period of the user's choosing).

Is there a preference for this, or can one be added?

Comment 1

[Marco Bonardo [:mak] (Away Apr 25 - May 5)]

No, there is no official preference, the hard limit is based on the database file size and, unless there has been a dataloss due to corruption two months ago, the only way to reduce history length is either a huge change in browsing behavior (you suddenty start browsing 10 hours a day instead of 1) or we start saving more data.
We have telemetry over that, and afaict it doesn't show a major reduction in the stored history size, there is a small reduction but indeed we added a few more things and we're comfortable with that (We're storing on average a bit more than a year of history). Nothing as big as what you are seeing though, so there must be more to this case.
Do you have a places.sqlite.corrupt in your profile folder?
If you order history in the Library by most recent date, what is the oldest date you see?

Comment 2

[Botond Ballo [:botond]]

(In reply to Marco Bonardo [:mak] from comment #1)

Do you have a places.sqlite.corrupt in your profile folder?

Nope

If you order history in the Library by most recent date, what is the oldest date you see?

December 20 2021, so not even a full two months of history.

Comment 3

[Marco Bonardo [:mak] (Away Apr 25 - May 5)]

Could you please open chrome://browser/content/places/interactionsViewer.html go to Places Stats section, and get a screenshot of those stats?
And yes, we should make it easier to get stats from about:support, I'll file a bug for that.

Comment 4

[Marco Bonardo [:mak] (Away Apr 25 - May 5)]

could you please also tell me the value of any pref under places.history.expiration.* branch?

Comment 5

[Botond Ballo [:botond]]

(In reply to Marco Bonardo [:mak] from comment #3)

Could you please open chrome://browser/content/places/interactionsViewer.html go to Places Stats section, and get a screenshot of those stats?

This page does not load for me. I should mention that the profile in question is running in a 91.5esr installation.

(In reply to Marco Bonardo [:mak] from comment #4)

could you please also tell me the value of any pref under places.history.expiration.* branch?

There aren't any.

Comment 6

[Marco Bonardo [:mak] (Away Apr 25 - May 5)]

(In reply to Botond Ballo [:botond] from comment #5)

(In reply to Marco Bonardo [:mak] from comment #3)

Could you please open chrome://browser/content/places/interactionsViewer.html go to Places Stats section, and get a screenshot of those stats?

This page does not load for me. I should mention that the profile in question is running in a 91.5esr installation.

Ah yes, we added that page after 91.
Could you please run Database Integrity from about:support Places Database section and attach here results from running it?
Unfortunately, being an older version it will have a lot less details, but hopefully something useful.

Comment 7

[Botond Ballo [:botond]]

Attachment: "Verify Integrity" output

Comment 8

[Botond Ballo [:botond]]

Attachment: "Verify Integrity" output

Attached is the output of the integrity verification.

Comment 9

[Marco Bonardo [:mak] (Away Apr 25 - May 5)]

Ok, something's very wrong here:

Places.sqlite size is 20480KiB
History can store a maximum of 24265 unique pages

That number is very low, I'd expect it to be > 100k in most cases. Because that number is low, Places will try to reduce history into that limit by expiring the oldest stuff.

How much memory does your system have?
Could you please run this code in the browser console and report results: Services.sysinfo.getProperty("memsize")

How much free space on disk does your profile folder have?
Could you please run this code in the browser console and report results: PlacesUtils.history.DBConnection.databaseFile.QueryInterface(Ci.nsIFile).diskSpaceAvailable

Comment 10

[Botond Ballo [:botond]]

(In reply to Marco Bonardo [:mak] from comment #9)

How much memory does your system have?
Could you please run this code in the browser console and report results: Services.sysinfo.getProperty("memsize")

33551912960

That's 32GB, which is what I would expect.

How much free space on disk does your profile folder have?
Could you please run this code in the browser console and report results: PlacesUtils.history.DBConnection.databaseFile.QueryInterface(Ci.nsIFile).diskSpaceAvailable

882970624

That's 842MB, which seems low (I have more free disk space than that). I'm guessing this one is the determining factor for the history size limit?

Comment 11

[Botond Ballo [:botond]]

How much free space on disk does your profile folder have?
Could you please run this code in the browser console and report results: PlacesUtils.history.DBConnection.databaseFile.QueryInterface(Ci.nsIFile).diskSpaceAvailable

882970624

That's 842MB, which seems low (I have more free disk space than that). I'm guessing this one is the determining factor for the history size limit?

It appears that I'm using profile-sync-daemon to run my profile from a RAM disk, and the RAM disk indeed has 842MB of free space.

Comment 12

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:mak, could you have a look please?

For more information, please visit auto_nag documentation.

Comment 13

[Marco Bonardo [:mak] (Away Apr 25 - May 5)]

(In reply to Botond Ballo [:botond] from comment #10)

How much free space on disk does your profile folder have?
Could you please run this code in the browser console and report results: PlacesUtils.history.DBConnection.databaseFile.QueryInterface(Ci.nsIFile).diskSpaceAvailable

882970624

That's 842MB, which seems low (I have more free disk space than that). I'm guessing this one is the determining factor for the history size limit?

Yes, this is the limiting factor, we keep a smaller database if the available space is limited. We use 2% as the threshold, so on your ~800MB it would be 17MB. A 4GB space would reach our standard db size, but 4GB may be a lot for a ram disk.
Maybe it's not the right approach and we should just half the target db size on small disks, it would still cause a restriction but not so drastic.
A similar issue happens on profile folders of users with a limited quota.

I'd suggest we simplify this and just half DATABASE_MAX_SIZE value if the calculated memory or disk threshold is < DATABASE_MAX_SIZE.
The code is at
https://searchfox.org/mozilla-central/rev/211649f071259c4c733b4cafa94c44481c5caacc/toolkit/components/places/PlacesExpiration.jsm#691-695

We should probably also print out the memory and disk size on about:support, I'm not sure why we don't.

Comment 14

[gliu20]

Hi there! I'd be open to taking this issue. From what I understand, the cause of the "data loss" is that the calculated optimal database size is too small. This size is currently calculated with:

let optimalDatabaseSize = Math.min(
      (memSizeBytes * DATABASE_TO_MEMORY_PERC) / 100,
      (diskAvailableBytes * DATABASE_TO_DISK_PERC) / 100,
      DATABASE_MAX_SIZE
    );

I'd suggest we simplify this and just half DATABASE_MAX_SIZE value if the calculated memory or disk threshold is < DATABASE_MAX_SIZE.
If we detect that the memory or disk is small, would it be possible that half DATABASE_MAX_SIZE exceeds the memory or the disk space available? Or can we safely assume that DATABASE_MAX_SIZE being 75MiB is never going to realistically exceed that? If we do have to account for this scenario, I think we end up back to where we were; we have to take into account the disk space available and the memory size, so I'm not sure how much code simplification we can do in this case.

I also noticed that MEMSIZE_FALLBACK_BYTES and DISKSIZE_FALLBACK_BYTES are both 256 MiB. This means that any time diskAvailableBytes or memSizeBytes is set to the fallback value, we also end up in a situation with an unreasonably low database size (~10 MiB or ~5 MiB).

Comment 15

[gliu20]

Attachment: Bug 1753729 - Add system memory and disk size and placeDB page limit to about:support. r?#fluent-reviewers,Gijs

Depends on D140912

Comment 16

[gliu20]

(In reply to gliu20 from comment #15)

Created attachment 9267529 [details]
Bug 1753729 - Add system memory and disk size and placeDB page limit to about:support. r?#fluent-reviewers,Gijs

Depends on D140912

I'm still new to contributing, but it seems like I made a child revision. I just removed it.

Comment 17

[avada]

(In reply to Marco Bonardo [:mak] from comment #6)

(In reply to Botond Ballo [:botond] from comment #5)

(In reply to Marco Bonardo [:mak] from comment #3)

Could you please open chrome://browser/content/places/interactionsViewer.html go to Places Stats section, and get a screenshot of those stats?

This page does not load for me. I should mention that the profile in question is running in a 91.5esr installation.

Ah yes, we added that page after 91.
Could you please run Database Integrity from about:support Places Database section and attach here results from running it?
Unfortunately, being an older version it will have a lot less details, but hopefully something useful.

I have FF97 (dev edition) running and that page doesn't load for me either.

(In reply to Marco Bonardo [:mak] from comment #9)

Ok, something's very wrong here:

Places.sqlite size is 20480KiB
History can store a maximum of 24265 unique pages

That number is very low, I'd expect it to be > 100k in most cases. Because that number is low, Places will try to reduce history into that limit by expiring the oldest stuff.

https://bugzilla.mozilla.org/attachment.cgi?id=9267952

For me it is lower...: "History can store a maximum of 20936 unique pages"
Actually lower than the amount of bookmark records: "+ Table moz_bookmarks has 22889 records"

Comment 18

[avada]

(In reply to Marco Bonardo [:mak] from comment #13)

(In reply to Botond Ballo [:botond] from comment #10)

How much free space on disk does your profile folder have?
Could you please run this code in the browser console and report results: PlacesUtils.history.DBConnection.databaseFile.QueryInterface(Ci.nsIFile).diskSpaceAvailable

882970624

That's 842MB, which seems low (I have more free disk space than that). I'm guessing this one is the determining factor for the history size limit?

Yes, this is the limiting factor, we keep a smaller database if the available space is limited. We use 2% as the threshold, so on your ~800MB it would be 17MB. A 4GB space would reach our standard db size, but 4GB may be a lot for a ram disk.
Maybe it's not the right approach and we should just half the target db size on small disks, it would still cause a restriction but not so drastic.
A similar issue happens on profile folders of users with a limited quota.

I'd suggest we simplify this and just half DATABASE_MAX_SIZE value if the calculated memory or disk threshold is < DATABASE_MAX_SIZE.
The code is at
https://searchfox.org/mozilla-central/rev/211649f071259c4c733b4cafa94c44481c5caacc/toolkit/components/places/PlacesExpiration.jsm#691-695

We should probably also print out the memory and disk size on about:support, I'm not sure why we don't.

Seriously... I have no words, but expletives.
I had 850ish MB of free space, and Firefox shredded 90+% of my history and places.sqlite went from 60MB (Judging by a month old backup) to 15 so it saved a whopping 45 megabytes...

What is see is absolutely no reason to limit the database size on PC. Even SSDs start at hundreds of gigabytes and HDDs at terabyte sizes.
If the free space is getting low it would be more useful to show a warning. (Windows apparently doesn't nowadays, but keeps eating it up with updates.)

Comment 19

[gliu20]

Attachment: Bug 1753729 - Simplify logic for calculating optimal PlacesDB size. r?mak

Depends on D140918

Comment 20

[Worcester12345]

Adding insult to injury, Thunderbird is consuming a TON of space (many Gb) on the hard drive. It's bad enough with a non-SSD, but one that is low on space and swapping is insane.

Comment 21

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b6773b17a61c
Add system memory and disk size and placeDB page limit to about:support. r=Gijs,fluent-reviewers
https://hg.mozilla.org/integration/autoland/rev/9f4330cf340a
Simplify logic for calculating optimal PlacesDB size. r=mak

Comment 22

[Iulian Moraru]

https://hg.mozilla.org/mozilla-central/rev/b6773b17a61c
https://hg.mozilla.org/mozilla-central/rev/9f4330cf340a

Comment 23

[gliu20]

Clearing needsinfo since answered elsewhere