835796 - Negative value in about:memory storage/sqlite/other due to double-counting of shared-cache connections

Status: RESOLVED FIXED

(Toolkit :: Storage, defect, P3)

Description

[Tero Nieminen]

Attachment: negative-about-memory.png

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:18.0) Gecko/20100101 Firefox/18.0
Build ID: 20130116073211

Steps to reproduce:

Upgraded from FF 18.0.0 to the latest Firefox (18.0.1 En_Us Linux x86_64) mozilla.org binary distribution. Started my previously saved session (session manager add-on) with this firefox.


Actual results:

Noticed that about:memory page diplays some (usually one occasionally two) negative memory values with a red error message at the top of the about:memory page:

"WARNING: the following values are negative or unreasonably large.
 explicit/storage/sqlite/(13 tiny)/other
This indicates a defect in one or more memory reporters.  The invalid values are highlighted."

The negative memory values seem to appear appear under "Explicit Allocations -> storage -> sqlite -> (13 tiny) ->other [?!].


Expected results:

Up to FF 18.0.0 all the memory values were positive, like most likely ought to be under 18.0.1 also.

Comment 1

[Tero Nieminen]

I just yesterday noticed Similar negative memory valuesreported also in FF 17.0.5 ESR under 64-bit Centos 6.4 (a fresh install and Firefox from standard Centos repo):

"WARNING: the following values are negative or unreasonably large.
 explicit/storage/sqlite/(15 tiny)
 explicit/storage/sqlite/(15 tiny)/other
This indicates a defect in one or more memory reporters.  The invalid values are highlighted."

Comment 2

[Tero Nieminen]

Just updating the ongoing story: Negative memory values still seem to occur quite often (for me at least) even on Firefox 20.0.1. Did exactly that just a few minutes ago right at entering the about:memory page after swithching over to Mozilla.org FF 20.0.1:
---
WARNING: the following values are negative or unreasonably large.
 explicit/storage/sqlite/(13 tiny)/other
This indicates a defect in one or more memory reporters.  The invalid values are highlighted.
---

And then the particular highlighted memory entry line:
---
      │      └──-13.61 MB (-0.84%) ── other [?!]
---

I'll attach the complete memory list in a moment.


It is with a firefox session with quite a hefty number of windows and tabs and quite many addons also (no denying that) - maybe that contributes to the problem - maybe it only occurs while "pushing the limits". Anyway the error message literally states the problem is with the memory reporters (components of firefox memory management maybe?). I wouldn't think they ought to err in any ussage scenario, though.

Comment 3

[Tero Nieminen]

Attachment: A complete about memory listing of FF 20.0.1 with a negative memory value reported

Comment 4

[Tero Nieminen]

Negative memory values still occur (for me at least) in FF 21 (and also FF 17.0.6 ESR). Seems to occur more often when there are many windows/tabs in FF and always in sqlite/tiny or sqlite/other.

Comment 5

[Tero Nieminen]

(In reply to Tero Nieminen from comment #4)
> Negative memory values still occur (for me at least) in FF 21 (and also FF
> 17.0.6 ESR). Seems to occur more often when there are many windows/tabs in
> FF and always in sqlite/tiny or sqlite/other.

Even FF ESR 17.0.8 under RHEL Desktop 6.4 still exhibits negative memory values (at least for me). The error message says: "This indicates a defect in one or more memory reporters". I don't have any idea whether these memory reporters part FF or parts of add-ons, anyway, FF itself says that they are defective. Wheather that actually is a correct assesment is beyond my knowledge.

I don't have any more Idea of the cause for this, exept I've noticed it occurring for the past year in several different FF/Linux configurations (lately on RHEL/Centos 6.4 with distro supplied FF ESR) and that it dowes not necessarily occur every time I open the about:memory page (especially on just started Firefox), insteaf it occurs more consistently when there are more open pages/tabs in FF (and especially after beeing used for some time).

I haven't noticed any effect on the browser operation though: no crashes or profile corruption or anything). Maybe it also ought to be noted that I would consider myself a rather a FF heavy user: I've got quite a few add-ons and and a hefty number of tabs also (though most tabs in unloaded state). The memory size of my FF right now (as reported by top) is:
---
17855 ttn       20   0 4317m 2.6g  20m S  9.2 68.8 266:32.84 firefox            
---

The FF has been running for about a week since last start, but I haven't noticed any significant increase in the negative values over time. Also theyseem to stay the same even after "Minimise memory usage".

The heavy usage may have something to do with the persistence I see those negative memory values.

Below is a sample from RHEL Desktop (this time negative values appear under "sqlite/tiny" and "sqlite/other").

------ cut here ------
Main Process

WARNING: the following values are negative or unreasonably large.
 explicit/storage/sqlite/(14 tiny)
 explicit/storage/sqlite/(14 tiny)/other
This indicates a defect in one or more memory reporters.  The invalid values are highlighted.

Explicit Allocations

2,716.32 MB (100.0%) -- explicit
├──1,815.80 MB (66.85%) ++ window-objects
├────398.50 MB (14.67%) ── heap-unclassified
├────241.87 MB (08.90%) ++ js-non-window
├────180.78 MB (06.66%) ++ workers/workers()
├─────33.21 MB (01.22%) ── xpconnect
├─────30.82 MB (01.13%) -- storage
│     ├──29.14 MB (01.07%) -- sqlite
│     │  ├──67.27 MB (02.48%) -- CertPatrol.sqlite
│     │  │  ├──56.78 MB (02.09%) ── cache-used [180]
│     │  │  └──10.49 MB (00.39%) ++ (2 tiny)
│     │  └──-38.13 MB (-1.40%) -- (14 tiny) [?!]
│     │     ├────4.58 MB (00.17%) ++ places.sqlite
│     │     ├────0.79 MB (00.03%) ++ extensions.sqlite
│     │     ├────0.71 MB (00.03%) ++ addons.sqlite
│     │     ├────0.37 MB (00.01%) ++ formhistory.sqlite
│     │     ├────0.36 MB (00.01%) ++ webappsstore.sqlite
│     │     ├────0.35 MB (00.01%) ++ signons.sqlite
│     │     ├────0.28 MB (00.01%) ++ content-prefs.sqlite
│     │     ├────0.26 MB (00.01%) ++ readItLater.sqlite
│     │     ├────0.25 MB (00.01%) ++ chromeappsstore.sqlite
│     │     ├────0.20 MB (00.01%) ++ index.sqlite
│     │     ├────0.18 MB (00.01%) ++ cookies.sqlite
│     │     ├────0.14 MB (00.01%) ++ downloads.sqlite
│     │     ├────0.11 MB (00.00%) ++ permissions.sqlite
│     │     └──-46.69 MB (-1.72%) ── other [?!]
│     └───1.68 MB (00.06%) ++ prefixset
└─────15.32 MB (00.56%) ++ (10 tiny)
------ cut here ------

T:Tero

Comment 6

[Tero Nieminen]

Attachment: Firefox ESR 17.0.9 Troubleshooting information from a Red Hat Enterprise Linux Workstation release 6.4

Comment 7

[Tero Nieminen]

Negative memory values still occur (for me at least) in Firefox ESR 24.2.0 (under RHEL Desktop 6.5 and also in a non-identical FF configuration in a Centos 6.5, among others), ie. no change there.

Maybe the FF version number ought to be updated to reflect most recent FF version affected in the case headers, to prevent this case from falling into obsolence.

Below a sample memory value report from RHEL 6.5 and Firefox ESR 24.2.0 (firefox-24.2.0-1.el6_5.x86_64):
---
2,151.22 MB (100.0%) -- explicit
├──1,158.83 MB (53.87%) ++ window-objects
├────441.11 MB (20.51%) ++ js-non-window
├────298.89 MB (13.89%) ── heap-unclassified
├────148.06 MB (06.88%) ++ workers/workers()
├─────53.36 MB (02.48%) ++ (16 tiny)
├─────25.55 MB (01.19%) -- storage
│     ├──24.33 MB (01.13%) -- sqlite
│     │  ├──76.72 MB (03.57%) -- CertPatrol.sqlite
│     │  │  ├──71.01 MB (03.30%) ── cache-used [98]
│     │  │  └───5.70 MB (00.27%) ++ (2 tiny)
│     │  └──-52.38 MB (-2.44%) -- (13 tiny) [?!]
│     │     ├────7.65 MB (00.36%) ++ places.sqlite
│     │     ├────0.83 MB (00.04%) ++ extensions.sqlite
│     │     ├────0.71 MB (00.03%) ++ addons.sqlite
│     │     ├────0.34 MB (00.02%) ++ formhistory.sqlite
│     │     ├────0.30 MB (00.01%) ++ webappsstore.sqlite
│     │     ├────0.28 MB (00.01%) ++ content-prefs.sqlite
│     │     ├────0.26 MB (00.01%) ++ readItLater.sqlite
│     │     ├────0.24 MB (00.01%) ++ cookies.sqlite
│     │     ├────0.23 MB (00.01%) ++ signons.sqlite
│     │     ├────0.14 MB (00.01%) ++ :memory:
│     │     ├────0.14 MB (00.01%) ++ downloads.sqlite
│     │     ├────0.10 MB (00.00%) ++ permissions.sqlite
│     │     └──-63.60 MB (-2.96%) ── other [?!]
│     └───1.22 MB (00.06%) ++ prefixset
└─────25.41 MB (01.18%) ++ add-ons
---

Unfortunatey, I haven't had the time to try and locate the cause for the negative values in any way.

I will also attach an up-to-date Troubleshooting information.

Comment 8

[Tero Nieminen]

Attachment: Troubleshooting information from 64-bit RHEL 6.5 Desktop/FF 24.2.0 ESR

Comment 9

[jihor]

Confirmed of FF 27 Stable:

├───22.02 MB (03.14%) -- storage
│   ├──20.60 MB (02.93%) -- sqlite
│   │  ├──19.52 MB (02.78%) -- viewmarks.sqlite
│   │  │  ├──19.49 MB (02.78%) ── cache-used [20]
│   │  │  └───0.03 MB (00.00%) ++ (2 tiny)
│   │  └───1.08 MB (00.15%) -- (11 tiny)
│   │      ├──7.00 MB (01.00%) ++ places.sqlite
│   │      ├──3.20 MB (00.46%) ++ history.sqlite
│   │      ├──2.18 MB (00.31%) ++ webappsstore.sqlite
│   │      ├──1.05 MB (00.15%) ++ cookies.sqlite
│   │      ├──0.38 MB (00.05%) ++ readItLater.sqlite
│   │      ├──0.28 MB (00.04%) ++ formhistory.sqlite
│   │      ├──0.27 MB (00.04%) ++ content-prefs.sqlite
│   │      ├──0.24 MB (00.03%) ++ signons.sqlite
│   │      ├──0.23 MB (00.03%) ++ healthreport.sqlite
│   │      ├──0.17 MB (00.02%) ++ permissions.sqlite
│   │      └──-13.93 MB (-1.98%) ── other [?!]

Comment 10

[Chris Morgan]

I’ve never got this before Firefox 30, and I don’t *think* it was immediately after upgrading to 30, but on my 64-bit Arch Linux system I have for the past two or three weeks been consistently getting slightly negative values for explicit/storage/sqlite/other, with a few Firefox restarts in between; the two values I have observed have been -0.50 MB and -0.42 MB.

Comment 11

[Tero Nieminen]

As of FF 31.1.0 (ESR) the negative memory values seem to clear themselves after doing a "minimize memory usage" and then "Measure". They still seem to appear (for me at least) initially every time Firefox is started and after being used for some time if Measure is done before Minimize.

Comment 12

[Tero Nieminen]

(In reply to Tero Nieminen from comment #11)
> As of FF 31.1.0 (ESR) the negative memory values seem to clear themselves
> after doing a "minimize memory usage" and then "Measure". They still seem to
> appear (for me at least) initially every time Firefox is started and after
> being used for some time if Measure is done before Minimize.

It appears that negative memory values reappear and start growing again very quickly after "minimize memory use" (ie. that cure is only temporary). It seems just creating a new empty FF window is enough to bring it up again, and visiting any web page there will make the negative values grow further.

Comment 13

[Tero Nieminen]

Apparently neither GC nor CC button alone (nor both in that sequence) clear the negative memory values, but Minimize memory usage button currently seems to.

I still have no idea whether this is a reporting bug only or is there something actually going wrong with memory allocation. I've had no crashes, though, that could indicate the latter. Maybe the bug is in the reporters just as Firefox itself has been suggesting all along. Below a fresh exerpt from about:memory a short while after Minimize memory usage (after which there was initially no negative memory values):

-----
Main Process

WARNING: the following values are negative or unreasonably large.

    explicit/(19 tiny)/storage/sqlite/(10 tiny)
    explicit/(19 tiny)/storage/sqlite/(10 tiny)/other 

This indicates a defect in one or more memory reporters. The invalid values are highlighted.
Explicit Allocations 
1,483.76 MB (100.0%) -- explicit
├────701.52 MB (47.28%) ++ window-objects
├────333.22 MB (22.46%) ++ js-non-window
├────181.01 MB (12.20%) ── heap-unclassified
├─────85.03 MB (05.73%) ++ add-ons
├─────55.88 MB (03.77%) ++ heap-overhead
├─────55.20 MB (03.72%) -- (19 tiny)
│     ├──14.05 MB (00.95%) ++ layout
│     ├──13.29 MB (00.90%) -- storage
│     │  ├──11.64 MB (00.78%) -- sqlite
│     │  │  ├──17.81 MB (01.20%) -- CertPatrol.sqlite
│     │  │  │  ├──15.14 MB (01.02%) ── cache-used [48]
│     │  │  │  └───2.67 MB (00.18%) ++ (2 tiny)
│     │  │  └──-6.17 MB (-0.42%) -- (10 tiny) [?!]
│     │  │     ├───3.38 MB (00.23%) ++ places.sqlite
│     │  │     ├───0.37 MB (00.02%) ++ cookies.sqlite
│     │  │     ├───0.30 MB (00.02%) ++ webappsstore.sqlite
│     │  │     ├───0.24 MB (00.02%) ++ healthreport.sqlite
│     │  │     ├───0.21 MB (00.01%) ++ content-prefs.sqlite
│     │  │     ├───0.20 MB (00.01%) ++ signons.sqlite
│     │  │     ├───0.14 MB (00.01%) ++ permissions.sqlite
│     │  │     ├───0.09 MB (00.01%) ++ formhistory.sqlite
│     │  │     ├───0.07 MB (00.00%) ++ readItLater.sqlite
│     │  │     └──-11.17 MB (-0.75%) ── other [?!]
│     │  └───1.66 MB (00.11%) ++ prefix-set
│     ├───8.50 MB (00.57%) ── atom-tables
│     ├───6.08 MB (00.41%) ++ dom
│     ├───4.08 MB (00.28%) ++ images
│     ├───2.55 MB (00.17%) ++ startup-cache
│     ├───1.64 MB (00.11%) ++ network
│     ├───1.47 MB (00.10%) ++ cycle-collector
│     ├───1.09 MB (00.07%) ── xpti-working-set
│     ├───0.66 MB (00.04%) ── preferences
│     ├───0.40 MB (00.03%) ++ xpcom
│     ├───0.38 MB (00.03%) ── history-links-hashtable
│     ├───0.30 MB (00.02%) ++ gfx
│     ├───0.24 MB (00.02%) ── cookie-service
│     ├───0.23 MB (00.02%) ── icu
│     ├───0.13 MB (00.01%) ── script-namespace-manager
│     ├───0.08 MB (00.01%) ── telemetry
│     ├───0.00 MB (00.00%) ++ media
│     └───0.00 MB (00.00%) ── spell-check
├─────52.60 MB (03.55%) ++ workers/workers()
└─────19.31 MB (01.30%) ++ xpconnect
-----

Comment 14

[Tero Nieminen]

Just a reminder to note that this is still ongoing: two years and counting... :)

Negative values still occurring on FF ESR 31.5.0 (64-bit) under RHEL 6.6 (x86_64) even after many FF versions, add-ons changed/updated/removed/added, settings changed etc. Now, just as of late, complete "Minimize memory usage" clears the negative values for a while - almost any activity brings it back again, though.

While gathering the details for this comment and writing this note, the negative values have grown from a few dozen megs to a few hundred megs (see below). Growing would seem to slow after some time (in any case I've never noticed it to reach amounts that would make any dent in current computers RAM - though I do wonder at times, what other consequences the negative values might have as I have for example seen a couple of times that FF cpu core hogging has ceased (for a while anyway) after "Minimize memory usage" (not nearly every time, but still often enough for it probably not to be just coincidental).

Below some details collected just now:

--- clip clip 1 ---
WARNING: the following values are negative or unreasonably large.

    explicit/(19 tiny)/storage/sqlite/(10 tiny)
    explicit/(19 tiny)/storage/sqlite/(10 tiny)/other 

This indicates a defect in one or more memory reporters. The invalid values are highlighted. 
--- clip clip 1 ---


--- clip clip 2 ---
├─────82.08 MB (02.52%) -- (19 tiny)
│     ├──17.88 MB (00.55%) -- storage
│     │  ├──15.94 MB (00.49%) -- sqlite
│     │  │  ├──35.65 MB (01.09%) ++ CertPatrol.sqlite
│     │  │  └──-19.71 MB (-0.60%) -- (10 tiny) [?!]
│     │  │     ├────2.38 MB (00.07%) ++ places.sqlite
│     │  │     ├────0.43 MB (00.01%) ++ cookies.sqlite
│     │  │     ├────0.24 MB (00.01%) ++ webappsstore.sqlite
│     │  │     ├────0.24 MB (00.01%) ++ formhistory.sqlite
│     │  │     ├────0.17 MB (00.01%) ++ signons.sqlite
│     │  │     ├────0.14 MB (00.00%) ++ content-prefs.sqlite
│     │  │     ├────0.14 MB (00.00%) ++ permissions.sqlite
│     │  │     ├────0.09 MB (00.00%) ++ healthreport.sqlite
│     │  │     ├────0.07 MB (00.00%) ++ readItLater.sqlite
│     │  │     └──-23.61 MB (-0.72%) ── other [?!]
│     │  └───1.94 MB (00.06%) ++ prefix-set
--- clip clip 2 ---


--- clip clip 3 ---
├─────81.35 MB (02.47%) -- (19 tiny)
│     ├──20.86 MB (00.63%) -- storage
│     │  ├──18.92 MB (00.57%) -- sqlite
│     │  │  ├──214.09 MB (06.49%) -- CertPatrol.sqlite
│     │  │  │  ├──208.26 MB (06.31%) ── cache-used [105]
│     │  │  │  └────5.83 MB (00.18%) ++ (2 tiny)
│     │  │  └──-195.17 MB (-5.91%) -- (10 tiny) [?!]
│     │  │     ├─────3.11 MB (00.09%) ++ places.sqlite
│     │  │     ├─────0.52 MB (00.02%) ++ cookies.sqlite
│     │  │     ├─────0.26 MB (00.01%) ++ content-prefs.sqlite
│     │  │     ├─────0.24 MB (00.01%) ++ webappsstore.sqlite
│     │  │     ├─────0.24 MB (00.01%) ++ formhistory.sqlite
│     │  │     ├─────0.20 MB (00.01%) ++ signons.sqlite
│     │  │     ├─────0.14 MB (00.00%) ++ permissions.sqlite
│     │  │     ├─────0.08 MB (00.00%) ++ healthreport.sqlite
│     │  │     ├─────0.07 MB (00.00%) ++ readItLater.sqlite
│     │  │     └──-200.04 MB (-6.06%) ── other [?!]
│     │  └───1.94 MB (00.06%) ++ prefix-set
--- clip clip 3 ---



I wonder how many people ever even go into about:memory - let alone witness something like this (might take quite heavy usage to ever even see it, or maybe not). Could it, maybe, show up in the health report (which I can't seem to get to stay activated).

Comment 15

[neopeeves]

Still a problem. Firefox becomes VERY laggy and unresponsive. 36.0.1 here. I know damn well you have enough information from the other people.

WARNING: the following values are negative or unreasonably large.

    explicit/storage/sqlite/(7 tiny)/other 

This indicates a defect in one or more memory reporters. The invalid values are highlighted.

Comment 16

[Andrew Sutherland [:asuth] (he/him)]

If using system SQLite, SQLite's memory usage is not hooked (enabled by the MOZ_STORAGE_MEMORY define), but the memory reporters will still be active.  It would make sense if this resulted in negative values, I think.  I believe many linux distribution builds like to try and use the system SQLite which will result in this scenario, and I think explains the observed comments.  (Noting that this bug is not going to impact the responsiveness of Firefox.)  Bug 938730 and bug 1026828 have some context here.

On my Ubuntu 14.04.2 system which does not use native SQLite (libmozsqlite3.so can be found in the distribution), the about:memory values are all positive and don't generate errors.

Comment 17

[Tero Nieminen]

I have yet to notice any meaningful difference in the negative values between RHEL FF ESR (now at 31.5.0-1) and Mozilla binaries (FF latest/36.0.1)), though admittedly, I haven't made a side-by-side comparison: both seem to exhibit quite similar negatives.

I just tested with latest Mozilla binary FF 36.0.1 and I readily see negatives after session restore completes. They will (for a year now) always go away for a while by doing a complete memory minimization, but opening a new tab to Google search (I guess any web address would do) will suffice to make them reappear, albeit initially small. Opening just a new empty tab does not seem to cause negatives (at least not immediately).

Comment 18

[Tero Nieminen]

Finally traced this problem to AddOn Certificate Patroll (that hasn't been updated for some years now (since 2011)). After disabling or removing it I have yet to witness any negative memory values.

Comment 19

[neopeeves]

(In reply to Tero Nieminen from comment #18)
> Finally traced this problem to AddOn Certificate Patroll (that hasn't been
> updated for some years now (since 2011)). After disabling or removing it I
> have yet to witness any negative memory values.

Would you like to share how to do that?

Comment 20

[Tero Nieminen]

(In reply to neopeeves from comment #19)
> (In reply to Tero Nieminen from comment #18)
> > Finally traced this problem to AddOn Certificate Patroll (that hasn't been
> > updated for some years now (since 2011)). After disabling or removing it I
> > have yet to witness any negative memory values.
> 
> Would you like to share how to do that?

Sorry, I was overly terse.

I had installed a Firefox AddOn called "Certificate Patrol", that watches out and notifies user every time a certificate of a web site changes, to alert the user to verify whether the change was appropriate, or if is someone trying MITM with false certificate or some such.

The idea of the AddOn seemed sound, but lately it has been more of a nuisance especially Google seems to have differing short lived (sha1-)certificates in different cloud nodes (and what not) and thus causing constant CertPatrol certificate change alerts to pop out.

I decided to try something different in it's place (Perspectives, actually) and just disabled CertPatrol. A bit later, I also noticed I'm not getting the negative memory values any more and thus thought CertPatrol to have been the culprit, for me anyway: how exactly, I don't know, but it does seem to use Sqlite for storage (that does not, of course, in any way prove that the bug would actually be in CertPatrol, but it could well be - being practically abandoned AddOn, and all).

Probably not very many people have certPatrol installed, so I guess, so there could quite well be other culprits for the negative memory values.

Anyway, just inactivating the CertPatrol AddOn in FF seem to have cleared the negative memory values in about memory page (for me, and for the time being anyway - fingers crossed it holds :). Different people may well have different causes for negative memory values, but since this was originally reported by me (with a slightly less specific subject line, though), I thought the case could now be closed, for my part anyway.

Does that answer your question?

Feel free to keep this open if you are still experiencing the negative memory values, but try to include any relevant details (if they differ from mine).

PS. For what it's worth, I've been mostly using a latest version of mozilla.org binary distribution of FF (all Linux/x86_64) instead of a one that comes with the distros I use (mostly RHEL6/Centos6)) with an occasional FF ESR thrown in for comparison (never any difference there).

Comment 21

[neopeeves]

Well I've never heard of it, let alone have it installed. So that's definitely not the reason my firefox goes to hell when looking at images.

Comment 22

[Tero Nieminen]

Back to unconfirmed, as is was, then...

T:Tero

Comment 23

[Eric Rahm [:erahm]]

We're seeing this for a Windows user as well in bug 1273735.

Comment 24

[Andrew Sutherland [:asuth] (he/him)]

It looks like our per-connection reporters don't account for the possibility of shared cache instances.  Seeing "cache-used [20]" for viewmarks.sqlite means 20 connections exist/are leaked, which is of course bad, but can also cause these errors to accumulate.  Likewise for "cache-used [55]" for CertPatrol.sqlite.

Our "other" memory is defined as:
  int64_t other = ::sqlite3_memory_used() - totalConnSize;
So when our totalConnSize gets erroneously large due to double-counting, the value goes negative.

We're not seeing this in general because many connections are either unshared or only opened once or a small enough number to cause errors but not go negative.

Note: I haven't replicated and have done only a very quick code inspection pass, but I think this is very likely to explain the problem.

Comment 25

[Eric Rahm [:erahm]]

This would be nice to fix, but seems somewhat rare.

Andrew, is it clear when a cache instance is shared? It seems like we could build up a list of caches and filter out duplicates before reporting. Another option is to just not report shared instances, which is unfortunate but probably better then overreporting.

For posterity the code we're looking at is here: https://dxr.mozilla.org/mozilla-central/rev/b69a5bbb5e40bd426e35222baa600b481e50d265/storage/mozStorageService.cpp#125-196

Comment 26

[Andrew Sutherland [:asuth] (he/him)]

(In reply to Eric Rahm [:erahm] from comment #25)
> Andrew, is it clear when a cache instance is shared?

Sometimes.  In the simplest cases, we have mozIStorageService.openDatabase which is explicitly shared, and mozIStorageService.openUnsharedDatabase which is explicitly unshared.  These flags are saved into mFlags on the connection.

It gets more complex in that openDatabaseWithFileURL is shared by default and that flag saved into mFlags.  However, the specified file URI may explicitly pass a "cache" parameter (https://sqlite.org/c3ref/open.html); we could extract that out to update our mFlags, but don't.

It gets even more complex because "ATTACH <database>" accepts URI's where an explicit "cache" mode can also be specified.

Practically speaking, if we want correct and maximally useful answers, the best thing to do is probably to ask the SQLite team to augment "PRAGMA database_list" or something like it to have the following columns:
* shareCount: number from pBt->nRef
* cacheUsed: sqlite3PagerMemUsed(sqlite3BtreePager(pBt))
We can then have our memory reporter go a step further and report each attached database to the connection and its cache usage rather than a sum.

Alternately, we can request that SQLITE_DBSTATUS_CACHE_USED gain a new friend SQLITE_DBSTATUS_PROPORTIONATE_CACHE_USED that instead of summing cacheUsed sums cacheUsed/shareCount.  In this case we don't need to otherwise change our memory reporters since the existing paths and summing should largely work out.

The extra dbstatus code doesn't seem that bad, so, :mak, what do you think about asking for that one?  The pragma might be a bit much, and it would really be to help you given that you're ATTACHing databases now.

(Note that although there is a global `sqlite3SharedCacheList` by necessity, I don't believe there is a global connection list known to SQLite at this time, and from a locking perspective it seems bad to go down the road of introducing one in order to support having a "global memory usage" virtual table implementation or similar, although it would be neat.)

Comment 27

[Andrew Sutherland [:asuth] (he/him)]

Note: And in the PRAGMA database_list case we'd have our memory reporter do the division so that the sums line up.

Comment 28

[Marco Bonardo [:mak] (Away Apr 25 - May 5)]

(In reply to Andrew Sutherland [:asuth] from comment #26)
> It gets more complex in that openDatabaseWithFileURL is shared by default
> and that flag saved into mFlags.  However, the specified file URI may
> explicitly pass a "cache" parameter (https://sqlite.org/c3ref/open.html); we
> could extract that out to update our mFlags, but don't.

sounds like this would be easy to do.

> It gets even more complex because "ATTACH <database>" accepts URI's where an
> explicit "cache" mode can also be specified.

Good catch, I didn't know about that, just found it in the documentation. Fwiw, we could also look for "cache=shared" in the query string and do something special only for ATTACH statements.

It sounds a little bit fragile, but this is not a critical issue, so it could do.

> Practically speaking, if we want correct and maximally useful answers, the
> best thing to do is probably to ask the SQLite team to augment "PRAGMA
> database_list" or something like it to have the following columns:
> * shareCount: number from pBt->nRef
> * cacheUsed: sqlite3PagerMemUsed(sqlite3BtreePager(pBt))
> We can then have our memory reporter go a step further and report each
> attached database to the connection and its cache usage rather than a sum.

Sounds like this may add some overhead (it should run the pragma on every known connection, get every result, check for the cache flag), but it only happen when we measure memory, it could work.

> Alternately, we can request that SQLITE_DBSTATUS_CACHE_USED gain a new
> friend SQLITE_DBSTATUS_PROPORTIONATE_CACHE_USED that instead of summing
> cacheUsed sums cacheUsed/shareCount.  In this case we don't need to
> otherwise change our memory reporters since the existing paths and summing
> should largely work out.

If it's feasible, this sounds perfect

> The pragma might be a bit much, and it would
> really be to help you given that you're ATTACHing databases now.

Yes, but I don't use uris, I just retain the cache status of the original connection, that is unshared fwiw. I don't see the benefits from that.

Summing up, I think the DBSTATUS idea is sound, provided it's not too complex for them to add it.
Alternatively we could track the shared status by parsing some of the queries and the API calls, it would be a little bit more fragile, but a decent workaround.

Comment 29

[Andrew Sutherland [:asuth] (he/him)]

Attachment: example patch introducing a SQLITE_DBSTATUS_PROPORTIONAL_CACHE_USED variant v1

:drh, see comment 26 for more context, but basically:
- Our current about:memory https://developer.mozilla.org/en-US/docs/Mozilla/Performance/about%3Amemory memory reporters aren't compensating for shared caches and it seems like it's cleanest to address in SQLite.
- We're wondering if it would be reasonable to add a "proportional" memory use estimation for SQLite's page cache that compensates for shared caches by equally divvying up responsibility among all connections.  The prior art is the "PSS" value in Linux's /proc/PID/smaps that works along similar lines.

I've attached a very simple draft patch I made to convince myself this isn't too Mozilla-specific.  We're also very open to other suggestions about how best to make our numbers more accurate.  Thanks!

PS: This is low priority.

Comment 30

[D. Richard Hipp]

(In reply to Andrew Sutherland [:asuth] from comment #29)
> Created attachment 8768269 [details] [diff] [review]

Your patch looks very clever, :asuth.  We'll need to do some testing and analysis, but I think it likely that we can land something like that in SQLite 3.14.0 for you.  I'll follow up to this bug when we know more.

Comment 31

[D. Richard Hipp]

(In reply to Andrew Sutherland [:asuth] from comment #29)
> - We're wondering if it would be reasonable to add a "proportional" memory
> use estimation for SQLite's page cache that compensates for shared caches by
> equally divvying up responsibility among all connections. 

Change to implement this are now on the SQLite trunk (https://www.sqlite.org/src/timeline?c=06cf2680) and should be included in the next SQLite release.  The name of the new sqlite3_db_status() call is slightly different: SQLITE_DBSTATUS_CACHE_USED_SHARED instead of SQLITE_DBSTATUS_PROPORTIONAL_CACHE_USED, but it works the same as outlined by :asuth above.

Comment 32

[Andrew Sutherland [:asuth] (he/him)]

Awesome, thank you!

Comment 33

[D. Richard Hipp]

Comment on attachment 8768269 [details] [diff] [review]
example patch introducing a SQLITE_DBSTATUS_PROPORTIONAL_CACHE_USED variant v1

Review of attachment 8768269 [details] [diff] [review]:
-----------------------------------------------------------------

Name changed to SQLITE_DBSTATUS_CACHE_USED_SHARED.  Also some minor technical details corrected.  But the suggestion has been implemented as requested.  A pre-release snapshot is available on the https://www.sqlite.org/download.html page.  Tentative documentation for the new API can be found at https://www.sqlite.org/draft/c3ref/c_dbstatus_options.html#sqlitedbstatuscacheusedshared

Comment 34

[Nicholas Nethercote [inactive]]

Great work here. Thank you all.

Comment 35

[Marco Bonardo [:mak] (Away Apr 25 - May 5)]

Attachment: Bug 835796 - Negative value in about:memory storage/sqlite/other due to double-counting of shared-cache connections.

Review commit: https://reviewboard.mozilla.org/r/90658/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/90658/

Comment 36

[Andrew Sutherland [:asuth] (he/him)]

Comment on attachment 8807519 [details]
Bug 835796 - Negative value in about:memory storage/sqlite/other due to double-counting of shared-cache connections.

https://reviewboard.mozilla.org/r/90658/#review90354

Wooo!

Comment 37

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/1fa12824f3d8
Negative value in about:memory storage/sqlite/other due to double-counting of shared-cache connections. r=asuth

Comment 38

[Phil Ringnalda (:philor)]

https://hg.mozilla.org/mozilla-central/rev/1fa12824f3d8