1408406 - Differences in storage sizes between data stored in Firefox and profile folder

Status: RESOLVED WONTFIX

(Core :: DOM: Core & HTML, defect, P2)

Description

[Ciprian Georgiu, Desktop QA]

Attachment: screenshot showing this issue - OS X 10.11.png

[Affected versions]:
- latest Nightly 58.0a1 (2017-10-13)
- Beta 57.0b8 (20171013042429)

[Affected platforms]:
- Mac OS X 10.11
- Ubuntu 16.04 x64

[Steps to reproduce]:
1. Start Firefox on a clean profile.
2. Go to youtube.com, persist the website and store some data.
3. Compare and observe the storage data size from about:preferences#privacy with the one from profile folder.

[Expected result]:
- The storage size has the exact size displayed in Firefox as it appears in profile folder.

[Actual result]:
- There are differences in storage sizes between data stored in Firefox and profile folder.

[Regression range]:
- I don't think that this is a regression, I can also repro this on an old Nightly build from 2017-04-25

[Additional notes]:
- please see the attached screenshot
- on Windows 10 x64 I cannot see this behavior, it seems that the data size stored in Firefox has the same size as in profile folder.

Comment 1

[Shawn Huang [:shawnjohnjr][:shuang] (as a happy gecko contributor)]

I think this is intentional. In #perferences Site Data, it also combines other storage data which is not managed by QuotaManager. Fischer, can you clarify this behavior first?

Comment 2

[Shawn Huang [:shawnjohnjr][:shuang] (as a happy gecko contributor)]

https://bugzilla.mozilla.org/show_bug.cgi?id=1348733#c27
"Based on the con-call meeting, the specs has been changed as below:
- exclude counting and removing http cache per site
- group sites based on host across scheme, port and origin attributes
- list sites using quota usage or appcache"

Comment 3

[Fischer [:Fischer]]

(In reply to Shawn Huang [:shawnjohnjr] from comment #1)
> I think this is intentional. In #perferences Site Data, it also combines
> other storage data which is not managed by QuotaManager. Fischer, can you
> clarify this behavior first?
Yes, Shawn as your comment 2.
About the difference of the usage displayed, there are many factors affecting that:
1. The gap is bigger if compared to the https+++www.some_website.com folder because that folder contains some more meat data
2. the gap is smaller if only compared to the https+++www.some_website.com/idb folder with a smaller usage
3. while displaying usage we will do some round-off
4. not sure if it is possible and guess it is quite hard to always fetch and calculate the *precise* usage. (And also what is the precise scope to calculate usage?)

However, at least the more data is stored, the larger usage is displayed. That is for sure.
Here are my observations:
1. 72.0KB in site data settings displayed,
     80KB for https+++www.some_website.com folder,
     74KB for https+++www.some_website.com/idb folder

2.  1.5GB in site data settings displayed,
   1.61GB for https+++www.some_website.com folder, 
   1.61GB for https+++www.some_website.com/idb folder

So I would guess this is more like about a UX question: Could we stand some tolerance on usage displayed?

@Shawn
How do yo think about the above usage display tolerance? I tend to reckon this is a kind of limitation and WONTFIX.

Comment 4

[Hsin-Yi Tsai (she/her) [:hsinyi]]

Marking as P2 for now and waiting for follow-up of comment 3.

Comment 5

[Shawn Huang [:shawnjohnjr][:shuang] (as a happy gecko contributor)]

(In reply to Fischer [:Fischer] from comment #3)
> (In reply to Shawn Huang [:shawnjohnjr] from comment #1)
> > I think this is intentional. In #perferences Site Data, it also combines
> > other storage data which is not managed by QuotaManager. Fischer, can you
> > clarify this behavior first?
> Yes, Shawn as your comment 2.
> About the difference of the usage displayed, there are many factors
> affecting that:
> @Shawn
> How do yo think about the above usage display tolerance? I tend to reckon
> this is a kind of limitation and WONTFIX.
Here is what I think:
1. I don't think users would really get confused since it's tricky to check folders under storage directory.
2. We just need to provide a way to let QA figure out storage folder (temporary or persistent storage size + extra storage space) is displayed. So QA can verify the correctness of size displaying. Can you list any quick way to check extra value? Like appCache.
3. I agree with you this is by design. But we just need to clarify things designed correctly.

Comment 6

[Shawn Huang [:shawnjohnjr][:shuang] (as a happy gecko contributor)]

Meanwhile, ni Mark for comments for Comment 3. Although I think this is just about verfication on size displaying.
Mark, can you comment on any impact for users?

Comment 7

[Fischer [:Fischer]]

(In reply to Shawn Huang [:shawnjohnjr] from comment #5)
> (In reply to Fischer [:Fischer] from comment #3)
> > (In reply to Shawn Huang [:shawnjohnjr] from comment #1)
> > > I think this is intentional. In #perferences Site Data, it also combines
> > > other storage data which is not managed by QuotaManager. Fischer, can you
> > > clarify this behavior first?
> > Yes, Shawn as your comment 2.
> > About the difference of the usage displayed, there are many factors
> > affecting that:
> > @Shawn
> > How do yo think about the above usage display tolerance? I tend to reckon
> > this is a kind of limitation and WONTFIX.
> Here is what I think:
> 1. I don't think users would really get confused since it's tricky to check
> folders under storage directory.
> 2. We just need to provide a way to let QA figure out storage folder
> (temporary or persistent storage size + extra storage space) is displayed.
> So QA can verify the correctness of size displaying. Can you list any quick
> way to check extra value? Like appCache.
For appCache, the simpler and the more accurate way to verify it is to only have appCache installed and check it out. So the SOP for appCache case would be:
1. Open Nightly with an new Profile
2. Go to the Site Data in about:preferences and make sure nothing there
3. Open the OfflineCache folder in the Profile, making sure that's an empty folder.
4. Visit a Site only store appCache and no other data (Don't persist that site either)
5. Go to the Site Data in about:preferences and make sure the website is there
6. Open the OfflineCache folder in the Profile, making sure that's not an empty folder.

If we verify the quota storage(ex. indexedDB) and appCache together, it is quite hard to calculate how many usage belongs to indexedDB and how many belongs to appCache.

Another way I could think of verifying the quota storage(ex. indexedDB) and appCache together is to store very small amount of indexedDB(like few KB only) then have a very large image(few MB) saved as appCache resource. Then to see if few MB usage is displayed.

> 3. I agree with you this is by design. But we just need to clarify things
> designed correctly.

Comment 8

[Shawn Huang [:shawnjohnjr][:shuang] (as a happy gecko contributor)]

(In reply to Fischer [:Fischer] from comment #7)
> (In reply to Shawn Huang [:shawnjohnjr] from comment #5)
> > (In reply to Fischer [:Fischer] from comment #3)
> > > (In reply to Shawn Huang [:shawnjohnjr] from comment #1)
> > > > I think this is intentional. In #perferences Site Data, it also combines
> > > > other storage data which is not managed by QuotaManager. Fischer, can you
> > > > clarify this behavior first? 
> Another way I could think of verifying the quota storage(ex. indexedDB) and
> appCache together is to store very small amount of indexedDB(like few KB
> only) then have a very large image(few MB) saved as appCache resource. Then
> to see if few MB usage is displayed.

Okay, then I think appCache might not play a big role here. It might be interesting to explain why on Windows 10 the data size stored in Firefox has the same size as in profile folder. We should keep in mind, that right now we should calculate appCache + storage folder size.

Comment 9

[Jan Varga [:janv]]

Here's my 2 cents:
The goal never has been to match profile/storage size on disk with size shown in FF UI.
Not all files count towards usage/quota. There are some metadata files, etc.
And not all quota clients will use physical size of sqlite databases. For example LocalStorage will likely just take key.length + value.length
Also, file systems can report different usages across operating systems and machines, there are too many factors affecting that.

If someone insists on showing physical usage (besides current "logical" size), we could provide that, but I don't see a reason for that.

Comment 10

[Shawn Huang [:shawnjohnjr][:shuang] (as a happy gecko contributor)]

Close this bug because different platforms might have different size as Comment 9 mentioned. I don't see any reason we should follow up further. Feel free to re-open.