1376444 - Full disk warnings are not displayed after clearing data from "Preferences"

Status: VERIFIED FIXED

(Core :: Storage: Quota Manager, defect, P1)

Description

[Ioana Crisan]

Attachment: index.html

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:56.0) Gecko/20100101 Firefox/56.0
Build ID: 20170627093404

Steps to reproduce:
1. Run the "python -m SimpleHTTPServer" command in terminal to create a http server
2. Add the attached HTML file in the folder where the server is running
3. Open the following url: http://localhost:8000
4. Click the "Persist me" button in order to persist the site
5. Click the "Start adding data" button to start storing data via indexeddb
6. Continue to store data until the disk runs out of space(e.g: 96%)

Expected results:
2 notifications should be displayed, one when the firefox-used space < 5GB and one when the firefox-used space >= 5GB

Actual results:
No notification is displayed.

Comment 1

[Shawn Huang [:shawnjohnjr][:shuang] (as a happy gecko contributor)]

(In reply to Ioana Crisan from comment #0)
> Created attachment 8881384 [details]
> index.html
> 
> User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:56.0) Gecko/20100101
> Firefox/56.0
> Build ID: 20170627093404
> 
> Steps to reproduce:
> 1. Run the "python -m SimpleHTTPServer" command in terminal to create a http
> server
> 2. Add the attached HTML file in the folder where the server is running
> 3. Open the following url: http://localhost:8000
If this is just http, I wonder that persist() can work properly, you might not be able to get navigator.storage.
Can you confirm this step?

> 4. Click the "Persist me" button in order to persist the site
> 5. Click the "Start adding data" button to start storing data via indexeddb
> 6. Continue to store data until the disk runs out of space(e.g: 96%)
> 
> Expected results:
> 2 notifications should be displayed, one when the firefox-used space < 5GB
> and one when the firefox-used space >= 5GB
> 
> Actual results:
> No notification is displayed.

Comment 2

[Ioana Crisan]

Attachment: firstTry.png

Comment 3

[Ioana Crisan]

Attachment: secondTry.png

Comment 4

[Ioana Crisan]

(In reply to Shawn Huang [:shawnjohnjr] from comment #1)

> > Steps to reproduce:
> > 1. Run the "python -m SimpleHTTPServer" command in terminal to create a http
> > server
> > 2. Add the attached HTML file in the folder where the server is running
> > 3. Open the following url: http://localhost:8000
> If this is just http, I wonder that persist() can work properly, you might
> not be able to get navigator.storage.
> Can you confirm this step?


It is possible to access the navigator.storage.persist() with http but only with localhost. You can try this or directly the bmoattachment linked to this issue.
After further investigation I noticed that I can get one of the warnings(when the firefox-used space >= 5GB) but only with a clean profile. Even though I cleared the stored site data from "Preferences" this only works once per profile. Please see the attached screenshots for more details(I run it directly from bmoattachment so it's https secured).

Comment 5

[Shawn Huang [:shawnjohnjr][:shuang] (as a happy gecko contributor)]

(In reply to Ioana Crisan from comment #4)
> (In reply to Shawn Huang [:shawnjohnjr] from comment #1)
> 
> > > Steps to reproduce:
> > > 1. Run the "python -m SimpleHTTPServer" command in terminal to create a http
> > > server
> > > 2. Add the attached HTML file in the folder where the server is running
> > > 3. Open the following url: http://localhost:8000
> > If this is just http, I wonder that persist() can work properly, you might
> > not be able to get navigator.storage.
> > Can you confirm this step?
> 
> 
> It is possible to access the navigator.storage.persist() with http but only
> with localhost. You can try this or directly the bmoattachment linked to
> this issue.
> After further investigation I noticed that I can get one of the
> warnings(when the firefox-used space >= 5GB) but only with a clean profile.
> Even though I cleared the stored site data from "Preferences" this only
> works once per profile. Please see the attached screenshots for more
> details(I run it directly from bmoattachment so it's https secured).

I see. I remembered we do have a timer to not disturb users to show notification often (i guess 20 mins). Fischer, can you confirm that?

Comment 6

[Fischer [:Fischer]]

(In reply to Shawn Huang [:shawnjohnjr] from comment #5)
> (In reply to Ioana Crisan from comment #4)
> > (In reply to Shawn Huang [:shawnjohnjr] from comment #1)
> > 
> > > > Steps to reproduce:
> > > > 1. Run the "python -m SimpleHTTPServer" command in terminal to create a http
> > > > server
> > > > 2. Add the attached HTML file in the folder where the server is running
> > > > 3. Open the following url: http://localhost:8000
> > > If this is just http, I wonder that persist() can work properly, you might
> > > not be able to get navigator.storage.
> > > Can you confirm this step?
> > 
> > 
> > It is possible to access the navigator.storage.persist() with http but only
> > with localhost. You can try this or directly the bmoattachment linked to
> > this issue.
> > After further investigation I noticed that I can get one of the
> > warnings(when the firefox-used space >= 5GB) but only with a clean profile.
> > Even though I cleared the stored site data from "Preferences" this only
> > works once per profile. Please see the attached screenshots for more
> > details(I run it directly from bmoattachment so it's https secured).
> 
> I see. I remembered we do have a timer to not disturb users to show
> notification often (i guess 20 mins). Fischer, can you confirm that?
Yes, in order to not bother users too much, the 2nd warning notification would be allowed after 20 mins of the 1st notification.
There is a pref controlling this time: "browser.storageManager.pressureNotification.minIntervalMS"
So it could be set to 1000 which will lower the warning notification interval to 1 sec.

Comment 7

[Ioana Crisan]

I am not able to receive the 2nd warning even though I expect more than 20 minutes from the 1st warning before breaching the global limit again.

Comment 8

[Fischer [:Fischer]]

(In reply to Ioana Crisan from comment #7)
> I am not able to receive the 2nd warning even though I expect more than 20 minutes from the 1st warning before breaching the global limit again.
Please try to store something again. That should trigger the space-checking mechanism again then trigger the warning notification. Thank you.

Comment 9

[Ioana Crisan]

(In reply to Fischer [:Fischer] from comment #8)
> (In reply to Ioana Crisan from comment #7)
> > I am not able to receive the 2nd warning even though I expect more than 20 minutes from the 1st warning before breaching the global limit again.
> Please try to store something again. That should trigger the space-checking
> mechanism again then trigger the warning notification. Thank you.

Yes it indeed re-triggers the notification the second time I try to store something else, using the pref you provided (setting it to 1000ms) and not clearing the data from preferences.

What I was trying to describe is a behavior involving clearing data and then trying to store it again.
So, not only the notification is not triggered, but it also permits the persisted site add data over the global limit, it may even take 100% of the available usage(system) on disk. Please see the linked video where the issue is reproduced: 
https://drive.google.com/open?id=0B_L6mNdlzO5TQndtT0pkcnBIclk

Note: I'm using the BuildID: 20170701181359

Also I'm not receiving The Full Disk warning notification described by this slide from UX specs:
https://mozilla.invisionapp.com/share/4Y87EJO39#/screens/218412145

Comment 10

[Shawn Huang [:shawnjohnjr][:shuang] (as a happy gecko contributor)]

Thank you so much for reporting this bug. I will investigate this week.

Comment 11

[Shawn Huang [:shawnjohnjr][:shuang] (as a happy gecko contributor)]

This issue seems to be related to a unknown issue.

Comment 12

[Shawn Huang [:shawnjohnjr][:shuang] (as a happy gecko contributor)]

Per discussion with Tom, this seems to be related to QuotaManager known issue.

Comment 13

[Shawn Huang [:shawnjohnjr][:shuang] (as a happy gecko contributor)]

(In reply to Shawn Huang [:shawnjohnjr] from comment #11)
> This issue seems to be related to a unknown issue.

s:/unknown/known

Comment 14

[Shawn Huang [:shawnjohnjr][:shuang] (as a happy gecko contributor)]

Attachment: Tuning storage limit so we can easily see storage pressure notification

Comment 15

[Shawn Huang [:shawnjohnjr][:shuang] (as a happy gecko contributor)]

Bug 1372116 Patch1 and Patch2 fix will be landed first but without landing test at the same time. Please retest again after fix landed in mozilla-central (should be tomorrow).

Comment 16

[Shawn Huang [:shawnjohnjr][:shuang] (as a happy gecko contributor)]

Set ni? for Comment 15.

Comment 17

[Ioana Crisan]

After bug #1372116 was fixed the persisted sites don't allow you to store more than 50% from the available disk space, but I still don't receive the warning when the Firefox reaches 90% from the available space.

Please confirm if I'm doing something wrong or this is indeed a bug.
 - My available disk space is 74G so the Firefox can use up to 37G to store data.
 - (90/100 *37G) = 33.3G, so I expect to receive the first warning when the the available disk space is 74G - 33.3G = 40.7G(~=41G)
 - second warning I expect to get when the available space is 37G

Comment 18

[Tom Tung [:tt, :ttung]]

(In reply to Ioana Crisan from comment #17)
Hi Ioana,

>  - (90/100 *37G) = 33.3G, so I expect to receive the first warning when the
> the available disk space is 74G - 33.3G = 40.7G(~=41G)

Could you check the usage used by Firefox through storage.estimate() or preference rather than observing the rest of overall usage? Besides, could you confirm the following things? Thanks!

- Firefox should warn users when it stores data more than 90% (33.3 G) from its space (50% from the available disk space, 37G in your case).
- Firefox checks its space only when the first write/store happen. Once it checks how much space it can use, it won't check the available space again until it is closed and opened in the next time. 

Note: the warning should happen while Firefox store data make its usage greater than 90% form its space. The warning won't happen when you write things on your available space though other applications.

Comment 19

[Ioana Crisan]

(In reply to Tom Tung [:tt] from comment #18)

Hi Tom,

> Could you check the usage used by Firefox through storage.estimate() or
> preference rather than observing the rest of overall usage? Besides, could
> you confirm the following things? Thanks!
I checked the usage from Preferences(via storage.estimate() I couldn't since the web page is persisted). 

> - Firefox should warn users when it stores data more than 90% (33.3 G) from
> its space (50% from the available disk space, 37G in your case).
Even though I increased the usage only with Firefox I received the warning only when the usage reached 100% from the allocated disk space for Firefox(in my case 37G)

> - Firefox checks its space only when the first write/store happen. Once it
> checks how much space it can use, it won't check the available space again
> until it is closed and opened in the next time.
From my tests this seems to be like you mentioned. 

> Note: the warning should happen while Firefox store data make its usage
> greater than 90% form its space. The warning won't happen when you write
> things on your available space though other applications.

Comment 20

[Shawn Huang [:shawnjohnjr][:shuang] (as a happy gecko contributor)]

I will dig a bit more. Thanks.

Comment 21

[Shawn Huang [:shawnjohnjr][:shuang] (as a happy gecko contributor)]

(In reply to Ioana Crisan from comment #19)
> (In reply to Tom Tung [:tt] from comment #18)
> 
> Hi Tom,
> 
> > Could you check the usage used by Firefox through storage.estimate() or
> > preference rather than observing the rest of overall usage? Besides, could
> > you confirm the following things? Thanks!
> I checked the usage from Preferences(via storage.estimate() I couldn't since
> the web page is persisted). 
> 
> > - Firefox should warn users when it stores data more than 90% (33.3 G) from
> > its space (50% from the available disk space, 37G in your case).
> Even though I increased the usage only with Firefox I received the warning
> only when the usage reached 100% from the allocated disk space for
> Firefox(in my case 37G)
I thing to be added.
As I investigated in Bug 1384492 Comment 6. If you tested using pref "browser.storageManager.pressureNotification.testStorageLimitKB" which indirectly set "dom.quotaManager.temporaryStorage.fixedLimit" in gecko, that leads a fixed value instead of using 50% available space. So pressure event only fired once, when the current usage space is full.

Comment 22

[Shawn Huang [:shawnjohnjr][:shuang] (as a happy gecko contributor)]

(In reply to Ioana Crisan from comment #19)
> (In reply to Tom Tung [:tt] from comment #18)
> 
> Hi Tom,
> 
> > Could you check the usage used by Firefox through storage.estimate() or
> > preference rather than observing the rest of overall usage? Besides, could
> > you confirm the following things? Thanks!
> I checked the usage from Preferences(via storage.estimate() I couldn't since
> the web page is persisted). 
> 
> > - Firefox should warn users when it stores data more than 90% (33.3 G) from
> > its space (50% from the available disk space, 37G in your case).
> Even though I increased the usage only with Firefox I received the warning
> only when the usage reached 100% from the allocated disk space for
> Firefox(in my case 37G)
> 

One thing to be added.
As I investigated in Bug 1384492 Comment 6. If you tested using pref "browser.storageManager.pressureNotification.testStorageLimitKB" which indirectly set "dom.quotaManager.temporaryStorage.fixedLimit" in gecko, that leads a fixed value instead of using 50% available space. So pressure event only fired once, when the current usage space is full.

Comment 23

[Tom Tung [:tt, :ttung]]

(In reply to Tom Tung [:tt] from comment #18)
> (In reply to Ioana Crisan from comment #17)
> - Firefox should warn users when it stores data more than 90% (33.3 G) from
Just discuss this with Shawn, we find out I somehow mix up the UX spec v1 and v1.5/v2. 

We should warn the user only when the disk is full in v1. It means Firefox cannot store any data by evicting the old temporary storage. In this case, it's 37G, since there is only a persistent storage. 

Sorry for the misunderstanding. Warning when usage > 90% is on UX spec v1.5/v2.

Besides, the message should suggest user cleaning their storage since their usage is greater than 5G.

Comment 24

[Tom Tung [:tt, :ttung]]

(In reply to Ioana Crisan from comment #19)
> Even though I increased the usage only with Firefox I received the warning
> only when the usage reached 100% from the allocated disk space for
> Firefox(in my case 37G)

So, I think it's reasonable as I said in comment 23.

Comment 25

[Shawn Huang [:shawnjohnjr][:shuang] (as a happy gecko contributor)]

(In reply to Ioana Crisan from comment #0)
> Created attachment 8881384 [details]
> index.html
> 
> User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:56.0) Gecko/20100101
> Firefox/56.0
> Build ID: 20170627093404
> Expected results:
> 2 notifications should be displayed, one when the firefox-used space < 5GB
> and one when the firefox-used space >= 5GB
If firefox-used space < 5GB, then warn user to free some disk space.
This is because this usage is small and not the main cause for space issue.
In order to avoid the bad and wrong impression among users that
firefox consumes disk space a lot, indicate users to clean up other disk space.


If firefox-used space >= 5GB, then guide users to about:preferences
to clear some data stored on firefox by websites.

The notification is triggered when the disk is "full" and the difference is that firefox occupied usage.

So I think the expectation is not correct. But can you confirm it again, Fischer, since I'm not the one implemented this.

Comment 26

[Fischer [:Fischer]]

(In reply to Shawn Huang [:shawnjohnjr] from comment #25)
> (In reply to Ioana Crisan from comment #0)
> > Created attachment 8881384 [details]
> > index.html
> > 
> > User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:56.0) Gecko/20100101
> > Firefox/56.0
> > Build ID: 20170627093404
> > Expected results:
> > 2 notifications should be displayed, one when the firefox-used space < 5GB
> > and one when the firefox-used space >= 5GB
> If firefox-used space < 5GB, then warn user to free some disk space.
> This is because this usage is small and not the main cause for space issue.
> In order to avoid the bad and wrong impression among users that
> firefox consumes disk space a lot, indicate users to clean up other disk
> space.
> 
> 
> If firefox-used space >= 5GB, then guide users to about:preferences
> to clear some data stored on firefox by websites.
> 
Yes, this is why 2 types of warnings, < 5G and >= 5GB. 

> The notification is triggered when the disk is "full" and the difference is
> that firefox occupied usage.
> 
> So I think the expectation is not correct. But can you confirm it again, Fischer, since I'm not the one implemented this.
Also by default, we don't want to bother user too much so within 20 mins only prompt once. If the 1st warning type was the type of < 5G, we would only see 1 warning for this type.

Comment 27

[Shawn Huang [:shawnjohnjr][:shuang] (as a happy gecko contributor)]

In summary, I think it's normal to see one notification. If everyone agree the behavior is correct, I will let the reporter decide whether this bug should be closed.

Comment 28

[Ioana Crisan]

(In reply to Shawn Huang [:shawnjohnjr] from comment #21)
> (In reply to Ioana Crisan from comment #19)
> > (In reply to Tom Tung [:tt] from comment #18)
> > 
> > Hi Tom,
> > 
> > > Could you check the usage used by Firefox through storage.estimate() or
> > > preference rather than observing the rest of overall usage? Besides, could
> > > you confirm the following things? Thanks!
> > I checked the usage from Preferences(via storage.estimate() I couldn't since
> > the web page is persisted). 
> > 
> > > - Firefox should warn users when it stores data more than 90% (33.3 G) from
> > > its space (50% from the available disk space, 37G in your case).
> > Even though I increased the usage only with Firefox I received the warning
> > only when the usage reached 100% from the allocated disk space for
> > Firefox(in my case 37G)
> I thing to be added.
> As I investigated in Bug 1384492 Comment 6. If you tested using pref
> "browser.storageManager.pressureNotification.testStorageLimitKB" which
> indirectly set "dom.quotaManager.temporaryStorage.fixedLimit" in gecko, that
> leads a fixed value instead of using 50% available space. So pressure event
> only fired once, when the current usage space is full.

Hi, sorry for the late response but I was on PTO.
I did not test using pref "browser.storageManager.pressureNotification.testStorageLimitKB". The pref I used is "browser.storageManager.pressureNotification.minIntervalMS" to lower the warning notification interval to 1 sec by setting it to 1000.

Comment 29

[Shawn Huang [:shawnjohnjr][:shuang] (as a happy gecko contributor)]

(In reply to Ioana Crisan from comment #28)
> (In reply to Shawn Huang [:shawnjohnjr] from comment #21)
> > (In reply to Ioana Crisan from comment #19)
> > > (In reply to Tom Tung [:tt] from comment #18)
> > > 
> > > Hi Tom,
> > > 
> > > > Could you check the usage used by Firefox through storage.estimate() or
> > > > preference rather than observing the rest of overall usage? Besides, could
> > > > you confirm the following things? Thanks!
> > > I checked the usage from Preferences(via storage.estimate() I couldn't since
> > > the web page is persisted). 
> > > 
> > > > - Firefox should warn users when it stores data more than 90% (33.3 G) from
> > > > its space (50% from the available disk space, 37G in your case).
> > > Even though I increased the usage only with Firefox I received the warning
> > > only when the usage reached 100% from the allocated disk space for
> > > Firefox(in my case 37G)
> > I thing to be added.
> > As I investigated in Bug 1384492 Comment 6. If you tested using pref
> > "browser.storageManager.pressureNotification.testStorageLimitKB" which
> > indirectly set "dom.quotaManager.temporaryStorage.fixedLimit" in gecko, that
> > leads a fixed value instead of using 50% available space. So pressure event
> > only fired once, when the current usage space is full.
> 
> Hi, sorry for the late response but I was on PTO.
> I did not test using pref
> "browser.storageManager.pressureNotification.testStorageLimitKB". The pref I
> used is "browser.storageManager.pressureNotification.minIntervalMS" to lower
> the warning notification interval to 1 sec by setting it to 1000.

I think even you did not set "browser.storageManager.pressureNotification.testStorageLimitKB", the notifcation only fires once.

Comment 30

[Ioana Crisan]

Hi Francis,

Could you please confirm what Tom said in comment #23? I'm confused because the v1 UX spec displays TBD for both notifications. Please see the link(https://mozilla.invisionapp.com/share/4Y87EJO39#/screens/179635747_1-0_Cover). Thank you!

Comment 31

[Tom Tung [:tt, :ttung]]

To be clear, the difference between [1] and [2] is that the content of notifications should be different between the current usage > 5G or <= 5G, when the full disk warning is presenting. Thus, only one notification shows up should be reasonable.

[1] https://mozilla.invisionapp.com/share/4Y87EJO39#/screens/213203147
[2] https://mozilla.invisionapp.com/share/4Y87EJO39#/screens/218412145

Comment 32

[Francis Lee [:frlee]]

Hi Ioana,

Tom is right, in Storage V1, we only deal with V1 spec. some further improvement will be handled in V1.5 or V2.

thank you

Comment 33

[Mark Liang(:mark_liang)]

(In reply to Ioana Crisan from comment #30)
> Hi Francis,
> 
> Could you please confirm what Tom said in comment #23? I'm confused because
> the v1 UX spec displays TBD for both notifications. Please see the
> link(https://mozilla.invisionapp.com/share/4Y87EJO39#/screens/179635747_1-
> 0_Cover). Thank you!

Sorry for the confusion, the "TBD" was for early stage UX explorations, I just remove them and updated the spec.

Thanks.

Comment 34

[Ioana Crisan]

Thank you all for the clarifications!

Comment 35

[Paul Silaghi, QA [:pauly]]

57.0a1 (2017-08-25)
The full disk warning is now properly displayed after clearing the site data in about:preferences#privacy. 
The problem is I'm seeing different notification's content across platforms:
Ubuntu 14.04: http://imgur.com/a/N0hdT
Win 10, OS X 10.12: http://imgur.com/a/JZxUe
Thoughts?

Comment 36

[Shawn Huang [:shawnjohnjr][:shuang] (as a happy gecko contributor)]

(In reply to Paul Silaghi, QA [:pauly] from comment #35)
> 57.0a1 (2017-08-25)
> The full disk warning is now properly displayed after clearing the site data
> in about:preferences#privacy. 
> The problem is I'm seeing different notification's content across platforms:
> Ubuntu 14.04: http://imgur.com/a/N0hdT
> Win 10, OS X 10.12: http://imgur.com/a/JZxUe
> Thoughts?

Because usage is less than 5GB and an another is larger than 5GB, as Comment 25, 31 mentioned.

Comment 37

[Paul Silaghi, QA [:pauly]]

Thanks, Shawn.
Verified fixed based on comment 36.