Clear site data doesn't remove the quota storage for different ports
Categories
(Toolkit :: Data Sanitization, defect, P2)
Tracking
()
Tracking | Status | |
---|---|---|
firefox78 | --- | verified |
People
(Reporter: tt, Assigned: tt)
References
(Blocks 1 open bug)
Details
(Whiteboard: [possible dupe of bug 1629658])
Attachments
(4 files)
See: https://bugzilla.mozilla.org/show_bug.cgi?id=1558478#c10
The storage from localhost is not removed after clearing cookies and site data
I could reproduce it, but not sure if I got the same thing:
STR:
- Open a python simple HTTP server
- Run a script with calling
storage.persist()
through localhost - Allow the permission
- Go to "about:preference" and clear the cookies and site data for the localhost
- The storage of the localhost is still there
My understanding of why this happens:
- QM creates
http+++localhost+8000
for the storage - When I click the "Manage Data" the port is stripped so it's
http://localhost
3.. ClearDataService asks QM to removehttp://localhost
rather thanhttp+++localhost+8000
So I guess it's because inconsistent of treating port between QM and ClearDataService. I'm not sure if it's an issue on QM or ClearDataService.
Question:
- Should QM removes the storage for a host disregarding the different ports or should ClearDataService differentiate the hosts between ports.
Assignee | ||
Comment 1•5 years ago
|
||
(In reply to Tom Tung [:tt, :ttung] from comment #0)
Question:
- Should QM remove the storage for a host disregarding the different ports or should ClearDataService differentiate the hosts between ports.
Comment 2•5 years ago
|
||
AFAIK, QMS provides a list of origins via listOrigins
or getUsage
and that should include the port. Later when a specific origin is cleared it must be done by calling QMS::clearStorageForPrincipal
which should again include the port.
What do you see in the panel when there is data for http://localhost and also for http://localhost:8000 ?
Just one item or two ?
Assignee | ||
Comment 3•5 years ago
|
||
(In reply to Jan Varga [:janv] from comment #2)
AFAIK, QMS provides a list of origins via
listOrigins
orgetUsage
and that should include the port. Later when a specific origin is cleared it must be done by callingQMS::clearStorageForPrincipal
which should again include the port.What do you see in the panel when there is data for http://localhost and also for http://localhost:8000 ?
Just one item or two ?
I saw only "localhost" on the panel and "http+++localhost+8000" on the disk.
I will check the code where we get the origins and show them on the panel.
Just for providing more info: the deletion was here
Comment 4•5 years ago
•
|
||
I meant that you would try to create/store data for both origins and the check what SiteManager does with that. If it merges the two or keep them separate.
Comment 5•5 years ago
•
|
||
(In reply to Tom Tung [:tt, :ttung] from comment #3)
Just for providing more info: the deletion was here
That sounds like a problem - deleteByHost
. The information about the port is lost. QMS provides an origin list (with ports), but it doesn't get the port back when asked for clearing an origin.
Assignee | ||
Comment 6•5 years ago
|
||
The issue seems to be caused by here. (The port gets stripped)
(In reply to Jan Varga [:janv] from comment #4)
I meant that you would try to create/store data for both origins and the check what SiteManager does with that. If it merges the two or keep them separate.
Will try that
Assignee | ||
Comment 7•5 years ago
|
||
(In reply to Tom Tung [:tt, :ttung] from comment #6)
(In reply to Jan Varga [:janv] from comment #4)
I meant that you would try to create/store data for both origins and the check what SiteManager does with that. If it merges the two or keep them separate.
Will try that
It merged together
Assignee | ||
Comment 8•5 years ago
|
||
(In reply to Tom Tung [:tt, :ttung] from comment #6)
The issue seems to be caused by here. (The port gets stripped)
Actually, this line causes the issue.
Assignee | ||
Comment 9•5 years ago
|
||
The issue is:
According to nsIURI, the host and port are different components.
While we want to clear usage for a host (says foo.com), QM clear the exact storage for the target host but doesn't clear the same host with a different port (says foo.com:8080).
To fix this, either we need to differentiate the storage with port (we are using just host now) or QM will need to clear storage for a host in different port.
Comment 10•5 years ago
|
||
(In reply to Tom Tung [:tt, :ttung] from comment #9)
or QM will need to clear storage for a host in different port.
I would like to avoid this if possible, all the directory locking around clearing is already quite complicated.
Ideally, QMS would only provide low level primitives for getting origins and clearing them.
Instead, _getOrInsertSite
in SiteDataManager.jsm can have an array of origins for each site (group of origins) which can later be used for clearing exact origins, but I didn't check all other possible dependencies.
Comment 11•5 years ago
|
||
I can imagine UI that would provide a tree view instead of the list view, so if there is more than one origin for given host, the user would be able to click on a site and unroll all origins for given site that way.
Assignee | ||
Comment 12•5 years ago
|
||
(In reply to Jan Varga [:janv] from comment #11)
I can imagine UI that would provide a tree view instead of the list view, so if there is more than one origin for given host, the user would be able to click on a site and unroll all origins for given site that way.
Changing UI might need UX to involve, but I guess I can make site
object carries the origins under that site. So that we propagate the list of origins under a host to clearDataService while clearing data in this scenario.
Comment 13•5 years ago
|
||
The UI change is not necessary right now (of course).
Assignee | ||
Comment 14•5 years ago
|
||
Comment 15•5 years ago
|
||
Sorry, before we continue fixing this I'd like to understand what the desired behavior is for clearing data from http://localhost:8000
. As you mentioned, in our official definition the host for that origin is "localhost", so I think our two options are:
- Fix the ClearDataService implementation of the QuotaCleaner to correctly clean all of "localhost" (I think this is actually what bug 1629658 is about, see my comment there)
- Decide that we only want to clear by origin in this case and update our UI and clearing code to always consider localhost by origin instead of host (so that you would have different entries for localhost:3000, localhost:8000, etc.)
I feel like 2) might provide a better user/developer experience but 1) is probably the fix we want in the short term.
Tom, what do you think? Should we approach this via my suggestion in bug 1629658?
Assignee | ||
Comment 16•5 years ago
|
||
(In reply to Johann Hofmann [:johannh] from comment #15)
Sorry, before we continue fixing this I'd like to understand what the desired behavior is for clearing data from
http://localhost:8000
.
Actually, I'm also curious about what should be the desired behavior from your perspective.
I think is that the storage of http://localhost:8000
should belong to http://localhost
given http://localhost:8000
wasn't show on the panel.
However, if you think that they are different storage, then the fix can also be showing both http://localhost
and http://localhost:8000
on the panel.
While I'm reproducing bug 1629658, I found both senglehardt.com
and test.senglehardt.com
were showed on the panel. So, from my viewpoint, (I might be wrong) if you only select the storage for senglehardt.com
to clear, then only the storage of senglehardt.com
should be removed.
As you mentioned, in our official definition the host for that origin is "localhost", so I think our two options are:
- Fix the ClearDataService implementation of the QuotaCleaner to correctly clean all of "localhost" (I think this is actually what bug 1629658 is about, see my comment there)
- Decide that we only want to clear by origin in this case and update our UI and clearing code to always consider localhost by origin instead of host (so that you would have different entries for localhost:3000, localhost:8000, etc.)
I feel like 2) might provide a better user/developer experience but 1) is probably the fix we want in the short term.
Tom, what do you think? Should we approach this via my suggestion in bug 1629658?
There is another solution and could be done in the short-term (just like this).
I describe the idea below:
Since the SiteDataManager
caches the data for the quota storage from QuotaManager
while collecting the storage in different modules in a set. We can query the sites for the selected hosts and then call clearSiteDataFromPrincipal
for each principal for those sites instead of calling clearSiteDataFromHost
for the hosts.
However, that's based on that the storage of test.senglehardt.com
should be removed when only senglehardt.com
was selected from the panel and both URLs were shown on the panel.
Does this solution sound good to you?
Comment 17•5 years ago
•
|
||
However, if you think that they are different storage, then the fix can also be showing both http://localhost and http://localhost:8000 on the panel.
To be honest I'm not 100% sure what to think here, but I would lean towards saying that in the case of localhost we should treat each port separately. I was mostly suggesting that the quickest way to temporarily fix it would be to clear all quota data for subdomains in ClearDataService.
About your separate approach:
The main thing that I want is that we have a clear decision in ClearDataService. We should either have clearSiteDataFromHost always clear data from sub-domains or never clear it. I would lean towards never, and then adjust calling sites that want to clear sub-domains to manually gather those ahead of time.
However that might not be something we need to do for this bug.
For this bug I'd like to have a clear behavior in ClearDataService for all quota data, not singling out localStorage. So I feel like the solution that I suggested (removing the "ls" parameter from https://searchfox.org/mozilla-central/rev/a4d62e09a4c46aef918667fa759bf9ae898dc258/toolkit/components/cleardata/ClearDataService.jsm#561 is a better approach.
Then we could keep that behavior but modify it to not do the baseDomain check and instead check for the same host (and maybe adding a separate case for localhost where we check hostPort). Probably in a separate bug.
Does that make sense to you? :)
Assignee | ||
Comment 18•5 years ago
|
||
(In reply to Johann Hofmann [:johannh] from comment #17)
To be honest I'm not 100% sure what to think here, but I would lean towards saying that in the case of localhost we should treat each port separately. I was mostly suggesting that the quickest way to temporarily fix it would be to clear all quota data for subdomains in ClearDataService.
I see where you coming from. Thanks for explaining that!
The temporary fix you proposed should fix the problem as well. I will submit a patch.
The main reason why I proposed another approach is that the whole flow is like:
SiteDataManager
asksQuotaManager
to provide a list of origin and usageSiteDataManager
transform the origins into principals and only show the hosts on the panel- A user selects some of them to be removed
SiteDataManager
askClearSiteData
to removed the selectedhost
sClearSiteData
askQuotaManager
to provide a list of origin again (because of thelocalstorage
)
QuotaManager
would traverse ${profile}/storage/*
at least 3 times (2 for listing, 1 for clearing) in this flow.
And I wanted to reduce the traverse into 2 times by using the cached data in SiteDataManager
so I proposed another quick fix. However, as you mentioned, if it's just a temporary fix, and it's cleaner (align the behaviors) for ClearDataService
, then I would just go for your approach.
Does that make sense to you? :)
Yes, that makes sense to me.
The only thing remaining is that the behavior might be different when the pref for lsng is on and off. I will consider if we want to align the behaviors no matter if the pref for lsng is on or off in this change.
Assignee | ||
Comment 19•5 years ago
|
||
Assignee | ||
Comment 20•5 years ago
|
||
Depends on D71715
Assignee | ||
Comment 21•5 years ago
|
||
Assignee | ||
Comment 22•5 years ago
|
||
Comment 23•5 years ago
|
||
Comment 24•5 years ago
|
||
Comment 25•5 years ago
|
||
Backed out for geckoview failures on org.mozilla.geckoview.test.StorageControllerTest.clearDataFromHost
backout: https://hg.mozilla.org/integration/autoland/rev/2caa5151bb02f264ced92871f7b17bfdc47591e6
failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=299821524&repo=autoland&lineNumber=11301
[task 2020-04-28T16:47:12.051Z] 16:47:12 INFO - TEST-START | org.mozilla.geckoview.test.StorageControllerTest.clearDataFromHost
[task 2020-04-28T16:47:42.208Z] 16:47:42 INFO - org.mozilla.geckoview.test | INSTRUMENTATION_STATUS: numtests=714
[task 2020-04-28T16:47:42.208Z] 16:47:42 INFO - org.mozilla.geckoview.test | INSTRUMENTATION_STATUS: stream=
[task 2020-04-28T16:47:42.208Z] 16:47:42 INFO - org.mozilla.geckoview.test | Error in clearDataFromHost(org.mozilla.geckoview.test.StorageControllerTest):
[task 2020-04-28T16:47:42.208Z] 16:47:42 INFO - org.mozilla.geckoview.test | org.mozilla.geckoview.test.util.UiThreadUtils$TimeoutException: Timed out after 30000ms
[task 2020-04-28T16:47:42.208Z] 16:47:42 INFO - org.mozilla.geckoview.test | at org.mozilla.geckoview.test.util.UiThreadUtils$TimeoutRunnable.run(UiThreadUtils.java:52)
[task 2020-04-28T16:47:42.208Z] 16:47:42 INFO - org.mozilla.geckoview.test | at android.os.Handler.handleCallback(Handler.java:751)
[task 2020-04-28T16:47:42.208Z] 16:47:42 INFO - org.mozilla.geckoview.test | at android.os.Handler.dispatchMessage(Handler.java:95)
[task 2020-04-28T16:47:42.209Z] 16:47:42 INFO - org.mozilla.geckoview.test | at org.mozilla.geckoview.test.util.UiThreadUtils.waitForCondition(UiThreadUtils.java:155)
[task 2020-04-28T16:47:42.209Z] 16:47:42 INFO - org.mozilla.geckoview.test | at org.mozilla.geckoview.test.util.UiThreadUtils.waitForResult(UiThreadUtils.java:80)
[task 2020-04-28T16:47:42.211Z] 16:47:42 INFO - org.mozilla.geckoview.test | at org.mozilla.geckoview.test.rule.GeckoSessionTestRule.waitForResult(GeckoSessionTestRule.java:2252)
[task 2020-04-28T16:47:42.211Z] 16:47:42 INFO - org.mozilla.geckoview.test | at org.mozilla.geckoview.test.StorageControllerTest.clearDataFromHost(StorageControllerTest.kt:208)
[task 2020-04-28T16:47:42.211Z] 16:47:42 INFO - org.mozilla.geckoview.test | at java.lang.reflect.Method.invoke(Native Method)
[task 2020-04-28T16:47:42.211Z] 16:47:42 INFO - org.mozilla.geckoview.test | at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
[task 2020-04-28T16:47:42.211Z] 16:47:42 INFO - org.mozilla.geckoview.test | at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
[task 2020-04-28T16:47:42.211Z] 16:47:42 INFO - org.mozilla.geckoview.test | at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
[task 2020-04-28T16:47:42.211Z] 16:47:42 INFO - org.mozilla.geckoview.test | at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
[task 2020-04-28T16:47:42.211Z] 16:47:42 INFO - org.mozilla.geckoview.test | at org.mozilla.geckoview.test.rule.GeckoSessionTestRule$2.lambda$evaluate$0$GeckoSessionTestRule$2(GeckoSessionTestRule.java:1284)
[task 2020-04-28T16:47:42.211Z] 16:47:42 INFO - org.mozilla.geckoview.test | at org.mozilla.geckoview.test.rule.-$$Lambda$GeckoSessionTestRule$2$sIbRNaZJgAu-QrUVWSGD8JbPSWM.run(lambda)
[task 2020-04-28T16:47:42.211Z] 16:47:42 INFO - org.mozilla.geckoview.test | at android.app.Instrumentation$SyncRunnable.run(Instrumentation.java:1950)
[task 2020-04-28T16:47:42.211Z] 16:47:42 INFO - org.mozilla.geckoview.test | at android.os.Handler.handleCallback(Handler.java:751)
[task 2020-04-28T16:47:42.211Z] 16:47:42 INFO - org.mozilla.geckoview.test | at android.os.Handler.dispatchMessage(Handler.java:95)
[task 2020-04-28T16:47:42.211Z] 16:47:42 INFO - org.mozilla.geckoview.test | at android.os.Looper.loop(Looper.java:154)
[task 2020-04-28T16:47:42.211Z] 16:47:42 INFO - org.mozilla.geckoview.test | at android.app.ActivityThread.main(ActivityThread.java:6077)
[task 2020-04-28T16:47:42.211Z] 16:47:42 INFO - org.mozilla.geckoview.test | at java.lang.reflect.Method.invoke(Native Method)
[task 2020-04-28T16:47:42.211Z] 16:47:42 INFO - org.mozilla.geckoview.test | at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:866)
[task 2020-04-28T16:47:42.211Z] 16:47:42 INFO - org.mozilla.geckoview.test | at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:756)
[task 2020-04-28T16:47:42.211Z] 16:47:42 INFO - org.mozilla.geckoview.test |
[task 2020-04-28T16:47:42.211Z] 16:47:42 INFO - org.mozilla.geckoview.test | INSTRUMENTATION_STATUS: id=AndroidJUnitRunner
[task 2020-04-28T16:47:42.211Z] 16:47:42 INFO - org.mozilla.geckoview.test | INSTRUMENTATION_STATUS: test=clearDataFromHost
[task 2020-04-28T16:47:42.211Z] 16:47:42 INFO - org.mozilla.geckoview.test | INSTRUMENTATION_STATUS: class=org.mozilla.geckoview.test.StorageControllerTest
[task 2020-04-28T16:47:42.211Z] 16:47:42 INFO - org.mozilla.geckoview.test | INSTRUMENTATION_STATUS: stack=org.mozilla.geckoview.test.util.UiThreadUtils$TimeoutException: Timed out after 30000ms
[task 2020-04-28T16:47:42.211Z] 16:47:42 INFO - org.mozilla.geckoview.test | at org.mozilla.geckoview.test.util.UiThreadUtils$TimeoutRunnable.run(UiThreadUtils.java:52)
[task 2020-04-28T16:47:42.211Z] 16:47:42 INFO - org.mozilla.geckoview.test | at android.os.Handler.handleCallback(Handler.java:751)
[task 2020-04-28T16:47:42.212Z] 16:47:42 INFO - org.mozilla.geckoview.test | at android.os.Handler.dispatchMessage(Handler.java:95)
[task 2020-04-28T16:47:42.212Z] 16:47:42 INFO - org.mozilla.geckoview.test | at org.mozilla.geckoview.test.util.UiThreadUtils.waitForCondition(UiThreadUtils.java:155)
[task 2020-04-28T16:47:42.212Z] 16:47:42 INFO - org.mozilla.geckoview.test | at org.mozilla.geckoview.test.util.UiThreadUtils.waitForResult(UiThreadUtils.java:80)
[task 2020-04-28T16:47:42.212Z] 16:47:42 INFO - org.mozilla.geckoview.test | at org.mozilla.geckoview.test.rule.GeckoSessionTestRule.waitForResult(GeckoSessionTestRule.java:2252)
[task 2020-04-28T16:47:42.212Z] 16:47:42 INFO - org.mozilla.geckoview.test | at org.mozilla.geckoview.test.StorageControllerTest.clearDataFromHost(StorageControllerTest.kt:208)
[task 2020-04-28T16:47:42.212Z] 16:47:42 INFO - org.mozilla.geckoview.test | at java.lang.reflect.Method.invoke(Native Method)
[task 2020-04-28T16:47:42.212Z] 16:47:42 INFO - org.mozilla.geckoview.test | at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
[task 2020-04-28T16:47:42.213Z] 16:47:42 INFO - org.mozilla.geckoview.test | at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
[task 2020-04-28T16:47:42.213Z] 16:47:42 INFO - org.mozilla.geckoview.test | at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
[task 2020-04-28T16:47:42.213Z] 16:47:42 INFO - org.mozilla.geckoview.test | at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
[task 2020-04-28T16:47:42.213Z] 16:47:42 INFO - org.mozilla.geckoview.test | at org.mozilla.geckoview.test.rule.GeckoSessionTestRule$2.lambda$evaluate$0$GeckoSessionTestRule$2(GeckoSessionTestRule.java:1284)
[task 2020-04-28T16:47:42.213Z] 16:47:42 INFO - org.mozilla.geckoview.test | at org.mozilla.geckoview.test.rule.-$$Lambda$GeckoSessionTestRule$2$sIbRNaZJgAu-QrUVWSGD8JbPSWM.run(lambda)
[task 2020-04-28T16:47:42.213Z] 16:47:42 INFO - org.mozilla.geckoview.test | at android.app.Instrumentation$SyncRunnable.run(Instrumentation.java:1950)
[task 2020-04-28T16:47:42.214Z] 16:47:42 INFO - org.mozilla.geckoview.test | at android.os.Handler.handleCallback(Handler.java:751)
[task 2020-04-28T16:47:42.214Z] 16:47:42 INFO - org.mozilla.geckoview.test | at android.os.Handler.dispatchMessage(Handler.java:95)
[task 2020-04-28T16:47:42.214Z] 16:47:42 INFO - org.mozilla.geckoview.test | at android.os.Looper.loop(Looper.java:154)
[task 2020-04-28T16:47:42.214Z] 16:47:42 INFO - org.mozilla.geckoview.test | at android.app.ActivityThread.main(ActivityThread.java:6077)
[task 2020-04-28T16:47:42.214Z] 16:47:42 INFO - org.mozilla.geckoview.test | at java.lang.reflect.Method.invoke(Native Method)
[task 2020-04-28T16:47:42.214Z] 16:47:42 INFO - org.mozilla.geckoview.test | at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:866)
[task 2020-04-28T16:47:42.214Z] 16:47:42 INFO - org.mozilla.geckoview.test | at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:756)
[task 2020-04-28T16:47:42.214Z] 16:47:42 INFO - org.mozilla.geckoview.test |
[task 2020-04-28T16:47:42.214Z] 16:47:42 INFO - org.mozilla.geckoview.test | INSTRUMENTATION_STATUS: current=565
[task 2020-04-28T16:47:42.215Z] 16:47:42 INFO - org.mozilla.geckoview.test | INSTRUMENTATION_STATUS_CODE: -2
[task 2020-04-28T16:47:42.215Z] 16:47:42 WARNING - TEST-UNEXPECTED-FAIL | org.mozilla.geckoview.test.StorageControllerTest.clearDataFromHost | org.mozilla.geckoview.test.util.UiThreadUtils$TimeoutException: Timed out after 30000ms
Updated•5 years ago
|
Assignee | ||
Comment 26•5 years ago
|
||
Sorry for the backout, I didn't check the tests in Android. The test failed at here and D71716 could be related to.
Assignee | ||
Comment 27•5 years ago
|
||
It looks like StorageControllerTest.sessionContext
is somewhat related.
Here is try for disabling sessionContext: https://treeherder.mozilla.org/#/jobs?repo=try&revision=bfa25682c3dc76858a17bdf6e575d844d6cf6aa7
It shows that clearDataFromHost didn't timeout if sessionContext is disabled.
Assignee | ||
Comment 28•5 years ago
|
||
We do clearStoragsForPrincipal based on the result from listOrigin, so it's fine
to not clear all stroages that match the prefix here. Also, since the passing
principals can contain origin attributes which is not allowed to be used with
aClearAll.
Assignee | ||
Updated•5 years ago
|
Comment 29•5 years ago
|
||
Comment 30•5 years ago
|
||
bugherder |
https://hg.mozilla.org/mozilla-central/rev/b2c0bab59ddc
https://hg.mozilla.org/mozilla-central/rev/379d30335465
https://hg.mozilla.org/mozilla-central/rev/7f275c977854
https://hg.mozilla.org/mozilla-central/rev/bb7b37cfeea0
Updated•5 years ago
|
Comment 32•5 years ago
|
||
Reproduced the initial issue using an old nightly build from 2020-04-13, verified that the issue is fixed now using latest Firefox Beta build 78.0b8 across platforms (Windows 10 64bit, macOS 10.15.6 beta2 and Ubuntu 18.04 64bit).
Description
•