Closed Bug 1557289 Opened 5 years ago Closed 4 years ago

Introduce a new minor storage upgrade on QM/IDB

Categories

(Core :: Storage: Quota Manager, task, P1)

task

Tracking

()

RESOLVED DUPLICATE of bug 1591123

People

(Reporter: tt, Assigned: tt)

References

(Blocks 3 open bugs)

Details

The plan for this upgrade is to:

  • Remove empty origin directories
  • Remove directory metadata files on the origin directories (Note: not .metadata-v2)
  • Wipe the origin if the idb inside contains wasm
  • Wipe the origin if fileId inside the IDB sqlite database file mismatches the physical files.

In today's meeting, I mentioned I would like to test if opening databases during an upgrade (initialization) takes too much time, I would like to take a fallback plan. The plan is only to upgrade the idb in the permanent repository and let idb in the default (& temporary) repository be evited when users storage is about to full.

Jan and Andrew suggested doing that on database maintenance rather than on initialization so that it won't have a performance impact. However, that might not guarantee the release of removing that completely. So, checking wasm during accessing the database might be needed (I probably lost something on my note because it still can not guarantee the release of completing the removing).

Andrew also suggested remove/block the database while a wasm entry is found. I guess I need to check if it's safe to remove a database while using because I'm not sure if a website uses more than one database and has a data storing in separate databases or two databases have a relationship with each other. If we remove an origin instead, then that wouldn't have this issue. However, it might raise other issues, like a service worker is using in that origin.

We probably need to discuss this on Allhands.

If we end up with not traversing idb databases during an upgrade, we need to remove idb/origin somewhere else [1] if their file_ids on the databases mismatches the filenames of structureClone files.

Because my next thing is to take care of bug 1519859, 1522188.

[1] Probably, remove the database/origin while that problem is found.

The telemetry shows there are not a lot of Nightly users having this issue, though. We probably have to add assertion to catch the reason why causing this issue.
https://telemetry.mozilla.org/new-pipeline/dist.html#!cumulative=0&end_date=2019-05-31&include_spill=0&keys=__none__!__none__!__none__&max_channel_version=nightly%252F69&measure=SCALARS_IDB.FAILURE.FILEINFO_ERROR&min_channel_version=nightly%252F66&processType=*&product=Firefox&sanitize=0&sort_by_value=0&sort_keys=submissions&start_date=2019-05-20&table=0&trim=1&use_submission_date=0

To elaborate more: My current plan is to at least check IDB databases for persistent/persisted origins during the upgrade and let the rest of them be checked while they are creating a storage connection (which means the next accesses) by a schema upgrade in IDB. (If the databases won't be accessed, they would be evicted eventually)

Let's discuss that next week and I will update the decision to here.

(In reply to Tom Tung [:tt, :ttung] from comment #0)

The plan for this upgrade is to:

  • Remove empty origin directories
  • Remove directory metadata files on the origin directories (Note: not .metadata-v2)

Might only remain these two for the upgrade

See Also: → 1581067
See Also: 1581067
Status: ASSIGNED → RESOLVED
Closed: 4 years ago
Resolution: --- → DUPLICATE
You need to log in before you can comment on or make changes to this bug.