1621916 - Ignore unknown files in client directories during temporary storage initialization

Status: RESOLVED FIXED

(Core :: Storage: Quota Manager, task, P1)

Description

[Jan Varga [:janv]]

This mostly involves IndexedDB client directory, but we should verify that other quota clients don't have this problem either.

Comment 1

[Jan Varga [:janv]]

Attachment: Bug 1621916 - Simplify test_unknownFiles.js by merging stage 1 and stage 2; r=#dom-workers-and-storage-reviewers

We plan to replace this test with more thorough tests in bug 1620277, but for
now it's still needed and will be part of a short term fix for ignoriging
unknown files and directories.

Comment 2

[Jan Varga [:janv]]

Attachment: Bug 1621916 - Remove redundant checks that were needed before we started ignoring unknown files and directories; r=#dom-workers-and-storage-reviewers

Depends on D75948

Comment 3

[Jan Varga [:janv]]

Attachment: Bug 1621916 - Convert test_unknownFiles.js to use async functions; r=#dom-workers-and-storage-reviewers

Depends on D75949

Comment 4

[Jan Varga [:janv]]

Attachment: Bug 1621916 - Convert test_unknownFiles.js to use quasi subtests; r=#dom-workers-and-storage-reviewers

Depends on D75950

Comment 5

[Jan Varga [:janv]]

Attachment: Bug 1621916 - Enhance test_unknownFiles.js to cover initTemporaryStorage method; r=#dom-workers-and-storage-reviewers

Depends on D75951

Comment 6

[Jan Varga [:janv]]

Attachment: Bug 1621916 - Update test_unknownFiles.js with unknown files for cache, sdb and ls quota client; r=#dom-workers-and-storage-reviewers

Comment 7

[Jan Varga [:janv]]

Attachment: Bug 1621916 - Extract profile creation from test_unknownFiles.js into make_unknownFiles.js and make it more sophisticated; r=#dom-workers-and-storage-reviewers

A generated profile now matches real world profiles:

• contains directory metadata
• contains real quota client specific files
• contains unique non-empty unknown files (instead of empty foo.bar files)
• contains unique non-empty unknown directories (instead of empty foo dirs)

There's now only one origin directory which contains unknown files and
directories. All methods that take a principal are verified to work with the
origin direcrtory.

Depends on D76095

Comment 8

[Jan Varga [:janv]]

Attachment: Bug 1621916 - Fix SimpleDB to ignore unknown files during initialization and usage calculation; r=#dom-workers-and-storage-reviewers

SimpleDB files on disk now use a suffix, so unknown files can be filtered out.

Depends on D76700

Comment 9

[Jan Varga [:janv]]

Attachment: Bug 1621916 - Run test_unknownFiles.js in multiple modes; r=#dom-workers-and-storage-reviewers

The functionality needs to be verified for all initialization states (modes)
since some methods don't automatically trigger temporary storage initialization
which results in calling different quota client methods (for example
GetUsageForOrigin calls either Client::InitOrigin or Client::GetUsageForOrigin).

Depends on D76701

Comment 10

[Jan Varga [:janv]]

Attachment: Bug 1621916 - Fix IndexedDB to ignore unknown files in the files and journals directory during initialization and usage calculation; r=#dom-workers-and-storage-reviewers

All unknown files in the files and journals directory are now properly filtered
out if the filename can't be converted to an integer.

Depends on D76702

Comment 11

[Jan Varga [:janv]]

Attachment: Bug 1621916 - Unify reporting of unknown files across quota manager, idb, sdb and ls quota client; r=#dom-workers-and-storage-reviewers

Depends on D76703

Comment 12

[Jan Varga [:janv]]

Attachment: Bug 1621916 - Add unknown files for Cache API morgue directory and its subdirectories; r=#dom-workers-and-storage-reviewers

Cache API is currently not consistent regarding unknown files. Some unknown
files are reported and ignored, other unknown files are reported and deleted or
just silently deleted without a warning.
Quota Manager, IndexedDB, SimpleDB and LocalStorage currently report and ignore
such files. We should try to unify this in future.

Depends on D76717

Comment 13

[Jan Varga [:janv]]

Attachment: Bug 1621916 - Rename GetBaseFilename to GetFilenameBase to match rest of the code; r=#dom-workers-and-storage-reviewers

Depends on D76718

Comment 14

[Jan Varga [:janv]]

Attachment: Bug 1621916 - Rename GetDatabaseFilename to GetDatabaseFilenameBase; r=#dom-workers-and-storage-reviewers

The method returns a database filename without any extension, so the method name
needs to express that.

Depends on D77068

Comment 15

[Jan Varga [:janv]]

Attachment: Bug 1621916 - Remove unnecessary local variables; r=#dom-workers-and-storage-reviewers

Depends on D77069

Comment 16

[Jan Varga [:janv]]

Attachment: Bug 1621916 - Remove unused argument in Client::InitOrigin; r=#dom-workers-and-storage-reviewers

Depends on D77070

Comment 17

[Jan Varga [:janv]]

Attachment: Bug 1621916 - Fix IndexedDB to ignore unknown files in the client directory during initialization and usage calculation; r=#dom-workers-and-storage-reviewers

Depends on D77072

Comment 18

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/509f8842782a
Simplify test_unknownFiles.js by merging stage 1 and stage 2; r=dom-workers-and-storage-reviewers,ttung
https://hg.mozilla.org/integration/autoland/rev/f3bc1c339aef
Remove redundant checks that were needed before we started ignoring unknown files and directories; r=dom-workers-and-storage-reviewers,sg

Comment 19

[Jan Varga [:janv]]

There will be some followup patches, but the core problem should be covered by the patches that have been already submitted.

Comment 20

[Razvan Maries]

https://hg.mozilla.org/mozilla-central/rev/509f8842782a
https://hg.mozilla.org/mozilla-central/rev/f3bc1c339aef

Comment 21

[Jan Varga [:janv]]

Attachment: Bug 1621916 - Provide a usable stack when requestFinished throws; r=#dom-workers-and-storage-reviewers

Depends on D77073

Comment 22

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b9737a318730
Convert test_unknownFiles.js to use async functions; r=dom-workers-and-storage-reviewers,sg
https://hg.mozilla.org/integration/autoland/rev/6a5e10dfd0d0
Convert test_unknownFiles.js to use quasi subtests; r=dom-workers-and-storage-reviewers,sg
https://hg.mozilla.org/integration/autoland/rev/3af8bec13325
Enhance test_unknownFiles.js to cover initTemporaryStorage method; r=dom-workers-and-storage-reviewers,sg
https://hg.mozilla.org/integration/autoland/rev/904cc259a98d
Update test_unknownFiles.js with unknown files for cache, sdb and ls quota client; r=dom-workers-and-storage-reviewers,ttung
https://hg.mozilla.org/integration/autoland/rev/3e78d35a061c
Extract profile creation from test_unknownFiles.js into make_unknownFiles.js and make it more sophisticated; r=dom-workers-and-storage-reviewers,ttung

Comment 23

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/cb80856ac40b
Fix SimpleDB to ignore unknown files during initialization and usage calculation; r=dom-workers-and-storage-reviewers,ttung

Comment 24

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b1a0dc2886ce
Run test_unknownFiles.js in multiple modes; r=dom-workers-and-storage-reviewers,ttung
https://hg.mozilla.org/integration/autoland/rev/5a575e559146
Fix IndexedDB to ignore unknown files in the files and journals directory during initialization and usage calculation; r=dom-workers-and-storage-reviewers,ttung
https://hg.mozilla.org/integration/autoland/rev/3dfe670a0bd2
Unify reporting of unknown files across quota manager, idb, sdb and ls quota client; r=dom-workers-and-storage-reviewers,ttung
https://hg.mozilla.org/integration/autoland/rev/3bee45f35de2
Add unknown files for Cache API morgue directory and its subdirectories; r=dom-workers-and-storage-reviewers,ttung
https://hg.mozilla.org/integration/autoland/rev/a235cbd69f6f
Rename GetBaseFilename to GetFilenameBase to match rest of the code; r=dom-workers-and-storage-reviewers,ttung
https://hg.mozilla.org/integration/autoland/rev/29086c8c3e4e
Rename GetDatabaseFilename to GetDatabaseFilenameBase; r=dom-workers-and-storage-reviewers,sg
https://hg.mozilla.org/integration/autoland/rev/24d3d987b60d
Remove unnecessary local variables; r=dom-workers-and-storage-reviewers,sg
https://hg.mozilla.org/integration/autoland/rev/c3f73781a871
Remove unused argument in Client::InitOrigin; r=dom-workers-and-storage-reviewers,sg

Comment 25

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d62da1a75701
Fix IndexedDB to ignore unknown files in the client directory during initialization and usage calculation; r=dom-workers-and-storage-reviewers,sg,ttung

Comment 26

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/mozilla-central/rev/b9737a318730
https://hg.mozilla.org/mozilla-central/rev/6a5e10dfd0d0
https://hg.mozilla.org/mozilla-central/rev/3af8bec13325
https://hg.mozilla.org/mozilla-central/rev/904cc259a98d
https://hg.mozilla.org/mozilla-central/rev/3e78d35a061c
https://hg.mozilla.org/mozilla-central/rev/cb80856ac40b
https://hg.mozilla.org/mozilla-central/rev/b1a0dc2886ce
https://hg.mozilla.org/mozilla-central/rev/5a575e559146
https://hg.mozilla.org/mozilla-central/rev/3dfe670a0bd2
https://hg.mozilla.org/mozilla-central/rev/3bee45f35de2
https://hg.mozilla.org/mozilla-central/rev/a235cbd69f6f
https://hg.mozilla.org/mozilla-central/rev/29086c8c3e4e
https://hg.mozilla.org/mozilla-central/rev/24d3d987b60d
https://hg.mozilla.org/mozilla-central/rev/c3f73781a871

Comment 27

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/d62da1a75701

Comment 28

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3f5baab34b1f
Provide a usable stack when requestFinished throws; r=dom-workers-and-storage-reviewers,sg,asuth

Comment 29

[Atila Butkovits]

Backed out for causing failure at QuotaRequests.cpp

Backout link: https://hg.mozilla.org/integration/autoland/rev/10ad7868f3ca27cb90db9bd1d392ff4d7852a0cd

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&selectedTaskRun=Duk9FBf6TyeUbiECq8qmfQ-0&resultStatus=testfailed%2Cbusted%2Cexception&revision=3f5baab34b1fb242a09bdef99007b5b95cf75a74

Log link: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=305898003&repo=autoland&lineNumber=26592

Comment 30

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/06ab04be7c08
Provide a usable stack when requestFinished throws; r=dom-workers-and-storage-reviewers,sg,asuth

Comment 31

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/06ab04be7c08