1348705 - Netflix error 32660 (.metadata file blocks)

Status: NEW

(Core :: Storage: Quota Manager, defect, P3)

Description

[kolAflash]

I tried to play a movie on https://netflix.com but I got the Netflix error 32660.

OS: Windows 10, 64 Bit
Firefox: 52, 64 Bit

I tracked the problem down to the file:
FIREFOX-PROFILE/storage/persistent/chrome/.metadata
It's 0 KB in size.
Workaround/manual fix: If I delete the ".metadata" file and restart Firefox, Netflix works.

After deleting the ".metadata" file and playing something on Netflix the following new folder appears:
FIREFOX-PROFILE/storage/default/https+++www.netflix.com/
I guess the ".metadata" file maybe somehow blocked the creation of this folder!?!?


Note:
This is not about MP4/h264 or DRM. Both is completely disabled and I had to make any changes to it to get Netflix running.

Comment 1

[Anthony Jones (:ajones, :kentuckyfriedtakahe, :k17e)]

Andrew - does storage/persistent/chrome/.metadata belong to you?

Comment 2

[kolAflash]

(In reply to Anthony Jones (:kentuckyfriedtakahe, :k17e) from comment #1)
> Andrew - does storage/persistent/chrome/.metadata belong to you?

I didn't create any files manually in the Firefox profile folder. (if that's what you mean!?)

So I guess Firefox created that file somehow.

Comment 3

[Andrew Overholt [:overholt]]

Jan thinks we have different code for handling corrupted .metadata in storage/chrome vs. storage/default (for content). He's going to take a look.

Comment 4

[Jan Varga [:janv]]

This is now tracked under meta bug 1482662.

Comment 5

[Jan Varga [:janv]]

Tom, if I remember correctly, the issue is that we don't do metadata recovery for origin directories in storage/permanent directory.

Comment 6

[Tom Tung [:tt, :ttung]]

I guess I might need to do some experiment and more investigation to understand the real issue behind.

Based on the description in comment #0, it seems like the directory metadata file in the permanent repository couldn't be read and it's strange because the size of it was 0 kb. Besides, the reporter mentioned that the workaround is to remove the file, so it means QM was able to recover itself if that strange file didn't exist.

Jan advised me that it might be related to metadata recovery for directories in permanent folders.

However, by reading the code:
The main initialization logic is in EnsureOriginIsInitializedInternal(), it basically does:
1. EnsureStorageIsInitialized
2. EnsureTemporaryStorageIsInitialized
3. EnsureOriginDirectory and InitializeOrigin

1 is related to upgrade storage version. For all the upgrades, they restore the directory metadata files if they couldn't be read.
2 touches the directory metadata, but it only does that for default and temporary repositories. So it's not related to this issue.
3 creates a directory metadata file if the directory of given origin hasn't been created. Also, it grabs information from the persistent origin directory metadata if the directory of given origin was created. If the persistent origin directory metadata fails to be read, then it would be restored in this case.

For now, I guess this issue is more like somehow FF couldn't restore the metadata file in the permanent repository (for instance, the metadata file is hold by other applications or os system), but it seems to be unlikely to happen.

So, I should confirm the code and the speculation I mentioned above by doing experiments.

Anyway, there are no obsolete files involved, so the fix here seems not be a part of the upcoming minor upgrade.

Comment 7

[Tom Tung [:tt, :ttung]]

- It seems that the issue happened the schema version reporter's profile was version 0_0 so that comment #0 mentioned "persistent/chrome" rather "permanent/chrome".
- I suspect that "persistent/chrome/.metadata" somehow blocked the storage upgrades in EnsureStorageIsInitialized() so that it couldn't be upgraded and the initialization failed because upgradeStroage0_0To1_0 doesn't recover the metadata file for persistent storage.
- Will confirm the second item later.

Comment 8

[Tom Tung [:tt, :ttung]]

(In reply to Tom Tung [:tt] from comment #7)
> - I suspect that "persistent/chrome/.metadata" somehow blocked the storage
> upgrades in EnsureStorageIsInitialized() so that it couldn't be upgraded and
> the initialization failed because upgradeStroage0_0To1_0 doesn't recover the
> metadata file for persistent storage.

After looking into the code, I don't think an unreadable metadata can block upgrades even if it's under permanent/persistent repository. Will have a test to verify that tomorrow. Now, I lean to reckon the issue for restoring metadata was fixed in another bug.

The test might verify:
1. Initializing (initOrigin()) a normal origin with default type and with a profile that has a broken metadata file under the permanent folder.
2. Initializing (initOrigin()) an origin that storing under the permanent folder and its metadata file was broken.
3. Initializing (init()) an old profile (version 0_0) with a broken metadata in the "persistent" folder.

I guess all of these three subtests should succeed.

Comment 9

[Tom Tung [:tt, :ttung]]

Attachment: A test for verifying chrom metadata file doesn't break the storage initialization (upgrade, origin initializations)

Comment 10

[Tom Tung [:tt, :ttung]]

IIUC, I need to check whether an abnormal metadata file living in an origin directory and in the persistent repository break the storage initialization.

The storage initialization touches the persistent repository's origin directory metadata files only when it's in an upgrade or origin initialization. However, it seems that current QuotaManager does metadata file recoveries even for persistent type origin directories.

I wrote a test (attachment 9022873 [details] [diff] [review]) to verify the behavior of initialization when there is an empty metadata file in the permanent repository.

This test verifies three conditions when there is an empty metadata file in a permanent repository:
- The upgrades of storage to the newest version don't fail
- The initialization of another normal (default) origin doesn't fail
- The initialization of that persistent origin doesn't fail

If anything wasn't neglected, then that probably means the original issue has been resolved. I tested it locally and all the tests passed.

Jan, could you take a look and help me to check if there are some cases that I might miss?

Comment 11

[Firefox Bug Husbandry Bot]

See bug 1547409. Moving webcompat whiteboard tags to keywords.

Comment 12

[Jan Varga [:janv]]

The test looks good. However, if want to be 100% sure that empty .metadata/.metadata-v2 doesn't break things, we need to test it in all upgrade functions. Currently, the metadata is probably recovered during execution of UpgradeStorageFrom0_0To1_0 method. Other upgrade methods don't try to recover it, so the metadata recovery is untested there.
There's also the hacky downgrade stuff that can cause problems in theory.

Comment 13

[Tom Tung [:tt, :ttung]]

We need to update the test based on the comment 12