Closed Bug 1659053 Opened 4 years ago Closed 4 years ago

Updating from 78.0.2 to 79.0 corrupts add-ons, add-on settings

Categories

(Core :: Storage: Quota Manager, defect, P2)

Product:
Core
Component:
Storage: Quota Manager
Version:
79 Branch
Type:
defect

Tracking

()

Status:
VERIFIED FIXED
Milestone:
82 Branch
Tracking Flags:
Tracking Status
firefox81 --- verified
firefox82 --- verified

People

(Reporter: vaevkspoxwe, Assigned: tt)

References

(Blocks 1 open bug)

Details

Attachments

(2 files)

Bug 1659053 - Restore or remove the origin directory if its directory name is inconsistent with the origin in the directory metadta file;
47 bytes, text/x-phabricator-request
RyanVM
: approval-mozilla-beta+
Details | Review
Bug 1659053 - Have a test to verify the fix;
47 bytes, text/x-phabricator-request
RyanVM
: approval-mozilla-beta+
Details | Review

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:79.0) Gecko/20100101 Firefox/79.0

Steps to reproduce:

Updated Firefox 78.0.2 to 79.0. (Win10 x64)

I'm running Firefox in a virtualized environment, and I can reproduce the issue as often as needed.

I have made sure all add-ons are at their newest version prior to updating Firefox. I have experimented with disabling (or uninstalling) add-ons while on 78.0.2 and enabling (or reinstalling) them after the update to 79.0, but the results were the same.

Actual results:

Updating my Firefox installation from 78.0.2 to 79.0 corrupts the previously installed add-ons and their configuration. Affected add-ons include uBlock Origin and Cookie AutoDelete.

The issue breaks the user interface of the add-ons, for example in the following ways:

  • The uMatrix table is completely blank and clicking the buttons does not do anything
  • Cookie AutoDelete: clicking its icon will not even bring up the popup
  • uBlock Origin: some interface elements are missing entirely
  • Dark Reader: instead of the interface elements it just says "Loading, please wait"

The issue also affects add-on functionality. After updating to 79.0, I cannot even access any web sites until I disable uMatrix, which now prevents the loading of all sites (I just get the "Loading..." animation though, not the "blocked" message).

======================

This person seems to have experienced the same issue: https://blog.mozilla.org/addons/2020/07/09/changes-to-storage-sync-in-firefox-79/#comment-226984

I'm not using CCleaner or similar programs. I'm also not using a Firefox account/sync.

Expected results:

Add-ons should remain functional after updating Firefox, and if that is not possible for some reason, a warning should be displayed instead of leaving the browser in an unusable state.

Bugbug thinks this bug should belong to this component, but please revert this change in case of error.

Component: Untriaged → Add-ons Manager
Product: Firefox → Toolkit

I was not able to reproduce this issue on Win10x64 after updating from FF78. All addons previously installed (uMatrix, uBlock origin, Cookie Autodelete) worked after updating and restarting Firefox.

Please let me know if there are some specific steps needed in order to reproduce or if possible, attach a video of the issue in order to see if I am missing something when trying to reproduce.

Thanks.

This is not a general issue affecting every FF78 installation.

On one of my systems I can reliably reproduce the issue by performing an update to FF79. No further steps are required.

As this particular FF installation has been in use for years, the individual steps taken to get from a clean installation to where I am now are not traceable. The motivation behind this bug report is to offer help with figuring out what makes some installations prone to the issue.

Component: Add-ons Manager → Storage
Product: Toolkit → WebExtensions

is there anything that we changed between 78 and 79 at an IndexedDB or QuotaManager level that could trigger this kind of issue for some users?
(also what kind of details we may ask to the reporter to collect to investigate what may be going on, I see that QA cannot reproduce it and so it may be triggered by something that is only happening for some user profiles)

Flags: needinfo?(ttung)

(In reply to Luca Greco [:rpl] [:luca] [:lgreco] from comment #4)

is there anything that we changed between 78 and 79 at an IndexedDB or QuotaManager level that could trigger this kind of issue for some users?

We have many patches that were landed around June in IndexedDB and QuotaManager. It's hard to tell at the moment.

(also what kind of details we may ask to the reporter to collect to investigate what may be going on, I see that QA cannot reproduce it and so it may be triggered by something that is only happening for some user profiles)

It'd be useful to differentiate the issue from initialization failures or IndexedDB's/DOM Cache's issues as an initial step.
We often ask the user to visit https://firefox-storage-test.glitch.me/ and share the result with us. If all the storage APIs in the website are working functionally, then it's an issue in IndexedDB/DOM Cache. Also, it'd be nice to ask them to share the result in about:telemetry#search=QM_FIRST_INITIALIZATION_ATTEMPT right after visiting the website.

We are working on getting more information through telemetry and consoles on storage-related failures in QuotaManager. Meanwhile, we have noticed that there are corrupted indexedDB databases in some users' profiles and we are discussing how to handle corrupted SQLite databases properly in bug 1659464.

Flags: needinfo?(ttung)

(In reply to usanbe from comment #0)
Hi,

Thanks for reporting!

Could you:

That would help us to figure out if you have storage initialization failures or storage-related API failures. Thanks in advance!

Flags: needinfo?(vaevkspoxwe)

with 78.0.2 (= before updating)

https://firefox-storage-test.glitch.me/

Overview:
Storage is working. This is your first visit or all storage was automatically cleared.
Specific Subsystem Statuses:

LocalStorage
    Good: Totally Working. (fullyOperational)
QuotaManager
    Good: Totally Working. (fullyOperational)
IndexedDB
    Good: Totally Working. (fullyOperational)
Cache API
    Good: Totally Working. (fullyOperational)

Debug Info:

{
  "v": 1,
  "curVersion": 78,
  "prevVersion": 0,
  "ls": {},
  "qm": {
    "lastWorkedIn": 78
  },
  "idb": {
    "persistentCreatedIn": 78,
    "persistentLastOpenedIn": 78,
    "clearDetectedIn": 0
  },
  "cache": {
    "firstCacheCreatedIn": 78,
    "unpaddedOpaqueCreatedIn": 0,
    "paddedOpaqueCreatedIn": 78
  }
}

about:telemetry#search=QM_FIRST_INITIALIZATION_ATTEMPT

TemporaryOrigin
1 sample, average = 1, sum = 1

0 |  0  0%
1 |#########################  1  100%
2 |  0  0%

with 79 (= after updating)

https://firefox-storage-test.glitch.me/

Storage is broken. This is your first visit or all storage was automatically cleared.
Specific Subsystem Statuses:

LocalStorage
    Good: Totally Working. (fullyOperational)
QuotaManager
    Bad: Totally Broken. (fullyBroken)
IndexedDB
    Bad: Totally Broken. (fullyBroken)
Cache API
    Bad: Totally Broken. (fullyBroken)

Debug Info:

storage.estimate() threw: Internal error while estimating storage usage
Failed to create "persistent" IDB.
Failed to create "transient" IDB.

{
  "v": 1,
  "curVersion": 79,
  "prevVersion": 0,
  "ls": {},
  "qm": {
    "lastWorkedIn": 0
  },
  "idb": {
    "persistentCreatedIn": 0,
    "persistentLastOpenedIn": 0,
    "clearDetectedIn": 0
  },
  "cache": {
    "firstCacheCreatedIn": 0,
    "unpaddedOpaqueCreatedIn": 0,
    "paddedOpaqueCreatedIn": 0
  }
}

about:telemetry#search=QM_FIRST_INITIALIZATION_ATTEMPT


DefaultRepository
1 sample, average = 0, sum = 0

0 |#########################  1  100%
1 |  0  0%


TemporaryRepository
1 sample, average = 1, sum = 1

0 |  0  0%
1 |#########################  1  100%
2 |  0  0%


TemporaryStorage
1 sample, average = 0, sum = 0

0 |#########################  1  100%
1 |  0  0%


Storage
1 sample, average = 1, sum = 1

0 |  0  0%
1 |#########################  1  100%
2 |  0  0%



PersistentOrigin
1 sample, average = 1, sum = 1

0 |  0  0%
1 |#########################  1  100%
2 |  0  0%
Flags: needinfo?(vaevkspoxwe)

(In reply to usanbe from comment #7)
Thanks for the quick reply and for sharing this data with us! That accelerates the process a lot!

So, in your case, it's clear that there is an issue in temporary storage initialization after upgrading to 79 and it's related to default repository ({$profile}/storage/default/).

In this case, would you mind sharing your profile (files that under {$profile}/storage/default/ [1]) with us for debugging?

If you have some concerns with that, then would you mind running a debug build in treeherder [2] and sharing the log in the console after visiting https://firefox-storage-test.glitch.me/?

[1] You can file the file path for {$profile} in "about:profiles" page. It's totally understandable if you don't feel comfortable sharing that with us, but, in case of you are okay with that, please zip the files under {$profile}/storage/default/ and email that to me. We will figure out the issue and try to resolve that as soon as possible.

[2] The URL for treeherder is "https://treeherder.mozilla.org/#/jobs?repo=mozilla-release".
Since you are Windows x64, you can find a debug build for that in "https://treeherder.mozilla.org/#/jobs?repo=mozilla-release&selectedTaskRun=GuraQUORRrm-dwzP5uIesg.0".
Please check the "Artifacts" tab in the panel below, there is "target.zip".
Please download the zip file, unzip it and then run the firefox inside. You will find a console for the firefox while running it.

Assignee: nobody → ttung
Component: Storage → Storage: Quota Manager
Flags: needinfo?(vaevkspoxwe)
Product: WebExtensions → Core

Thank you for pointing me towards {$profile}/storage/default/. Although I cannot share the entire folder, I have been able to identify the exact (sub)directories that caused the issue, and I will share them with you by email (Subject: Bug 1659053) Should you still require treeherder output or any additional data/information, please let me know.

To give a quick summary of the directories' contents; one of them is named http+++example.org. Inside it are two files, .metadata and .metadata-v2, both of which contain http://example.org. This trailing dot seems to prevent Firefox from deleting the directory when the browser is being closed. The issue may be related to https://bugzilla.mozilla.org/show_bug.cgi?id=1645943

From a user's perspective, I was able to solve the problem as follows:

  • Close Firefox
  • Open {$profile}/storage/default/
  • Delete subdirectories starting with http+++ and https+++
  • Restart Firefox

I now have Firefox 79 running, and https://firefox-storage-test.glitch.me/ says:

Overview:

Storage is working. This is your first visit or all storage was automatically cleared.

Specific Subsystem Statuses:

LocalStorage
    Good: Totally Working. (fullyOperational)
QuotaManager
    Good: Totally Working. (fullyOperational)
IndexedDB
    Good: Totally Working. (fullyOperational)
Cache API
    Good: Totally Working. (fullyOperational)

Debug Info:

{
  "v": 1,
  "curVersion": 79,
  "prevVersion": 79,
  "ls": {},
  "qm": {
    "lastWorkedIn": 79
  },
  "idb": {
    "persistentCreatedIn": 79,
    "persistentLastOpenedIn": 79,
    "clearDetectedIn": 0
  },
  "cache": {
    "firstCacheCreatedIn": 0,
    "unpaddedOpaqueCreatedIn": 0,
    "paddedOpaqueCreatedIn": 79
  }
}
Flags: needinfo?(vaevkspoxwe)

(In reply to usanbe from comment #9)

Thank you for pointing me towards {$profile}/storage/default/. Although I cannot share the entire folder, I have been able to identify the exact (sub)directories that caused the issue, and I will share them with you by email (Subject: Bug 1659053) Should you still require treeherder output or any additional data/information, please let me know.

To give a quick summary of the directories' contents; one of them is named http+++example.org. Inside it are two files, .metadata and .metadata-v2, both of which contain http://example.org. This trailing dot seems to prevent Firefox from deleting the directory when the browser is being closed. The issue may be related to https://bugzilla.mozilla.org/show_bug.cgi?id=1645943

Thanks for the information and the email! Then the issue is much clear now!

It looks like an issue that the origin in metadata is inconsistent with its origin directory name. It should be fixed by bug 1645943 or its parent bug (because after that we should be able to create folders with trailing dot), but maybe it's a leftover that had been created before these bugs were resolved. I need to look into this deeper.

Would you mind checking the value of "dom.quotaManager.useDOSDevicePathSyntax" in "about:config" page?

Flags: needinfo?(vaevkspoxwe)

(In reply to Tom Tung [:tt, :ttung] from comment #11)

Would you mind checking the value of "dom.quotaManager.useDOSDevicePathSyntax" in "about:config" page?

The value of dom.quotaManager.useDOSDevicePathSyntax is true.

but maybe it's a leftover that had been created before these bugs were resolved.

The directories and files were created in 2017.

Flags: needinfo?(vaevkspoxwe)

(In reply to usanbe from comment #12)

(In reply to Tom Tung [:tt, :ttung] from comment #11)

Would you mind checking the value of "dom.quotaManager.useDOSDevicePathSyntax" in "about:config" page?

The value of dom.quotaManager.useDOSDevicePathSyntax is true.

but maybe it's a leftover that had been created before these bugs were resolved.

The directories and files were created in 2017.

Thanks for the information again! So, I think my guess is correct. That's something which was created before bug 1645943.

Ok, I can reproduce the issue by applying the origin directories from the reportor.
The failure log on my machine is:

[Parent 43009, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/tomtung/Work/mozilla-central/dom/cache/FileUtils.cpp, line 810
[Parent 43009, QuotaManager IO] WARNING: 'NS_FAILED(mozilla::dom::cache::LockedDirectoryPaddingGet( dir, &paddingSize))', file /Users/tomtung/Work/mozilla-central/dom/cache/QuotaClient.cpp, line 404
[Parent 43009, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/tomtung/Work/mozilla-central/dom/cache/QuotaClient.cpp, line 439
[Parent 43009, QuotaManager IO] WARNING: 'usageInfoOrErr.isErr()', file /Users/tomtung/Work/mozilla-central/dom/quota/ActorsParent.cpp, line 5293
[Parent 43009, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/tomtung/Work/mozilla-central/dom/quota/ActorsParent.cpp, line 5175
[Parent 43009, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/tomtung/Work/mozilla-central/dom/quota/ActorsParent.cpp, line 4557
[Parent 43009, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/tomtung/Work/mozilla-central/dom/quota/ActorsParent.cpp, line 7055
[Parent 43009, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/tomtung/Work/mozilla-central/dom/quota/ActorsParent.cpp, line 6855
[Parent 43009, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/tomtung/Work/mozilla-central/dom/quota/ActorsParent.cpp, line 6825
[Parent 43009, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/tomtung/Work/mozilla-central/dom/indexedDB/ActorsParent.cpp, line 21041
[Parent 43009, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/tomtung/Work/mozilla-central/dom/indexedDB/ActorsParent.cpp, line 20888

So, there are two factors to run into this issue:

  1. Visit a website with a trailing dot for its origin on Windows
  2. That website uses DOM Cache and the padding file is broken/empty

Maybe we should somehow update the incorrect origin string in the directory metadata file so that we wouldn't pass an incorrect origin string to quota client and cause the temporary storage initialization to fail.

I need to think about this more.

Severity: -- → S3
Priority: -- → P2

Depends on D88025

Should D88025 read "... inconsistent ..." ?

(In reply to Jens Stutte [:jstutte] (REO for FF 81) from comment #16)

Should D88025 read "... inconsistent ..." ?

Ah, right. I will correct that. Thanks for catching that!

Attachment #9171685 - Attachment description: Bug 1659053 - Remove the origin directory if its directory name is consistent with the origin in the directory metadta file; → Bug 1659053 - Remove the origin directory if its directory name is inconsistent with the origin in the directory metadta file;
Attachment #9171685 - Attachment description: Bug 1659053 - Remove the origin directory if its directory name is inconsistent with the origin in the directory metadta file; → Bug 1659053 - Restore or remove the origin directory if its directory name is inconsistent with the origin in the directory metadta file;
Blocks: 1661711
Pushed by ttung@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/428baa54b916
Restore or remove the origin directory if its directory name is inconsistent with the origin in the directory metadta file; r=dom-workers-and-storage-reviewers,janv
https://hg.mozilla.org/integration/autoland/rev/40d2481528c7
Have a test to verify the fix; r=dom-workers-and-storage-reviewers,janv

https://hg.mozilla.org/mozilla-central/rev/428baa54b916
https://hg.mozilla.org/mozilla-central/rev/40d2481528c7

Status: UNCONFIRMED → RESOLVED
Closed: 4 years ago
status-firefox82: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → 82 Branch

Comment on attachment 9171685 [details]
Bug 1659053 - Restore or remove the origin directory if its directory name is inconsistent with the origin in the directory metadta file;

Beta/Release Uplift Approval Request

  • User impact if declined: If their origin's name stored in the directory metadata file mismatches the origin directory name, they couldn't be able to initialize their temporary storage. That means they cannot use IndexedDB, DOM Cache, ServiceWorker.

They can have such problematic storage by visiting a site that has an origin with a trailing dot and not visiting the site that has a similar origin without a trailing dot in FF75- (75 or earlier).

  • Is this code covered by automated tests?: Yes
  • Has the fix been verified in Nightly?: Yes
  • Needs manual test from QE?: Yes
  • If yes, steps to reproduce: On Windows,

[1] you can find the file path for ${profile} in "about:profiles"

  • List of other uplifts needed: None
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky): This patch only add a support for handling a specific edge case and has a test. So, the risk is low.
  • String changes made/needed:
Attachment #9171685 - Flags: approval-mozilla-beta?
Attachment #9171686 - Flags: approval-mozilla-beta?
Flags: qe-verify+
QA Whiteboard: [qa-triaged]

Comment on attachment 9171685 [details]
Bug 1659053 - Restore or remove the origin directory if its directory name is inconsistent with the origin in the directory metadta file;

Approved for 81.0b5.

Attachment #9171685 - Flags: approval-mozilla-beta? → approval-mozilla-beta+
Attachment #9171686 - Flags: approval-mozilla-beta? → approval-mozilla-beta+

I tried verifying this issue with steps from Comment 21, on Windows 10 and observed the followings:
After visiting on FX75 build the "https://firefox-storage-test.glitch.me./" website, the "{profile}/storage/default/https+++firefox-storage-test.glitch.me/" folder is created (without a dot at the end). I have deleted this folder and continue by installing the latest Beta 81.0b5 and visit "https://firefox-storage-test.glitch.me/". I can confirm that all the outputs are green.
Tom, is this behavior correct?

@usanbe, considering that this is related to a particular firefox profile and I could not reproduce the initial issue related in the first comment, could you please confirm that the issue was fixed in your profile on Nightly 82 and Beta 81 builds?

Flags: needinfo?(ttung)
Flags: needinfo?(vaevkspoxwe)

(In reply to Brindusa Tot[:brindusat] from comment #24)

I tried verifying this issue with steps from Comment 21, on Windows 10 and observed the followings:
After visiting on FX75 build the "https://firefox-storage-test.glitch.me./" website, the "{profile}/storage/default/https+++firefox-storage-test.glitch.me/" folder is created (without a dot at the end). I have deleted this folder and continue by installing the latest Beta 81.0b5 and visit "https://firefox-storage-test.glitch.me/". I can confirm that all the outputs are green.
Tom, is this behavior correct?

@usanbe, considering that this is related to a particular firefox profile and I could not reproduce the initial issue related in the first comment, could you please confirm that the issue was fixed in your profile on Nightly 82 and Beta 81 builds?

Ah, the STR in comment #21 is incorrect. Sorry about that, and I list the correct one below:
If yes, steps to reproduce: On Windows,

Flags: needinfo?(ttung)

(In reply to Brindusa Tot[:brindusat] from comment #24)

@usanbe, considering that this is related to a particular firefox profile and I could not reproduce the initial issue related in the first comment, could you please confirm that the issue was fixed in your profile on Nightly 82 and Beta 81 builds?

I have just installed nightly 82.0a1 (2020-09-01) over the 78.0.2 installation my initial report was based on, and the previously reproducible issue did not occur.

As far as I can tell, the issue has been fixed.

Flags: needinfo?(vaevkspoxwe)

Verified again based on steps from comment 25 and I can confirm the fix on Beta 81.0b5 and latest Nightly build 82.0a1 (Build ID 20200904094341).

Status: RESOLVED → VERIFIED
QA Whiteboard: [qa-triaged]
status-firefox81: fixed → verified
status-firefox82: fixed → verified
Flags: qe-verify+
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: