Open Bug 1482662 Opened 3 years ago Updated 1 month ago

[meta] The War on Storage Initialization Failures

Categories

(Core :: Storage: Quota Manager, defect, P1)

defect

Tracking

()

ASSIGNED
Tracking Status
firefox63 --- affected

People

(Reporter: janv, Assigned: janv)

References

(Depends on 17 open bugs, Blocks 4 open bugs)

Details

(Keywords: meta)

Attachments

(1 file, 3 obsolete files)

No description provided.
Depends on: 1423917
Depends on: 1395102
Depends on: CVE-2018-12401
Depends on: 1348705
Depends on: 1436128
Depends on: 1436188
Depends on: 1425146
Depends on: 1345585
Depends on: 1395133
Depends on: 1482812
Priority: -- → P2
Depends on: 1481716
Keywords: meta
Summary: The War on Storage Initialization Failures → The War on Storage Initialization Failures [meta]
Duplicate of this bug: 1426119
Depends on: 944918
Blocks: 1487479
Depends on: 1487779
See Also: → 1471375
Depends on: 1493262
Depends on: 1491637
Status of this project this week:

[1] I plan to finish patches in bug 1423917 and send them to review this week and they should cover:
- A minor QuotaManager upgrade (mainly for cleaning obsolete origins)
  - Obsolete origins:
    - about+xxx+number (was found in bug 1422456)
    - Whitelist ".*", "desktop.ini", "Desktop.ini", and "Thumbs.db" (was reported in bug 1493262, bug 1423917, bug 944918)
    - Recognize chrome protocol
    - Ignore invalid origins during ungrades

[2] Since it's are hard to identify all the issues that affecting QM initializing, we should see the result after finishing bug 1436188 and its followup (bug 1487779).

I haven't carefully checked about bug 1436128, bug 1395133, bug 1348705, and bug 1267349, but I don't think they affects many users. Will take care of them after completing [1] and [2].
Add the bug for having a nicer error message when indexedDB finds an unexpected file in the indexedDB directory.
Depends on: 1502077
Status of this project this week:

1. Most patches in the bug 1423917 have sent to review. The rest of the patches are ready too, but they might be affected by changes in bug 1491637 and bug 1436188, so I haven't sent them yet.

2. Bug 1348705 seems to be fixed in another issue. Will try to verify that by a simple test.

So:

High priority issues or issues that having patches:
Bug 1436188 has patches and is waiting for feedback. Bug 1487779 is the follow-up bug for it.
Bug 1491637 has patches and is waiting for review and feedback.

Bug 944918, 1423917, 1493262 should be fixed in bug 1423917. However, since the range of them are a little vague to me, so that I will monitor whether they are fixed by checking the data in Bug 1436188.

Bug 1436128 should be taken care in the local storage next generation.

Others:
Bug 1267349 and 1502077 is related to having a proper error message so their priorities are low to me.
Bug 1395133 is to add a test so that its priority is low to me.
Depends on: 1504535
Depends on: 1519822
See Also: → 1517531
Depends on: 1322569
Summary: The War on Storage Initialization Failures [meta] → [meta] The War on Storage Initialization Failures
No longer depends on: 944918
See Also: → 1541370
No longer blocks: Session_managers
Depends on: 1557289

LSNG is going to be disabled in 70.0.1 :(
It looks like there are too many quota manager storage initialization failures in the wild and since LSNG depends on quota manager and given widespread use of LS API, sites become unusable for many users. We need to revive telemetry for storage initialization failures on Nightly and possibly uplift that to beta too. We need to reduce storage initialization failures the same way as we reduced quota manager shutdown hangs. Then we can finally ship LSNG.

Assignee: nobody → jvarga
Status: NEW → ASSIGNED
Priority: P2 → P1
Depends on: 1592204

I found a telemetry for Nightly 69:
https://telemetry.mozilla.org/new-pipeline/dist.html#!cumulative=0&end_date=2019-07-07&include_spill=0&keys=internal!external!__none__!__none__&max_channel_version=nightly%252F69&measure=QM_INIT_TELEMETRY_ERROR&min_channel_version=nightly%252F63&os=Windows_NT&processType=parent&product=Firefox&sanitize=1&sort_by_value=0&sort_keys=submissions&start_date=2019-05-21&table=0&trim=1&use_submission_date=0

It seems there were some other problems still (not just unknown files/directories). The situation on beta and release can be (slightly) different.

Anyway, I propose to add a new simple telemetry that reflects temporary storage initialization success rate. I guess that can be uplifted to all channels given simple patch and simplicity of the statistics.

Depends on: 1592934
Depends on: 1594075
Depends on: 1594080
Blocks: 1541370
See Also: 1541370
Depends on: 1593365
No longer depends on: 1395133
Blocks: 1395133
Depends on: 1595913
Depends on: 1598555
Blocks: 1599979
Depends on: 1600352

Some of the bugs tracked here need to be revised with respect to the decision to ignore unknown files and directories in current QM version and also with respect to QM v4, I'll start cleaning it up following days.

See Also: → IndexedDB-SM
Depends on: 1619325
Depends on: 1619592
Depends on: 1619891
Depends on: 1619893
Depends on: 1619895
Depends on: 1619896
Depends on: 1616003
Depends on: 1606318
Depends on: 1617842
Depends on: 1615552

The bug dependency was messy, so I cleaned it up a bit. In general, this is the main meta bug for addressing storage initialization failures. Bug 1593365 is a sub project that is happening in parallel to fixing issues in the current storage version.

There are also new bugs that cover individual keys of the QM_FIRST_INITIALIZATION_ATTEMPT telemetry probe. For example storage upgrade fixes should be tracked under bug 1619891.

I will try to clean up the bug dependency even more.

Depends on: 1536796
No longer depends on: 1536796
No longer depends on: 1594075
No longer depends on: 1617842, 1616003
No longer depends on: 1615552
No longer depends on: 1619325
No longer depends on: 1592204
No longer depends on: 1619592
Severity: normal → S3
Depends on: 1635351
Blocks: IndexedDB-SM
Depends on: 1645312
No longer depends on: 1645312
Depends on: 1685498
Attachment #9196363 - Attachment mime type: text/x-python → text/plain

For build id 20210109092457, the current failures according to the script are:

Initialization::TemporaryStorage

    2447 dom/cache/FileUtils.cpp:762                        NS_ERROR_FILE_NOT_FOUND # not propagated
         dom/cache/QuotaClient.cpp:374                      NS_ERROR_FILE_NOT_FOUND
         not propagated                                                         
    1132 dom/quota/ActorsParent.cpp:2711                    NS_ERROR_FILE_NOT_FOUND # Bug 1679541 (will add call sites)
         dom/quota/ActorsParent.cpp:5016                    NS_ERROR_FILE_NOT_FOUND
     638 dom/quota/ActorsParent.cpp:5342                    NS_ERROR_FAILURE    # Bug 1686041 (call to IsDirectory)
         dom/quota/ActorsParent.cpp:5424                    NS_ERROR_FAILURE    
         dom/quota/ActorsParent.cpp:5251                    NS_ERROR_FAILURE    
         dom/quota/ActorsParent.cpp:5255                    NS_ERROR_FAILURE    
     312 dom/quota/ActorsParent.cpp:4552                                        # Bug 1686697
         dom/quota/ActorsParent.cpp:4590                    NS_ERROR_FAILURE    
     301 dom/localstorage/ActorsParent.cpp:1189                                 # not propagated
         not propagated                                                         
     239 dom/quota/ActorsParent.cpp:4572                                        # Bug 1685098 (will add QM_TRY calls after this non-propagated failure)
         dom/quota/ActorsParent.cpp:4590                    NS_ERROR_FAILURE    
     224 dom/indexeddb/ActorsParent.cpp:5929                NS_ERROR_FILE_ACCESS_DENIED # Bug 1686060
         dom/indexeddb/ActorsParent.cpp:6098                NS_ERROR_FILE_ACCESS_DENIED
         dom/indexeddb/ActorsParent.cpp:13110               NS_ERROR_FILE_ACCESS_DENIED
         dom/indexeddb/ActorsParent.cpp:13137               NS_ERROR_UNEXPECTED 
         dom/indexeddb/ActorsParent.cpp:13036               NS_ERROR_UNEXPECTED 
         dom/quota/ActorsParent.cpp:5384                    NS_ERROR_UNEXPECTED 
         dom/quota/ActorsParent.cpp:5424                    NS_ERROR_UNEXPECTED 
         dom/quota/ActorsParent.cpp:5251                    NS_ERROR_UNEXPECTED 
         dom/quota/ActorsParent.cpp:5255                    NS_ERROR_UNEXPECTED 
     150 dom/quota/ActorsParent.cpp:1136                    NS_ERROR_FAILURE    # Bug 1686041 (call to IsDirectory, this is actually in dom/localstorage/ActorsParent.cpp)
         dom/localstorage/ActorsParent.cpp:8774             NS_ERROR_FAILURE    
         dom/quota/ActorsParent.cpp:5384                    NS_ERROR_FAILURE    
         dom/quota/ActorsParent.cpp:5424                    NS_ERROR_FAILURE    
         dom/quota/ActorsParent.cpp:5251                    NS_ERROR_FAILURE    
         dom/quota/ActorsParent.cpp:5255                    NS_ERROR_FAILURE    
     129 dom/quota/ActorsParent.cpp:2697                    NS_ERROR_FILE_IS_LOCKED # Bug 1685098 (will add intermediate call sites)
         dom/quota/ActorsParent.cpp:10556                   NS_ERROR_FILE_IS_LOCKED
         dom/quota/ActorsParent.cpp:5217                    NS_ERROR_FILE_IS_LOCKED
         dom/quota/ActorsParent.cpp:5255                    NS_ERROR_FILE_IS_LOCKED
     120 dom/indexeddb/ActorsParent.cpp:13122                                   # not propagated
         not propagated                                                         
     115 dom/quota/ActorsParent.cpp:5259                    NS_ERROR_ABORT      # shutdown
     102 dom/localstorage/ActorsParent.cpp:1174             NS_ERROR_FILE_NOT_FOUND # not propagated
         not propagated                                                         
      74 dom/quota/QuotaCommon.h:1172                       NS_ERROR_FAILURE    # Bug 1686041 (call to GetDirectoryEntries)
         dom/indexeddb/ActorsParent.cpp:13476               NS_ERROR_FAILURE    
         dom/indexeddb/ActorsParent.cpp:13077               NS_ERROR_FAILURE    
         dom/indexeddb/ActorsParent.cpp:13036               NS_ERROR_FAILURE    
         dom/quota/ActorsParent.cpp:5384                    NS_ERROR_FAILURE    
         dom/quota/ActorsParent.cpp:5424                    NS_ERROR_FAILURE   # not propagated immediately in Nightly! 
         dom/quota/ActorsParent.cpp:1136                    NS_ERROR_FAILURE    
         dom/localstorage/ActorsParent.cpp:8774             NS_ERROR_FAILURE    # Bug 1686041 (call to IsDirectory, this is actually in dom/localstorage/ActorsParent.cpp)
         dom/quota/ActorsParent.cpp:5384                    NS_ERROR_FAILURE    
         dom/quota/ActorsParent.cpp:5424                    NS_ERROR_FAILURE    
         dom/quota/ActorsParent.cpp:5251                    NS_ERROR_FAILURE    
         dom/quota/ActorsParent.cpp:5255                    NS_ERROR_FAILURE    
      67 dom/indexeddb/ActorsParent.cpp:5929                NS_ERROR_FILE_ACCESS_DENIED  # Bug 1686060
         dom/indexeddb/ActorsParent.cpp:6098                NS_ERROR_FILE_ACCESS_DENIED
         dom/indexeddb/ActorsParent.cpp:13110               NS_ERROR_FILE_ACCESS_DENIED
         dom/indexeddb/ActorsParent.cpp:13137               NS_ERROR_UNEXPECTED 
         dom/indexeddb/ActorsParent.cpp:13036               NS_ERROR_UNEXPECTED 
         dom/quota/ActorsParent.cpp:5384                    NS_ERROR_UNEXPECTED 
         dom/quota/ActorsParent.cpp:5424                    NS_ERROR_UNEXPECTED 
         dom/quota/ActorsParent.cpp:4583                    NS_ERROR_UNEXPECTED 
         dom/quota/ActorsParent.cpp:4590                    NS_ERROR_UNEXPECTED 
      47 dom/cache/QuotaClient.cpp:45                                           # shutdown
         dom/cache/QuotaClient.cpp:50                       NS_ERROR_ABORT      
         dom/cache/QuotaClient.cpp:107                      NS_ERROR_ABORT      
         dom/cache/QuotaClient.cpp:416                      NS_ERROR_ABORT      
         dom/cache/QuotaClient.cpp:47                       NS_ERROR_ABORT      
         dom/cache/QuotaClient.cpp:50                       NS_ERROR_ABORT      
         dom/cache/QuotaClient.cpp:450                      NS_ERROR_ABORT      
         dom/quota/ActorsParent.cpp:5384                    NS_ERROR_ABORT      
         dom/quota/ActorsParent.cpp:5424                    NS_ERROR_ABORT      
         dom/quota/ActorsParent.cpp:5428                    NS_ERROR_ABORT      
         dom/quota/ActorsParent.cpp:5251                    NS_ERROR_ABORT      
         dom/quota/ActorsParent.cpp:5255                    NS_ERROR_ABORT      
      24 dom/quota/QuotaCommon.h:1172                       NS_ERROR_FAILURE    # Bug 1686041 (call to GetDirectoryEntries)
         dom/indexeddb/ActorsParent.cpp:13476               NS_ERROR_FAILURE    
         dom/indexeddb/ActorsParent.cpp:13077               NS_ERROR_FAILURE    
         dom/indexeddb/ActorsParent.cpp:13036               NS_ERROR_FAILURE    
         dom/quota/ActorsParent.cpp:5384                    NS_ERROR_FAILURE    
         dom/quota/ActorsParent.cpp:5424                    NS_ERROR_FAILURE    
         dom/quota/ActorsParent.cpp:5251                    NS_ERROR_FAILURE    
         dom/quota/ActorsParent.cpp:5255                    NS_ERROR_FAILURE    
      20 dom/quota/ActorsParent.cpp:5428                    NS_ERROR_ABORT      
         dom/quota/ActorsParent.cpp:5251                    NS_ERROR_ABORT      
         dom/quota/ActorsParent.cpp:5255                    NS_ERROR_ABORT      
      15 dom/indexeddb/ActorsParent.cpp:5977                NS_ERROR_FILE_ACCESS_DENIED
         dom/indexeddb/ActorsParent.cpp:13130               NS_ERROR_FILE_ACCESS_DENIED
         dom/indexeddb/ActorsParent.cpp:13134               NS_ERROR_UNEXPECTED 
      12 dom/localstorage/ActorsParent.cpp:1174             NS_ERROR_FILE_TARGET_DOES_NOT_EXIST
         not propagated                                                         
      10 dom/quota/ActorsParent.cpp:4533                    NS_ERROR_ILLEGAL_VALUE
         dom/quota/ActorsParent.cpp:4590                    NS_ERROR_ILLEGAL_VALUE
       7 dom/quota/ActorsParent.cpp:2697                    NS_ERROR_FILE_IS_LOCKED
         dom/quota/ActorsParent.cpp:10556                   NS_ERROR_FILE_IS_LOCKED
         dom/quota/ActorsParent.cpp:4569                    NS_ERROR_FILE_IS_LOCKED
         dom/quota/ActorsParent.cpp:4590                    NS_ERROR_FILE_IS_LOCKED
       6 dom/cache/QuotaClient.cpp:45                                           
         dom/cache/QuotaClient.cpp:50                       NS_ERROR_ABORT      
         dom/cache/QuotaClient.cpp:450                      NS_ERROR_ABORT      
         dom/quota/ActorsParent.cpp:5384                    NS_ERROR_ABORT      
         dom/quota/ActorsParent.cpp:5424                    NS_ERROR_ABORT      
         dom/quota/ActorsParent.cpp:5428                    NS_ERROR_ABORT      
         dom/quota/ActorsParent.cpp:5251                    NS_ERROR_ABORT      
         dom/quota/ActorsParent.cpp:5255                    NS_ERROR_ABORT      
       5 dom/cache/QuotaClient.cpp:45                                           
         dom/cache/QuotaClient.cpp:50                       NS_ERROR_ABORT      
         dom/cache/QuotaClient.cpp:107                      NS_ERROR_ABORT      
         dom/cache/QuotaClient.cpp:416                      NS_ERROR_ABORT      
         dom/cache/QuotaClient.cpp:47                       NS_ERROR_ABORT      
         dom/cache/QuotaClient.cpp:50                       NS_ERROR_ABORT      
         dom/cache/QuotaClient.cpp:450                      NS_ERROR_ABORT      
         dom/quota/ActorsParent.cpp:5384                    NS_ERROR_ABORT      
         dom/quota/ActorsParent.cpp:5424                    NS_ERROR_ABORT      
         dom/quota/ActorsParent.cpp:5428                    NS_ERROR_ABORT      
         dom/quota/ActorsParent.cpp:4583                    NS_ERROR_ABORT      
         dom/quota/ActorsParent.cpp:4590                    NS_ERROR_ABORT      
       4 dom/quota/ActorsParent.cpp:5428                    NS_ERROR_ABORT      
         dom/quota/ActorsParent.cpp:4583                    NS_ERROR_ABORT      
         dom/quota/ActorsParent.cpp:4590                    NS_ERROR_ABORT      
       2 dom/cache/QuotaClient.cpp:45                                           
         dom/cache/QuotaClient.cpp:50                       NS_ERROR_ABORT      
         dom/cache/QuotaClient.cpp:450                      NS_ERROR_ABORT      
         dom/quota/ActorsParent.cpp:5384                    NS_ERROR_ABORT      
         dom/quota/ActorsParent.cpp:5424                    NS_ERROR_ABORT      
         dom/quota/ActorsParent.cpp:5251                    NS_ERROR_ABORT      
         dom/quota/ActorsParent.cpp:5255                    NS_ERROR_ABORT      
       2 dom/cache/QuotaClient.cpp:45                                           
         dom/cache/QuotaClient.cpp:50                       NS_ERROR_ABORT      
         dom/cache/QuotaClient.cpp:107                      NS_ERROR_ABORT      
         dom/cache/QuotaClient.cpp:416                      NS_ERROR_ABORT      
         dom/cache/QuotaClient.cpp:47                       NS_ERROR_ABORT      
         dom/cache/QuotaClient.cpp:50                       NS_ERROR_ABORT      
         dom/cache/QuotaClient.cpp:450                      NS_ERROR_ABORT      
         dom/quota/ActorsParent.cpp:5384                    NS_ERROR_ABORT      
         dom/quota/ActorsParent.cpp:5424                    NS_ERROR_ABORT      
         dom/quota/ActorsParent.cpp:5251                    NS_ERROR_ABORT      
         dom/quota/ActorsParent.cpp:5255                    NS_ERROR_ABORT      
       2 dom/quota/QuotaCommon.h:1172                       NS_ERROR_FAILURE       # Bug 1686041 (call to GetDirectoryEntries)
         dom/indexeddb/ActorsParent.cpp:13476               NS_ERROR_FAILURE    
         dom/indexeddb/ActorsParent.cpp:13077               NS_ERROR_FAILURE    
         dom/indexeddb/ActorsParent.cpp:13036               NS_ERROR_FAILURE    
         dom/quota/ActorsParent.cpp:5384                    NS_ERROR_FAILURE    
         dom/quota/ActorsParent.cpp:5424                    NS_ERROR_FAILURE    
         dom/quota/ActorsParent.cpp:1136                    NS_ERROR_FAILURE    
         dom/localstorage/ActorsParent.cpp:8774             NS_ERROR_FAILURE    
       1 dom/quota/ActorsParent.cpp:2711                    NS_ERROR_FILE_IS_LOCKED # Bug 1679541 (will add call sites)
         dom/quota/ActorsParent.cpp:5016                    NS_ERROR_FILE_IS_LOCKED
       1 dom/indexeddb/ActorsParent.cpp:5929                NS_ERROR_FILE_ACCESS_DENIED # Bug 1686060
         dom/indexeddb/ActorsParent.cpp:6098                NS_ERROR_FILE_ACCESS_DENIED
         dom/indexeddb/ActorsParent.cpp:13110               NS_ERROR_FILE_ACCESS_DENIED


Initialization::Storage

      77 dom/quota/ActorsParent.cpp:2711                    NS_ERROR_FILE_NOT_FOUND
         dom/quota/ActorsParent.cpp:10382                   NS_ERROR_FILE_NOT_FOUND
      67 dom/indexeddb/ActorsParent.cpp:12934                                   
         not propagated                                                         

I'm going to file individual bugs for the top items here, and include the bug numbers in the list.

Depends on: 1686041
Depends on: 1679541
Depends on: 1686051
Depends on: 1686060

@janv, I guess all failures involving NS_ERROR_ABORT can be ignored, as they are due to a user request for shutdown during storage initialization?

Flags: needinfo?(jvarga)

(In reply to Simon Giesecke [:sg] [he/him] from comment #11)

@janv, I guess all failures involving NS_ERROR_ABORT can be ignored, as they are due to a user request for shutdown during storage initialization?

Yes, I think I raised that during reviews that we need to do something about that.

Flags: needinfo?(jvarga)

(In reply to Jan Varga [:janv] from comment #12)

(In reply to Simon Giesecke [:sg] [he/him] from comment #11)

@janv, I guess all failures involving NS_ERROR_ABORT can be ignored, as they are due to a user request for shutdown during storage initialization?

Yes, I think I raised that during reviews that we need to do something about that.

Sorry, I don't remember exactly. What do you think should be done about the aborts?

The first thing that came into my mind is to add a quota module specific abort error and then detect that in LogError.
All our NS_ABORT_ERROR cases which are used just for cancelling an operation (either by a user request or by shutdown condition) can be then converted to the new error.

xpcom/base/ErrorList.py

+modules["QUOTA_MANAGER"] = Mod(44)

+# 44: NS_ERROR_MODULE_QUOTA_MANAGER
+# =======================================================================
+with modules["QUOTA_MANAGER"]:

  • errors["NS_ERROR_QM_ABORT"] = FAILURE(1)

+# =======================================================================

I planned to add such error codes anyway.

Updated script with autodetection of split points.

Attachment #9196558 - Attachment is obsolete: true
Attachment #9196558 - Attachment is obsolete: false
Attachment #9196363 - Attachment is obsolete: true
Depends on: 1686697

On the latest Nightly builds, the top 4 propagated issues are all blocked by Bug 1686041 and show a NS_ERROR_FAILURE from some nsLocalFile method.

The errors, leaving out the non-propagated and NS_ERROR_ABORT ones are:

Initialization::TemporaryStorage

    1546 dom/quota/QuotaCommon.h:1181                       NS_ERROR_FAILURE    # Blocked by Bug 1686041 
         dom/quota/ActorsParent.cpp:5068                    NS_ERROR_FAILURE    
         dom/quota/ActorsParent.cpp:4530                    NS_ERROR_FAILURE    
         dom/quota/ActorsParent.cpp:6532                    NS_ERROR_FAILURE    
    1136 dom/cache/QuotaClient.cpp:63                       NS_ERROR_FAILURE    # Blocked by Bug 1686041 
         dom/cache/QuotaClient.cpp:47                       NS_ERROR_FAILURE    
         dom/cache/QuotaClient.cpp:50                       NS_ERROR_FAILURE    
         dom/cache/QuotaClient.cpp:106                      NS_ERROR_FAILURE    
         dom/cache/QuotaClient.cpp:428                      NS_ERROR_FAILURE    
         dom/cache/QuotaClient.cpp:47                       NS_ERROR_FAILURE    
         dom/cache/QuotaClient.cpp:50                       NS_ERROR_FAILURE    
         dom/cache/QuotaClient.cpp:462                      NS_ERROR_FAILURE    
         dom/quota/ActorsParent.cpp:5193                    NS_ERROR_FAILURE    
         dom/quota/ActorsParent.cpp:5233                    NS_ERROR_FAILURE    
    1136 dom/quota/ActorsParent.cpp:5060                    NS_ERROR_FAILURE    # Blocked by Bug 1686041 
         dom/quota/ActorsParent.cpp:5064                    NS_ERROR_FAILURE    
    1133 dom/quota/ActorsParent.cpp:4530                    NS_ERROR_FAILURE    # Blocked by Bug 1686041 
         dom/quota/ActorsParent.cpp:6532                    NS_ERROR_FAILURE    
     161 dom/quota/ActorsParent.cpp:5060                    NS_ERROR_FILE_TARGET_DOES_NOT_EXIST # Tail of the two issues below
         dom/quota/ActorsParent.cpp:5064                    NS_ERROR_FILE_TARGET_DOES_NOT_EXIST
      81 dom/indexeddb/ActorsParent.cpp:12766               NS_ERROR_FILE_TARGET_DOES_NOT_EXIST # Bug 1687256
         dom/indexeddb/ActorsParent.cpp:12776               NS_ERROR_FILE_TARGET_DOES_NOT_EXIST
         dom/indexeddb/ActorsParent.cpp:13191               NS_ERROR_FILE_TARGET_DOES_NOT_EXIST
         dom/indexeddb/ActorsParent.cpp:13036               NS_ERROR_FILE_TARGET_DOES_NOT_EXIST
         dom/quota/ActorsParent.cpp:5193                    NS_ERROR_FILE_TARGET_DOES_NOT_EXIST
         dom/quota/ActorsParent.cpp:5233                    NS_ERROR_FILE_TARGET_DOES_NOT_EXIST
      80 dom/cache/QuotaClient.cpp:63                       NS_ERROR_FILE_TARGET_DOES_NOT_EXIST # Bug 1687254
         dom/cache/QuotaClient.cpp:47                       NS_ERROR_FILE_TARGET_DOES_NOT_EXIST
         dom/cache/QuotaClient.cpp:50                       NS_ERROR_FILE_TARGET_DOES_NOT_EXIST
         dom/cache/QuotaClient.cpp:106                      NS_ERROR_FILE_TARGET_DOES_NOT_EXIST
         dom/cache/QuotaClient.cpp:428                      NS_ERROR_FILE_TARGET_DOES_NOT_EXIST
         dom/cache/QuotaClient.cpp:47                       NS_ERROR_FILE_TARGET_DOES_NOT_EXIST
         dom/cache/QuotaClient.cpp:50                       NS_ERROR_FILE_TARGET_DOES_NOT_EXIST
         dom/cache/QuotaClient.cpp:462                      NS_ERROR_FILE_TARGET_DOES_NOT_EXIST
         dom/quota/ActorsParent.cpp:5193                    NS_ERROR_FILE_TARGET_DOES_NOT_EXIST
         dom/quota/ActorsParent.cpp:5233                    NS_ERROR_FILE_TARGET_DOES_NOT_EXIST
      80 dom/quota/ActorsParent.cpp:4530                    NS_ERROR_FILE_TARGET_DOES_NOT_EXIST # Maybe blocked by https://phabricator.services.mozilla.com/D102181?
         dom/quota/ActorsParent.cpp:6532                    NS_ERROR_FILE_TARGET_DOES_NOT_EXIST
       8 dom/cache/FileUtils.cpp:479                        NS_ERROR_FILE_ACCESS_DENIED
       6 dom/cache/FileUtils.cpp:479                        NS_ERROR_FILE_DIR_NOT_EMPTY
       5 dom/quota/ActorsParent.cpp:4530                    NS_ERROR_FILE_IS_LOCKED
         dom/quota/ActorsParent.cpp:6532                    NS_ERROR_FILE_IS_LOCKED
       4 dom/quota/ActorsParent.cpp:2585                    NS_ERROR_FILE_IS_LOCKED
         dom/quota/ActorsParent.cpp:11170                   NS_ERROR_FILE_IS_LOCKED
         dom/quota/ActorsParent.cpp:10023                   NS_ERROR_FILE_IS_LOCKED
         dom/quota/ActorsParent.cpp:11158                   NS_ERROR_FILE_IS_LOCKED
         dom/quota/ActorsParent.cpp:4835                    NS_ERROR_FILE_IS_LOCKED
         dom/quota/ActorsParent.cpp:4908                    NS_ERROR_FILE_IS_LOCKED
         dom/quota/ActorsParent.cpp:5024                    NS_ERROR_FILE_IS_LOCKED
         dom/quota/ActorsParent.cpp:5064                    NS_ERROR_FILE_IS_LOCKED
       3 dom/quota/QuotaCommon.h:1181                       NS_ERROR_FAILURE    
         dom/quota/ActorsParent.cpp:5068                    NS_ERROR_FAILURE    
       2 dom/quota/ActorsParent.cpp:4530                    NS_ERROR_FAILURE    
       1 dom/cache/QuotaClient.cpp:63                       NS_ERROR_FILE_TARGET_DOES_NOT_EXIST
       1 dom/quota/ActorsParent.cpp:464                     NS_ERROR_FILE_NO_DEVICE_SPACE
         dom/quota/ActorsParent.cpp:4544                    NS_ERROR_FILE_NO_DEVICE_SPACE
         dom/quota/ActorsParent.cpp:6532                    NS_ERROR_FILE_NO_DEVICE_SPACE
       1 dom/localstorage/ActorsParent.cpp:998              NS_ERROR_FILE_IS_LOCKED
         dom/localstorage/ActorsParent.cpp:976              NS_ERROR_FILE_IS_LOCKED
         dom/localstorage/ActorsParent.cpp:8365             NS_ERROR_FILE_IS_LOCKED
         dom/localstorage/ActorsParent.cpp:8371             NS_ERROR_FILE_IS_LOCKED
         dom/localstorage/ActorsParent.cpp:8379             NS_ERROR_FILE_IS_LOCKED
         dom/quota/ActorsParent.cpp:5193                    NS_ERROR_FILE_IS_LOCKED
         dom/quota/ActorsParent.cpp:5233                    NS_ERROR_FILE_IS_LOCKED
       1 dom/quota/ActorsParent.cpp:5060                    NS_ERROR_FILE_IS_LOCKED
         dom/quota/ActorsParent.cpp:5064                    NS_ERROR_FILE_IS_LOCKED
       1 dom/cache/QuotaClient.cpp:63                       NS_ERROR_FAILURE    
         dom/cache/QuotaClient.cpp:47                       NS_ERROR_FAILURE    
         dom/cache/QuotaClient.cpp:50                       NS_ERROR_FAILURE    
       1 dom/cache/QuotaClient.cpp:63                       NS_ERROR_FAILURE    
       1 dom/quota/QuotaCommon.h:1181                       NS_ERROR_FAILURE    

Initialization::Storage

      none!
Depends on: 1687253

Updated the script to sort by distinct client counts, and filter out NS_ERROR_ABORT pseudo-failures.

Attachment #9196558 - Attachment is obsolete: true

Results from this weekend's builds (leaving out the non-propagated and NS_ERROR_ABORT ones):

Initialization::TemporaryStorage

 Clients Sessions Location                                           Result              
       3        8 dom/cache/FileUtils.cpp:479                        NS_ERROR_FILE_DIR_NOT_EMPTY  # caller is not instrumented, this is not propagated
       1      152 dom/localstorage/ActorsParent.cpp:8330             NS_ERROR_FILE_IS_LOCKED 
                  dom/quota/ActorsParent.cpp:5187                    NS_ERROR_FILE_IS_LOCKED
                  dom/quota/ActorsParent.cpp:5227                    NS_ERROR_FILE_IS_LOCKED
                  dom/quota/ActorsParent.cpp:5054                    NS_ERROR_FILE_IS_LOCKED
                  dom/quota/ActorsParent.cpp:5058                    NS_ERROR_FILE_IS_LOCKED
       1      151 dom/quota/ActorsParent.cpp:4529                    NS_ERROR_FILE_IS_LOCKED # tail from the stack above
                  dom/quota/ActorsParent.cpp:6526                    NS_ERROR_FILE_IS_LOCKED
       1        5 dom/cache/FileUtils.cpp:479                        NS_ERROR_FILE_ACCESS_DENIED # caller is not instrumented, this is not propagated
       1       73 dom/quota/QuotaCommon.cpp:180                      WIN32(0x570)        # 0x570 is ERROR_FILE_CORRUPT, maybe this is running on VMware?
                  dom/cache/QuotaClient.cpp:64                       WIN32(0x570)        
                  dom/cache/QuotaClient.cpp:49                       WIN32(0x570)        
                  dom/cache/QuotaClient.cpp:52                       WIN32(0x570)        
                  dom/cache/QuotaClient.cpp:111                      WIN32(0x570)        
                  dom/cache/QuotaClient.cpp:423                      WIN32(0x570)        
                  dom/cache/QuotaClient.cpp:49                       WIN32(0x570)        
                  dom/cache/QuotaClient.cpp:52                       WIN32(0x570)        
                  dom/cache/QuotaClient.cpp:457                      WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5187                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5227                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5054                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5058                    WIN32(0x570)        
       1       73 dom/quota/ActorsParent.cpp:4529                    WIN32(0x570)        # tail from the stack above
                  dom/quota/ActorsParent.cpp:6526                    WIN32(0x570)        
       1        1 dom/cache/FileUtils.cpp:501                        NS_ERROR_FILE_ACCESS_DENIED # caller is not instrumented, this is not propagated

Initialization::Storage

 Clients Sessions Location                                           Result              
       1     3999 dom/quota/ActorsParent.cpp:576                                         
                  dom/quota/ActorsParent.cpp:6185                    NS_ERROR_FILE_CORRUPTED
       1        1 dom/quota/ActorsParent.cpp:576                                         

The previous results were very distorted as they were sorted by session numbers, and it seems that some errors are associated with a huge number of sessions per client. Now the sorting is by unique clients that are affected. As can be seen, only a small number of clients was affected. But we will need to monitor the results over a work week as well.

All errors that are shown above, which are actually propagated, affect only a single client. Before taking further action, we should wait for the results of some more days.

FWIW, it seems that while we know NS_ERROR_FILE_CORRUPTED in our code, it is not mapped from any WIN32 error message in nsLocalFileWin.cpp.

We now have first results from beta/aurora 86.0b1. There are the propagating, non-abort stacks so far that affect more than 1 client:

Initialization::TemporaryStorage

 Clients Sessions Location                                           Result              
       2      313 dom/localstorage/ActorsParent.cpp:956              WIN32(0x570)        # A
                  dom/localstorage/ActorsParent.cpp:8320             WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5187                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5227                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5231                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5234                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5054                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5058                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5062                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5065                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:4529                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:4532                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:6526                    WIN32(0x570)        
       2     1502 dom/quota/ActorsParent.cpp:462                     NS_ERROR_FILE_CORRUPTED # B
                  dom/quota/ActorsParent.cpp:4543                    NS_ERROR_FILE_CORRUPTED
                  dom/quota/ActorsParent.cpp:6526                    NS_ERROR_FILE_CORRUPTED

Initialization::Storage

 Clients Sessions Location                                           Result              
       4     1563 dom/quota/ActorsParent.cpp:462                     NS_ERROR_FILE_CORRUPTED # C
                  dom/quota/ActorsParent.cpp:6303                    NS_ERROR_FILE_CORRUPTED
       3        3 dom/quota/ActorsParent.cpp:546                     NS_ERROR_FILE_CORRUPTED # D
                  dom/quota/ActorsParent.cpp:6200                    NS_ERROR_FILE_CORRUPTED
       3     3534 dom/quota/ActorsParent.cpp:546                     NS_ERROR_FILE_CORRUPTED # E
                  dom/quota/ActorsParent.cpp:5985                    NS_ERROR_FILE_CORRUPTED
                  dom/quota/ActorsParent.cpp:6208                    NS_ERROR_FILE_CORRUPTED

A is a case where we get the Windows System Error ERROR_FILE_CORRUPT when checking a path (of a usage file in this case) for being a directory.

B-E are all cases where executing a SQL statement results in NS_ERROR_FILE_CORRUPTED. It's conceivable that these are cases where a file operation failed with ERROR_FILE_CORRUPTED as well, but inside SQLite code. Cases B and C are inside InvalidateCache, which is only called if the cache was deemed usable before, so it's very unlikely this is caused by internal actions. Cases D and E are inside InitializeLocalStorageArchive on the first CREATE TABLE statement on a freshly created database, which also seems to indicate this is an externally caused condition.

In all of these cases, the question is if this is just a transient condition, which might go away by retrying.

Everything related to the local storage archive (for example InitializeLocalStorageArchive) which returns NS_ERROR_FILE_CORRUPTED should probably trigger removal of the archive and creating an empty one. The archive is created from the database used by the old localStorage implementation. We don't do any vacuuming of the archive after its copied from the old database because that could slow down entire initialization.

Might we see here also the consequences of bug 1588498 in some cases? Something like: QM Shutdown hang -> Unclean SQLite shutdown -> broken database file?

(In reply to Jens Stutte [:jstutte] from comment #23)

Might we see here also the consequences of bug 1588498 in some cases? Something like: QM Shutdown hang -> Unclean SQLite shutdown -> broken database file?

Yes, but that should be rare since SQLite should be able to recover from a crash in most of the cases. Here's a description of how to corrupt an SQLite database file: https://www.sqlite.org/howtocorrupt.html

It also depends on the setting of the "synchronous" flag:
https://www.sqlite.org/pragma.html#pragma_synchronous

For example, the storage.sqlite database which is very important for QM, has this set to "EXTRA". The database is small and we don't do a lot of writes.

IDB: synchronous = NORMAL (until dom.indexedDB.fullSynchronous is set to true which will set the pragma to FULL)
DOMCACHE: synchronous = default setting (I think)
LSNG: synchronous = FULL

We should also start thinking if we somehow can get QM_TRY data from Beta (not only early Beta) or even Release.
There are two main issues, increased binary size and possibly huge amount of data.
However the telemetry reporting is currently scoped to storage initialization only and maybe it's possible to enable it for a fraction of clients.

(In reply to Jan Varga [:janv] from comment #25)

We should also start thinking if we somehow can get QM_TRY data from Beta (not only early Beta) or even Release.
There are two main issues, increased binary size and possibly huge amount of data.
However the telemetry reporting is currently scoped to storage initialization only and maybe it's possible to enable it for a fraction of clients.

If we have a pref for it, we can ask for an experiment.

(In reply to Jan Varga [:janv] from comment #25)

We should also start thinking if we somehow can get QM_TRY data from Beta (not only early Beta) or even Release.

Late beta maybe, but why? Is there a different population for late beta than early beta?

However, I don't think we should do this for Release. We should only consider this if we really need to, i.e. when we notice that Release population suffers from issues that we couldn't see in Beta. But we don't have any indication for that right now.

There are two main issues, increased binary size and possibly huge amount of data.

There's also a performance impact (independent of whether telemetry reporting is active or not).

However the telemetry reporting is currently scoped to storage initialization only and maybe it's possible to enable it for a fraction of clients.

I don't think that's the critical point. When it's in the build, it can be activated unconditionally.

(In reply to Jan Varga [:janv] from comment #22)

Everything related to the local storage archive (for example InitializeLocalStorageArchive) which returns NS_ERROR_FILE_CORRUPTED should probably trigger removal of the archive and creating an empty one. The archive is created from the database used by the old localStorage implementation. We don't do any vacuuming of the archive after its copied from the old database because that could slow down entire initialization.

Ok, I didn't look at that closely enough yet. I was assuming that a newly created storage archive is empty, but if it's not that can better explain the NS_ERROR_FILE_CORRUPTED errors. I will reanalyze the beta failure today or tomorrow and then file a bug for this issue.

(In reply to Simon Giesecke [:sg] [he/him] from comment #27)

However, I don't think we should do this for Release. We should only consider this if we really need to, i.e. when we notice that Release population suffers from issues that we couldn't see in Beta. But we don't have any indication for that right now.

I think otherwise. It happened multiple times in the past that quite big issues manifested only on Release. For example LSNG on Release which had to be disabled by a dot release. There was extensive testing before that, even manual testing by QA. The other one was the issue with the old localStorage implementation, bug 1683299.
Experimental data from Release could help with eliminating unexpected surprises after enabling LSNG on Release again.
We could enable QM_TRY only in Release 87 and then disable it. I think we should explore options here more, especially if the performance impact is so big/noticeable.

Depends on: 1689636

These are the top beta failures, affecting more than 3 clients:

Initialization::TemporaryStorage

 Clients Sessions Location                                           Result
      28    35726 dom/quota/ActorsParent.cpp:462                     NS_ERROR_FILE_CORRUPTED  # Bug 1689636
                  dom/quota/ActorsParent.cpp:4543                    NS_ERROR_FILE_CORRUPTED
                  dom/quota/ActorsParent.cpp:6526                    NS_ERROR_FILE_CORRUPTED
      15       24 dom/indexeddb/ActorsParent.cpp:13441               NS_ERROR_FILE_NOT_FOUND # Bug 1689655
                  dom/indexeddb/ActorsParent.cpp:13491               NS_ERROR_FILE_NOT_FOUND
                  dom/indexeddb/ActorsParent.cpp:13092               NS_ERROR_FILE_NOT_FOUND
                  dom/indexeddb/ActorsParent.cpp:13051               NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:5187                    NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:5227                    NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:5231                    NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:5234                    NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:5054                    NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:5058                    NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:5062                    NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:5065                    NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:4529                    NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:4532                    NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:6526                    NS_ERROR_FILE_NOT_FOUND
      12     2226 dom/quota/QuotaCommon.h:1174                       WIN32(0x570)        
                  dom/cache/FileUtilsImpl.h:83                       WIN32(0x570)        
                  dom/cache/QuotaClient.cpp:109                      WIN32(0x570)        
                  dom/cache/QuotaClient.cpp:49                       WIN32(0x570)        
                  dom/cache/QuotaClient.cpp:52                       WIN32(0x570)        
                  dom/cache/QuotaClient.cpp:111                      WIN32(0x570)        
                  dom/cache/QuotaClient.cpp:423                      WIN32(0x570)        
                  dom/cache/QuotaClient.cpp:49                       WIN32(0x570)        
                  dom/cache/QuotaClient.cpp:52                       WIN32(0x570)        
                  dom/cache/QuotaClient.cpp:457                      WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5187                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5227                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5231                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5234                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5054                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5058                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5062                    WIN32(0x570)
                  dom/quota/ActorsParent.cpp:5065                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:4529                    WIN32(0x570)
                  dom/quota/ActorsParent.cpp:4532                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:6526                    WIN32(0x570)        
       8       76 dom/cache/FileUtilsImpl.h:42                       NS_ERROR_FILE_NOT_FOUND # Bug 1689656
                  dom/cache/FileUtilsImpl.h:83                       NS_ERROR_FILE_NOT_FOUND
                  dom/cache/QuotaClient.cpp:109                      NS_ERROR_FILE_NOT_FOUND
                  dom/cache/QuotaClient.cpp:49                       NS_ERROR_FILE_NOT_FOUND
                  dom/cache/QuotaClient.cpp:52                       NS_ERROR_FILE_NOT_FOUND
                  dom/cache/QuotaClient.cpp:111                      NS_ERROR_FILE_NOT_FOUND
                  dom/cache/QuotaClient.cpp:423                      NS_ERROR_FILE_NOT_FOUND
                  dom/cache/QuotaClient.cpp:49                       NS_ERROR_FILE_NOT_FOUND
                  dom/cache/QuotaClient.cpp:52                       NS_ERROR_FILE_NOT_FOUND
                  dom/cache/QuotaClient.cpp:457                      NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:5187                    NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:5227                    NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:5231                    NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:5234                    NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:5054                    NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:5058                    NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:5062                    NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:5065                    NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:4529                    NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:4532                    NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:6526                    NS_ERROR_FILE_NOT_FOUND
       8       34 dom/quota/ActorsParent.cpp:2557                    NS_ERROR_FAILURE    
                  dom/quota/ActorsParent.cpp:11175                   NS_ERROR_FAILURE    
                  dom/quota/ActorsParent.cpp:10028                   NS_ERROR_FAILURE    
                  dom/quota/ActorsParent.cpp:11163                   NS_ERROR_FAILURE    
                  dom/quota/ActorsParent.cpp:4834                    NS_ERROR_FAILURE    
                  dom/quota/ActorsParent.cpp:4907                    NS_ERROR_FAILURE    
                  dom/quota/ActorsParent.cpp:5018                    NS_ERROR_FAILURE    
                  dom/quota/ActorsParent.cpp:5058                    NS_ERROR_FAILURE    
                  dom/quota/ActorsParent.cpp:5062                    NS_ERROR_FAILURE    
                  dom/quota/ActorsParent.cpp:5065                    NS_ERROR_FAILURE    
                  dom/quota/ActorsParent.cpp:4529                    NS_ERROR_FAILURE
                  dom/quota/ActorsParent.cpp:4532                    NS_ERROR_FAILURE    
                  dom/quota/ActorsParent.cpp:6526                    NS_ERROR_FAILURE
       8       12 dom/quota/ActorsParent.cpp:462                     NS_ERROR_FILE_CORRUPTED # Bug 1689636
                  dom/quota/ActorsParent.cpp:4543                    NS_ERROR_FILE_CORRUPTED
       7     2621 dom/localstorage/ActorsParent.cpp:956              WIN32(0x570)        
                  dom/localstorage/ActorsParent.cpp:8320             WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5187                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5227                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5231                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5234                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5054                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5058                    WIN32(0x570)
                  dom/quota/ActorsParent.cpp:5062                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5065                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:4529                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:4532                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:6526                    WIN32(0x570)        
       7     2329 dom/indexeddb/ActorsParent.cpp:13441               WIN32(0x570)        
                  dom/indexeddb/ActorsParent.cpp:13491               WIN32(0x570)        
                  dom/indexeddb/ActorsParent.cpp:13092               WIN32(0x570)        
                  dom/indexeddb/ActorsParent.cpp:13051               WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5187                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5227                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5231                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5234                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5054                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5058                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5062                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5065                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:4529                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:4532                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:6526                    WIN32(0x570)        
       6     1028 dom/cache/FileUtilsImpl.h:42                       WIN32(0x570)
                  dom/cache/FileUtilsImpl.h:83                       WIN32(0x570)        
                  dom/cache/QuotaClient.cpp:109                      WIN32(0x570)
                  dom/cache/QuotaClient.cpp:49                       WIN32(0x570)        
                  dom/cache/QuotaClient.cpp:52                       WIN32(0x570)        
                  dom/cache/QuotaClient.cpp:111                      WIN32(0x570)        
                  dom/cache/QuotaClient.cpp:423                      WIN32(0x570)        
                  dom/cache/QuotaClient.cpp:49                       WIN32(0x570)        
                  dom/cache/QuotaClient.cpp:52                       WIN32(0x570)        
                  dom/cache/QuotaClient.cpp:457                      WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5187                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5227                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5231                    WIN32(0x570)
                  dom/quota/ActorsParent.cpp:5234                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5054                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5058                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5062                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5065                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:4529                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:4532                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:6526                    WIN32(0x570)        
       6       31 dom/quota/QuotaCommon.cpp:194                      NS_ERROR_STORAGE_IOERR
                  dom/quota/ActorsParent.cpp:4477                    NS_ERROR_STORAGE_IOERR
                  dom/quota/ActorsParent.cpp:4494                    NS_ERROR_STORAGE_IOERR
                  dom/quota/ActorsParent.cpp:6526                    NS_ERROR_STORAGE_IOERR
       6       14 dom/cache/QuotaClient.cpp:419                      NS_ERROR_FILE_NOT_FOUND # Bug 1689657
                  dom/cache/QuotaClient.cpp:49                       NS_ERROR_FILE_NOT_FOUND
                  dom/cache/QuotaClient.cpp:52                       NS_ERROR_FILE_NOT_FOUND
                  dom/cache/QuotaClient.cpp:457                      NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:5187                    NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:5227                    NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:5231                    NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:5234                    NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:5054                    NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:5058                    NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:5062                    NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:5065                    NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:4529                    NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:4532                    NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:6526                    NS_ERROR_FILE_NOT_FOUND
       6     1868 dom/quota/QuotaCommon.h:1174                       WIN32(0x570)        
                  dom/indexeddb/ActorsParent.cpp:13491               WIN32(0x570)        
                  dom/indexeddb/ActorsParent.cpp:13092               WIN32(0x570)        
                  dom/indexeddb/ActorsParent.cpp:13051               WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5187                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5227                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5231                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5234                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5054                    WIN32(0x570)
                  dom/quota/ActorsParent.cpp:5058                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5062                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5065                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:4529                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:4532                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:6526                    WIN32(0x570)        
       5      688 dom/quota/QuotaCommon.h:1209                       WIN32(0x570)        
                  dom/cache/QuotaClient.cpp:52                       WIN32(0x570)        
                  dom/cache/QuotaClient.cpp:457                      WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5187                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5227                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5231                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5234                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5054                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5058                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5062                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5065                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:4529                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:4532                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:6526                    WIN32(0x570)        
       5     1806 dom/quota/QuotaCommon.cpp:180                      WIN32(0x570)
                  dom/cache/QuotaClient.cpp:64                       WIN32(0x570)        
                  dom/cache/QuotaClient.cpp:49                       WIN32(0x570)        
                  dom/cache/QuotaClient.cpp:52                       WIN32(0x570)        
                  dom/cache/QuotaClient.cpp:111                      WIN32(0x570)        
                  dom/cache/QuotaClient.cpp:423                      WIN32(0x570)        
                  dom/cache/QuotaClient.cpp:49                       WIN32(0x570)        
                  dom/cache/QuotaClient.cpp:52                       WIN32(0x570)        
                  dom/cache/QuotaClient.cpp:457                      WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5187                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5227                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5231                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5234                    WIN32(0x570)
                  dom/quota/ActorsParent.cpp:5054                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5058                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5062                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5065                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:4529                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:4532                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:6526                    WIN32(0x570)        
       5        7 dom/quota/ActorsParent.cpp:4850                    NS_ERROR_FAILURE    
                  dom/quota/ActorsParent.cpp:4909                    NS_ERROR_FAILURE    
                  dom/quota/ActorsParent.cpp:5018                    NS_ERROR_FAILURE    
                  dom/quota/ActorsParent.cpp:5058                    NS_ERROR_FAILURE    
                  dom/quota/ActorsParent.cpp:5062                    NS_ERROR_FAILURE    
                  dom/quota/ActorsParent.cpp:5065                    NS_ERROR_FAILURE    
                  dom/quota/ActorsParent.cpp:4529                    NS_ERROR_FAILURE    
                  dom/quota/ActorsParent.cpp:4532                    NS_ERROR_FAILURE    
                  dom/quota/ActorsParent.cpp:6526                    NS_ERROR_FAILURE

Initialization::Storage

 Clients Sessions Location                                           Result              
      37    21340 dom/quota/ActorsParent.cpp:462                     NS_ERROR_FILE_CORRUPTED   # Bug 1689636
                  dom/quota/ActorsParent.cpp:6303                    NS_ERROR_FILE_CORRUPTED
      24    28339 dom/quota/ActorsParent.cpp:576                                         # Bug 1689680
                  dom/quota/ActorsParent.cpp:6185                    NS_ERROR_FILE_CORRUPTED
      19    50884 dom/quota/ActorsParent.cpp:546                     NS_ERROR_FILE_CORRUPTED # Bug 1689680
                  dom/quota/ActorsParent.cpp:5985                    NS_ERROR_FILE_CORRUPTED
                  dom/quota/ActorsParent.cpp:6208                    NS_ERROR_FILE_CORRUPTED
      18       19 dom/quota/ActorsParent.cpp:546                     NS_ERROR_FILE_CORRUPTED # Bug 1689680
                  dom/quota/ActorsParent.cpp:6200                    NS_ERROR_FILE_CORRUPTED
      14     2665 dom/quota/ActorsParent.cpp:504                     NS_ERROR_STORAGE_IOERR
                  dom/quota/ActorsParent.cpp:5761                    NS_ERROR_STORAGE_IOERR
                  dom/quota/ActorsParent.cpp:5931                    NS_ERROR_STORAGE_IOERR
                  dom/quota/ActorsParent.cpp:6176                    NS_ERROR_STORAGE_IOERR
      12     4122 dom/quota/ActorsParent.cpp:10848                   NS_ERROR_FILE_ACCESS_DENIED
                  dom/quota/ActorsParent.cpp:10028                   NS_ERROR_FILE_ACCESS_DENIED
                  dom/quota/ActorsParent.cpp:10580                   NS_ERROR_FILE_ACCESS_DENIED
                  dom/quota/ActorsParent.cpp:5416                    NS_ERROR_FILE_ACCESS_DENIED
                  dom/quota/ActorsParent.cpp:5440                    NS_ERROR_FILE_ACCESS_DENIED
                  dom/quota/ActorsParent.cpp:6147                    NS_ERROR_FILE_ACCESS_DENIED
       8     2339 dom/quota/ActorsParent.cpp:6070                    NS_ERROR_STORAGE_BUSY
       7        7 dom/quota/ActorsParent.cpp:576                                         # Bug 1689680
       7       13 dom/quota/ActorsParent.cpp:546                     NS_ERROR_FILE_CORRUPTED # Bug 1689680
                  dom/quota/ActorsParent.cpp:5985                    NS_ERROR_FILE_CORRUPTED
       6        6 dom/quota/ActorsParent.cpp:5817                    NS_ERROR_FILE_IS_LOCKED
       6        6 dom/quota/ActorsParent.cpp:5931                    NS_ERROR_FILE_IS_LOCKED
                  dom/quota/ActorsParent.cpp:6176                    NS_ERROR_FILE_IS_LOCKED
       6     1033 dom/quota/ActorsParent.cpp:6070                    NS_ERROR_STORAGE_IOERR
       5        8 dom/quota/ActorsParent.cpp:546                     NS_ERROR_FILE_CORRUPTED # Bug 1689680
       4     1077 dom/quota/ActorsParent.cpp:5792                    WIN32(0x17)         
                  dom/quota/ActorsParent.cpp:5931                    WIN32(0x17)         
                  dom/quota/ActorsParent.cpp:6176                    WIN32(0x17)         
       4        4 dom/quota/ActorsParent.cpp:546                     NS_ERROR_FAILURE    # Bug 1689680
                  dom/quota/ActorsParent.cpp:6200                    NS_ERROR_FAILURE    
Depends on: 1689680

As an update, these are the failures of beta over the weekend, showing propagating failures affecting more than 3 clients. I couldn't query the numbers the full week for some reason (the query then took excessively long). I can try to combine the results, but maybe that's not necessary:


Initialization::TemporaryStorage

 Clients Sessions Location                                           Result
      55    67859 dom/quota/ActorsParent.cpp:462                     NS_ERROR_FILE_CORRUPTED # Bug 1689636
                  dom/quota/ActorsParent.cpp:4543                    NS_ERROR_FILE_CORRUPTED
                  dom/quota/ActorsParent.cpp:6526                    NS_ERROR_FILE_CORRUPTED
      20       47 dom/quota/ActorsParent.cpp:462                     NS_ERROR_FILE_CORRUPTED # Bug 1689636
                  dom/quota/ActorsParent.cpp:4543                    NS_ERROR_FILE_CORRUPTED
      17     2115 dom/cache/FileUtilsImpl.h:42                       WIN32(0x570)  # Bug 1690234
                  dom/cache/FileUtilsImpl.h:83                       WIN32(0x570)        
                  dom/cache/QuotaClient.cpp:109                      WIN32(0x570)
                  dom/cache/QuotaClient.cpp:49                       WIN32(0x570)        
                  dom/cache/QuotaClient.cpp:52                       WIN32(0x570)
                  dom/cache/QuotaClient.cpp:111                      WIN32(0x570)
                  dom/cache/QuotaClient.cpp:423                      WIN32(0x570)
                  dom/cache/QuotaClient.cpp:49                       WIN32(0x570)        
                  dom/cache/QuotaClient.cpp:52                       WIN32(0x570)
                  dom/cache/QuotaClient.cpp:457                      WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5187                    WIN32(0x570)
                  dom/quota/ActorsParent.cpp:5227                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5231                    WIN32(0x570)
                  dom/quota/ActorsParent.cpp:5234                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5054                    WIN32(0x570)
                  dom/quota/ActorsParent.cpp:5058                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5062                    WIN32(0x570)
                  dom/quota/ActorsParent.cpp:5065                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:4529                    WIN32(0x570)
                  dom/quota/ActorsParent.cpp:4532                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:6526                    WIN32(0x570)
      14       20 dom/quota/ActorsParent.cpp:462                     NS_ERROR_FILE_CORRUPTED # Bug 1689636
      14     2952 dom/quota/QuotaCommon.h:1174                       WIN32(0x570)  # Bug 1690234
                  dom/cache/FileUtilsImpl.h:83                       WIN32(0x570)        
                  dom/cache/QuotaClient.cpp:109                      WIN32(0x570)
                  dom/cache/QuotaClient.cpp:49                       WIN32(0x570)        
                  dom/cache/QuotaClient.cpp:52                       WIN32(0x570)
                  dom/cache/QuotaClient.cpp:111                      WIN32(0x570)        
                  dom/cache/QuotaClient.cpp:423                      WIN32(0x570)
                  dom/cache/QuotaClient.cpp:49                       WIN32(0x570)        
                  dom/cache/QuotaClient.cpp:52                       WIN32(0x570)
                  dom/cache/QuotaClient.cpp:457                      WIN32(0x570)
                  dom/quota/ActorsParent.cpp:5187                    WIN32(0x570)
                  dom/quota/ActorsParent.cpp:5227                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5231                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5234                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5054                    WIN32(0x570)
                  dom/quota/ActorsParent.cpp:5058                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5062                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5065                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:4529                    WIN32(0x570)
                  dom/quota/ActorsParent.cpp:4532                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:6526                    WIN32(0x570)
      13       22 dom/indexeddb/ActorsParent.cpp:13441               NS_ERROR_FILE_NOT_FOUND # Bug 1689655
                  dom/indexeddb/ActorsParent.cpp:13491               NS_ERROR_FILE_NOT_FOUND
                  dom/indexeddb/ActorsParent.cpp:13092               NS_ERROR_FILE_NOT_FOUND
                  dom/indexeddb/ActorsParent.cpp:13051               NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:5187                    NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:5227                    NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:5231                    NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:5234                    NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:5054                    NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:5058                    NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:5062                    NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:5065                    NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:4529                    NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:4532                    NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:6526                    NS_ERROR_FILE_NOT_FOUND
      13       40 dom/cache/FileUtilsImpl.h:42                       NS_ERROR_FILE_NOT_FOUND # Bug 1689656
                  dom/cache/FileUtilsImpl.h:83                       NS_ERROR_FILE_NOT_FOUND
                  dom/cache/QuotaClient.cpp:109                      NS_ERROR_FILE_NOT_FOUND
                  dom/cache/QuotaClient.cpp:49                       NS_ERROR_FILE_NOT_FOUND
                  dom/cache/QuotaClient.cpp:52                       NS_ERROR_FILE_NOT_FOUND
                  dom/cache/QuotaClient.cpp:111                      NS_ERROR_FILE_NOT_FOUND
                  dom/cache/QuotaClient.cpp:423                      NS_ERROR_FILE_NOT_FOUND
                  dom/cache/QuotaClient.cpp:49                       NS_ERROR_FILE_NOT_FOUND
                  dom/cache/QuotaClient.cpp:52                       NS_ERROR_FILE_NOT_FOUND
                  dom/cache/QuotaClient.cpp:457                      NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:5187                    NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:5227                    NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:5231                    NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:5234                    NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:5054                    NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:5058                    NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:5062                    NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:5065                    NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:4529                    NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:4532                    NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:6526                    NS_ERROR_FILE_NOT_FOUND
      10     6350 dom/indexeddb/ActorsParent.cpp:13441               WIN32(0x570) # Bug 1690234
                  dom/indexeddb/ActorsParent.cpp:13491               WIN32(0x570)
                  dom/indexeddb/ActorsParent.cpp:13092               WIN32(0x570)
                  dom/indexeddb/ActorsParent.cpp:13051               WIN32(0x570)
                  dom/quota/ActorsParent.cpp:5187                    WIN32(0x570)
                  dom/quota/ActorsParent.cpp:5227                    WIN32(0x570)
                  dom/quota/ActorsParent.cpp:5231                    WIN32(0x570)
                  dom/quota/ActorsParent.cpp:5234                    WIN32(0x570)
                  dom/quota/ActorsParent.cpp:5054                    WIN32(0x570)
                  dom/quota/ActorsParent.cpp:5058                    WIN32(0x570)
                  dom/quota/ActorsParent.cpp:5062                    WIN32(0x570)
                  dom/quota/ActorsParent.cpp:5065                    WIN32(0x570)
                  dom/quota/ActorsParent.cpp:4529                    WIN32(0x570)
                  dom/quota/ActorsParent.cpp:4532                    WIN32(0x570)
                  dom/quota/ActorsParent.cpp:6526                    WIN32(0x570)
      10     2905 dom/quota/QuotaCommon.h:1174                       WIN32(0x570) # Bug 1671932 (This is not happening during a directory scan but when initializing a specific origin)
                  dom/indexeddb/ActorsParent.cpp:13491               WIN32(0x570)        
                  dom/indexeddb/ActorsParent.cpp:13092               WIN32(0x570)
                  dom/indexeddb/ActorsParent.cpp:13051               WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5187                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5227                    WIN32(0x570)
                  dom/quota/ActorsParent.cpp:5231                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5234                    WIN32(0x570)
                  dom/quota/ActorsParent.cpp:5054                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5058                    WIN32(0x570)
                  dom/quota/ActorsParent.cpp:5062                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5065                    WIN32(0x570)
                  dom/quota/ActorsParent.cpp:4529                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:4532                    WIN32(0x570)
                  dom/quota/ActorsParent.cpp:6526                    WIN32(0x570)        
       9     2575 dom/localstorage/ActorsParent.cpp:956              WIN32(0x570) # Bug 1671932 (This is not happening during a directory scan but when initializing a specific origin)
                  dom/localstorage/ActorsParent.cpp:8320             WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5187                    WIN32(0x570)
                  dom/quota/ActorsParent.cpp:5227                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5231                    WIN32(0x570)
                  dom/quota/ActorsParent.cpp:5234                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5054                    WIN32(0x570)
                  dom/quota/ActorsParent.cpp:5058                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5062                    WIN32(0x570)
                  dom/quota/ActorsParent.cpp:5065                    WIN32(0x570)
                  dom/quota/ActorsParent.cpp:4529                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:4532                    WIN32(0x570)
                  dom/quota/ActorsParent.cpp:6526                    WIN32(0x570)        
       8       13 dom/cache/QuotaClient.cpp:419                      NS_ERROR_FILE_NOT_FOUND # Bug 1689657
                  dom/cache/QuotaClient.cpp:49                       NS_ERROR_FILE_NOT_FOUND
                  dom/cache/QuotaClient.cpp:52                       NS_ERROR_FILE_NOT_FOUND
                  dom/cache/QuotaClient.cpp:457                      NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:5187                    NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:5227                    NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:5231                    NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:5234                    NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:5054                    NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:5058                    NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:5062                    NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:5065                    NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:4529                    NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:4532                    NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:6526                    NS_ERROR_FILE_NOT_FOUND
       8       47 dom/quota/ActorsParent.cpp:2557                    NS_ERROR_FAILURE # TODO Creating directory metadata
                  dom/quota/ActorsParent.cpp:11175                   NS_ERROR_FAILURE    
                  dom/quota/ActorsParent.cpp:10028                   NS_ERROR_FAILURE    
                  dom/quota/ActorsParent.cpp:11163                   NS_ERROR_FAILURE    
                  dom/quota/ActorsParent.cpp:4834                    NS_ERROR_FAILURE
                  dom/quota/ActorsParent.cpp:4907                    NS_ERROR_FAILURE    
                  dom/quota/ActorsParent.cpp:5018                    NS_ERROR_FAILURE
                  dom/quota/ActorsParent.cpp:5058                    NS_ERROR_FAILURE    
                  dom/quota/ActorsParent.cpp:5062                    NS_ERROR_FAILURE    
                  dom/quota/ActorsParent.cpp:5065                    NS_ERROR_FAILURE
                  dom/quota/ActorsParent.cpp:4529                    NS_ERROR_FAILURE    
                  dom/quota/ActorsParent.cpp:4532                    NS_ERROR_FAILURE
                  dom/quota/ActorsParent.cpp:6526                    NS_ERROR_FAILURE    
       7      139 dom/quota/QuotaCommon.cpp:194                      NS_ERROR_STORAGE_IOERR # TODO Reading QM Cache
                  dom/quota/ActorsParent.cpp:4477                    NS_ERROR_STORAGE_IOERR
                  dom/quota/ActorsParent.cpp:4494                    NS_ERROR_STORAGE_IOERR
                  dom/quota/ActorsParent.cpp:6526                    NS_ERROR_STORAGE_IOERR
       7      545 dom/quota/ActorsParent.cpp:462                     NS_ERROR_FILE_ACCESS_DENIED # Bug 1689636 (strange error at this point)
                  dom/quota/ActorsParent.cpp:4543                    NS_ERROR_FILE_ACCESS_DENIED
                  dom/quota/ActorsParent.cpp:6526                    NS_ERROR_FILE_ACCESS_DENIED
       6        7 dom/localstorage/ActorsParent.cpp:493              NS_ERROR_STORAGE_IOERR # Bug 1690291
                  dom/localstorage/ActorsParent.cpp:8359             NS_ERROR_STORAGE_IOERR
                  dom/localstorage/ActorsParent.cpp:8372             NS_ERROR_STORAGE_IOERR
                  dom/localstorage/ActorsParent.cpp:8380             NS_ERROR_STORAGE_IOERR
                  dom/quota/ActorsParent.cpp:5187                    NS_ERROR_STORAGE_IOERR
                  dom/quota/ActorsParent.cpp:5227                    NS_ERROR_STORAGE_IOERR
                  dom/quota/ActorsParent.cpp:5231                    NS_ERROR_STORAGE_IOERR
                  dom/quota/ActorsParent.cpp:5234                    NS_ERROR_STORAGE_IOERR
                  dom/quota/ActorsParent.cpp:5054                    NS_ERROR_STORAGE_IOERR
                  dom/quota/ActorsParent.cpp:5058                    NS_ERROR_STORAGE_IOERR
                  dom/quota/ActorsParent.cpp:5062                    NS_ERROR_STORAGE_IOERR
                  dom/quota/ActorsParent.cpp:5065                    NS_ERROR_STORAGE_IOERR
                  dom/quota/ActorsParent.cpp:4529                    NS_ERROR_STORAGE_IOERR
                  dom/quota/ActorsParent.cpp:4532                    NS_ERROR_STORAGE_IOERR
                  dom/quota/ActorsParent.cpp:6526                    NS_ERROR_STORAGE_IOERR
       6      147 dom/quota/ActorsParent.cpp:462                     NS_ERROR_FILE_NO_DEVICE_SPACE # Bug 1689636
                  dom/quota/ActorsParent.cpp:4543                    NS_ERROR_FILE_NO_DEVICE_SPACE
                  dom/quota/ActorsParent.cpp:6526                    NS_ERROR_FILE_NO_DEVICE_SPACE
       5       21 dom/quota/ActorsParent.cpp:4850                    NS_ERROR_FAILURE # TODO Reading directory metadata
                  dom/quota/ActorsParent.cpp:4909                    NS_ERROR_FAILURE
                  dom/quota/ActorsParent.cpp:5018                    NS_ERROR_FAILURE
                  dom/quota/ActorsParent.cpp:5058                    NS_ERROR_FAILURE
                  dom/quota/ActorsParent.cpp:5062                    NS_ERROR_FAILURE
                  dom/quota/ActorsParent.cpp:5065                    NS_ERROR_FAILURE
                  dom/quota/ActorsParent.cpp:4529                    NS_ERROR_FAILURE
                  dom/quota/ActorsParent.cpp:4532                    NS_ERROR_FAILURE
                  dom/quota/ActorsParent.cpp:6526                    NS_ERROR_FAILURE
       5        6 dom/quota/ActorsParent.cpp:462                     NS_ERROR_STORAGE_IOERR # Bug 1689636
                  dom/quota/ActorsParent.cpp:4543                    NS_ERROR_STORAGE_IOERR
                  dom/quota/ActorsParent.cpp:6526                    NS_ERROR_STORAGE_IOERR
       5     1751 dom/quota/QuotaCommon.cpp:180                      WIN32(0x570)  # Bug 1690234
                  dom/cache/QuotaClient.cpp:64                       WIN32(0x570)
                  dom/cache/QuotaClient.cpp:49                       WIN32(0x570)
                  dom/cache/QuotaClient.cpp:52                       WIN32(0x570)
                  dom/cache/QuotaClient.cpp:111                      WIN32(0x570)        
                  dom/cache/QuotaClient.cpp:423                      WIN32(0x570)
                  dom/cache/QuotaClient.cpp:49                       WIN32(0x570)        
                  dom/cache/QuotaClient.cpp:52                       WIN32(0x570)
                  dom/cache/QuotaClient.cpp:457                      WIN32(0x570)
                  dom/quota/ActorsParent.cpp:5187                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5227                    WIN32(0x570)
                  dom/quota/ActorsParent.cpp:5231                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5234                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5054                    WIN32(0x570)
                  dom/quota/ActorsParent.cpp:5058                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5062                    WIN32(0x570)
                  dom/quota/ActorsParent.cpp:5065                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:4529                    WIN32(0x570)
                  dom/quota/ActorsParent.cpp:4532                    WIN32(0x570)
                  dom/quota/ActorsParent.cpp:6526                    WIN32(0x570)        
       5        7 dom/quota/QuotaCommon.h:1174                       WIN32(0x570) # Bug 1690234
                  dom/cache/FileUtils.cpp:476                        WIN32(0x570)        
       4        4 dom/cache/FileUtilsImpl.h:42                       WIN32(0x570) # Bug 1690234
                  dom/cache/FileUtilsImpl.h:83                       WIN32(0x570)
                  dom/cache/QuotaClient.cpp:109                      WIN32(0x570)
                  dom/cache/QuotaClient.cpp:49                       WIN32(0x570)
                  dom/cache/QuotaClient.cpp:52                       WIN32(0x570)
                  dom/cache/QuotaClient.cpp:111                      WIN32(0x570)
                  dom/cache/QuotaClient.cpp:423                      WIN32(0x570)
                  dom/cache/QuotaClient.cpp:49                       WIN32(0x570)
                  dom/cache/QuotaClient.cpp:52                       WIN32(0x570)
                  dom/cache/QuotaClient.cpp:457                      WIN32(0x570)
                  dom/quota/ActorsParent.cpp:5187                    WIN32(0x570)
       4        6 dom/quota/QuotaCommon.h:1174                       WIN32(0x570) # Bug 1690234
                  dom/cache/FileUtilsImpl.h:83                       WIN32(0x570)        
                  dom/cache/QuotaClient.cpp:109                      WIN32(0x570)
                  dom/cache/QuotaClient.cpp:49                       WIN32(0x570)        
                  dom/cache/QuotaClient.cpp:52                       WIN32(0x570)
                  dom/cache/QuotaClient.cpp:111                      WIN32(0x570)        
       4        9 dom/cache/DBSchema.cpp:517                         NS_ERROR_FILE_READ_ONLY # TODO Strange error at this point, shouldn't we know this when opening the database? NOTE Stack is continued below
                  dom/cache/DBAction.cpp:209                         NS_ERROR_FILE_READ_ONLY
       4        9 dom/cache/QuotaClient.cpp:141                      NS_ERROR_FILE_READ_ONLY
                  dom/cache/QuotaClient.cpp:391                      NS_ERROR_FILE_READ_ONLY
                  dom/cache/QuotaClient.cpp:401                      NS_ERROR_FILE_READ_ONLY
                  dom/quota/ActorsParent.cpp:5187                    NS_ERROR_FILE_READ_ONLY
                  dom/quota/ActorsParent.cpp:5227                    NS_ERROR_FILE_READ_ONLY
                  dom/quota/ActorsParent.cpp:5231                    NS_ERROR_FILE_READ_ONLY
                  dom/quota/ActorsParent.cpp:5234                    NS_ERROR_FILE_READ_ONLY
                  dom/quota/ActorsParent.cpp:5054                    NS_ERROR_FILE_READ_ONLY
                  dom/quota/ActorsParent.cpp:5058                    NS_ERROR_FILE_READ_ONLY
                  dom/quota/ActorsParent.cpp:5062                    NS_ERROR_FILE_READ_ONLY
                  dom/quota/ActorsParent.cpp:5065                    NS_ERROR_FILE_READ_ONLY
                  dom/quota/ActorsParent.cpp:4529                    NS_ERROR_FILE_READ_ONLY
                  dom/quota/ActorsParent.cpp:4532                    NS_ERROR_FILE_READ_ONLY
                  dom/quota/ActorsParent.cpp:6526                    NS_ERROR_FILE_READ_ONLY

Initialization::Storage

 Clients Sessions Location                                           Result
      39    18152 dom/quota/ActorsParent.cpp:462                     NS_ERROR_FILE_CORRUPTED # Bug 1689636
                  dom/quota/ActorsParent.cpp:6303                    NS_ERROR_FILE_CORRUPTED
      36    49133 dom/quota/ActorsParent.cpp:576                                                                    # Bug 1689680
                  dom/quota/ActorsParent.cpp:6185                    NS_ERROR_FILE_CORRUPTED
      24     5573 dom/quota/ActorsParent.cpp:10848                   NS_ERROR_FILE_ACCESS_DENIED # TODO Upgrade 0.0 to 1.0, renaming origin directory
                  dom/quota/ActorsParent.cpp:10028                   NS_ERROR_FILE_ACCESS_DENIED
                  dom/quota/ActorsParent.cpp:10580                   NS_ERROR_FILE_ACCESS_DENIED
                  dom/quota/ActorsParent.cpp:5416                    NS_ERROR_FILE_ACCESS_DENIED
                  dom/quota/ActorsParent.cpp:5440                    NS_ERROR_FILE_ACCESS_DENIED
                  dom/quota/ActorsParent.cpp:6147                    NS_ERROR_FILE_ACCESS_DENIED
      21    52789 dom/quota/ActorsParent.cpp:546                     NS_ERROR_FILE_CORRUPTED # Bug 1689680
                  dom/quota/ActorsParent.cpp:5985                    NS_ERROR_FILE_CORRUPTED
                  dom/quota/ActorsParent.cpp:6208                    NS_ERROR_FILE_CORRUPTED
      11       12 dom/quota/ActorsParent.cpp:576                                      # Bug 1689680
       9        9 dom/quota/ActorsParent.cpp:5817                    NS_ERROR_FILE_IS_LOCKED # TODO CreateLocalStorageArchiveConnectionFromWebAppsStore, renaming temp file to archive, probably (implicit) close failed before
                  dom/quota/ActorsParent.cpp:5931                    NS_ERROR_FILE_IS_LOCKED
                  dom/quota/ActorsParent.cpp:6176                    NS_ERROR_FILE_IS_LOCKED
       9        9 dom/quota/ActorsParent.cpp:546                     NS_ERROR_FILE_CORRUPTED # Bug 1689680
       8      226 dom/quota/ActorsParent.cpp:6070                    NS_ERROR_STORAGE_IOERR # Bug 1690291
       7        9 dom/quota/ActorsParent.cpp:546                     NS_ERROR_FILE_CORRUPTED # Bug 1689680
                  dom/quota/ActorsParent.cpp:6200                    NS_ERROR_FILE_CORRUPTED
       7      418 dom/quota/ActorsParent.cpp:504                     NS_ERROR_STORAGE_IOERR # Bug 1690291
                  dom/quota/ActorsParent.cpp:5761                    NS_ERROR_STORAGE_IOERR
                  dom/quota/ActorsParent.cpp:5931                    NS_ERROR_STORAGE_IOERR
                  dom/quota/ActorsParent.cpp:6176                    NS_ERROR_STORAGE_IOERR
       6        9 dom/quota/ActorsParent.cpp:546                     NS_ERROR_FILE_CORRUPTED # Bug 1689680
                  dom/quota/ActorsParent.cpp:5985                    NS_ERROR_FILE_CORRUPTED
       6        6 dom/quota/ActorsParent.cpp:462                     NS_ERROR_FILE_CORRUPTED # Bug 1689636
       5        6 dom/quota/ActorsParent.cpp:5901                    NS_ERROR_STORAGE_IOERR # Bug 1690291
                  dom/quota/ActorsParent.cpp:6176                    NS_ERROR_STORAGE_IOERR
       4       11 dom/quota/ActorsParent.cpp:5901                    NS_ERROR_STORAGE_BUSY
                  dom/quota/ActorsParent.cpp:6176                    NS_ERROR_STORAGE_BUSY
Depends on: 1690234
Depends on: 1690291
Depends on: 1690560

Errors from the weekend Nightlies so far, after some fixes landed:

Initialization::TemporaryStorage

 Clients Sessions Location                                           Result              
       1        1 dom/cache/QuotaClient.cpp:419                      NS_ERROR_FILE_NOT_FOUND # Bug 1687321 
                  dom/cache/QuotaClient.cpp:49                       NS_ERROR_FILE_NOT_FOUND
                  dom/cache/QuotaClient.cpp:52                       NS_ERROR_FILE_NOT_FOUND
                  dom/cache/QuotaClient.cpp:457                      NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:5296                    NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:5336                    NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:5163                    NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:5167                    NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:4667                    NS_ERROR_FILE_NOT_FOUND
                  dom/quota/ActorsParent.cpp:6554                    NS_ERROR_FILE_NOT_FOUND
       1        1 dom/cache/FileUtilsImpl.h:42                       WIN32(0x570)        # Bug 1687321 
                  dom/cache/FileUtilsImpl.h:83                       WIN32(0x570)        
                  dom/cache/QuotaClient.cpp:109                      WIN32(0x570)        
                  dom/cache/QuotaClient.cpp:49                       WIN32(0x570)        
                  dom/cache/QuotaClient.cpp:52                       WIN32(0x570)        
                  dom/cache/QuotaClient.cpp:111                      WIN32(0x570)        
                  dom/cache/QuotaClient.cpp:423                      WIN32(0x570)        
                  dom/cache/QuotaClient.cpp:49                       WIN32(0x570)        
                  dom/cache/QuotaClient.cpp:52                       WIN32(0x570)        
                  dom/cache/QuotaClient.cpp:457                      WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5296                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5336                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5163                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:5167                    WIN32(0x570)        

Both of these issues have already fixes that have not landed yet.

Errors from the weekend Nightlies:

Initialization::TemporaryStorage

 Clients Sessions Location                                           Result              
       2        2 dom/quota/ActorsParent.cpp:552                     NS_ERROR_FILE_NO_DEVICE_SPACE (TS.A)
                  dom/quota/ActorsParent.cpp:555                     NS_ERROR_FILE_NO_DEVICE_SPACE
                  dom/quota/ActorsParent.cpp:4200                    NS_ERROR_FILE_NO_DEVICE_SPACE
                  dom/quota/ActorsParent.cpp:6156                    NS_ERROR_FILE_NO_DEVICE_SPACE
       1        4 dom/quota/QuotaCommon.cpp:201                      NS_ERROR_FILE_CORRUPTED (TS.B)
                  dom/localstorage/ActorsParent.cpp:2881             NS_ERROR_FILE_CORRUPTED
                  dom/localstorage/ActorsParent.cpp:2882             NS_ERROR_FILE_CORRUPTED
                  dom/localstorage/ActorsParent.cpp:8139             NS_ERROR_FILE_CORRUPTED
                  dom/localstorage/ActorsParent.cpp:8147             NS_ERROR_FILE_CORRUPTED
                  dom/localstorage/ActorsParent.cpp:8155             NS_ERROR_FILE_CORRUPTED
                  dom/quota/ActorsParent.cpp:4810                    NS_ERROR_FILE_CORRUPTED
                  dom/quota/ActorsParent.cpp:4882                    NS_ERROR_FILE_CORRUPTED
                  dom/quota/ActorsParent.cpp:4678                    NS_ERROR_FILE_CORRUPTED
                  dom/quota/ActorsParent.cpp:4701                    NS_ERROR_FILE_CORRUPTED
                  dom/quota/ActorsParent.cpp:4186                    NS_ERROR_FILE_CORRUPTED
                  dom/quota/ActorsParent.cpp:6156                    NS_ERROR_FILE_CORRUPTED
       1        3 dom/localstorage/ActorsParent.cpp:973              WIN32(0x570)        (TS.C)
                  dom/localstorage/ActorsParent.cpp:8095             WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:4810                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:4882                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:4678                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:4701                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:4186                    WIN32(0x570)        
                  dom/quota/ActorsParent.cpp:6156                    WIN32(0x570)        

Initialization::Storage

 Clients Sessions Location                                           Result              
       3        4 dom/quota/ActorsParent.cpp:5932                    NS_ERROR_STORAGE_BUSY (S.A)

Some details:

  • TS.A
    • Module: Quota Manager
    • Description: In InvalidateCache, deleting the cache failed with NS_ERROR_FILE_NO_DEVICE_SPACE, subsequently also setting the invalid flag failed, probably because the physical device has no space left for writing the journal.
    • Permanence: Temporary, but might require user intervention to resolve
    • Mitigation options: Evict some origins, Notify storage pressure
  • TS.B
    • Module: Local Storage
    • Description: After loading the usage file failed, we encounter an NS_ERROR_FILE_CORRUPTED error in GetUsage.
    • Permanence: Probably permanent
    • Mitigation options: Clear affected origin
  • TS.C
    • Module: Local Storage
    • Description: Checking existence of the usage file fails due to a ERROR_FILE_CORRUPT error.
    • Permanence: Unknown
    • Mitigation options: Treat usage file as non-existent (but probably updating the usage file later will fail as well)
  • S.A
    • Module: Quota Manager
    • Description: Opening the storage database file fails with NS_ERROR_STORAGE_BUSY.
    • Permanence: Probably temporary
    • Mitigation options: Retry

(In reply to Simon Giesecke [:sg] [he/him] from comment #33)

Some details:

  • TS.A
    • Module: Quota Manager
    • Description: In InvalidateCache, deleting the cache failed with NS_ERROR_FILE_NO_DEVICE_SPACE, subsequently also setting the invalid flag failed, probably because the physical device has no space left for writing the journal.
    • Permanence: Temporary, but might require user intervention to resolve
    • Mitigation options: Evict some origins, Notify storage pressure

Yes, it seems we can move https://searchfox.org/mozilla-central/rev/f47a4b67643b3048ef9a2e2ac0c34edf6d1ebff3/dom/quota/ActorsParent.cpp#4199-4201 to https://searchfox.org/mozilla-central/rev/f47a4b67643b3048ef9a2e2ac0c34edf6d1ebff3/dom/quota/ActorsParent.cpp#6166
But I will have to double check during review.

  • TS.B
    • Module: Local Storage
    • Description: After loading the usage file failed, we encounter an NS_ERROR_FILE_CORRUPTED error in GetUsage.
    • Permanence: Probably permanent
    • Mitigation options: Clear affected origin

Yes, luckily, the temporary storage is only being initialized (not in use) so we can just add orElse here: https://searchfox.org/mozilla-central/rev/f47a4b67643b3048ef9a2e2ac0c34edf6d1ebff3/dom/quota/ActorsParent.cpp#4672
The orElse will remove entire origin directory.
However, I wonder if we should also use the new CallWithDelayedRetriesIfAccessDenied method there.

I'll take a look at the rest in a minute.

(In reply to Simon Giesecke [:sg] [he/him] from comment #33)

  • TS.C
    • Module: Local Storage
    • Description: Checking existence of the usage file fails due to a ERROR_FILE_CORRUPT error.
    • Permanence: Unknown
    • Mitigation options: Treat usage file as non-existent (but probably updating the usage file later will fail as well)

Ok, this is again FS corruption and we don't have a good solution for that yet...

  • S.A
    • Module: Quota Manager
    • Description: Opening the storage database file fails with NS_ERROR_STORAGE_BUSY.
    • Permanence: Probably temporary
    • Mitigation options: Retry

Yes, we actually do that in IDB (but because some of our threads may be causing it), for example https://searchfox.org/mozilla-central/rev/f47a4b67643b3048ef9a2e2ac0c34edf6d1ebff3/dom/indexedDB/ActorsParent.cpp#6886

It seems that CallWithDelayedRetriesIfAccessDenied needs to be generalized even more.

Depends on: 1694365
Depends on: 1647323
Depends on: 1694935

Updated script with more recent build id

Attachment #9198601 - Attachment is obsolete: true

Errors from the weekend Nightlies:
build ids: 20210227094458/20210227214359/20210228103709/20210228215216
changeset: https://hg.mozilla.org/mozilla-central/rev/1add9a1bbfacd4ff1bee961fa8886b8cd0566d23
query: https://sql.telemetry.mozilla.org/queries/78426/source

Initialization::TemporaryStorage

 Clients Sessions Location                                           Result              
       4        6 dom/cache/FileUtils.cpp:487                        NS_ERROR_FILE_DIR_NOT_EMPTY
       3       29 dom/cache/FileUtils.cpp:487                        NS_ERROR_FILE_ACCESS_DENIED
       1       30 dom/indexeddb/ActorsParent.cpp:5794                NS_ERROR_FILE_ACCESS_DENIED
                  dom/indexeddb/ActorsParent.cpp:12823               NS_ERROR_FILE_ACCESS_DENIED
                  dom/indexeddb/ActorsParent.cpp:12827               NS_ERROR_UNEXPECTED 
       1        1 dom/quota/ActorsParent.cpp:544                     NS_ERROR_FILE_ACCESS_DENIED
       1        2 dom/cache/FileUtils.cpp:519                        NS_ERROR_FILE_ACCESS_DENIED
       1        1 dom/quota/QuotaCommon.h:1212                       WIN32(0x570)
                  dom/cache/QuotaClient.cpp:52                       WIN32(0x570)
                  dom/cache/QuotaClient.cpp:478                      WIN32(0x570)
                  dom/quota/ActorsParent.cpp:4825                    WIN32(0x570)
                  dom/quota/ActorsParent.cpp:4897                    WIN32(0x570)
                  dom/quota/ActorsParent.cpp:4693                    WIN32(0x570)
                  dom/quota/ActorsParent.cpp:4716                    WIN32(0x570)
                  dom/quota/ActorsParent.cpp:4187                    WIN32(0x570)
                  dom/quota/ActorsParent.cpp:6172                    WIN32(0x570)

I'll double check if I got the "not_propagate" list right.

It seems, all these errors (except the last one) are related to the interference with anti-viruses, so we should probably add the retrying there.

Depends on: 1700915
Depends on: 1702411
Depends on: 1703840
No longer depends on: 1703840
You need to log in before you can comment on or make changes to this bug.