1953860 - QM: L2 quota info cache

Status: ASSIGNED

(Core :: Storage: Quota Manager, task, P1)

Description

[Jan Varga [:janv]]

Even when async storage initialization is done, when the build ID changes (e.g., for nightly users or release users upgrading to a new version) or after a crash occurs, the whole storage currently needs to be scanned because the existing quota info cache is ignored/invalidated. This can lead to high CPU and disk activity for several minutes.

Previously, we considered only two options:

• Using the existing quota info cache, which is stored in-memory during runtime and then serialized and persisted in a database during shutdown, so it can be deserialized and restored on the next startup. However, this approach is unreliable in cases of unclean shutdowns (such as crashes or forced app terminations).
• Performing a full scan of the entire storage to reconstruct the usage data from scratch when the existing quota info cache is unavailable.

However, we now plan to introduce an L2 quota info cache that leverages existing per-origin directory metadata files to store:

• Usage information.
• A new "accessed" flag, which helps determine whether a full scan is needed.

Proposed Approach:

• Before any access to an origin directory, the "accessed" flag in its metadata file will be set to true.
• On the next startup, the flag will be read:
  • If false, the cached usage information will be trusted instead of performing a deep scan.
  • If true, a full scan of the origin directory will be performed.
    • The new usage information will be stored back in the metadata file, and the "accessed" flag will be reset to false.
• This ensures that origins that haven't been modified retain their cached usage information, significantly reducing unnecessary full scans.

Importance of L2 quota info cache on Android:

• On Android, the existing quota info cache is often unusable due to the way apps are "closed".
  • Apps are frequently force-stopped by the system rather than cleanly shut down, leading to unclean shutdowns.
  • This causes the existing quota info cache to be lost or become unreliable, forcing a full scan on every startup.
• The L2 quota info cache will mitigate this issue by persisting usage data across restarts, significantly reducing the frequency of full scans on Android.

Expected Benefits:

• Reduces unnecessary CPU and disk activity on startup.
• Minimizes full scans by relying on L2 quota info cache whenever possible.
• Ensures correctness by only scanning origins that were actively accessed.
• Greatly improves performance on Android, where the existing quota info cache is unreliable due to frequent unclean shutdowns.