Open Bug 1395705 Opened 7 years ago Updated 8 days ago

Figure out what storage can be moved to 'Local' path on Windows and what can remain in 'Roaming'

Categories

(Core :: Storage: Quota Manager, defect, P3)

Product:
Core
Component:
Storage: Quota Manager
Version:
55 Branch
Type:
defect

Tracking

()

Status:
UNCONFIRMED

People

(Reporter: rainer.meier, Unassigned)

References

Details

(Whiteboard: DWS_NEXT) 

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:55.0) Gecko/20100101 Firefox/55.0
Build ID: 20170824053622

Steps to reproduce:

Run Firefox on Windows (Currently using version 55.0.3). Brwowse some pages like youtube using offline storage.


Actual results:

"%AppData%\Mozilla\Firefox\Profiles\<profile>.default\storage" gets populated with lots of "cache type" data which can be safely removed.


Expected results:

As the "storage" folder contains mainly volatile data it should be stored where caches are stored. For example at "%LocalAppData%\Mozilla\Firefox\Profiles\<profile>.default\storage". Especially as the "storage" folder will build up hundreds of MB in volatile data.

Keep in mind that the %AppData% folder in professional environment might be roamed (ie. roaming profiles feature). All data in %AppData% by default is synchronized to a central roaming profile store. So when the user logs off the profile is stored on the share and when the user logs on to a different host the profile is downloaded.
Local Data should be stored in %LocalAppData% as this gets not roamed (ie. not synced to file share and downloaded on raoming host).

With the current implementation users complain about logon/logoff times being very long as the host synchronizes the Mozilla folder from the share with hundreds or thousands of small files which takes ages.

The only work-around known to me is to exclude %AppData%\Mozilla from roaming but this leads to bad user experience as users will start with a new (empty) Firefox profile on a new host and the profiles are not roamed when they log on to different host.
Despite this issue Firefox roaming works very well. Users can log on to any host in the network and get their profile including settings, themes, extensions and passwords synchronized. But synchronizing a cache folder is a no-go for this type of roaming profiles. Unfortunately Microsoft does not include any possibility to exclude wildcards from roaming. It would be possible to exclude "%AppData%\Mozilla\Firefox\Profiles\<profile>.default\storage" but unfortunately the <profile> string is random and therefore this approach does not work without knowing the directory name. So administrators are locked.

Proposed workaround:
Firefox should store the storage folder where temporary files are stored (these ones are properly located under %LocalAppData% folder).
In addition it might helpful to have a setting available to redirect the storage folder (e.g. relative to %AppData%\Mozilla\profiles\<profile>.default) so it could be configured to something like "[ProfD]../../../../Local/Thunderbird/Profiles/storage" like I did in Thunderbird for the IMAP cache folders. Unfortunately I was unable to find any information about the storage folder and potential configuration yet.

The current implementation render Firefox unusable in corporate environments with roaming profiles and settings synchronization. Please seriously consider fixing this soon; I would like to keep Firefox installed in those environments as using IE or chrome is not my favourite solution but increasing customer complaints and network load force me to think about this :(.
Component: Untriaged → Startup and Profile System
Product: Firefox → Toolkit
Not sure about the right component, but this seems like Core stuff.
Component: Startup and Profile System → DOM
Product: Toolkit → Core
The location of these files is determine by our quota system.  Jan, what do you think?
Component: DOM → DOM: Quota Manager
Flags: needinfo?(jvarga)
Jim, do you know about profile location stuff like this?
Flags: needinfo?(jmathies)
Priority: -- → P2
The argument here is valid, cache data shouldn't be in Roaming AppData. I'm not familiar with the data stored here, but looking at it locally, it sure seems like this isn't best suited to the roaming part of the profile. We store most cached data over in the Local path.

http://searchfox.org/mozilla-central/rev/2ef8bd8a46a02c68ddbb1d5f25fa254dd7be1fbd/xpcom/io/nsAppDirectoryServiceDefs.h#67
Flags: needinfo?(jmathies)
Summary: Profile "storage" stored in wrong location on Windows → Figure out what storage can be moved to 'Local' path on Windows and what can remain in 'Roaming'
What do other browsers do here?
Well, Chrome is pretty special here as it stores it's complete installation including binaries in %LocalAppData% and therefore does not roam. However Chrome tries to put all data in the cloud and therefore the settings will just be downloaded via cloud if you log on to a different machine.

IE/Edge store settings in HKCU portion of the registry or in %AppData% (Favorites). Most temporary and volatile data is stored in "%LocalAppData%\Microsoft\Internet Explorer" here and does not unnecessary roam. This includes caches and indexed DB.

Many people did not completely understand the purpose of splitting up application data into roaming and local. This concept basically splits up the data into 3 categories:

- Temporary: Usually temporary data is stored at %TEMP% which is a sub-folder of %LocalAppData% and contains all sorts of temporary and volatile data. Temporary files by definition can be erased and will/can be unavailable on next application startup.

- Local profile data: Data which is permanently stored on the local machine but is basically volatile/replacable data. Most applications just rebuild data stored in %LocalAppData% when deleted. Typical examples are web caches. Caches are nice to have but they can be re-built any time without losing data. As long as the user logs on to the same machine again the local data will remain on the system (except if policy to erase profile on logout is set). So in >90% of logins when the user is using the same machine every day the local profile data remains available as a persistent storage.

- Roaming profile data: Data which is supposed to follow the user across system boundaries. In corporate environments roaming profiles (or application virtualization) usually syncs the %AppData% (roaming) content to the server. When logging on to the same machine again the profile is synchronized. In case the user logs on to a different machine the %AppData% contents are downloaded from the server (note: %LocalAppData% is not synced to servers and therefore not available at logon to a new host). So applications can expect %AppData% to be fully populated while %LocalAppData% might be lost at any time


I think web application caches and web caches in general are of class "local profile data". If the user logs on to the same system it's nice to have them available but it does not matter if they are deleted. If deleted it might affect first access to a page (cache) or offline capabilities but no user data is lost.
I think implementation-wise Firefox behaves properly. If the storage folder is deleted, it's just re-created and populated. It is just stored in wrong location where roaming profiles force Windows to synchronize the storage folder to the file server. As there might be a lot of files in this store this can lead to lots of network traffic on logon/logoff. If Firefox would store the files in %LocalAppData% just like the web cache it will still be available as long as the user logs on to the same node (which also applies to non-corporate environments with local profiles). Only in case the user roams between multiple machines the %LocalAppData% contents are lost and need to be rebuilt.

Firefox should for sure still store persistent user configuration data in %AppData%. This includes bookmarks, browsing history, prefs.js and add-on configuration files.

Regarding extensions: They used to be a problem for roaming profiles too as they often cluttered the profile extensions/ directory. However most users don't deploy lots of plugins and nowadays they are packed in xpi package where most packages are less than 1MB in size. Syncing big XPI files is less of a problem on logout since the files are not changing and checking timestamp or checksum of a few large files is quickly done. Synchronizing thousands of small files however is a problem on typical SMB shares (mainly due to latency).
Theoretically it would be possible to move extensions to %LocalAppData% too but then Firefox would have to re-download them on first run when %LocalAppData% is cleared. Moreover Firefox cannot download all extensions as some might not be on addons.mozilla.org but installed manually. At least those would have to be stored in the %AppData% part in order not to get lost when roaming. So I belive extensions/ is placed correctly unless split up into remote extensions (official "store", could be re-downloaded and therefore go to local appdata) and local extensions (XPI locally added, would have to go to roaming). I recommend not to change anything here.

Also the datareporting/ folder is a folder which should go to %LocalAppData% as the data stored there as of my knowledge is not valuable for the user and it's ok to delete/lose when roaming.

Same for crashes/ folder. There is no point in syncing crashdumps from one host to another host.

Also the gmp* folders should not be stored in %AppData% from my point of view as they contain the cisco H.264 plugin which is downloaded automatically. It's easy just to re-download it if it vanishes from %LocalAppData% if the user eventually roamed. If the user roams between 2 machines it would be downloaded once on each machine only.
Another issue here is that in corporate environments loading dll's from user directories is often forbidden for malware protection but I understood it's a license issue why those libs are not included in Firefox installer and placed in %ProgramFiles% folder.


The jetpack/ folder is placed correclty I believe. Jetpack extensions are similar to legacy extensions in extensions/ folder. They might be re-downloaded but it might make sense to roam then. For sure the jetpack extension data/settings should roam.

Telemetry data should always be stored in %LocalAppData%. This is not data the user is interested in syncing and storing on the server to roam across machines. It might even make more sense to collect telemetry per machine. Not sure if it makes sense that Firefox telemetry data is often synced to different hosts. If you collect performance data there it makes absolute no sense to roam them to different hardware.


There might be more components in Firefox which need a similar decision and you might of course have different opinion on each of them. But you probably got my view on %AppData% and %LocalAppData% separation.
Note, if quota /storage directory is moved to a Local directory, then we must also move the serviceworker registrar data file.  There may be other per-origin storage (localstorage, cookies, etc) that should also be moved.  We don't want to "delete" only part of a site's data if the user moves to another machine on the same intranet.
I've had requests via the policy work to put more stuff in Local instead of Roaming. See:

https://github.com/mozilla/policy-templates/issues/121

'We had problems with archive folders from pings - this is what i remember. These where getting bigger and bigger. A quick check showed that toolkit.telemetry.archive.enabled is still true even if telemetry is diabled."

"We are using roaming profiles and those folder have a negative impact on opening/closing account session.

It have a negative impact too when we must backup the server that store roaming profiles as there are a huge amount of small files (that we do not need) to read.

Anyway, why mozilla do not "move" those folders on "local" (C:\Users<username>\AppData\Local\Mozilla\Firefox\Profiles\xxxxxx.default) like "OfflineCache" folder ?"

So it definitely seems that we should be putting more non volatile stuff in Local versus Roaming.
adding to DWS_NEXT
Flags: needinfo?(jvarga)
Whiteboard: DWS_NEXT

:mconca, needinfo-ing you for product/feature processing of this request.

As I understand the problem, there are 2 use-cases that would like the storage moved to other places:

  1. Simplifying backups. Bug 1339696 primarily seems to be requesting a workaround for a backup program, but in general I understand that historically backup programs frequently do not have a fun time dealing with SQLite databases unless they have low-level file-system access so that they only need to record page-level deltas to databases.
  2. For actually roaming profiles, there can be a lot of storage usage in the profile.

I understand the concern on this bug to be primarily about the latter, which I think is likely considered a (Windows) enterprise-style feature.

I understand the current scenario to be that we do store HTTP cache data as local, non-roamed storage. However, our content-visible storage (cookies, LocalStorage, IndexedDB, Cache API, ServiceWorkers, etc.) is stored in roaming storage.

From a product perspective, I think the key questions are how/whether to suppose this use-case. The primary engineering concerns are that this is about content-visible storage, not a best-effort cache. This means that if using a page that uses storage APIs to store data, like to store a draft of an email message locally, then that user data won't roam between machines. This is most similar to separate Firefox installations synchronized via sync. This also has privacy implications as all of our mechanisms for clearing/forgetting site data are strictly local. (And indeed, synchronizing a "delete the data for example.com" is potentially contrary to existing privacy goals, as a tombstone to explicitly clear a site is a record of that site.)

It's possible that there's another alternative to help improve the use-case for this scenario beyond moving storage entirely from roaming to non-roaming. Assuming the roaming profile mechanism doesn't copy all of the data all of the time, but only (potentially naive) deltas, we might be able to re-architect so that Firefox profiles cause smaller deltas to be observed by the roaming mechanism.

It's also possible that we could tune and/or help tune eviction heuristics for the QuotaManager-managed storage so that there is less data persistently kept around in general. Currently QuotaManager limits site storage based on free disk space by default, evicting data only when we QM-managed storage crosses a free-space related threshold. Other heuristics could trigger eviction more aggressively.

Flags: needinfo?(mconca)

Copying a bit from https://bugzilla.mozilla.org/show_bug.cgi?id=1554460: https://snag.gy/7Ri52b.jpg

:mconca On my local machine, personal install of Firefox it appears that the roaming data is predominantly cache and temp files, at least according to the file/folder names. Digging a bit in to the storage folder (see screenshot above), I see default and temporary. Right off the bat temporary should not be roaming. Glancing through default, it appears to be a folder for each site, with a large cache folder inside each. cache files should all be in %LOCALAPPDATA%, as they don't need to roam across a network. I also see extensions and backups in there, which are similarly not user-config, and thus not appropriate for %APPDATA%, as well as Crash Reports, which are definitely not something that should roam/be backed up.

After spending a few minutes digging through the profile folder, everything I see that is of significant size is not appropriate for roaming storage. I suspect somewhere in there are config files that are appropriate for roaming (like bookmarks, preferences, etc.) but those files are likely measured in kilobytes.

If you are going to be storing browser local storage in roaming, then I think there needs to be a much more aggressive eviction policy. Just because computer A has 750GB of free space doesn't mean that computer B that it is roaming to has 750GB of free space, not to mention that roaming GBs of data is incredibly destructive to a network or backup system, even if it is doing delta syncs.

My recommendation would be to either move everything to %LOCALAPPDATA% if splitting up the data appropriately is too hard or not possible, or extracting out everything except small config type data like bookmarks and preferences (not browsing history, cookies, local storage, caches, temp files, etc.) into %LOCALAPPDATA%.

While the idea of roaming/backing up all of the user's various transient data sounds cool, I don't think it is reasonable for most of the world (data caps, limited bandwidth, etc.). I think the question that should be asked is, "Would we be willing to provide cloud storage for this data and/or would users be willing to pay for cloud storage for this data?" If not, then it probably doesn't belong in the roaming profile and instead should be stored on local disk only. As a user, I can assert with confidence that I would not be willing to pay for cloud storage of Browser Local Storage, unbounded history, caches, etc. I would be willing to pay for cloud storage of bookmarks, browser settings, current tabs (and maybe very recently closed tabs) extension manifests (but not extension data, I can re-download that).

(In reply to Micah Zoltu from comment #13)

:mconca On my local machine, personal install of Firefox it appears that the roaming data is predominantly cache and temp files, at least according to the file/folder names. Digging a bit in to the storage folder (see screenshot above), I see default and temporary. Right off the bat temporary should not be roaming. Glancing through default, it appears to be a folder for each site, with a large cache folder inside each. cache files should all be in %LOCALAPPDATA%, as they don't need to roam across a network. I also see extensions and backups in there, which are similarly not user-config, and thus not appropriate for %APPDATA%, as well as Crash Reports, which are definitely not something that should roam/be backed up.

To clarify:

  • "storage/temporary/" is only used by the asm.js cache and that's going away. (Previously we also allowed IndexedDB to specify explicit storage levels, but that functionality has been fully deprecated.)
  • storage/default/ORIGIN/cache/ is the Cache API (https://w3c.github.io/ServiceWorker/#cache-objects) that is exposed to content, as well as the underlying storage mechanism for ServiceWorkers. Per spec and practical considerations, it needs to be coherent with IndexedDB state and all data hung off of ServiceWorker registrations. Ideally it's also consistent with LocalStorage.
  • Although I think the point above about extensions is about the XPIs themselves, note that extension origins stored under storage/default do include extension configuration data (especially since bug 1474562 but even before that they could use normal storage APIs). That said, WebExtensions does have a concept of local/synchronized data and Firefox I believe is in the process of implementing support for that (or recently implemented it?)

(In reply to Andrew Sutherland [:asuth] (he/him) from comment #12)

From a product perspective, I think the key questions are how/whether to suppose this use-case. The primary engineering concerns are that this is about content-visible storage, not a best-effort cache. This means that if using a page that uses storage APIs to store data, like to store a draft of an email message locally, then that user data won't roam between machines. This is most similar to separate Firefox installations synchronized via sync. This also has privacy implications as all of our mechanisms for clearing/forgetting site data are strictly local. (And indeed, synchronizing a "delete the data for example.com" is potentially contrary to existing privacy goals, as a tombstone to explicitly clear a site is a record of that site.)

I think there are multiple reasons to pursue the idea of storing most, perhaps all, content-visible data locally in the non-roamed storage area.

  1. Privacy - As mentioned, having local site data automatically migrated between machines by the OS probably isn't ideal from a data privacy point-of-view, particularly in the "delete all of my data" use case. Limiting content-visible data to the local machine only, where the browser can confidently manage/remove that data, seems more in-line with Mozilla's mission to protect user data.
  2. Fx Accounts - Mozilla is encouraging more and more users to create a Firefox account, which allows the browser to synchronize data via cloud storage and is more in line with what Chrome does. Already, having an account synchronizes bookmarks and extension data (for those extensions that use the storage.sync API), and it is easy to imagine the service being able to sync other types of data. It also forces the privacy question - if Firefox isn't comfortable sync'ing certain data via Accounts, it probably shouldn't be occuring via roaming, either.
  3. Platform Consistency - Only Windows users are seeing the "benefit" of a roamed profile. While that represents the majority of Firefox users, particularly for enterprise, it creates an inconsistent experience across operating systems. Using Fx Accounts as the synchronization mechanism, however, benefits all platform, regardless of OS, and is inclusive of mobile.
  4. Performance - While this was the original reason this bug was filed, I actually think it is the least important from a strategic point-of-view. Protecting user privacy via Fx Accounts is the leading motivation, with platform consistency and performance benefits making the case for doing this even stronger.

It's possible that there's another alternative to help improve the use-case for this scenario beyond moving storage entirely from roaming to non-roaming. Assuming the roaming profile mechanism doesn't copy all of the data all of the time, but only (potentially naive) deltas, we might be able to re-architect so that Firefox profiles cause smaller deltas to be observed by the roaming mechanism.

Sounds challenging, and doesn't address any of the above-mentioned issues except performance. ROI is low.

It's also possible that we could tune and/or help tune eviction heuristics for the QuotaManager-managed storage so that there is less data persistently kept around in general. Currently QuotaManager limits site storage based on free disk space by default, evicting data only when we QM-managed storage crosses a free-space related threshold. Other heuristics could trigger eviction more aggressively.

Also only addresses performance, and as pointed out in comment 13, different computers will have different amounts of free disk space.

Flags: needinfo?(mconca)

Thank you for the detailed and quick feedback!

(In reply to Mike Conca [:mconca] from comment #15)

I think there are multiple reasons to pursue the idea of storing most, perhaps all, content-visible data locally in the non-roamed storage area.

I'm not sure I parsed this right. I understand there to be a few levels here you might mean:

  • A. Each computer has its own non-roamed profile. Any information sharing happens via Fx Account.
  • B. The profile roams, but the roaming data is only history / passwords, just like Fx Account would sync. Cookies and sessionstore data doesn't roam, so ex: gmail logins are logically distinct as far as google is concerned.
  • C. Option B plus cookies and sessionstore roam so that the user sees their session do a best-effort migration. This might cause some weirdness for sites where ServiceWorkers are involved and their behavior differs from hitting the URL without the seviceworker installed.
  1. Privacy - As mentioned, having local site data automatically migrated between machines by the OS probably isn't ideal from a data privacy point-of-view, particularly in the "delete all of my data" use case. Limiting content-visible data to the local machine only, where the browser can confidently manage/remove that data, seems more in-line with Mozilla's mission to protect user data.

To be clear, my concern about user expectations for their data concerns the following situation if we're talking about option B or C above and we make only simple changes to our implementation:

  1. Users browses engagement-ring.example.com on computer A. This creates a roamed history entry and non-roamed site data stored under storage/default/http+++example.com. Let's assume the site uses IndexedDB to offline products the user looked at.
  2. User browses engagement-ring.example.com on computer B via history entry, which also creates non-roamed site data.
  3. Still on computer B, user realizes they don't want that in their history popping up, so they use "forget about this site" to clear both the roamed history entry for engagement-ring.example.com and the non-roamed site data. As far as they can see, they cleared all that data.
  4. User goes back to computer A. If they go to about:preferences, they see there is still site data for engagement-ring.example.com. And/or if they go back to the site, they can see it clearly has remembered data about the user.

In my mental model of user expectations, I think they would find step 4 surprising.

Having said that, there are ways to make it so that at step 4 when the user starts up firefox on computer A so that:

  • The data for engagement-ring.example.com is stored in such a way that there's no data directly in the clear on disk.
  • The data is automatically purged in the background.

While that would involve some engineering effort, we are potentially already going to be doing things related to this in our work for SessionStorage and IndexedDB-on-PrivateBrowsing, so it's a direction we can head in if we know that's where we want to go with the product.

(In short, the idea would be that the data in storage/default/ORIGIN would instead be storage/by-uuid/UUID where the mapping from origin-to-uuid happens in a database that is roamed. The data on disk would also be encrypted using information found in the database. So when we startup on computer A, we would no longer have the mapping for engagement-ring.example.com and would discard it.)

(In reply to Andrew Sutherland [:asuth] (he/him) from comment #16)

Thank you for the detailed and quick feedback!

(In reply to Mike Conca [:mconca] from comment #15)

I think there are multiple reasons to pursue the idea of storing most, perhaps all, content-visible data locally in the non-roamed storage area.

I'm not sure I parsed this right. I understand there to be a few levels here you might mean:

  • A. Each computer has its own non-roamed profile. Any information sharing happens via Fx Account.

This is basically where I would like Firefox to head, primarily for all of the odd use-case reasons and exceptions you outlined for options B and C. Option A is simple, straight-forward, privacy-protecting and performant. To be clear, though, I'm not saying this should be our highest priority. I'd just like to see the browser head this direction.

I am disappointed to see that the Storage is still being included in Roaming and not Local. I have a user with a slack folder that alone has more than 2gb of storage, and we have a profile quota limit which this user keeps hitting.

My advise to the user will have to be to use chrome, since this was last discussed 3 years ago and Mozilla are clearly not interested in fixing it.

Thanks.

I am hitting this problem more and more.

Please could you let us know if this is actively being considered?

Alternatively, could you let me know if a way of fixing the profile string, as we can only exclude this (which is what we do for other things like Teams/Signal/Zoom/Skype). So I'd like to exclude c:\users\username\appdata\roaming\mozilla\Firefox\Profiles\fixed-profile-esr\storage

Unfortunately we cannot use a wildcard in place of the profile string, so the only options available are for us to wait for you to fix it, or for you to enable us to fix the string for everybody.

Thanks

John Peachey
Senior Windows Systems Administrator
Department of Computer Science
Oxford University.

Flags: needinfo?(mconca)
Flags: needinfo?(bugmail)

I do not have a real solution for this. But if you really want to continue using Firefox in it's current state I might have a workaround for you which might work at least in some cases.
You can use the following power shell script to exclude unwanted profile folders of Firefox and Thunderbird (excludeMozillaFolders.ps1):

# This script is used to exclude Mozilla storage folder from roaming profiles.
# The storage folder can be re-created by Mozilla products like
# Firefox/Thunderbird on startup.
# There is no need for this data to move across machines.
# Some cached data of some extensions and offline page caches are lost on
# roaming but they are considered to be re-creatable.
#
# Since the path is dynamic they can't be excluded via GPO as no wildcards are
# supported. But HKCU\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon
# containing a user-specified exclude list (ExcludeProfileDirs) attribute which
# is merged to the GPO-excluded list.
#
# This script can be included as logoff script and applied via GPO to add those
# profile storage folders dynamically on logoff.
#
# Note: Depending on execution policy you might have to run a CMD script and
# execute this powershell script as follows:
# powershell -ExecutionPolicy RemoteSigned -File "%~dp0updateProfileExcludes.ps1"

$profilePaths = (Get-ItemProperty -Path "HKCU:\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon").ExcludeProfileDirs.Split(";")

# Remove previous listings of Mozilla (Firefox/Thunderbird) profiles.
$profilePaths = @($profilePaths) -inotmatch "^AppData\\.*\\Profiles\\.*\\.*$"

# Path to Mozilla profiles.
$profileLocationFirefox = $env:APPDATA + "\Mozilla\Firefox\Profiles\*\storage"
$profileLocationTHunderbird = $env:APPDATA + "\Thunderbird\Profiles\*\storage"

# Get a list of profile paths.
Set-Location "$env:USERPROFILE"
# Firefox storage.

$profiles = @()
# Firefox.
$profiles += @(Get-Item -Path "$env:APPDATA\Mozilla\Firefox\Profiles\*\bookmarkbackups" -ErrorAction Ignore)
$profiles += @(Get-Item -Path "$env:APPDATA\Mozilla\Firefox\Profiles\*\crashes" -ErrorAction Ignore)
$profiles += @(Get-Item -Path "$env:APPDATA\Mozilla\Firefox\Profiles\*\datareporting" -ErrorAction Ignore)
$profiles += @(Get-Item -Path "$env:APPDATA\Mozilla\Firefox\Profiles\*\gmp*" -ErrorAction Ignore)
$profiles += @(Get-Item -Path "$env:APPDATA\Mozilla\Firefox\Profiles\*\minidumps" -ErrorAction Ignore)
$profiles += @(Get-Item -Path "$env:APPDATA\Mozilla\Firefox\Profiles\*\sessionstore-backups" -ErrorAction Ignore)
$profiles += @(Get-Item -Path "$env:APPDATA\Mozilla\Firefox\Profiles\*\shader-cache" -ErrorAction Ignore)
$profiles += @(Get-Item -Path "$env:APPDATA\Mozilla\Firefox\Profiles\*\storage" -ErrorAction Ignore)
# Files are not supported, but the following files would not be
# needed tool.
# $profiles += @(Get-Item -Path "$env:APPDATA\Mozilla\Firefox\Profiles\*\favicons.sqlite*")
# $profiles += @(Get-Item -Path "$env:APPDATA\Mozilla\Firefox\Profiles\*\storage-sync-v2.sqlite*")
# $profiles += @(Get-Item -Path "$env:APPDATA\Mozilla\Firefox\Profiles\*\webappsstore.sqlite*")

# Thunderbird.
$profiles += @(Get-Item -Path "$env:APPDATA\Thunderbird\Profiles\*\crashes" -ErrorAction Ignore)
$profiles += @(Get-Item -Path "$env:APPDATA\Thunderbird\Profiles\*\datareporting" -ErrorAction Ignore)
$profiles += @(Get-Item -Path "$env:APPDATA\Thunderbird\Profiles\*\gmp" -ErrorAction Ignore)
$profiles += @(Get-Item -Path "$env:APPDATA\Thunderbird\Profiles\*\ImapMail" -ErrorAction Ignore)
$profiles += @(Get-Item -Path "$env:APPDATA\Thunderbird\Profiles\*\shader-cache" -ErrorAction Ignore)
$profiles += @(Get-Item -Path "$env:APPDATA\Thunderbird\Profiles\*\storage" -ErrorAction Ignore)

# $profilePaths = Get-Item -Path $profileLocationFirefox

$profilePaths += $profiles | ForEach-Object {($_.FullName | Resolve-Path -Relative).TrimStart(".\")}

$excludeProfileDirs = $profilePaths -join ";"
# Write-Host "Exclude: $excludeProfileDirs"

Set-ItemProperty -Path "HKCU:\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon" -Name "ExcludeProfileDirs" -Value "$excludeProfileDirs"

This script will exclude all "bookmarkbackups", "crashes", "datareporting", "gmp", "minidumps", sessionstore-backups", "shader-cache" and "storage" folders from roaming profiles. Unfortunately you can't exclude volatile plain files like favicons.sqlite or webappsstore.sqlite with this method as you can exclude only complete folders.

The script can be assigned in GPO to the user to run at logof (standard logoff script for users). The script is written so it will first clear all previous folders from the user-specific exclude list to avoid the list to grow indefinitely in case the profile folder is changed (ie. a new profile is created by the user) or the profile folder is renamed. Also this avoids the directories to be added multiple times so the exclude list would grow indefinitely. So the script will also take into account if the user has multiple profiles. As long as they are located in the default Profiles folder.

Note: You should perhaps consider NOT to wipe the local profiles early or during logoff. If the profile remains on the client the excluded folders remain on the client and all data is kept. If the profile is deleted (via expiry, or policy to clean it at logoff or when the user logs on to a different machine) then those folders are sipmply missing and Firefox is re-creating them. This results in losing the store (temporary) data. For exmample for Whatsapp web and presumably for applications like slack you might lose the synchronized history and Firefox needs to sync again with the servers to re-download this data. However this is expected and in my tests not causing any issues rather than a slightly slower page load on first use after profile re-build.

I still hope Mozilla is fixing this in the future and moving such temporary data directly to the %LocalAppData% tree as it is essentially temporary data which can be re-created any time. If the user is always using the same machine (most used use-case I believe) then either using my script or a permanent fix moving the data to %LocalAppData% would still preserve this data unless either manually cleared or moving to another machine.

Thanks a lot Rainer

I've tested that out and it seems to be doing the job very well.

I hope Mozilla take note of this and someone there cares about fixing this issue, but for anyone else reading this then Rainer has indeed done a fine job of making this problem go away for me.

Thanks again.

John.

Flags: needinfo?(rainer.meier)
Flags: needinfo?(mconca)
Flags: needinfo?(bugmail)
Severity: normal → S3
Priority: P2 → P3

Thank you. This might also give an indication which folders (potentially) can be moved to %LocalAppData% safely.
John has set the needinfo flag for me but I do not see an additional question. So I am clearing it.

Flags: needinfo?(rainer.meier)

My apologies for the incorrect flag -- I'm a bugzilla newbie and appreciate your help ;)

No worries. Thanks for the feedback and for participating. The more people are showing interest in bug reports the more likely it gets for them to get fixed.

Huge space taken up by Firefox in Roaming Profiles (roughly 200 MB) was the reason why we stopped using Firefox on our high school.

(In reply to Franta Hanzlik from comment #25)

Huge space taken up by Firefox in Roaming Profiles (roughly 200 MB) was the reason why we stopped using Firefox on our high school.

Hi Franta

Rainer's powershell script solution from 11 months ago which we apply at logoff has completely solved our issue. While this is a workaround compared to the amount of effort I have to spend with other problems you put this in place once and that solves your problem. I highly recommend you implement this - feel free to reach out to me if you have any issues in doing so.

All the best, John.

Nothing new from my side to help that topic get solved but we had the same problems in a 3000 windows client company with roaming user profiles enabled. Storage folder is too huge, resulting in long log on & off times. Please move the location of it to %localappdata% the sooner the better or give us admins a tool to set it via GPO

(In reply to s.osdoba from comment #27)

Nothing new from my side to help that topic get solved but we had the same problems in a 3000 windows client company with roaming user profiles enabled. Storage folder is too huge, resulting in long log on & off times. Please move the location of it to %localappdata% the sooner the better or give us admins a tool to set it via GPO

If you look at this comment https://bugzilla.mozilla.org/show_bug.cgi?id=1395705#c20 it has a powershell script that we have been using very successfully to solve the issue. While this is a workaround it has 100% solved my problem. Frustrating as it is that Mozilla haven't fixed it - there is a pretty good solution there...

(In reply to John Peachey from comment #28)

(In reply to s.osdoba from comment #27)
If you look at this comment https://bugzilla.mozilla.org/show_bug.cgi?id=1395705#c20 it has a powershell script that we have been using very successfully to solve the issue. While this is a workaround it has 100% solved my problem. Frustrating as it is that Mozilla haven't fixed it - there is a pretty good solution there...

I saw it and will try to implement it but i still have the hope that Mozilla will solve it so we dont have to apply workarounds. Yes i still believe in Santa Claus :)

(In reply to s.osdoba from comment #29)

(In reply to John Peachey from comment #28)

(In reply to s.osdoba from comment #27)
If you look at this comment https://bugzilla.mozilla.org/show_bug.cgi?id=1395705#c20 it has a powershell script that we have been using very successfully to solve the issue. While this is a workaround it has 100% solved my problem. Frustrating as it is that Mozilla haven't fixed it - there is a pretty good solution there...

I saw it and will try to implement it but i still have the hope that Mozilla will solve it so we dont have to apply workarounds. Yes i still believe in Santa Claus :)

You mean he's not real?!! :(

To implement is so simple - the script doesn't need any edits. Make sure you apply it as a user policy to run the script on logoff...

Any problems we're all here to help :)

Just for the record: favicons.sqlite is tied to places.sqlite (bookmarks, history); if they get out of step, your bookmarks will look ugly.

Duplicate of this bug: 1892936
You need to log in before you can comment on or make changes to this bug.