1429838 - After restoring profile to a different location, paths in extensions.json are incorrect

Status: NEW

(Toolkit :: Add-ons Manager, defect, P2)

Description

[Elhem Enohpi]

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:59.0) Gecko/20100101 Firefox/59.0
Build ID: 20180111100722

Steps to reproduce:

1. Follow the instructions for "backing up your profile" and "restoring to a different location" here:
https://support.mozilla.org/en-US/kb/back-and-restore-information-firefox-profiles
2. Delete or rename the original profile folder.


Actual results:

After starting Firefox with the new profile, the file "extensions.json" still contains paths to the old profile folder. This causes problems like missing icons from some (but not all) extensions in the Add-ons manager in all current versions of Firefox, and, at least in Firefox 56, extensions may fail to operate. There are no error messages.

If the original folder is not deleted, the user may not notice anything wrong for some time.

Affects Firefox 56 (and possibly older) through Firefox Nightly 59.0a1 (2018-01-11).


Expected results:

Paths should point to the correct location, and extensions' functionality should not be affected. Firefox should detect a change in profile location, and the extensions.json file (and any other necessary files) should be regenerated, or at least, the instructions above should include a step of manually deleting the file so it will be regenerated.

Comment 1

[Ciprian Muresan [:cmuresan], Formerly of Ecosystem QA - please needinfo @mcoman for things regarding the team]

User Agent 	Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:59.0) Gecko/20100101 Firefox/59.0
Build ID 	20180116100114

Hi Elhem,

I haven't managed to reproduce the issue you are experiencing on Nightly 59 using Mac 10.13. 

I've installed several extensions (uBlock Origin, FoxyTab, Enhancer for YouTube, Firefox Notes, Test Pilot, Forecasefox and a theme)
Opened my profile folder
Saved it to a different location
Deleted the profile from the Firefox folder
Started Firefox 56
Created a new profile
Opened Firefox 56 with the new profile
Closed the browser
Copied the contents of my previous profile
Pasted over the contents of the new profile
Replaced all conflicting files
Opened the Firefox 56 with the new profile

After doing the above and all the extensions were installed, in working condition and extensions.json had updated the file paths for the installed extensions.

Did you have issues with specific extensions? If so, could you please list them here?

Comment 2

[Elhem Enohpi]

Hi Ciprian,

It's not clear what you did in step 1. In what version of Firefox did you create the initial profile, and install the extensions? If it wasn't Firefox 56, then the extensions.json file will be automatically regenerated when you open it in a different version of Firefox, and you won't see the problem.

If that's the case, please follow again the exact steps above, using only one version of Firefox.

Comment 3

[Ciprian Muresan [:cmuresan], Formerly of Ecosystem QA - please needinfo @mcoman for things regarding the team]

User Agent 	Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:59.0) Gecko/20100101 Firefox/59.0
Build ID 	20180117100129

Thank you for the clarification Elhem. 

I've used the same Firefox version while following the same steps and I can reproduce the issue now. Extensions paths point to a folder which I have deleted. The add-ons that I have used do not seem affected, but I have seen that I've lost some of the experiments icons in about:addons->Extensions.

I've managed to reproduce this with Firefox 56.0.2 and Nightly 59.0a1.

Comment 4

[Andrew Swan [:aswan]]

This came up on IRC last week and there was some disagreement about whether this is supposed to be supported or not.
Kris, :zombie, anything to say here for the record?

Comment 5

[Tomislav Jovanovic :zombie]

My stance is that we should if not "support", at least avoid completely breaking when the profile is moved to a different path.  

Backing up and restoring the "roaming" profile across machines has been a reliable way to move your firefox data for decades, even documented on SUMO:

https://support.mozilla.org/af/kb/back-and-restore-information-firefox-profiles#w_restoring-to-a-different-location

We also have several tools, some of them "officially supported", that use various versions of profile "cloning", like web-ext and mozregression (and not to mention potentially lots of one-off scripts in automation).

I would be fine with "this is too much work at the moment, and not a priority", but would disagree with a WONTFIX.

Comment 6

[Robert AB]

And here we have another problem - more incompatibility between extension data in different profiles. This regression is caused by introduction of IndexedDB databases:
https://www.reddit.com/r/firefox/comments/ar78wr/firefox_extensions_to_use_different_storage_type/egmdkul

comment 1 by u/Thuringwethon:

I really dislike the change.

The main problem is that user has lost any control over extension's data.

• It's no longer readable. It's hidden inside ugly sqlite file and unreadable even with SQLit viewer
• It no longer can be imported into new profile. Not unless the extension itself provides such opportunity.
• The only option to access stored data is debugging extension - and from end user perspective this is not an acceptable solution

While performance improvements are nice, it's totally not worth it. I miss the days when extensions data could be accessed from about:config.

comment 2 by u/df63fe67b24cb116e92a:

Fully agreed. The first is bad enough but there are some solutions like accessing indexedDB via developer tools in browser. The second is just truly awful and will make restoring/backing up my firefox profile 10x more complicated. The internal UUID used changes with each firefox install so the extension settings can't simply be copied between installs.

Even chrome with it's opaque storage format for extension settings is far better in this regard as it uses a static extension id for the extension storage folder.

https://www.reddit.com/r/firefox/comments/ar3mps/extensions_in_firefox_66/egnizj6/?context=3

Comment 7

[Robert AB]

I talking in Bug 1429838 comment 6 about important regression making this bug even worse. Why was this categorized as off-topic?

Comment 8

[David Durst [:ddurst]]

(In reply to Robert Ab from comment #7)

I talking in Bug 1429838 comment 6 about important regression making this bug even worse. Why was this categorized as off-topic?

In general, adding noise to a bug is not useful; in this case, it's categorized as off-topic because it's not adding anything actionable to this bug.

This bug should be focused on reporting the issue and information that can help guide the solution. In this case, the issue has been identified, and this bug has been prioritized.

Comment 9

[nis]

This is definitely something that needs to be fixed in some way.

It is "well known" that Firefox (and Thunderbird) profiles can just be copied to a new OS/computer/etc. There's also the case of restoring from a backup after a reinstall (where the username may not be the same). As such this has the potential to cause problems for quite a few users.

In fact, I myself is currently in the process of setting up my computer to dual-boot Windows and Linux, and intended to simply sync the profile between the two so that I would always have the same FF environment (and no, FF Sync won't cut it - it has its own problems). Luckily I stumbled upon this bug before I got that far.

My suggestion would be to have a number of base-path variables such as:

file:///C:/Program%20Files/Mozilla%20Firefox/browser/features/[e-mail]!/ -> $install_dir/browser/features/[e-mail]!/

file:///C:/Users/nis/AppData/Roaming/Mozilla/Firefox/Profiles/b204735w.default-1560856491827/extensions/[e-mail]!/ -> $profile_dir/extensions/[e-mail]!/

Comment 10

[FirefoxPortableIsNotReallyPortable]

This bug affects every software that uses Firefox Extensions API: Firefox, Thunderbird, all their "portable" versions (which due to this bug are not portable at all) and all their forks and derivatives.

Comment 11

[Georg Koppen]

My current plan for Tor Browser is to patch parseDB() in XPIProvider.jsm adding an additional sanity check making sure the paths of the extensions exist. If not the DB is recreated. Would that be an acceptable approach for getting merged into Firefox?

Comment 13

[Georg Koppen]

(In reply to Georg Koppen from comment #11)

My current plan for Tor Browser is to patch parseDB() in XPIProvider.jsm adding an additional sanity check making sure the paths of the extensions exist. If not the DB is recreated. Would that be an acceptable approach for getting merged into Firefox?

That's not sufficient anymore (at least not for Firefox 68 and probably later versions, too), see: https://trac.torproject.org/projects/tor/ticket/27604#comment:19.

Comment 14

[Alex Catarineu (Tor Browser dev)]

We have fixed this in Tor Browser with this patch: https://gitweb.torproject.org/tor-browser.git/commit/?h=tor-browser-68.2.0esr-9.5-1&id=7d5f879bba1e81ee64576e882f9d8916a2bc471e. The issues are also reproducible in latest Firefox when changing profile folder (tested using -p /newprofilepath cmdline arguments). We would like to have some feedback on the patch and the findings, as there are some parts we're not completely sure of. I'm adding some comments from the Tor Browser ticket:

I think there are (at least) three different issues here:

One is the problem of extensions being broken that mcs mentioned, which I think was introduced by bug 1512436. More concretely, the problem seems to be the new rootURI field in addonStartup.json which is absolute, unlike the path one which is serialized as relative: https://searchfox.org/mozilla-esr68/rev/22bae08f58d48ff86e02d5bbd12e5630af148d6f/toolkit/mozapps/extensions/internal/XPIProvider.jsm#555. I think this was needed to have built-in addons with resource:// rootURI like resource://gre/modules/themes/default/. Fixed this by always overriding it [https://searchfox.org/mozilla-esr68/rev/22bae08f58d48ff86e02d5bbd12e5630af148d6f/toolkit/mozapps/extensions/internal/XPIProvider.jsm#488 here] if this.file exists (should not happen with addons with resource:// rootURIs).

The other is some special treatment that langpacks get due to bug 1492459. These are flagged as missing early [https://searchfox.org/mozilla-esr68/rev/22bae08f58d48ff86e02d5bbd12e5630af148d6f/toolkit/mozapps/extensions/internal/XPIProvider.jsm#502 here] (currentModifiedTime is set [https://searchfox.org/mozilla-esr68/source/toolkit/mozapps/extensions/AddonManagerStartup.cpp#445 here] only for langpacks) if the (old) path of the extension xpi does not exist. If that's the case the langpack is removed in https://searchfox.org/mozilla-esr68/rev/715f10032bb8be971ff0e9846e12be58afad4950/toolkit/mozapps/extensions/internal/XPIDatabase.jsm#3143. That seems to fallback to English, but on browser restart it's completely broken, showing the same error message mentioned in comment:29 and in bugzilla. I'm not so sure what's the best way to fix this, in the proposed patch I'm checking for the (new) extension file existing before flagging it as missing.

Third is the issue of extensions.json (and possibly webext.sc.lz4) paths not being updated, but I'm not so sure this has functionality impact if the two previous issues are fixed. In any case, I think making scanForChanges return true when the path of some addon location has changed will do the trick here and trigger an update of extensions.json. I verified that webext.sc.lz4 paths are also updated, although I did not investigate what exactly in the code is updating the latter.

BTW, a small thing not changed in the Tor Browser patch that I saw while looking at the code: I think file should not be part of JSON_FIELDS in https://searchfox.org/mozilla-central/rev/35873cfc312a6285f54aa5e4ec2d4ab911157522/toolkit/mozapps/extensions/internal/XPIProvider.jsm#456. It's not serialized and even if it was I think there could be some kind race condition, as this.file is set in the path setter: https://searchfox.org/mozilla-central/rev/35873cfc312a6285f54aa5e4ec2d4ab911157522/toolkit/mozapps/extensions/internal/XPIProvider.jsm#519.

Comment 15

[BugBot [:suhaib / :marco/ :calixte]]

Bugbug thinks this bug is a regression, but please revert this change in case of error.

Comment 17

[steffen.schweikhard]

Is any more effort being put into solving this issue? portableapps.com says that this issue blocks moving to any Thunderbird version beyond 60.9.1:

https://portableapps.com/node/60496

Since this more and more leaves me with a vulnerable mail client, I am now strongly considering to ditch Thunderbird as a portable mail client. This will lead to moving to a different solution also with non-portable installations at my place.

My assumption is that this affects a whole bunch of users, and I'd greatly appreciate progress on this matter. Thanks.

Comment 18

[Alex Catarineu (Tor Browser dev)]

Attachment: Bug 1429838 - Fix addon issues when relocating profile directory

• Recompute XPIState rootURI, which is stored as an absolute path
  for non-builtin addons.

• Avoid langpacks XPIState being set as missing when profile folder
  is relocated, due to UpdateLastModifiedTime() in
  AddonManagerStartup.cpp (bug 1492459).

• Return true in XPIProvider scanForChanges() if a location path
  changes, to trigger an update of stored paths in extension.json
  and webext.sc.lz4.

Comment 22

[Masatoshi Kimura [:emk]]

Is it possible to workaround manually, for example, by deleting some files and/or executing some code snippet in Browser Console? This bug makes it impossible to moving the installation folder due to dedicated profile per installation path. I tried to move the installation folder due to bug 1676840 and hit this bug.

Comment 23

[Niels]

Any News on this in 3 Years? And like in the last Comment, even a Workaround would be appreciated

Comment 24

[david.gittler]

I don't understand why Mozilla is avoiding this bug like the plague, but I can contribute a workaround for a very specific workload.
I originally opened another bug but it was marked as duplicate of this one: https://bugzilla.mozilla.org/show_bug.cgi?id=1654491

This bug is actually meant for moving and restoring profiles, which is just a tad related to what I do, so it probably won't help you if you are not doing the same thing as I am. I am managing a Terminal-Server-Farm for many users and a bunch of them are also able to log in. This is used for instance a warehouse where people use not-personalized accounts for just scanning barcodes to a computer, for instance. And they may also need a browser or the barcode scanning software might actually be a web page. This can result in the following scenario:

A user runs multiple windows isloated windows sessions on the same machine, and those sessions from one user share the same user profile file system. This will happen, doesn't matter if you use MS RDP or Citrix, if you use classic roaming profiles, FSLogix or Citrix UPM. If one account spawns two sessions on the same host, they will share C:\Users\username\AppData\ .

For Firefox this means multiple instances of firefox.exe that are technically isloated from each other access the same profile path, and thats a big no-no. The 2nd attempt will tell you the browser is in use, and yeah, bad luck.

This is where I came up with the idea of moving the profile do a session-specific path via script and modify profiles.ini via script. This works until you run into extensions, and there is the overlap with this bug here. But i cannot block extensions from users, I actually need them, because I have a bunch of different languages on the same serverfarm and langpacks in firefox are extensions.

I invested A LOT of time in this workaround but it works and is reliable. However it needs FSLogix to work like intended. I put all my scripts in one zip will and attach it to this bug.
Any competent windows/terminalserver-admin should be able to work with it. :) It also contains an exe file, the only thing that I found on the internet that actually can decompile and compile thos mozlz4 files 100% like firefox does. There is also a readme.txt in for further instructions.

Comment 25

[david.gittler]

Attachment: firefox_profilemanagement_terminalservers.zip

this is a very hacky workaround for terminal-server installations where one user might spawn multiple sessions on the same machne and all of them should be able to use firefox.

referenced in comment https://bugzilla.mozilla.org/show_bug.cgi?id=1429838#c24

Comment 26

[Apollinaris]

Is there anything an ordinary user could do to speed up the resolution of this issue?

Comment 29

[Luca Greco [:rpl] [:luca] [:lgreco]]

Bug 1704147 is another similar issue triggered by having the absolute path to the extensions in extensions.json.
The STR from Bug 1704147 shows that this issue may also affect builtin addons when is the application directory to have been moved to a different path.

Comment 30

[Rafael Ristovski]

A quick manual fix:

1. navigate to about:profiles
2. Open the folder listed as Root Directory under the currently used profile ("This is the profile in use...")
3. Remove the file addonStartup.json.lz4
4. Restart firefox twice.

Comment 31

[BugBot [:suhaib / :marco/ :calixte]]

The bug assignee didn't login in Bugzilla in the last 7 months.
:mixedpuppy, could you have a look please?
For more information, please visit auto_nag documentation.

Comment 32

[C0oo1D]

Firefox 99.0, after creating new profile and sync it - bug is present.
Quick manual fix from Rafael Ristovski (comment #30) works with relocated profile, but paths in fixed that way extensions.json still absolute.

Comment 34

[Pier Angelo Vendrame]

Hello!

I am a Tor Browser developer and I'm willing to start looking at this again.

From what I understand, the comments on the patch were addressed, and I do not see new ones.

So, please let me know if I can do anything to forward the merging of this patch.

Thanks!

Comment 35

[Nick Rishel [:nrishel]]

NI @nalexander for followup.

Comment 36

[BugBot [:suhaib / :marco/ :calixte]]

The severity field for this bug is relatively low, S3. However, the bug has 8 duplicates and 246 votes.
:robwu, could you consider increasing the bug severity?

For more information, please visit auto_nag documentation.

Comment 37

[BugBot (nomail) [:suhaib / :marco/ :calixte]]

The last needinfo from me was triggered in error by recent activity on the bug. I'm clearing the needinfo since this is a very old bug and I don't know if it's still relevant.

Comment 39

[BugBot [:suhaib / :marco/ :calixte]]

Clear a needinfo that is pending on an inactive user.

Inactive users most likely will not respond; if the missing information is essential and cannot be collected another way, the bug maybe should be closed as INCOMPLETE.

For more information, please visit BugBot documentation.

Comment 41

[github]

I'm having the same problem while creating a portable Firefox installation in my USB stick.
I'm starting firefox with programs/firefox-117.0.r20230912013654-x86_64.AppImage --profile data/firefox, but the profile path is hardcoded inside extensions.json, causing all extensions to stop working after moving installation to another path / another computer.

Comment 43

[pg_78]

Shouldn't the support page mentioned in the original issue (https://support.mozilla.org/en-US/kb/back-and-restore-information-firefox-profiles) be changed?

As of now, it still implies that "Restore to a different location" is a valid procedure when in fact - because of this bug - it will not work correctly and will frustrate a user who trusts the documentation.

Comment 44

[pg_78]

From my testing, a fairly straightforward workaround is to delete the value of the preference "extensions.databaseSchema" - either (1) by modifying prefs.js prior to starting Firefox with the new profile, or (2) by starting Firefox, deleting the pref in about:config, and restarting.

This gave me a working profile, where Firefox had automatically modified the paths in extensions.json to their correct new locations.

I can't verify this works in all circumstances, but it does seem to correctly regenerate the paths in extensions.json whether the previous path they point to still exists (i.e. cloning a profile on the same machine) or not (i.e. moving a profile or copying it to a different machine).

Comment 46

[.]

Attachment: about_support

I ran into a similar problem today with Nightly and lost all my web browser extensions. They showed up in about:addons as installed and enabled, but none of them exhibited any functionality until a few uninstalls and reinstalls of each extension.

STR:
1. Create a new web browser profile and install a common extension; e.g., Ublock Origin.
2. Exit the web browser, then move the web browser profile directory to a different location (where Nightly will not locate it).
3. Start the web browser profile manager, create a new web browser profile, then exit the web browser.
4. Following the instructions at https://support.mozilla.org/en-US/kb/back-and-restore-information-firefox-profiles, overwrite the contents of the new web browser profile with those of the existing profile.
5. Start the web browser with the (modified) new web browser profile. Check the extensions list in about:addons and whether the installed extension still exhibits any functionality.

Expected:
- The list of extensions in about:addons reflects the actual state of the web browser profile by either being empty or listing functional add-ons.

Actual:
- The list of extensions in about:addons remains the same, but all extensions have lost all functionality. In the case of the example extension mentioned above, the icon will have disappeared from the toolbar and no content filtering will occur.

------

I found a log message in the Nightly browser console that quietly hints the extensions are now dead, but the average end-user would probably not notice for a while.
```
1753826398089	addons.xpi	WARN	Exception running bootstrap method startup on [e-mail]: Error: Error while loading 'jar:file:///home/<HOME_DIR>/.mozilla/firefox/98nrz3vj.delme0/extensions/[e-mail]!/manifest.json' (NS_ERROR_FAILURE)(resource://gre/modules/Extension.sys.mjs:1162:20) JS Stack trace: readJSON/</<@Extension.sys.mjs:1162:20
[e-mail]:124:18
```

Comment 47

[.]

I ran into a similar problem today with Nightly and lost all my web browser extensions. They showed up in about:addons as installed and enabled, but none of them exhibited any functionality until a few uninstalls and reinstalls of each extension.

STR:

1. Create a new web browser profile and install a common extension; e.g., Ublock Origin.
2. Exit the web browser, then move the web browser profile directory to a different location (where Nightly will not locate it).
3. Start the web browser profile manager, create a new web browser profile, then exit the web browser.
4. Following the instructions at https://support.mozilla.org/en-US/kb/back-and-restore-information-firefox-profiles, overwrite the contents of the new web browser profile with that of the existing profile.
5. Start the web browser with the (modified) new web browser profile. Check the extensions list in about:addons and whether the installed extension still exhibits any functionality.

Expected:

• The list of extensions in about:addons reflects the actual state of the web browser profile by either being empty or listing functional add-ons.

Actual:

• The list of extensions in about:addons remains the same, but all extensions have lost all functionality. In the case of the example extension mentioned above, the icon will have disappeared from the toolbar and no content filtering will occur.

---

I found a log message in the Nightly browser console that quietly hints the extensions are now dead, but the average end-user would probably not notice for a while.

1753826398089	addons.xpi	WARN	Exception running bootstrap method startup on [e-mail]: Error: Error while loading 'jar:file:///home/<HOME_DIR>/.mozilla/firefox/98nrz3vj.delme0/extensions/[e-mail]!/manifest.json' (NS_ERROR_FAILURE)(resource://gre/modules/Extension.sys.mjs:1162:20) JS Stack trace: readJSON/</<@Extension.sys.mjs:1162:20
[e-mail]:124:18

Comment 48

[.]

Attachment: about_support