1504547 - Developer Edition (64.0b5) removes all web-extensions

Status: RESOLVED INCOMPLETE

(Toolkit :: Add-ons Manager, defect)

Description

[shine [:shine]]

After the update to 64.0b5, all my web extensions were removed. Although, the older non-compatible addons were still retained.

I reproduced this by opening an old profile that had compatible web extensions. After opening it with Developer Edition, all the web extensions were gone leaving only the non-compatible addons behind.

I then rsync'd my extensions from my default profile on Firefox (just to be sure that it was the browser's fault and not anything else and before reporting this bug) and restarted the browser again, same result. This last test was done on 64.0b6 and the bug is still there in the latest version as well.

Comment 1

[:Felipe Gomes (needinfo for replies!)]

[Tracking Requested - why for this release]:
Extensions being removed

Comment 2

[Andrew Swan [:aswan]]

Given that we don't have any other similar reports, I suspect this is something specific to your profile or something else on your computer.  Can you set the preference "extensions.logging.enabled" to true in about:config, then go through the steps that cause this to happen, then open the browser console and copy its contents into an attachment on this bug?

Comment 3

[shine [:shine]]

Funnily enough, I can't view the browser console with the preference 'extensions.logging.enabled' set to true. However, I can see at least the header bar with the Clear Log icon and the Filter Output icon and text field when the preference is set to false (default).

However, to show a demonstration of the bug, I've taken a screencast and am attaching it to this bug. I hope this helps.

2 other things that I noticed while debugging this was that the 'Switch to Tab' feature doesn't work anymore. With the upgrade to 64.0b8, now there are a few extensions that remain from the valid list, but all of them are disabled. I have to manually enable them. However, most of my active addons (Privacy Badger, Disconnect, Test Pilot, Containers, etc) are still being deleted.

In order to clear confusion, I also did a few other tests to confirm that this was not a specific case.
* I created a fresh profile, rsync'd the extensions from my default profile and started the profile - FAIL
* Downloaded latest version of Firefox Developer Edition from firefox.com/developer and tried opening the default developer edition profile as well as the fresh profile I created for this test; both with the extensions from my default profile rsync'd in - FAIL.

Comment 4

[shine [:shine]]

Attachment: Screencast from Sunday 11 November 2018 02:42:03 IST.webm

Comment 5

[shine [:shine]]

Attachment: Screencast from Sunday 11 November 2018 02:46:59 IST.webm

Comment 6

[Andrew Swan [:aswan]]

(In reply to shine [:shine] from comment #3)
> * I created a fresh profile, rsync'd the extensions from my default profile
> and started the profile - FAIL

Do I have this correct: you used the Firefox profile manager to create a new profile, and did not change any preferences in this new profile.  Then you copied some extensions into the extensions/ directory inside the profile and restarted the browser, at which point some of those extensions were deleted?  From the screencast, it looks like at least some of those extensions are unpacked (ie directories), these are no longer supported.  But it looks like some packed extensions (ie, xpi files) may also have been affected.

Since I can't reproduce this here, we'll need some help from you narrowing this down to a repeatable set of steps for reproduction.  Can you try repeating this sequence of events with a subset of the extensions to narrow it down to one extension (or a small number of extensions) that cause the browser console to become unusable?

Comment 7

[shine [:shine]]

(In reply to Andrew Swan [:aswan] from comment #6)
> (In reply to shine [:shine] from comment #3)
> > * I created a fresh profile, rsync'd the extensions from my default profile
> > and started the profile - FAIL
> 
> Do I have this correct: you used the Firefox profile manager to create a new
> profile, and did not change any preferences in this new profile.  Then you
> copied some extensions into the extensions/ directory inside the profile and
> restarted the browser, at which point some of those extensions were deleted?
Correct.

> From the screencast, it looks like at least some of those extensions are
> unpacked (ie directories), these are no longer supported.  
I knew this but since I was using rsync, I just took everything and didn't want to put in a complicated exclude pattern.

> Since I can't reproduce this here, we'll need some help from you narrowing
> this down to a repeatable set of steps for reproduction.  Can you try
> repeating this sequence of events with a subset of the extensions to narrow
> it down to one extension (or a small number of extensions) that cause the
> browser console to become unusable?
Unfortunately, after a lot of testing, I couldn't get the right combination of extensions that would make the browser console usable. Here are the tests I tried :
1. remove each of the remaining extensions one-by-one that are left-over after the mass-deletion during browser start.
2. remove all of the remaining extensions together that are left-over after the mass-deletion during browser start.
3. remove each extension one-by-one including the ones that are freshly rsync'd.
4. remove extensions that looked weird by name - those that had curly braces {} in the name and others that had a 'jid' prefix
5. remove extensions from 2 and 4.

Notes :
During these tests, I made sure that there were only packaged .xpi extensions in the <profile>/extensions/ directory.
I rsync'd the same set of extensions from my default profile before each test.

Is there some log file that I could tail and see what's going wrong? Could the browser console be logged to a file or something?

Comment 8

[shine [:shine]]

Attachment: extensions.default.list

I'm attaching the list of extensions that were rsync'd each time before each test.

Comment 9

[Andrew Swan [:aswan]]

If you set the preference browser.dom.window.dump.enabled to true, the logs should be written to stdout (or maybe it is stderr, I can't remember)

Comment 10

[Julien Cristau [:jcristau]]

(In reply to Andrew Swan [:aswan] from comment #9)
> If you set the preference browser.dom.window.dump.enabled to true, the logs
> should be written to stdout (or maybe it is stderr, I can't remember)

Comment 11

[shine [:shine]]

Attachment: browser.dom.window.dump

I couldn't recognize anything useful out of that dump. Maybe you could identify something?

PS : the dump gets written to stdout.

Comment 12

[Andrew Swan [:aswan]]

I'm not sure what to make of that last attachment.  For one thing, it includes a bunch of system add-ons that are no longer part of Firefox 64.  What Firefox version are you running?  And where did you get it from?
Also, was this from a run in which some extensions were unexpectedly deleted from your profile?  I would expect to see evidence of that in the logs, and there is none.

Comment 13

[shine [:shine]]

(In reply to Andrew Swan [:aswan] from comment #12)
> I'm not sure what to make of that last attachment.  For one thing, it
> includes a bunch of system add-ons that are no longer part of Firefox 64. 
> What Firefox version are you running?  
I'm actually running Firefox Developer Edition 64.0b12 now.

> And where did you get it from?
Since my initial bunch of extensions were removed in the first place, I am rsync'ing the extensions from a profile that Firefox ESR uses from Debian. That's my default profile actually.

> Also, was this from a run in which some extensions were unexpectedly deleted
> from your profile?  I would expect to see evidence of that in the logs, and
> there is none.
Yes, this was running from the same browser (even though it is a more upgraded version now than when this bug was initially reported). That's also what I meant when I said that I couldn't make anything useful of that dump.
However, the issue of deletion is still happening. I could create more screencasts to prove my point, but then again, it would be pretty much the same thing.

Is there something more that we can do to further debug this issue?

Comment 14

[Andrew Swan [:aswan]]

If you're really running Firefox 64.0b12, something is very wrong with your Firefox installation.  
These messages specifically should never show up in Firefox 64:
1543090843213	addons.xpi	DEBUG	Existing add-on [e-mail] in app-system-defaults
1543090843214	addons.xpi	DEBUG	Existing add-on [e-mail] in app-system-defaults
1543090843214	addons.xpi	DEBUG	Existing add-on [e-mail] in app-system-defaults

So, how exactly did you install Firefox on this machine?

Comment 15

[shine [:shine]]

I've had this instance downloaded as a tarball a long time back - probably ever since Firefox Developer Edition was first introduced. It has been auto-updating ever since.

At least the profiles have been.

I might have re-downloaded the source tarball when the auto-updater would get stuck in a restart-loop or something, but the contents of the profile directory has been the same. I have files from as old as Sep 3 2014.

Comment 16

[Andrew Swan [:aswan]]

Again, the logs suggest that either you're not actually running 64 or you've got a bad install.
Can you reproduce this if you download a fresh copy of 64 and run that instead of the copy that's been updated from an older version?

Comment 17

[Julien Cristau [:jcristau]]

As it sounds like this is a special case with an individual install and not something that's likely to affect too many users of 64 I'm going to stop tracking for that release.

Comment 18

[shine [:shine]]

(In reply to Andrew Swan [:aswan] from comment #16)
> Again, the logs suggest that either you're not actually running 64 or you've
> got a bad install.
> Can you reproduce this if you download a fresh copy of 64 and run that
> instead of the copy that's been updated from an older version?

I already tried that too. I had the same results. Check last point in comment #3.

Could it be that there could be some file that is triggering this from within my profile directory? The source of the browser and the profile directory are placed in different locations.

Comment 19

[shine [:shine]]

Attachment: Screenshot from 2018-11-29 05-16-57.png

proof that I'm running the latest version of Firefox Developer Edition

Comment 20

[Andrew Swan [:aswan]]

I'm not really sure what's going on, in the only code path I know of where the addon manager deletes xpi files from the profile, it emits a log message that includes the string "Add-on (id) is invalid", this message is not present in your logs.
Are you still launching browser instances from about:profiles as in your video?
We could try to have a Mozilla QA person reproduce this but given the lack of other similar reports and the inconsistencies between the log output and what you're describing, this seems very likely to be something in your local environment or testing methodology.

Furthermore, I should note that manipulating the contents of the profile directly is not a supported method for installing extensions, if you want a way to automatically install extensions into certain Firefox profiles, please use the policy feature:
https://support.mozilla.org/en-US/products/firefox-enterprise/policies-enterprise

Comment 21

[shine [:shine]]

(In reply to Andrew Swan [:aswan] from comment #20)
> I'm not really sure what's going on, in the only code path I know of where
> the addon manager deletes xpi files from the profile, it emits a log message
> that includes the string "Add-on (id) is invalid", this message is not
> present in your logs.
I was expecting something like that from the logs as well. Or something more direct saying that add-on manager was deleting add-ons.

> Are you still launching browser instances from about:profiles as in your
> video?
Yes.

I think I did it once directly from the command line after enabling browser.dom.window.dump only to get the window dump log. Otherwise, it has always been from about:profiles, since that is the easiest way to start multiple profile windows.

> We could try to have a Mozilla QA person reproduce this but given the lack
> of other similar reports and the inconsistencies between the log output and
> what you're describing, this seems very likely to be something in your local
> environment or testing methodology.
If you could tell me what or where to look, I could do it. I'm fairly well-versed in the command-line as well.

> Furthermore, I should note that manipulating the contents of the profile
> directly is not a supported method for installing extensions
I don't do it on a regular basis. It was only to reproduce this bug that I was rsync'ing my existing add-ons from a different profile that I was using as my default.

Comment 22

[Andrew Swan [:aswan]]

Krupa, could we get some help from QA reproducing this?

Comment 23

[krupa raj [:krupa--use this to needinfo]]

Vlad, please help reproduce this issue.

Comment 24

[Vlad Jiman]

Hello, unfortunately I was unable to reproduce this issue on Ubuntu 18.04.1 and Ubuntu 16.04, using multiple FF versions and about 15 installed addons, some of which were marked as incompatible by some earlier versions of Firefox, prior to updating.

Some of the versions I've used in order to try to reproduce: 52.0, 59.0b12, 60.0b8, 62.0, 63.0. 

Tests have also been run on existing and new profiles but the addons were still present in the about:addons page after each case. Please let me know if there's something I'm missing or any other info that could help me reproduce the problem. Thanks

Comment 25

[shine [:shine]]

Hello Vlad,

Thank you for trying to reproduce this issue.

> Some of the versions I've used in order to try to reproduce: 52.0, 59.0b12,
> 60.0b8, 62.0, 63.0. 
Actually, this issue was reported for Firefox Developer Edition which is supposed to be on the aurora update channel. While this issue was reported for v64.0b5. I'm still able to reproduce this on 65.0b4. Could you perhaps try with the latest Firefox Developer Edition version and see if you're able to reproduce this issue?

Comment 26

[Andrew Swan [:aswan]]

Vlad reported that he tried this with different versions, which I believe includes 64 or 65 or both.  I don't believe this is likely to be related to what release channel the browser is set to.

Given the lack of similar bug reports, this appears to be something specific to your local machine.  Without more information here, I'm afraid there's not much we can do.  Please feel free to re-open if you can provide additional information that might help us reproduce this.