Last Comment Bug 680802 - Upgrade Firefox when there is an add-on update waiting to be installed hides the add-on
: Upgrade Firefox when there is an add-on update waiting to be installed hides ...
Status: VERIFIED FIXED
[qa!]
: verified-aurora, verified-beta
Product: Toolkit
Classification: Components
Component: Add-ons Manager (show other bugs)
: 7 Branch
: All All
: -- major with 4 votes (vote)
: mozilla10
Assigned To: Dave Townsend [:mossop]
:
Mentors:
http://support.mozilla.com/en-US/kb/a...
: 601937 689752 689842 690020 (view as bug list)
Depends on:
Blocks: 653637 689906
  Show dependency treegraph
 
Reported: 2011-08-21 15:18 PDT by Abhay
Modified: 2011-11-07 09:35 PST (History)
59 users (show)
dtownsend: in‑testsuite+
vlad.mozbugs: in‑litmus+
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---
+
fixed
+
fixed
fixed


Attachments
patch rev 1 (9.33 KB, patch)
2011-09-27 22:53 PDT, Dave Townsend [:mossop]
robert.strong.bugs: review+
Details | Diff | Review
patch rev 2 (9.35 KB, patch)
2011-09-28 10:13 PDT, Dave Townsend [:mossop]
robert.strong.bugs: review+
christian: approval‑mozilla‑aurora+
christian: approval‑mozilla‑beta+
christian: approval‑mozilla‑release+
Details | Diff | Review

Description Abhay 2011-08-21 15:18:23 PDT
User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:7.0) Gecko/20100101 Firefox/7.0
Build ID: 20110816154714

Steps to reproduce:

Upgrade from 6.0 to 7.0 on beta channel.


Actual results:

NoScript addon was uninstalled


Expected results:

Addons should not have been affected
Comment 1 Robert Strong [:rstrong] (use needinfo to contact me) 2011-08-22 10:58:40 PDT
Are you sure it isn't listed in the bottom section of the add-ons manager?
Comment 2 Abhay 2011-08-22 11:32:31 PDT
Yeah pretty sure. I checked about:addons first thing.
Comment 3 Robert Strong [:rstrong] (use needinfo to contact me) 2011-08-22 11:34:55 PDT
Please double check. The installer and the updater both don't touch the user's profile so it is extremely unlikely that the add-on was by removed by either of them. It is possible that the add-ons manager code which does have code that touches the extensions in the user's profile could have removed it.
Comment 4 Abhay 2011-08-22 12:47:49 PDT
I am a s/w developer myself and understand the importance of repros. So I double checked it on both my machines.


I have the same situation on my laptop so next time I restart FF I will come back and report on it when the update gets applied on the restart
Comment 5 Robert Strong [:rstrong] (use needinfo to contact me) 2011-08-30 14:29:43 PDT
Have you had a chance to check yet?
Comment 6 patrik.nyborg 2011-09-27 11:26:38 PDT
I can confirm this (6.0.2=>7.0 release channel).

Mozilla/5.0 (Windows NT 6.1; WOW64; rv:7.0) Gecko/20100101 Firefox/7.0
and
Mozilla/5.0 (Windows NT 6.1; rv:7.0) Gecko/20100101 Firefox/7.0
Comment 7 Graham Gordon 2011-09-27 11:35:54 PDT
I had a similar issue. While I was running the Firefox 6.0beta Linux build, I went to Tools>Addons and noticed that I had a pending update for NoScript (green stripes, etc). I ignored it and went on browsing, thinking that I'll wait till I restart my browser for that update to occur (I generally leave my browser open till the end of the day, then shut it down before shutting down my computer). However, some hours later, I noticed that Firefox was wanting to update to the 7.0beta, so I decided to let it update and restart the browser. It was once that I restarted the browser and it was running as 7.0beta, that I noticed that NoScript was missing. Its .xpi file was in the extension's directory in my profile, but it wasn't listed as an addon by Firefox (other extensions were fine). I then reinstalled NoScript, with no further issues, and the extension had updated fine since.

This was on a Debian Squeeze (32bit) machine.
Comment 8 Robert Strong [:rstrong] (use needinfo to contact me) 2011-09-27 12:26:48 PDT
That sounds like the add-ons manager is not handling add-on updates properly when the application version changes after an application update. What ever the case, application update doesn't touch the profile, modify extensions, etc. so moving over to add-ons manager.
Comment 9 Dave Townsend [:mossop] 2011-09-27 14:32:28 PDT
Confirmed, this line is throwing NS_ERROR_UNEXPECTED for some reason: http://mxr.mozilla.org/mozilla-central/source/toolkit/mozapps/extensions/XPIProvider.jsm#1172
Comment 10 Giorgio Maone [:mao] 2011-09-27 16:18:53 PDT
So, if I understand it right, the downloaded file is kept in the profile (comment 7). 
Does this mean that another Firefox update can detect the mess and fix it?
Comment 11 Giorgio Maone [:mao] 2011-09-27 16:25:54 PDT
And could someone figure out whether it's better removing latest add-on version from AMO or declaring it incompatible with Fx 7?
Comment 12 Kris Maglione [:kmag] 2011-09-27 16:34:43 PDT
Giorgio: I'd recommend removing it. The AMO update check mechanism is supposed to Do The Right Thing in circumstances where the newest version is not marked compatible with a given version and an older is, but that's unfortunately not reliable. If you do remove it, just let me know when you've re-uploaded and I'll review it right away.
Comment 13 jah 2011-09-27 16:37:41 PDT
Had the same problem today. Went from latest release of v6 to v7 on WinXP SP3 and there was no incompatibility warning regarding noscript, it was silently uninstalled.  I'm not sure which version of noscript I was using before it was removed, but it was one of the dev channel releases.
Comment 14 Graham Gordon 2011-09-27 16:39:05 PDT
(In reply to Giorgio Maone from comment #10)
> So, if I understand it right, the downloaded file is kept in the profile
> (comment 7).

Yes, I noticed that the new .xpi file was in the profile (in the extensions directory), yet wasn't installed and also missing in about:addons after the browser update and restart.
Comment 15 Dave Townsend [:mossop] 2011-09-27 16:48:32 PDT
I've figured out what the problem here is and it will likely affect any user that has a new add-on waiting to install or an update to an existing add-on waiting to be installed at the time they update from anything earlier than Firefox 7 to Firefox 7 or later.

The problem stems from how the add-on install process works. We download the XPI and store it in a staging directory. We also load the information from install.rdf into a JS object. Any updated compatibility information (as well as other things I'm sure I'm not remembering) is applied to this JS object and then it is written alongside the XPI as a JSON text file. After restarting Firefox loads the JSON data rather than needing to parse install.rdf and do compatibility checks a second time.

Normally that works out fine however if the version of Firefox changes in the meantime and new fields are now supported in install.rdf the loaded JSON data won't contain those new fields, best case the install/update will complete but the database won't contain the correct data from install.rdf, worst case the install fails. In this case the latter is happening. Bug 653637 introduced the new fields that is causing this.

I have a hack that sort of solves the problem but I have a strong feeling that it may break things. The better solution is probably to do a better job of passing the data between Firefox instances and I have a few ideas of how to do that that I need to investigate further.
Comment 16 Dave Townsend [:mossop] 2011-09-27 16:56:21 PDT
(In reply to Graham Gordon from comment #14)
> (In reply to Giorgio Maone from comment #10)
> > So, if I understand it right, the downloaded file is kept in the profile
> > (comment 7).
> 
> Yes, I noticed that the new .xpi file was in the profile (in the extensions
> directory), yet wasn't installed and also missing in about:addons after the
> browser update and restart.

That is a strange thing and it has the weird side effect that if the user installs a different add-on then NoScript would suddenly be detected and installed again.
Comment 17 Giorgio Maone [:mao] 2011-09-27 16:58:51 PDT
(In reply to Dave Townsend (:Mossop) from comment #15)

> I have a hack that sort of solves the problem but I have a strong feeling
> that it may break things. The better solution is probably to do a better job
> of passing the data between Firefox instances and I have a few ideas of how
> to do that that I need to investigate further.

"Solve" and "solution" like in just "this won't happen again on next major update" or also "next minor update will complete broken installation/updates"?
Comment 18 Dave Townsend [:mossop] 2011-09-27 17:09:22 PDT
(In reply to Giorgio Maone from comment #17)
> (In reply to Dave Townsend (:Mossop) from comment #15)
> 
> > I have a hack that sort of solves the problem but I have a strong feeling
> > that it may break things. The better solution is probably to do a better job
> > of passing the data between Firefox instances and I have a few ideas of how
> > to do that that I need to investigate further.
> 
> "Solve" and "solution" like in just "this won't happen again on next major
> update" or also "next minor update will complete broken
> installation/updates"?

The code changes would mean that it won't happen again the next time we introduce new fields to install.rdf. I suspect that the next update to Firefox would make it re-detect the present XPI in the profile already.
Comment 19 Kris Maglione [:kmag] 2011-09-27 17:18:15 PDT
(In reply to Dave Townsend (:Mossop) from comment #18)
> The code changes would mean that it won't happen again the next time we
> introduce new fields to install.rdf. I suspect that the next update to
> Firefox would make it re-detect the present XPI in the profile already.

Can we confirm that? If it is the case, is there something else we can do to automatically re-install the add-ons that were uninstalled from this bug for users with automatic updates? If not, can we push out a minor release with this fixed so that it happens anyway? Millions of users having their favorite add-ons disappear is not good for Firefox's public image.
Comment 20 Dave Townsend [:mossop] 2011-09-27 17:22:30 PDT
(In reply to Kris Maglione [:kmag] from comment #19)
> (In reply to Dave Townsend (:Mossop) from comment #18)
> > The code changes would mean that it won't happen again the next time we
> > introduce new fields to install.rdf. I suspect that the next update to
> > Firefox would make it re-detect the present XPI in the profile already.
> 
> Can we confirm that? If it is the case, is there something else we can do to
> automatically re-install the add-ons that were uninstalled from this bug for
> users with automatic updates? If not, can we push out a minor release with
> this fixed so that it happens anyway? Millions of users having their
> favorite add-ons disappear is not good for Firefox's public image.

I'm working with the release drivers as we speak to figure out the best path forwards
Comment 21 Henrik Skupin (:whimboo) 2011-09-27 17:23:00 PDT
Dave, is there anything QA could help at this stage?
Comment 22 Dave Townsend [:mossop] 2011-09-27 17:28:45 PDT
(In reply to Kris Maglione [:kmag] from comment #19)
> (In reply to Dave Townsend (:Mossop) from comment #18)
> > The code changes would mean that it won't happen again the next time we
> > introduce new fields to install.rdf. I suspect that the next update to
> > Firefox would make it re-detect the present XPI in the profile already.
> 
> Can we confirm that? If it is the case, is there something else we can do to
> automatically re-install the add-ons that were uninstalled from this bug for
> users with automatic updates? If not, can we push out a minor release with
> this fixed so that it happens anyway? Millions of users having their
> favorite add-ons disappear is not good for Firefox's public image.

I've also just confirmed that this works, it reverts back to the previous version of NoScript, but that'd then automatically updated in the next day.
Comment 23 Dave Townsend [:mossop] 2011-09-27 17:29:16 PDT
(In reply to Henrik Skupin (:whimboo) from comment #21)
> Dave, is there anything QA could help at this stage?

Not right now thanks
Comment 24 christian 2011-09-27 17:37:42 PDT
(In reply to Dave Townsend (:Mossop) from comment #22)
> (In reply to Kris Maglione [:kmag] from comment #19)
> > (In reply to Dave Townsend (:Mossop) from comment #18)
> > > The code changes would mean that it won't happen again the next time we
> > > introduce new fields to install.rdf. I suspect that the next update to
> > > Firefox would make it re-detect the present XPI in the profile already.
> > 
> > Can we confirm that? If it is the case, is there something else we can do to
> > automatically re-install the add-ons that were uninstalled from this bug for
> > users with automatic updates? If not, can we push out a minor release with
> > this fixed so that it happens anyway? Millions of users having their
> > favorite add-ons disappear is not good for Firefox's public image.
> 
> I've also just confirmed that this works, it reverts back to the previous
> version of NoScript, but that'd then automatically updated in the next day.

A) I'm not exactly sure this is millions of users. Yes, we have reports in this bug but we aren't seeing widespread problems yet (admittedly could be due to a lag in feedback channels)

B) Does uninstalling a different add-on trigger the "hidden" ones to show up / reinstall as well?
Comment 25 Kris Maglione [:kmag] 2011-09-27 17:44:48 PDT
(In reply to Christian Legnitto [:LegNeato] from comment #24)
> A) I'm not exactly sure this is millions of users. Yes, we have reports in
> this bug but we aren't seeing widespread problems yet (admittedly could be
> due to a lag in feedback channels)

I'm basing my assumption on the fact that this seems to be 100%
reproducable when an add-on update was downloaded just before a
Firefox update. NoScript has over 2 million users, and Giorgio
pushed out an update several hours before the Firefox release team
did. There have also been reports of affected AdBlock Plus users. So
this easily has the potential to affect millions of users and I'd
rather err on the side of caution.
Comment 26 Dave Townsend [:mossop] 2011-09-27 17:46:03 PDT
(In reply to Christian Legnitto [:LegNeato] from comment #24)
> B) Does uninstalling a different add-on trigger the "hidden" ones to show up
> / reinstall as well?

Yes
Comment 27 Justin Scott [:fligtar] 2011-09-27 18:08:56 PDT
(In reply to Christian Legnitto [:LegNeato] from comment #24)
> A) I'm not exactly sure this is millions of users. Yes, we have reports in
> this bug but we aren't seeing widespread problems yet (admittedly could be
> due to a lag in feedback channels)
> 

I wouldn't expect widespread reports of this for days or weeks -- most users won't notice their add-ons are gone immediately.

The day before and of a Firefox release is always the busiest time for putting out add-on updates.
Comment 28 Giorgio Maone [:mao] 2011-09-27 18:22:11 PDT
I removed the latest versions of my add-on both from the beta and the dev channel, but as a more general stop-gap measure couldn't AMO temporarily stop answering add-on update pings which come from Gecko 4-6.x?
Comment 29 Justin Scott [:fligtar] 2011-09-27 18:25:55 PDT
(In reply to Giorgio Maone from comment #28)
> I removed the latest versions of my add-on both from the beta and the dev
> channel, but as a more general stop-gap measure couldn't AMO temporarily
> stop answering add-on update pings which come from Gecko 4-6.x?

We'd have to devise a pretty complicated solution in order to do that, because most add-ons aren't explicitly compatible with Firefox releases anymore and depend on AMO update responses saying they are compatible.
Comment 30 Kris Maglione [:kmag] 2011-09-27 18:38:30 PDT
(In reply to Justin Scott [:fligtar] from comment #29)
> (In reply to Giorgio Maone from comment #28)
> > I removed the latest versions of my add-on both from the beta and the dev
> > channel, but as a more general stop-gap measure couldn't AMO temporarily
> > stop answering add-on update pings which come from Gecko 4-6.x?
> 
> We'd have to devise a pretty complicated solution in order to do that,
> because most add-ons aren't explicitly compatible with Firefox releases
> anymore and depend on AMO update responses saying they are compatible.

I think he means pings coming from Gecko 4-6. I.e., for requests where currentAppVersion is less than 7, return no results until this issue has been sorted out.
Comment 31 Mardeg 2011-09-27 20:26:41 PDT
*** Bug 689842 has been marked as a duplicate of this bug. ***
Comment 32 Patrick Elliott 2011-09-27 20:42:44 PDT
Just to be clear, I ran into this too. Its not uncommon for Firefox to state that some addons are "incompatible", and it will, usually, update some of them, when installing the new version. In my case, I got a list, which I mostly didn't pay attention to, except that I know it included Linkification (self adjusted, because its one of those that half the time doesn't update at all), and HTTPS-Everywhere, which I very recently added. I don't know if Noscript was "in the list" or not, but a number where listed as, "Current version is not compatible, check for updates?" As soon as the client loaded, every single thing that was in that list was just flat missing. Once I "installed" another plugin, as suggested by someone here, and restarted, they all reappeared.

It should be noted that, for me, installing a new Noscript "failed" to make this happen, or make it reappear, I had to install a completely different plugin, picked semi-randomly out of the ones available. So, something in the process of checking for compatibility and updating the list of new versions to load, failed. Its possible it effected only very recent additions, since it killed ones that I installed and/or manually updated, the last time I got such an update message as well. Everything installed "prior" to that update, stayed. And I know Noscript updated since then, I added the HTTPS one after that, and I also hand patched Linkification, to make it compatible (since I didn't figure what it did what likely broken between versions). I will bet donuts to dollars that everything else listed was in the same category, with the exception of a few that really are out of date, and I just haven't gotten rid of (or can't... I just love ones that refuse to let you uninstall them...).
Comment 33 Dave Townsend [:mossop] 2011-09-27 20:56:59 PDT
(In reply to Patrick Elliott from comment #32)
> Just to be clear, I ran into this too. Its not uncommon for Firefox to state
> that some addons are "incompatible", and it will, usually, update some of
> them, when installing the new version. In my case, I got a list, which I
> mostly didn't pay attention to, except that I know it included Linkification
> (self adjusted, because its one of those that half the time doesn't update
> at all), and HTTPS-Everywhere, which I very recently added. I don't know if
> Noscript was "in the list" or not, but a number where listed as, "Current
> version is not compatible, check for updates?" As soon as the client loaded,
> every single thing that was in that list was just flat missing. Once I
> "installed" another plugin, as suggested by someone here, and restarted,
> they all reappeared.

The list you refer to, was it something that displayed while you were still running Firefox 6, part of the UI telling you that a new version of Firefox was available, or was it something that displays after you first started Firefox 7?
Comment 34 Patrick Elliott 2011-09-27 21:27:18 PDT
Hmm. Hard to say, actually. I think the list was shown when starting the new one, maybe, then there was an error, during the actual start up, in a sort of off color, yellowish, box, during the step where updates to the addons would normally happen, which said that there had been a problem installing new version of the addons. The original list is the one that always shows up when patching versions, and a conflict happens, so I am pretty sure it happened when ever that does normally. But the error happened "during" the update for new addons, so.. I think that happens during start up, not in the old one.

In any case, I think Firefox does realize something went wrong, hence the error message, during the step where download of the addons would happen, after starting up, but it didn't know what to do about it.
Comment 35 Dave Townsend [:mossop] 2011-09-27 21:31:36 PDT
(In reply to Patrick Elliott from comment #34)
> Hmm. Hard to say, actually. I think the list was shown when starting the new
> one, maybe, then there was an error, during the actual start up, in a sort
> of off color, yellowish, box, during the step where updates to the addons
> would normally happen, which said that there had been a problem installing
> new version of the addons. The original list is the one that always shows up
> when patching versions, and a conflict happens, so I am pretty sure it
> happened when ever that does normally. But the error happened "during" the
> update for new addons, so.. I think that happens during start up, not in the
> old one.

The bug we have here happens before the UI you're talking about appears so the UI simply wouldn't contain add-ons that got hit by it. You're also saying you lost a lot of add-ons, some of which definitely wouldn't have had any updates waiting to install before you upgraded. So either we don't yet correctly understand this bug, or there is a different bug that you hit.
Comment 36 Chris Beard 2011-09-27 21:51:57 PDT
Our China team reports that they are seeing this across their entire user base as their localized distribution of Firefox relies upon a bundle of add-ons to package and deliver their local market adaptations -- and we updated all of those add-ons this week ahead of the release
Comment 37 Patrick Elliott 2011-09-27 22:08:25 PDT
Well, as I said, my guess is that it may effect addons that "only" where updated "recently", like within weeks of the new patch. The bug may be happening before that UI appears, but the UI itself, in my case, just happened to catch that something went odd. I am not entirely certain which one though. Thinking on it.. I may have seen one, on shutdown, that said they where not compatible, then one on start, which showed "others" which where not up to date (legitimately), but not the ones that had gone missing, and *then* a message, telling me that something went badly wrong. But I am not 100% certain of the order.

But, now that you mention it, its likely the bug, and the initial list, happened in the process of the first one closing down. In any case, all the "lost" ones where, as with Chris, *recent*, and they all popped up, and without "incompatibility" messages, as soon as a new addon was put in the queue, and a new restart was done.
Comment 38 Dave Townsend [:mossop] 2011-09-27 22:53:34 PDT
Created attachment 562976 [details] [diff] [review]
patch rev 1

This is the patch that seems like the best solution. We still cache the entire manifest when preparing to install (for backwards compatibility) however on the startup it completely loads the install manifest from install.rdf (thus parsing it and getting all the fields that the new Firefox supports) and then also loads the cached manifest and copies across only certain appropriate properties. If any of the properties expected seem to be missing then we just ignore and carry on. This should guarantee a safe and working add-on install.

There is a small performance penalty associated here because we have to read install.rdf on startup for every new add-on. I haven't measured the cost of that yet but I expect it to be low and as it is only for the case when a new add-on is installed I'd expect it to be worth it for correcting this.

I think the only risk here is that if I have missed any properties to copy across. That would appear as the add-on changing state in some way on upgrade (maybe getting disabled etc.) but I am growing confident that I have them all. I'll be double checking that in the morning but figured I might as well put the patch up now as adding properties to that list won't really change the review. We can also be over-zealous about listing properties there (maybe even just try to copy all properties) because the new code will simply ignore any it doesn't find and the cached properties will always be the same or newer than those loaded from install.rdf.

The testcase from bug 664833 already checks installing an update when the DB schema and app version changes. The only thing missing was to make the cached manifest have a necessary property missing so I updated the test to do that. Without this fix the test fails in the same way that I've been seeing in this bug.

I've pushed this to the try server, if people want to test builds they will be showing up at http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/dtownsend@mozilla.com-1cb2d7c3b19b
Comment 39 Dave Townsend [:mossop] 2011-09-27 22:57:45 PDT
(In reply to Dave Townsend (:Mossop) from comment #38)
> The testcase from bug 664833 already checks installing an update when the DB
> schema and app version changes. The only thing missing was to make the
> cached manifest have a necessary property missing so I updated the test to
> do that. Without this fix the test fails in the same way that I've been
> seeing in this bug.

Forgot to add, the testcase here is good and I think an accurate test for this, however it suffers from the same problem that much of our automated tests for updates do, it doesn't actually test moving from one version of Firefox to another in reality. So we certainly want a litmus test for this scenario.
Comment 40 Robert Strong [:rstrong] (use needinfo to contact me) 2011-09-27 23:06:33 PDT
Comment on attachment 562976 [details] [diff] [review]
patch rev 1

Looks like a decent approach.

btw: there is a bug to test app updates after the nightly build. Perhaps a similar test for this case should be done at the same time.
Comment 41 Henrik Skupin (:whimboo) 2011-09-28 03:25:55 PDT
(In reply to Dave Townsend (:Mossop) from comment #39)
> tests for updates do, it doesn't actually test moving from one version of
> Firefox to another in reality. So we certainly want a litmus test for this
> scenario.

We already have a litmus test for it. Sadly it's only part of the major update test scenario. Since we do minor updates now, those tests haven't been executed. We should create a test group for minor updates and include all appropriate tests from the MU test group. I have already started this discussion so we can hopefully proceed on it quickly.
Comment 42 Steffen Wilberg 2011-09-28 03:33:10 PDT
This has been reported by several users in a German IT news site:
the update deleted Adblock Plus (http://www.heise.de/newsticker/foren/S-Firefox-Update-loescht-Adblock-Plus/forum-210502/msg-20851562/read/)
Comment 43 Loic 2011-09-28 04:00:48 PDT
I guess bug 689752 (NoScript icon disappeared from add-on manager after auto-updating to Firefox 7) is a dupe of this bug.

Issue reported here http://forums.informaction.com/viewtopic.php?p=31326#p31326 with NoScript.
Comment 44 Henrik Skupin (:whimboo) 2011-09-28 04:12:16 PDT
*** Bug 689752 has been marked as a duplicate of this bug. ***
Comment 45 Wladimir Palant 2011-09-28 04:50:34 PDT
I did some testing and can confirm that uninstalling another extension resolves the problem - as does any other change to your extensions (enabling/disabling an extension, updating some extension etc). The add-on isn't really uninstalled and stays in the extensions/ directory, the update is actually applied correctly. It is only that the add-on manager temporarily "forgets" about it. I also verified that a minor browser update fixes the issue automatically - it forces the add-on manager to check the list of add-ons again.

So far I have only seen one report of this issue (https://addons.mozilla.org/addon/adblock-plus/reviews/313717/), no idea how many Adblock Plus users are affected. Is removing the update from AMO really the best course of action? I am pretty certain that I will receive lots of complains about Adblock Plus not being compatible with Firefox 7 then - install.rdf of Adblock Plus 1.3.9 lists Firefox 7.0a1 as maxVersion which means trouble even if AMO says something different.
Comment 46 VanillaMozilla 2011-09-28 07:08:58 PDT
*** Bug 601937 has been marked as a duplicate of this bug. ***
Comment 47 Anthony Hughes (:ashughes) [GFX][QA][Mentor] 2011-09-28 07:20:52 PDT
Just want to get a clear set of steps to reproduce this bug so QA can quickly verify the fix. Please confirm these steps:

1) Install Firefox <7.0
2) Install an add-on which is compatible to Firefox 7.0 but not the latest version of the add-on
3) Restart Firefox
4) Check for and download the update for the add-on (DO NOT INSTALL IT)
5) Check for and download the update for Firefox 7.0.1
6) Restart Firefox to install the update

Expected Result:
Firefox and add-ons updated

Please confirm the above steps. Also, what is the expected update path / test for:
 * 4.* users
 * 5.* users
 * 6.* users
 * 7.0 users
 * 4b*/ 5b* / 6b* users
 * 3.6 Major Update
Comment 48 Dave Townsend [:mossop] 2011-09-28 08:16:20 PDT
(In reply to Anthony Hughes, Mozilla QA (irc: ashughes) from comment #47)
> Just want to get a clear set of steps to reproduce this bug so QA can
> quickly verify the fix. Please confirm these steps:
> 
> 1) Install Firefox <7.0
> 2) Install an add-on which is compatible to Firefox 7.0 but not the latest
> version of the add-on
> 3) Restart Firefox
> 4) Check for and download the update for the add-on (DO NOT INSTALL IT)
> 5) Check for and download the update for Firefox 7.0.1
> 6) Restart Firefox to install the update
> 
> Expected Result:
> Firefox and add-ons updated

These look like the correct STR to me.

I would also suggest a second set of steps for verification:

1) Install Firefox <7.0
2) Install an add-on which is compatible to Firefox 7.0 but not the latest version of the add-on
3) Restart Firefox
4) Check for and download the update for the add-on (DO NOT INSTALL IT)
5) Exit and install Firefox 7.0
6) Check that the add-on is now missing from the add-ons manager
7) Check for and download the update for Firefox 7.0.1
8) Restart Firefox to install the update

Expected Result:
The add-on that disappeared in step 6 should have reappeared.
Comment 49 Maniac Vlad Florin (:vladmaniac) 2011-09-28 09:27:08 PDT
Can confirm the fix for upgrading from 6.0 to try server build 
http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/dtownsend@mozilla.com-1cb2d7c3b19b/
Comment 50 Chimel 2011-09-28 10:04:44 PDT
I lost AddBlock Plus, had to reinstall manually.
http://www.reddit.com/r/firefox/comments/ktvyl/upgrading_to_firefox_7_uninstalls_adblock_plus/

In addition to fixing the bug, can you:
1) Add this test case (add-on that has a pending update) to your normal test cases
2) Notify users of any add-on that was disabled or removed after a Firefox upgrade

Given the new fast dev cycle between upgrades, it is really important to log every anomaly instead of just ignoring it.

I would have liked to test the patch, but my add-ons are now up to date.
It would be great if you could create a test add-on that users can update locally to try and repro this type of bug, instead of relying on actual add-ons that may or may not have been updated.
Comment 51 Dave Townsend [:mossop] 2011-09-28 10:13:43 PDT
Created attachment 563099 [details] [diff] [review]
patch rev 2

Made the minor change of adding applyBackgroundUpdates to the list of properties copied across, otherwise identical, probably doesn't need re-review but better safe than sorry
Comment 52 Dave Townsend [:mossop] 2011-09-28 11:47:23 PDT
Pushed to trunk: https://hg.mozilla.org/mozilla-central/rev/80b55a615ac9
Comment 53 Dave 2011-09-28 12:08:02 PDT
*** Bug 690020 has been marked as a duplicate of this bug. ***
Comment 54 christian 2011-09-28 12:32:47 PDT
Comment on attachment 563099 [details] [diff] [review]
patch rev 2

Approved everywhere. Land all the places!
Comment 55 christian 2011-09-28 12:34:19 PDT
To be clear we'll want to land this on the "default" branch of:

releases/mozilla-aurora
releases/mozilla-beta
releases/mozilla-release

as well. Thanks!
Comment 57 Henrik Skupin (:whimboo) 2011-09-28 13:45:08 PDT
(In reply to Chimel from comment #50)
> It would be great if you could create a test add-on that users can update
> locally to try and repro this type of bug, instead of relying on actual
> add-ons that may or may not have been updated.

I have created an extension which demonstrates that and will be compatible with all future versions of Firefox. We will use it for our Litmus test which needs to be added for this particular case. You can find it here:

https://www.mozqa.com/data/firefox/addons/update-nonrestartless/
Comment 58 Justin Scott [:fligtar] 2011-09-28 14:06:49 PDT
Since it hasn't been linked here yet, in addition to the fix going into Firefox, we've also created a workaround to recover any hidden add-ons.

http://blog.mozilla.com/addons/2011/09/28/issue-discovered-with-firefox-add-on-upgrades/
http://support.mozilla.com/en-US/kb/add-ons-hidden-after-updating-firefox-7
Comment 59 stefan.blumenrath 2011-09-28 15:15:27 PDT
(In reply to Justin Scott [:fligtar] from comment #58)
> Since it hasn't been linked here yet, in addition to the fix going into
> Firefox, we've also created a workaround to recover any hidden add-ons.

Great, unfortunaly this won't work in SeaMonkey. I don't dare just to change the install.rdf - would that help?

(I've lost AB, but was able to reinstall, after reinstalling some filters disappeared).
Comment 60 Tony Mechelynck [:tonymec] 2011-09-28 15:27:45 PDT
(In reply to stefan.blumenrath from comment #59)
> (In reply to Justin Scott [:fligtar] from comment #58)
> > Since it hasn't been linked here yet, in addition to the fix going into
> > Firefox, we've also created a workaround to recover any hidden add-ons.
> 
> Great, unfortunaly this won't work in SeaMonkey. I don't dare just to change
> the install.rdf - would that help?
> 
> (I've lost AB, but was able to reinstall, after reinstalling some filters
> disappeared).

My guess is that the following, placed where it belongs in install.rdf, ought to be enough:

      <!-- SeaMonkey version -->
        <em:targetApplication>
            <Description>
                <em:id>{92650c4d-4b8e-4d2a-b7eb-24ecf4f6b63a}</em:id>
                <em:minVersion>2.4a1</em:minVersion>
                <em:maxVersion>2.7.*</em:maxVersion>
            </Description>
        </em:targetApplication>

but Mossop knows better than me if there's no other subtle caveat.
Comment 61 Justin Wood (:Callek) 2011-09-28 15:28:44 PDT
(In reply to stefan.blumenrath from comment #59)
> (In reply to Justin Scott [:fligtar] from comment #58)
> > Since it hasn't been linked here yet, in addition to the fix going into
> > Firefox, we've also created a workaround to recover any hidden add-ons.
> 
> Great, unfortunaly this won't work in SeaMonkey. I don't dare just to change
> the install.rdf - would that help?

That would work, but there is also a version on people.m.o that installs into SeaMonkey just fine: https://people.mozilla.com/%7Edtownsend/shared/bug680802/hotfix.xpi

The trick is the addon has toolkit@ entry, but AMO needs a specific app entry, sadly.
Comment 62 Justin Scott [:fligtar] 2011-09-28 15:33:53 PDT
You don't need to make any changes for that to work with SeaMonkey; simply installing, removing, or enabling/disabling any add-on and restarting Firefox will fix the issue.
Comment 63 Tony Mechelynck [:tonymec] 2011-09-28 16:05:07 PDT
Update: the version at AMO now supports SeaMonkey too.
Comment 64 Cristian Nicolae (:xti) 2011-09-29 03:01:34 PDT
I've performed the following steps:

1. Install Firefox 6.0
2. Install an add-on which is compatible to Firefox 7.0 but not the latest version of the add-on (Try URL Fixer by installing Version 3.0)
3. Restart Firefox to install the add-on
4. Check for and download the update for the add-on (DO NOT INSTALL IT BY RESTARTING), it should update to 3.1
5. Exit or kill Firefox 6.0 and install Firefox 7.0.1 by replacing the install
6. Check that the add-on is not missing from the add-ons manager, and that it is active and usable.

It seems that this issue is not reproducible any more. I've verified it with both restartless (e.g., an add-on such as Cleary) and restart add-ons (e.g, Persona's), but works fine.

--
Mozilla/5.0 (Android;Linux armv7l;rv:7.0.1)Gecko/20110928
Firefox/7.0.1 Fennec/7.0.1
Device: Samsung Google Nexus S
OS: Android 2.3
Comment 65 Vlad [QA] 2011-09-29 07:03:11 PDT
Setting resolution to verified Fixed on:

Mozilla/5.0 (X11; Linux i686; rv:7.0.1) Gecko/20100101 Firefox/7.0.1
Mozilla/5.0 (X11; Linux x86_64; rv:7.0.1) Gecko/20100101 Firefox/7.0.1

Mozilla/5.0 (Windows NT 5.0; rv:7.0.1) Gecko/20100101 Firefox/7.0.1
Mozilla/5.0 (Windows NT 5.1; rv:7.0.1) Gecko/20100101 Firefox/7.0.1
Mozilla/5.0 (Windows NT 6.0; rv:7.0.1) Gecko/20100101 Firefox/7.0.1


Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:7.0.1) Gecko/20100101 Firefox/7.0.1
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:7.0.1) Gecko/20100101 Firefox/7.0.1

Mozilla/5.0 (Windows NT 6.1; rv:7.0.1) Gecko/20100101 Firefox/7.0.1
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:7.0.1) Gecko/20100101 Firefox/7.0.1

This bug is no longer reproducible on Firefox 7.0.1.
Comment 66 Anthony Hughes (:ashughes) [GFX][QA][Mentor] 2011-09-29 07:06:22 PDT
(In reply to Vlad [QA] from comment #65)
> This bug is no longer reproducible on Firefox 7.0.1.

Could you comment the steps you used to verify this (similar to how Cristian did in comment 64)? This way we have 100% visibility into the fix verification.

Thanks
Comment 67 Dave Townsend [:mossop] 2011-09-29 07:59:12 PDT
I don't see it explicitly called out, did we also verify the steps in comment 48? I want to be sure we've fixing the problem for people that already saw it as well as for new updaters
Comment 68 Aaron Train [:aaronmt] 2011-09-29 08:06:06 PDT
(In reply to Dave Townsend (:Mossop) from comment #67)
> I don't see it explicitly called out, did we also verify the steps in
> comment 48? I want to be sure we've fixing the problem for people that
> already saw it as well as for new updaters

Yes. I will update our mobile notes; 6.0 -> 7.0 -> 7.0.1 update process resolves the issue
Comment 69 Henrik Skupin (:whimboo) 2011-09-29 11:50:13 PDT
Also can we get someone from the China team to verify the fix? Also for partner builds I would kinda like to see a confirmation that everything is ok. Carsten?
Comment 70 juan becerra [:juanb] 2011-09-29 14:05:03 PDT
I've been trying to reproduce the problem using builds from firefox.com.cn, however I don't know where to get older versions of the bundled addons such that I can check for addon updates in order to reproduce the problem.

What I've observed is that when I do an update from 6.0.2 to the latest version, there is a dialog for addon updates, which it then downloads and the application starts up fine with no missing addons.
Comment 71 a_geek 2011-09-30 01:14:48 PDT
Maybe the addon, and 7.0.1, only work if one does the upgrade in one go?

I recently upgraded to 7.0 and lost a number of addons, then reinstalled some from addons.mozilla.org, then, today, found that a number of addons were disabled again. Seeing that, I exited FF7 and installed the upgrade to 7.0.1 (systemwide), then restarted FF again. But my addons are still missing. Then I also installed this "addon recovery tool" addon and followed the instructions, but my addons are still hidden/not there. But addons.sqlite in the profile still lists them. It would be nice if FF would recover all the addons automatically.

This is with FF 7.0.1 (Mozilla/5.0 (X11; Linux i686; rv:7.0.1) Gecko/20100101 Firefox/7.0.1).

Imho, the bug is not fixed.
Comment 72 Steffen Wilberg 2011-09-30 02:46:06 PDT
(In reply to a_geek from comment #71)
> I recently upgraded to 7.0 and lost a number of addons, then reinstalled
> some from addons.mozilla.org, then, today, found that a number of addons
> were disabled again.
Are they present in the Addons-Manager and marked as disabled?
Or are they hidden, not visible in the Addons-Manager?

Can you find them in your profile folder (about:support has a button to open it), in the "extensions" subfolder?
Comment 73 a_geek 2011-09-30 07:38:44 PDT
They are not visible in the addons manager. I've reinstalled most of them in the meantime because I needed them, but they were present in the addons.sqlite file. For those that I didn't reinstall yet, they are not listed under "Extensions" on "about:support". They have also been removed from the extensions subdir of the profile, as a cursory look indicates, but are (were?) still present in addons.sqlite.
Comment 74 Maniac Vlad Florin (:vladmaniac) 2011-09-30 08:46:40 PDT
Litmus test https://litmus.mozilla.org/show_test.cgi?id=33510
Comment 75 Anthony Hughes (:ashughes) [GFX][QA][Mentor] 2011-09-30 09:34:34 PDT
(In reply to a_geek from comment #73)
> They are not visible in the addons manager. I've reinstalled most of them in
> the meantime because I needed them, but they were present in the
> addons.sqlite file. For those that I didn't reinstall yet, they are not
> listed under "Extensions" on "about:support". They have also been removed
> from the extensions subdir of the profile, as a cursory look indicates, but
> are (were?) still present in addons.sqlite.

I'm thinking this is a separate issue which is a fallout case from this bug. Should a new bug be filed and investigated?
Comment 76 Dave Townsend [:mossop] 2011-09-30 18:09:13 PDT
(In reply to Anthony Hughes, Mozilla QA (irc: ashughes) from comment #75)
> (In reply to a_geek from comment #73)
> > They are not visible in the addons manager. I've reinstalled most of them in
> > the meantime because I needed them, but they were present in the
> > addons.sqlite file. For those that I didn't reinstall yet, they are not
> > listed under "Extensions" on "about:support". They have also been removed
> > from the extensions subdir of the profile, as a cursory look indicates, but
> > are (were?) still present in addons.sqlite.
> 
> I'm thinking this is a separate issue which is a fallout case from this bug.
> Should a new bug be filed and investigated?

Yeah almost all the symptoms listed are different to those that this bug causes so we should get a new bug filed. I wouldn't suspect it was fallout from this though as apparently some add-ons went missing in Firefox 7.0.
Comment 77 a_geek 2011-10-01 03:03:34 PDT
I didn't mean to confuse things. If I understand you correctly, I should submit a new bug. But please note that I currently can't reproduce the problem, as I have destroyed most of the original evidence (by running installs and upgrades). Any suggestions for components etc., please?
Comment 78 Henrik Skupin (:whimboo) 2011-10-01 05:56:22 PDT
(In reply to a_geek from comment #77)
> installs and upgrades). Any suggestions for components etc., please?

Same product and component as this bug, please.
Comment 79 Adam Williamson 2011-10-02 10:09:27 PDT
The (first) workaround link above is no use because the information has been removed from the page and replaced with an instruction to update to 7.0.1. Folks, just because you've released 7.0.1 doesn't mean your downstreams all have too. Many Linux users run Firefox from distro packages, not direct from upstream, and not all downstreams have pushed out the 7.0.1 update yet. Removing the workaround text is plain rude.

The later workaround link in comment #61 did work for me, if anyone else is looking for a workaround.
Comment 80 a_geek 2011-10-03 10:37:43 PDT
(In reply to Henrik Skupin (:whimboo) from comment #78)
> Same product and component as this bug, please.

Done:

https://bugzilla.mozilla.org/show_bug.cgi?id=691402
Comment 81 Henrik Skupin (:whimboo) 2011-10-03 23:09:36 PDT
(In reply to Adam Williamson from comment #79)
> The (first) workaround link above is no use because the information has been
> removed from the page and replaced with an instruction to update to 7.0.1.

About which workaround link you are talking? Can you please tell us the comment number or the URL directly? Thanks.
Comment 83 Loic 2011-10-04 04:29:35 PDT
(In reply to Henrik Skupin (:whimboo) from comment #81)
> (In reply to Adam Williamson from comment #79)
> > The (first) workaround link above is no use because the information has been
> > removed from the page and replaced with an instruction to update to 7.0.1.
> 
> About which workaround link you are talking? Can you please tell us the
> comment number or the URL directly? Thanks.

The workaround has been removed at the revision #17583 of the webpage:
http://support.mozilla.com/en-US/kb/add-ons-hidden-after-updating-firefox-7/compare?locale=en-US&to=17583&from=17582
Comment 84 Verdi [:verdi] 2011-10-05 04:29:43 PDT
(In reply to Adam Williamson from comment #79)
> Many Linux users run Firefox from distro packages, not direct from
> upstream, and not all downstreams have pushed out the 7.0.1 update yet.
> Removing the workaround text is plain rude.

Hi Adam,
I didn't intend to be rude by removing the workaround from the article. I was trying to simplify the instructions but obviously missed a circumstance where the workaround is still needed. I'm in the process of correcting that now. Thanks for bringing this to my attention.
Comment 85 Vlad [QA] 2011-10-11 06:30:21 PDT
I have followed the steps from comment47 and comment 48 in order to verify this on Firefox 7.0.1 (comment65).
If the target milestone set for this bug is mozilla 10, in order to verify this bug, it's enough if I update from an older nightly build to the latest one and check that no add-ons are missing or I have to follow some additional steps? (the same applies to Aurora as well).
Thanks.
Comment 86 Henrik Skupin (:whimboo) 2011-10-11 06:34:47 PDT
(In reply to Vlad [QA] from comment #85)
> If the target milestone set for this bug is mozilla 10, in order to verify
> this bug, it's enough if I update from an older nightly build to the latest
> one and check that no add-ons are missing or I have to follow some
> additional steps? (the same applies to Aurora as well).

You should update from a Nightly builds with a lower version number, means install a Nightly 9.0a1 build first and update to 10.0a1 for your check.
Comment 87 Vlad [QA] 2011-11-07 09:35:33 PST
I have verified using the following steps:

Note: Install Firefox 7.0.1 and change the update channel to "releasetest"
1. Start Firefox 7.0.1.
2. Install NoScript 2.1.4. RESTART.
3. After restart, install Ab Block Plus 1.3.10. DO NOT RESTART.
4. Go to Tools > Add-ons > Extensions. AbBlock should require to restart Firefox.DO NOT RESTART
5. In Add-ons Manager, check for updates. NoScript should update to the latest version and should require to restart Firefox. DO NOT RESTART.
6. Go to Menu > About Firefox check for updates. 

Expected results:
After updating to Firefox 8 RC, both add-ons should be present in the Add-ons Manager, NoScript should be updated to the latest version (2.1.8)

The same behavior applied to Aurora and Nightly. After updating from 8.0a2 to 9.0a2 and 9.0a1 to 10.0a1, both add-ons were present in Add-ons Manager.

I have verified on:
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:8.0) Gecko/20100101 Firefox/8.0
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:9.0a2) Gecko/20111107 Firefox/9.0a2
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:10.0a1) Gecko/20111107 Firefox/10.0a1

Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:8.0) Gecko/20100101 Firefox/8.0
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:9.0a2) Gecko/20111107 Firefox/9.0a2
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:10.0a1) Gecko/20111107 Firefox/10.0a1

Mozilla/5.0 (X11; Linux x86_64; rv:8.0) Gecko/20100101 Firefox/8.0
Mozilla/5.0 (X11; Linux x86_64; rv:9.0a2) Gecko/20111107 Firefox/9.0a2
Mozilla/5.0 (X11; Linux x86_64; rv:10.0a1) Gecko/20111107 Firefox/10.0a1

Mozilla/5.0 (Windows NT 5.1; rv:8.0) Gecko/20100101 Firefox/8.0
Mozilla/5.0 (Windows NT 5.1; rv:9.0a2) Gecko/20111107 Firefox/9.0a2
Mozilla/5.0 (Windows NT 5.1; rv:10.0a1) Gecko/20111107 Firefox/10.0a1

Considering this, setting resolution to Verified Fixed across platforms.

Note You need to log in before you can comment on or make changes to this bug.