Last Comment Bug 590978 - Blocklist personas plus 1.6
: Blocklist personas plus 1.6
Status: RESOLVED FIXED
:
Product: Toolkit
Classification: Components
Component: Blocklisting (show other bugs)
: unspecified
: x86 All
: -- critical (vote)
: ---
Assigned To: Michael Morgan [:morgamic]
:
Mentors:
https://addons.mozilla.org/en-US/fire...
: 594940 (view as bug list)
Depends on: 594940
Blocks: 590608
  Show dependency treegraph
 
Reported: 2010-08-26 11:06 PDT by Ryan Doherty (:rdoherty)
Modified: 2016-03-07 15:30 PST (History)
17 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---
.10+
.10-fixed


Attachments
v1, blocklist with personas entry (3.85 KB, text/plain)
2010-09-09 15:17 PDT, Michael Morgan [:morgamic]
robert.strong.bugs: review+
Details
v1, blocklist.sql to run (249 bytes, application/octet-stream)
2010-09-09 15:18 PDT, Michael Morgan [:morgamic]
no flags Details
v2, removed target application :) (3.73 KB, text/plain)
2010-09-09 15:25 PDT, Michael Morgan [:morgamic]
robert.strong.bugs: review+
Details
v2, simpler sql yum (90 bytes, text/plain)
2010-09-09 15:25 PDT, Michael Morgan [:morgamic]
no flags Details
v3, changed to fit 3.6 -> 3.6.* (3.91 KB, text/plain)
2010-09-13 11:24 PDT, Michael Morgan [:morgamic]
robert.strong.bugs: review+
dveditz: approval1.9.2.10+
Details
v3, updated blocklist sql (90 bytes, text/plain)
2010-09-13 11:25 PDT, Michael Morgan [:morgamic]
no flags Details
blocklist sql (275 bytes, text/plain)
2010-09-13 16:34 PDT, Michael Morgan [:morgamic]
no flags Details

Description Ryan Doherty (:rdoherty) 2010-08-26 11:06:42 PDT
Preemptively filing a bug to blocklist Personas plus 1.6 due to incompatibility with many other add-ons (bug 590823)

version: 1.6
guid: personas@christopher.beard

Let's get the changes ready and QA'd, but not push yet to production.
Comment 1 Michael Morgan [:morgamic] 2010-08-26 11:22:20 PDT
We probably want to avoid blocklisting personas and get the udpate out asap, but I'm overstating the obvious there.
Comment 2 Michael Morgan [:morgamic] 2010-08-26 11:23:11 PDT
Which version ranges do you need to block?  Fx 3.6.* and higher or?
Comment 3 Ryan Doherty (:rdoherty) 2010-08-26 11:48:53 PDT
(In reply to comment #1)
> We probably want to avoid blocklisting personas and get the udpate out asap,
> but I'm overstating the obvious there.

Yep. No eta on a fix yet, so we want to have alternate plans ready.

(In reply to comment #2)
> Which version ranges do you need to block?  Fx 3.6.* and higher or?

3.0 +
Comment 4 Dave Garrett 2010-08-26 12:12:22 PDT
Seeing as the 1.6 update breaks the extension manager, I don't know if an update to fix this will be able to install correctly. Blocklisting may or may not be needed even with a quick fix, but I don't know for sure.
Comment 5 Dave Garrett 2010-09-08 22:57:08 PDT
More bug reports on Bugzilla, new problems reported in bug 590608, and I'm still getting error reports via email and forums.

AMO stats indicate version 1.6 got up to 1.1 million users before bug 590823 was fully complete. If a new and better version isn't pushed out to them within the next week or so to fix this, I recommend blocklisting this version in some form be seriously considered. There was no ASAP update...
Comment 6 Mike Kaply [:mkaply] (Out June 27-July 5) 2010-09-09 06:48:42 PDT
This needs to happen now.

I've looked into this and Personas 1.6 overrides portions of the extension manager with a custom extension manager.

Then when addons try to access the extensions manager, they fail.

And the addons window won't come up to disable Personas Plus.

This is a very serious problem impacting tons of my users and I'm tired of getting bug reports on this.

Please do this now.
Comment 7 Dave Garrett 2010-09-09 07:04:36 PDT
It sounds like we must then have a full hard blocklist, not soft. (i.e. user cannot opt to override the block) In other words, am I right in assuming that affected users won't be able to update to a 1.6.x with things broken, thus it _has_ to be blocked for them to get any update?
Comment 8 Mike Kaply [:mkaply] (Out June 27-July 5) 2010-09-09 07:05:57 PDT
Actually, will even the blacklisting work? Does it depend on extensionmanager.js?
Comment 9 Dave Garrett 2010-09-09 07:06:50 PDT
Er... crap. Would be good to know that....
Comment 10 Dave Garrett 2010-09-09 07:08:00 PDT
Upping severity due to the level of stupidity discovered in this version.
Comment 11 Dave Garrett 2010-09-09 07:10:12 PDT
Well, some people can get around some of the breakage some of the time as discussed in other bugs and elsewhere. Hopefully even if blocklisting is affected they'll get it working at least once, which is all we need to kill it.
Comment 12 Mike Kaply [:mkaply] (Out June 27-July 5) 2010-09-09 07:10:41 PDT
The blocklist code is in nsExtensionManager.js.

There's a very good chance that it won't work either.

I'm not sure what can be done here short of pushing out a new version of
Firefox that explicitly breaks Personas 1.6.
Comment 13 Dave Garrett 2010-09-09 07:11:51 PDT
Does blocklisting work under Firefox safe mode?
Comment 14 Dave Garrett 2010-09-09 07:18:43 PDT
A blocklist entry is also needed just to prevent any people who aren't having issues with 1.6 in the meantime from tripping over the problem when installing an extension that causes total failure.
Comment 15 Dave Garrett 2010-09-09 07:25:44 PDT
(In reply to comment #2)
> Which version ranges do you need to block?  Fx 3.6.* and higher or?

In light of what it's doing I say block for all versions, all applications.
Comment 16 Ryan Doherty (:rdoherty) 2010-09-09 10:23:49 PDT
All of our testing we've done has only found issues with Google Toolbar and Personas Plus. Google Toolbar has an update out with >50% of their users updated. We'll test and wait for higher numbers before releasing the update again.
Comment 17 Ryan Doherty (:rdoherty) 2010-09-09 10:28:54 PDT
Marking as wontfix for now as our testing concludes the Google Toolbar update fixes the issue.
Comment 18 Dave Garrett 2010-09-09 11:35:12 PDT
Good, lets hope that Google gets a fully updated user base to get the problem population down low enough to avoid this specific interaction. There are other problems with this extension, but if the numbers are low enough then avoiding blocklisting a Mozilla Labs extension for a million people would be nice. Much better solution...

Any idea what Google's update rate is? In other words, any idea how much longer this will this continue to be a commonly reported problem?
Comment 19 Ryan Doherty (:rdoherty) 2010-09-09 11:45:53 PDT
(In reply to comment #18)
> Any idea what Google's update rate is? In other words, any idea how much longer
> this will this continue to be a commonly reported problem?

It's pretty fast, in 5 days they achieved about 50%. By the 13th I think it will be high enough to re-release 1.6. They are keeping me updated with their update rates.
Comment 20 Dave Garrett 2010-09-09 11:54:31 PDT
(In reply to comment #19)
> It's pretty fast, in 5 days they achieved about 50%. By the 13th I think it
> will be high enough to re-release 1.6. They are keeping me updated with their
> update rates.

Great progress. However, I'd rather not have a re-release of 1.6 as it can interact and cause problems with other extensions, though less common. Bug 590608 already has a patch working on a 1.6.next which will be better to push.
Comment 21 Mike Kaply [:mkaply] (Out June 27-July 5) 2010-09-09 12:29:36 PDT
Sorry guys, you're wrong

Go grab this.

http://canadiens.nhl.com/club/page.htm?id=52859

Install it. Then install Personas 1.6.

Watch the breakage.

It's not a Google Toolbar only problem.
Comment 22 Mike Kaply [:mkaply] (Out June 27-July 5) 2010-09-09 12:33:14 PDT
Reopening as Google Toolbar is NOT the only one broken.

What Personas is doing is wrong and it has broken 60 or 70 of our addons, in addition to all the other ones mentioned in 590608.
Comment 23 Nils Maier [:nmaier] 2010-09-09 12:48:41 PDT
Blocklisting addons is a responsibility of amo-admins, last time I checked.

mkaply is right in his assessment that Google Toolbar is unlikely the only "victim" here.
Comment 24 Jorge Villalobos [:jorgev] 2010-09-09 13:30:29 PDT
Ideally, Personas 1.6 should be released ASAP with the patch from bug 590608, assuming it fixes the problem (I bet it does). You probably need to use version 1.6.1 instead, so that all users experiencing this problem get the automatic update.

I don't think it's a good idea to re-release 1.6 as is, considering the breakage it's causing to multiple add-ons, and considering that there seems to a simple solution.
Comment 25 Mike Kaply [:mkaply] (Out June 27-July 5) 2010-09-09 13:48:24 PDT
The problem is that users might not get 1.6.1 because Personas 1.6 when combined with any of these addons breaks the actual extension manager...
Comment 26 Mike Kaply [:mkaply] (Out June 27-July 5) 2010-09-09 13:59:12 PDT
I've verified that if your Firefox is in this state, you can't install any XPIs.

So updates are useless. The user stays broke.
Comment 27 Jorge Villalobos [:jorgev] 2010-09-09 14:16:41 PDT
(In reply to comment #25)
> The problem is that users might not get 1.6.1 because Personas 1.6 when
> combined with any of these addons breaks the actual extension manager...

Right, but only a (hopefully small) subset of users have Personas Plus 1.6 + one of the add-ons that cause the bad combo in the Add-ons Manager. Pushing 1.6.1 would reduce those 1M to significantly less.

That is, unless Personas Plus breaks updates altogether by itself.
Comment 28 Mike Kaply [:mkaply] (Out June 27-July 5) 2010-09-09 14:21:54 PDT
Google Toolbar is one of the addons in question. I imagine that's a pretty popular addon.

The scope of this problem is completely unknown.

I hope in the future, you guys won't allow addons on AMO that replace the extension manager.
Comment 29 Nick Nguyen [:osunick] 2010-09-09 14:58:56 PDT
Mossop is investigating the impact of the problem and whether or not blocklist will be effective when Personas Plus 1.6 and a conflicting extension is installed.
Comment 30 Nick Nguyen [:osunick] 2010-09-09 14:59:36 PDT
Morgamic,

Can you get a blocklist patch ready for QA and review ASAP?
Comment 31 Nick Nguyen [:osunick] 2010-09-09 15:02:02 PDT
Blocking should be 1.6 only for all versions of Firefox.  (it's the only way to be sure).

Do not push the blocklist update until we hear from mossop.
Comment 32 Michael Morgan [:morgamic] 2010-09-09 15:02:42 PDT
Okay.
Comment 33 Michael Morgan [:morgamic] 2010-09-09 15:17:01 PDT
Created attachment 473766 [details]
v1, blocklist with personas entry
Comment 34 Michael Morgan [:morgamic] 2010-09-09 15:18:16 PDT
Created attachment 473768 [details]
v1, blocklist.sql to run
Comment 35 Robert Strong [:rstrong] (use needinfo to contact me) 2010-09-09 15:22:13 PDT
Comment on attachment 473766 [details]
v1, blocklist with personas entry

>    <emItem id="personas@christopher.beard">
>      <versionRange minVersion="1.6" maxVersion="1.6">
>        <targetApplication id="{ec8030f7-c20a-464f-9b0e-13a3a9e97384}">
>        </targetApplication>
>      </versionRange>
>    </emItem>
Should blocklist this for all apps so just remove the targetApplication.

r=me with that
Comment 36 Michael Morgan [:morgamic] 2010-09-09 15:25:09 PDT
Created attachment 473771 [details]
v2, removed target application :)
Comment 37 Michael Morgan [:morgamic] 2010-09-09 15:25:39 PDT
Created attachment 473772 [details]
v2, simpler sql yum
Comment 38 Dave Garrett 2010-09-09 15:36:04 PDT
(In reply to comment #28)
> The scope of this problem is completely unknown.

Hit the nail on the head there. I'm pretty sure Google Toolbar is the bulk of the triggers, or at least it's 99% of the triggers I've seen. Don't really know about the rest.

I'm confused as to comment 19 then, as supposedly Google was at least getting theirs updated.

If pushing a 1.6.1 with a fix still breaks things, then yes please, block. Standard procedure is to get the fix out first to minimize the needed blocks, however. In other words a fix should be out by last week.
Comment 39 Dave Garrett 2010-09-09 16:00:52 PDT
(In bug 590608 comment #64)
> (In reply to bug 590608 comment #63)
> > Bug 590978 comment 19 said that Google was getting this thing updated (possibly
> > through their own voodoo). If that's not the case, then yeah, blocklist time.
> > Ryan, which is it?
> 
> Google is updating their add-on with a fix that makes their add-on work, but
> there's no way to track if the users being updated also have Personas Plus
> installed. 

In lieu of this, scratch my all-is-well mood in comment 18.

Whatever. Only question left then, I guess, is if we're waiting for a Personas fix push before blocklisting or blocklisting first. Can the fix get rushed out fast enough?

Though, bug 594940.
Comment 40 Ryan Doherty (:rdoherty) 2010-09-09 16:03:51 PDT
(In reply to comment #39)
> Whatever. Only question left then, I guess, is if we're waiting for a Personas
> fix push before blocklisting or blocklisting first. Can the fix get rushed out
> fast enough?

We're testing now.
Comment 41 Dave Garrett 2010-09-09 16:13:29 PDT
(In reply to comment #40)
> (In reply to comment #39)
> > Whatever. Only question left then, I guess, is if we're waiting for a Personas
> > fix push before blocklisting or blocklisting first. Can the fix get rushed out
> > fast enough?
> 
> We're testing now.

Ok, good.

Even though we all want this fixed asap, I suggest the necessary evil of waiting a day at the most to allow users that can update naturally to do so, thus reducing the number of users to hit the glaring blocklist dialog. (just a suggestion; feel free to ignore me and just kill it now ;)
Comment 42 Dave Townsend [:mossop] 2010-09-09 17:28:17 PDT
(In reply to comment #29)
> Mossop is investigating the impact of the problem and whether or not blocklist
> will be effective when Personas Plus 1.6 and a conflicting extension is
> installed.

Here is what I have discovered.

When the extensions are conflicting extension updates are totally broken. The Blocklist service works in that it will download the new XML but it does not get properly applied so the extension remains enabled.

However when an application update occurs during startup all extensions are disabled and compatibility and blocklist updates do get applied.

To cut a long story short, if we push the blocklist update then the next application update will make the blocklist apply to Personas. This obviously relies on users using Firefox such that a blocklist update is downloaded and then applying the Firefox update.
Comment 43 Ryan Doherty (:rdoherty) 2010-09-09 17:41:11 PDT
(In reply to comment #42)
> Here is what I have discovered.
> 
> When the extensions are conflicting extension updates are totally broken. The
> Blocklist service works in that it will download the new XML but it does not
> get properly applied so the extension remains enabled.
> 
> However when an application update occurs during startup all extensions are
> disabled and compatibility and blocklist updates do get applied.
> 
> To cut a long story short, if we push the blocklist update then the next
> application update will make the blocklist apply to Personas. This obviously
> relies on users using Firefox such that a blocklist update is downloaded and
> then applying the Firefox update.

Crap.

So, we'll need to message users through every channel we have with instructions on how to disable Personas Plus and download the updated version. We'll have an update ready as soon as we can.
Comment 44 Ryan Doherty (:rdoherty) 2010-09-09 17:42:34 PDT
Also, I don't see a point in blocklisting now if it won't fix it for anyone experiencing issues. It would only serve to annoy users who aren't having problems. We can blocklist after we have an update and it has reasonable uptake.
Comment 45 Nick Nguyen [:osunick] 2010-09-09 17:48:44 PDT
I think we should try to accelerate QA for 1.6.1 (stephend is helping) and try to push out an update over the weekend.  Then we block on Monday.  We should also talk to the Firefox folks to see what we can do to prevent this in the next maintenance release of Firefox.

It looks like Google has their own update mechanism and can update their extension even when the EM is disabled.  Ryan, can you confirm this wtih them?
Comment 46 Ryan Doherty (:rdoherty) 2010-09-09 17:53:43 PDT
(In reply to comment #45)
> I think we should try to accelerate QA for 1.6.1 (stephend is helping) and try
> to push out an update over the weekend.  Then we block on Monday.  We should
> also talk to the Firefox folks to see what we can do to prevent this in the
> next maintenance release of Firefox.

Sounds good.

 
> It looks like Google has their own update mechanism and can update their
> extension even when the EM is disabled.  Ryan, can you confirm this wtih them?

They are updating their users, but not all their users are also using Personas Plus. I'll contact them about their update mechanism.
Comment 47 Dave Townsend [:mossop] 2010-09-09 17:57:27 PDT
Getting the blocklist out well in advance of the next app update will mean that more users have downloaded the new blocklist so it is ready to apply when they update.
Comment 48 Nils Maier [:nmaier] 2010-09-09 18:13:36 PDT
(In reply to comment #44)
> Also, I don't see a point in blocklisting now if it won't fix it for anyone
> experiencing issues. It would only serve to annoy users who aren't having
> problems. We can blocklist after we have an update and it has reasonable
> uptake.

Not serving a blocklist now will annoy users who install an "incompatible" addon tomorrow :(
Comment 49 Robert Strong [:rstrong] (use needinfo to contact me) 2010-09-09 21:33:21 PDT
Just talked with Dave regarding this also breaking app update. I believe there is a workaround though. By serving the update with the same version number as the current version of the application and a different build ID then app update won't check for incompatible add-ons
http://mxr.mozilla.org/mozilla1.9.2/source/toolkit/mozapps/update/src/nsUpdateService.js.in#1391

and

http://mxr.mozilla.org/mozilla1.9.2/source/toolkit/mozapps/update/content/updates.js#472

I haven't verified this will work but it should.
Comment 50 Dave Townsend [:mossop] 2010-09-09 21:38:21 PDT
As I was just posting:

Unfortunately I've done some more testing and have hit a problem with what I
said earlier. The extension manager is a pretty critical component that a bunch
of services rely on to function correctly. One of those services is application
update. When application update discovers a new update it will normally check
with the extension manager to see whether the new version will make any
extensions incompatible. With Personas plus leaving the EM broken this causes
the app update check to fail with a strange malformed XML error. Neither
automatic or manual checks will find and install a normal new Firefox version.

Rob's solution looks like it should work though it would appear a bit hacky.
Comment 51 Robert Strong [:rstrong] (use needinfo to contact me) 2010-09-09 21:44:28 PDT
btw: the user could still be presented with the correct version number by using the version property in the update snippet which is what is displayed to the user while extensionVersion is what is used to check if we should check add-on compatibility. The one thing this will do is not notify the user if an extension that is enabled and compatible with the current version will be disabled due to being incompatible with the new version.
Comment 52 Dave Townsend [:mossop] 2010-09-09 21:50:07 PDT
For the trick from comment 42 to work the application version would have to change for the new blocklist to apply, so unless we want to get into writing some custom code into the update we'd have to offer an updated version advertised as the same version as they were running.
Comment 53 Robert Strong [:rstrong] (use needinfo to contact me) 2010-09-09 21:54:45 PDT
The application version would change. The advertised version via the update snippet would not.
Comment 54 Robert Strong [:rstrong] (use needinfo to contact me) 2010-09-09 22:00:12 PDT
To clarify, the update snippet would have an extensionVersion attribute that is the same (used to check whether extension compatibility should be checked) and a version attribute that is the real / new version (used in the UI to display the minor update version). The version that is updated to would be the real / new version. That should work shouldn't it?
Comment 55 Dave Townsend [:mossop] 2010-09-09 22:02:47 PDT
Sounds like it yes
Comment 56 Robert Strong [:rstrong] (use needinfo to contact me) 2010-09-09 22:06:54 PDT
Filed bug 595078 to lessen the possibility of other components breaking app update.
Comment 57 Jorge Villalobos [:jorgev] 2010-09-09 22:07:05 PDT
(In reply to comment #50)
> As I was just posting:
> 
> Unfortunately I've done some more testing and have hit a problem with what I
> said earlier. The extension manager is a pretty critical component that a bunch
> of services rely on to function correctly. One of those services is application
> update. When application update discovers a new update it will normally check
> with the extension manager to see whether the new version will make any
> extensions incompatible. With Personas plus leaving the EM broken this causes
> the app update check to fail with a strange malformed XML error. Neither
> automatic or manual checks will find and install a normal new Firefox version.

Holy crap.

I think a bug should be filed to find a way to better separate the EM from the blocklist and the regular app update. I know a script hanging like this is hard to predict and code around, but those 2 features are crucial for cases like this, or if a malicious developer decides to write an add-on that messes with the EM.

On my side, I'll make sure that we take a stricter stance on making changes to the EM and (no offense here) stop trusting our add-ons so much. It makes no sense to bypass a free QA revision, even if the bug was difficult to trigger and probably wouldn't have been detected.
Comment 58 Robert Strong [:rstrong] (use needinfo to contact me) 2010-09-09 22:11:12 PDT
Stricter changes regarding overriding components or for that matter anything would be best. For example, if this add-on overrode app update then we would be lost.
Comment 59 Jorge Villalobos [:jorgev] 2010-09-09 22:14:41 PDT
We already reject add-ons that make any changes to the EM update mechanism, but that's as far as we go right now.
Comment 60 Dave Townsend [:mossop] 2010-09-09 22:18:16 PDT
(In reply to comment #57)
> (In reply to comment #50)
> > As I was just posting:
> > 
> > Unfortunately I've done some more testing and have hit a problem with what I
> > said earlier. The extension manager is a pretty critical component that a bunch
> > of services rely on to function correctly. One of those services is application
> > update. When application update discovers a new update it will normally check
> > with the extension manager to see whether the new version will make any
> > extensions incompatible. With Personas plus leaving the EM broken this causes
> > the app update check to fail with a strange malformed XML error. Neither
> > automatic or manual checks will find and install a normal new Firefox version.
> 
> Holy crap.
> 
> I think a bug should be filed to find a way to better separate the EM from the
> blocklist and the regular app update. I know a script hanging like this is hard
> to predict and code around, but those 2 features are crucial for cases like
> this, or if a malicious developer decides to write an add-on that messes with
> the EM.

App update I think we can cover ok, at least in this case. It's difficult to envision a way that we could make the blocklist service work without the extension service working too though.

The good news (at least as far as platform reliability goes) is that it is already much much harder to completely replace the extension manager in Firefox 4, I think there is more that could be done on that front though.
Comment 61 Dave Townsend [:mossop] 2010-09-09 22:38:57 PDT
(In reply to comment #60)
> The good news (at least as far as platform reliability goes) is that it is
> already much much harder to completely replace the extension manager in Firefox
> 4, I think there is more that could be done on that front though.

Filed bug 595081 on a way we can make it harder to override the add-ons manager in this way.
Comment 62 Dave Townsend [:mossop] 2010-09-10 00:03:50 PDT
I will have to test more but I think if bug 595092 is fixed then the problem with Personas 1.6 goes away, obviously we still need to be able to ship an update containing that.
Comment 63 Dave Garrett 2010-09-10 06:03:17 PDT
(In reply to comment #42)
> To cut a long story short, if we push the blocklist update then the next
> application update will make the blocklist apply to Personas. This obviously
> relies on users using Firefox such that a blocklist update is downloaded and
> then applying the Firefox update.

Isn't a blocklist file included in the Firefox update itself? Is it feasible to put the needed blocklist entry in there some time before pushing it out to everyone else?

Of course if the app update itself is hosed, this comment is a bit undershot by that, though I felt it worth mentioning.
Comment 64 [:Cww] 2010-09-10 07:01:46 PDT
Putting some solid numbers on comment 58:

The bulk of the users with this problem are probably on 3.6.8 or 3.5.11 and haven't gotten the updates from Tuesday.  So it seems like we should edit the updates (3.6.X>3.6.9 and 3.5.X>3.5.12) on the wire right now so that the extensionVersion is 3.6.8 and 3.5.11 respectively.

This won't cover for users who picked up Google Toolbar or something after updating to 3.6.9 or 3.5.12 -- in which case we'll need to do this again for 3.6.10 and 3.5.11.  In 3.6.10 and 3.5.11 we can also ship a blocklist for Personas Plus so that this won't carry forward any further.

3.0 users are probably hosed entirely but I don't think there are that many of them (may need metrics to check) -- We can lean on support for these edge cases.
Comment 65 Dave Townsend [:mossop] 2010-09-10 07:53:20 PDT
(In reply to comment #63)
> (In reply to comment #42)
> > To cut a long story short, if we push the blocklist update then the next
> > application update will make the blocklist apply to Personas. This obviously
> > relies on users using Firefox such that a blocklist update is downloaded and
> > then applying the Firefox update.
> 
> Isn't a blocklist file included in the Firefox update itself? Is it feasible to
> put the needed blocklist entry in there some time before pushing it out to
> everyone else?

There is a blocklist file shipped with the application but it is only used in the case that the user doesn't already have a blocklist in their profile folder.
Comment 66 Dave Garrett 2010-09-10 08:00:57 PDT
(In reply to comment #65)
> (In reply to comment #63)
> > (In reply to comment #42)
> > > To cut a long story short, if we push the blocklist update then the next
> > > application update will make the blocklist apply to Personas. This obviously
> > > relies on users using Firefox such that a blocklist update is downloaded and
> > > then applying the Firefox update.
> > 
> > Isn't a blocklist file included in the Firefox update itself? Is it feasible to
> > put the needed blocklist entry in there some time before pushing it out to
> > everyone else?
> 
> There is a blocklist file shipped with the application but it is only used in
> the case that the user doesn't already have a blocklist in their profile
> folder.

Shouldn't it automatically use whichever one is newer? In other words, can't this blocklist file included with the application update be used as a secondary blocklist update route?
Comment 67 Dave Townsend [:mossop] 2010-09-10 08:09:24 PDT
(In reply to comment #66)
> (In reply to comment #65)
> > (In reply to comment #63)
> > > (In reply to comment #42)
> > > > To cut a long story short, if we push the blocklist update then the next
> > > > application update will make the blocklist apply to Personas. This obviously
> > > > relies on users using Firefox such that a blocklist update is downloaded and
> > > > then applying the Firefox update.
> > > 
> > > Isn't a blocklist file included in the Firefox update itself? Is it feasible to
> > > put the needed blocklist entry in there some time before pushing it out to
> > > everyone else?
> > 
> > There is a blocklist file shipped with the application but it is only used in
> > the case that the user doesn't already have a blocklist in their profile
> > folder.
> 
> Shouldn't it automatically use whichever one is newer? In other words, can't
> this blocklist file included with the application update be used as a secondary
> blocklist update route?

At the time that support for the application file was added it was decided that that would be too slow to do since you can't rely on file modification times so you'd have to embed a timestamp in the XML and load and parse both on startup.  I have been considering whether there is an alternate method over the past day though.
Comment 68 Dave Garrett 2010-09-10 08:41:11 PDT
(In reply to comment #67)
> (In reply to comment #66)
> > (In reply to comment #65)
> > > (In reply to comment #63)
> > > > (In reply to comment #42)
> > > > > To cut a long story short, if we push the blocklist update then the next
> > > > > application update will make the blocklist apply to Personas. This obviously
> > > > > relies on users using Firefox such that a blocklist update is downloaded and
> > > > > then applying the Firefox update.
> > > > 
> > > > Isn't a blocklist file included in the Firefox update itself? Is it feasible to
> > > > put the needed blocklist entry in there some time before pushing it out to
> > > > everyone else?
> > > 
> > > There is a blocklist file shipped with the application but it is only used in
> > > the case that the user doesn't already have a blocklist in their profile
> > > folder.
> > 
> > Shouldn't it automatically use whichever one is newer? In other words, can't
> > this blocklist file included with the application update be used as a secondary
> > blocklist update route?
> 
> At the time that support for the application file was added it was decided that
> that would be too slow to do since you can't rely on file modification times so
> you'd have to embed a timestamp in the XML and load and parse both on startup. 
> I have been considering whether there is an alternate method over the past day
> though.

This is the sort of thing that would really benefit from a redundant backup update method, so if this can be added as an ability that'd be great.
Comment 69 Nils Maier [:nmaier] 2010-09-10 08:47:05 PDT
(In reply to comment #64)
> Putting some solid numbers on comment 58:
> 
> The bulk of the users with this problem are probably on 3.6.8 or 3.5.11 and
> haven't gotten the updates from Tuesday.  So it seems like we should edit the
> updates (3.6.X>3.6.9 and 3.5.X>3.5.12) on the wire right now so that the
> extensionVersion is 3.6.8 and 3.5.11 respectively.

Only 3.6* is affected, as the override won't be made for other versions.
See (buggy version):
http://hg.mozilla.org/labs/personas/file/2cbd2bcf31d9/client/components/nsPersonasExtensionManager.js#l465
Comment 70 Dave Garrett 2010-09-10 08:54:45 PDT
(In reply to comment #64)
> 3.0 users are probably hosed entirely but I don't think there are that many

(In reply to comment #69)
> Only 3.6* is affected, as the override won't be made for other versions.

Good news for 3.0 users then. Apparently not hosed entirely. :)
Comment 71 Nick Nguyen [:osunick] 2010-09-10 13:10:11 PDT
I still think we should block personas 1.6 for 3.0 as well, because once they upgrade to 3.6, they will get hosed if they have the right combination of add-ons.
Comment 72 Nick Nguyen [:osunick] 2010-09-10 13:25:36 PDT
Actually, after talking to fligtar and rereading mossop's assessment, the bug doesn't block the loading of the blocklist xml.  So we should revise this to block 3.6.* and higher.
Comment 73 Robert Strong [:rstrong] (use needinfo to contact me) 2010-09-10 14:47:46 PDT
Comment on attachment 473771 [details]
v2, removed target application :)

>    <emItem id="personas@christopher.beard">
>      <versionRange minVersion="1.6" maxVersion="1.6"/>
>    </emItem>
Per discussion this should only be blocklisted for Firefox 3.6.0 through 3.6.*
So this should be
    <emItem id="personas@christopher.beard">
      <versionRange minVersion="1.6" maxVersion="1.6"/>
        <targetApplication id="{ec8030f7-c20a-464f-9b0e-13a3a9e97384}">
           <versionRange minVersion="3.6" maxVersion="3.6.*"/>
        </targetApplication>
    </emItem>

r=me with that change since this might be time critical and I might not be around to review it quickly enough over the weekend.
Comment 74 christian 2010-09-10 15:47:01 PDT
(In reply to comment #64)
> Putting some solid numbers on comment 58:
> 
> The bulk of the users with this problem are probably on 3.6.8 or 3.5.11 and
> haven't gotten the updates from Tuesday.

We have ~32 million people on 3.6.9.

  So it seems like we should edit the
> updates (3.6.X>3.6.9 and 3.5.X>3.5.12) on the wire right now so that the
> extensionVersion is 3.6.8 and 3.5.11 respectively.

Which will do nothing for people on 3.6.9/3.5.12 with this problem but will allow people to upgrade FF releases, thus applying the new AMO blocklist. We need the AMO blocklist done a couple days before so it takes when they update and restart, right?

Also, I would imagine we need a whole build to change this, or are we able to do a "repack"?

> This won't cover for users who picked up Google Toolbar or something after
> updating to 3.6.9 or 3.5.12 -- in which case we'll need to do this again for
> 3.6.10 and 3.5.11.  

By "do this" you mean keep the EM compat version pegged at 3.6.8/3.5.11 so the update doesn't check extension compat, right? Is there any reason we rev it at all for point releases? I'm sure the conversation for that happened before me coming to moz...

> In 3.6.10 and 3.5.11 we can also ship a blocklist for
> Personas Plus so that this won't carry forward any further.

Yep, we should do this regardless. I've set this bug as blocking as it has the XML blocklist changes in it.
Comment 75 [:Cww] 2010-09-10 16:47:02 PDT
(In reply to comment #74)
>By "do this" you mean keep the EM compat version pegged at 3.6.8/3.5.11 so the
update doesn't check extension compat, right?

Yes, keep extensionVersion pegged at 3.6.8/3.5.11 for 3.6.10/3.5.13
Comment 76 Dave Garrett 2010-09-10 16:52:33 PDT
(In reply to comment #74)

See bug 595092 comment 6. According to Dave Townsend, the EM overriding doesn't affect 3.5.x so I'm pretty sure the blocking1.9.1.13+ here is unwarranted. Please clarify.
Comment 77 Dave Garrett 2010-09-10 16:54:04 PDT
typo: doesn't affect -> isn't done on
Comment 78 Robert Strong [:rstrong] (use needinfo to contact me) 2010-09-11 00:33:03 PDT
(In reply to comment #74)
> (In reply to comment #64)
>... 
>   So it seems like we should edit the
> > updates (3.6.X>3.6.9 and 3.5.X>3.5.12) on the wire right now so that the
> > extensionVersion is 3.6.8 and 3.5.11 respectively.
> 
> Which will do nothing for people on 3.6.9/3.5.12 with this problem but will
> allow people to upgrade FF releases, thus applying the new AMO blocklist. We
> need the AMO blocklist done a couple days before so it takes when they update
> and restart, right?
Correct.

> 
> Also, I would imagine we need a whole build to change this, or are we able to
> do a "repack"?
It would be just like any other release with the update snippet change being the only change from the normal process. 

One important item of note is the extensionVersion attribute in the update snippet must be the same as the client's app version that is being updated. So, the update snippet for 3.6.5 must have an extensionVersion of 3.6.5.

It might be possible to not provide the extensionVersion attribute in the update snippet to get the same result. I would need to verify this will work by reviewing the code paths and testing first before I would recommend this with any confidence though.

> > This won't cover for users who picked up Google Toolbar or something after
> > updating to 3.6.9 or 3.5.12 -- in which case we'll need to do this again for
> > 3.6.10 and 3.5.11.  
> 
> By "do this" you mean keep the EM compat version pegged at 3.6.8/3.5.11 so the
> update doesn't check extension compat, right? Is there any reason we rev it at
> all for point releases? I'm sure the conversation for that happened before me
> coming to moz...
They will need the same treatment as described above... so, 3.6.9 users would need an update with an extesionVersion of 3.6.9.

I'm fairly certain that the update snippets for users running 3.6.10 or 3.5.13 will be the next update after this happened where the update snippet would contain the correct extensionVersion attribute.
Comment 79 Robert Strong [:rstrong] (use needinfo to contact me) 2010-09-11 00:34:07 PDT
Ignore 3.5.x in my previous comment... I forgot this doesn't affect 3.5.x
Comment 80 Michael Morgan [:morgamic] 2010-09-13 11:24:14 PDT
Created attachment 474747 [details]
v3, changed to fit 3.6 -> 3.6.*
Comment 81 Michael Morgan [:morgamic] 2010-09-13 11:25:57 PDT
Created attachment 474750 [details]
v3, updated blocklist sql
Comment 82 Michael Morgan [:morgamic] 2010-09-13 15:09:08 PDT
Push bug is bug 596030.
Comment 83 Daniel Einspanjer [:dre] [:deinspanjer] 2010-09-13 15:17:14 PDT
Is there any chance that people hitting this problem could be pinging blocklist multiple times?
Comment 84 Michael Morgan [:morgamic] 2010-09-13 16:34:54 PDT
Created attachment 474900 [details]
blocklist sql
Comment 86 christian 2010-09-13 23:11:54 PDT
Comment on attachment 474747 [details]
v3, changed to fit 3.6 -> 3.6.*

a=LegNeato for 1.9.2.10. I'd like to get the change landed asap on mozilla-1.9.2 (on the GECKO1929_20100824_RELBRANCH branch). The sooner this gets in, the sooner we can start a build and kick the update out the door. Thanks!
Comment 87 Dave Townsend [:mossop] 2010-09-14 11:09:44 PDT
Pushed to 1.9.2: http://hg.mozilla.org/releases/mozilla-1.9.2/rev/e78dfe8281d5
And the relbranch: http://hg.mozilla.org/releases/mozilla-1.9.2/rev/18500db28f2c

There are a few more changes than jsut personas plus in there because we have been slack about importing blocklist changes to the tree, hopefully bug 426214 will resolve that.
Comment 88 Robert Strong [:rstrong] (use needinfo to contact me) 2010-09-14 19:43:49 PDT
btw: just landed Bug 595455 which makes it so if something causes the update to fail 10 times in a row during the background checks the user will be asked to check for a newer version.
Comment 89 Daniel Veditz [:dveditz] 2010-09-14 20:55:32 PDT
Comment on attachment 474747 [details]
v3, changed to fit 3.6 -> 3.6.*

correcting approval flag to match fixed-in release
Comment 90 kram1024 2010-09-17 01:33:11 PDT
also I suggest that those whom have a broken extensions manager can manually delete the offending extensions (through the file manager they are using, i.e. explorer or finder or even konqueror/dolphin or even nautilus) without firefox running, then restart firefox, I am sure mozilla staff can tell people how to do that and ultimately get the updates working again, btw .xpi files are actually zip files.
Comment 91 Dave Garrett 2010-09-17 05:35:35 PDT
(In reply to comment #90)
Firefox safe mode works fine to uninstall the broken Personas Plus 1.6 extension, and is the route I've been recommending to users.
http://support.mozilla.com/kb/Safe+Mode

Note that this bug is just for discussion about the blocklist implementation side of things. Its parent bug 590608 is where more general stuff here should be discussed, though this is not a support forum.

> btw .xpi files are actually zip files.

Yes, we know. A lot of things are secretly specialized zip files and even more things use its same compression algorithm, such as PNG image files for example.
Comment 92 Daniel Veditz [:dveditz] 2010-09-17 10:17:05 PDT
*** Bug 594940 has been marked as a duplicate of this bug. ***
Comment 93 Nils Maier [:nmaier] 2010-09-17 10:27:29 PDT
Was comment 49 implemented yet?

Note You need to log in before you can comment on or make changes to this bug.