827453 - Re-pack add-ons to avoid removal of private browsing service.

Status: RESOLVED FIXED

(Add-on SDK Graveyard :: General, defect, P1)

Description

[Jeff Griffiths (:canuckistani) (:⚡︎)]

Tracking bug for the effort to re-pack add-ons hosted on AMO with older versions of the SDK that would be broken by the removal of the private browsing service slated for Firefox 21.

Comment 1

[Jeff Griffiths (:canuckistani) (:⚡︎)]

Mental note, this should be flagged 'tracking-firefox20' when that tracking tag is added to bugzilla.

Comment 2

[Alex Keybl [:akeybl]]

Jeff - who will ultimately resolve this bug?

Comment 3

[Jeff Griffiths (:canuckistani) (:⚡︎)]

wontfixing this, it was solved instead by Ehsan adding a dummy service, the gory details are in bug 825471.

Comment 4

[(no longer active)]

Like I've said before, I'm hoping to be able to undo that change in Firefox 21, so we're going to need this repack.  Based on my understanding on the schedule of repacks (which happened in a bug which I can't find right now), the SDK team already has another repack scheduled for another task which matches with the release date of Firefox 21, so that sounded like a good time to do this as well.

Comment 5

[Jeff Griffiths (:canuckistani) (:⚡︎)]

Ehsan, re-packing is not a silver bullet for your problem, nor can I estimate right now at what point we can do an 'auto-repack'.

Why it's not a silver bullet:

* we can't re-pack all sdk-based add-ons, just the ones that use high-level apis
* not all sdk-based add-ons are hosted on AMO anyway, some of the more popular / visible ones like 1password and Yahoo Axis are hosted elsewhere. No matter what we'll break them.

I have created a timeline for landing the SDK in Firefox that addresses re-packing:

https://wiki.mozilla.org/Jetpack/SDK_Apis_In_Firefox_21#Add-on_Re-packing

In my mind we would re-pack existing add-ons to *strip out* SDK apis once Firefox 22 has *shipped*. We have a number of pieces to create, test and validate before I can be confident that the re-packing that we can do is reliable enough to go ahead. There are a number of moving pieces.

Help me understand better what the real trade-off is. What problems arise if the dummy service is left as-is and producing deprecation warnings for a while?

I'm also open to an explanation as to why 6 weeks is a sufficient deprecation period before removing the dummy service considering the known, vast amounts of breakage removing it will cause. I want 24 weeks, minimum, unless you can persuade me that the impact on Firefox users of leaving the dummy service in is really that dire.

Aside: it's also galling to me that the older implementation of the SDK breaks so fantastically when the service is removed. It sucks, but this is the situation we have.

Comment 6

[(no longer active)]

(In reply to comment #5)
> Ehsan, re-packing is not a silver bullet for your problem, nor can I estimate
> right now at what point we can do an 'auto-repack'.

I understand that.

> * not all sdk-based add-ons are hosted on AMO anyway, some of the more popular
> / visible ones like 1password and Yahoo Axis are hosted elsewhere. No matter
> what we'll break them.

Do we have a way to communicate changes to the SDK to the authors of those add-ons, to ask them to do a manual repack?  AFAIK this is not the first time that we're preparing a repack, so we must have had to solve this problem before.

> I have created a timeline for landing the SDK in Firefox that addresses
> re-packing:
> 
> https://wiki.mozilla.org/Jetpack/SDK_Apis_In_Firefox_21#Add-on_Re-packing
> 
> In my mind we would re-pack existing add-ons to *strip out* SDK apis once
> Firefox 22 has *shipped*. We have a number of pieces to create, test and
> validate before I can be confident that the re-packing that we can do is
> reliable enough to go ahead. There are a number of moving pieces.

OK, this actually got me curious as to why I used to think that we can hope for a repack aligning with the release of Firefox 21, and it seems like over in bug 825471 comment 12, _you_ told me that.  ;-)  Now I understand that repacks are a painful process, all I'm asking for is to have an idea on where this repack is necessary.  Note that the add-on SDK is the only thing keeping the old service from being removed completely.

> Help me understand better what the real trade-off is. What problems arise if
> the dummy service is left as-is and producing deprecation warnings for a while?

Basically, anything that uses the private browsing service will be broken starting from Firefox 20.  If we hypothetically had a way to keep the old API and make it no-op or something, we would be masking those breakages completely (i.e., your add-on would continue to run without raising any exceptions and such and it would start to violate the users' privacy with absolutely no sign of anything going bad.)

The more of the API we keep for backwards compat, the higher the chances of the breakages would be.  I would like to be very explicit in what works and what doesn't in add-ons, and throwing as soon as the add-on makes any attempt to access the service is the only way to ensure that.

Also, note that the current nsIPrivateBrowsingService doesn't have any methods or attributes, and it will throw as soon as you attempt to access anything off of it.  There are no deprecation warnings being reported.  Now, that _could_ mask the breakage of an add-on if it just accesses the service and store a reference to it in a global variable and then attempt to actually access its attributes from a different code path that the author does not fully test, which means masking an actual breakage.

> I'm also open to an explanation as to why 6 weeks is a sufficient deprecation
> period before removing the dummy service considering the known, vast amounts of
> breakage removing it will cause. I want 24 weeks, minimum, unless you can
> persuade me that the impact on Firefox users of leaving the dummy service in is
> really that dire.

I hope the above is enough explanation on why the current situation is bad.  To make things more clear, we didn't deprecate nsIPrivateBrowsingService, we just hacked our way around the original bug in the SDK, which was the necessary evil to prevent all Jetpack based add-ons from breaking.  We generally do not have any deprecation policy before we remove an interface or a service, and sometimes we end up having to do drastic hacks like we've done in this case.

> Aside: it's also galling to me that the older implementation of the SDK breaks
> so fantastically when the service is removed. It sucks, but this is the
> situation we have.

I absolutely understand, and sorry if I come off as being harsh, that is not my intention.  But please note that the current situation is extremely bad as far as the Private Browsing module is concerned, and as the module owner I think that is my responsibility and failure.

Comment 7

[Jeff Griffiths (:canuckistani) (:⚡︎)]

(In reply to :Ehsan Akhgari from comment #6)
...
> > * not all sdk-based add-ons are hosted on AMO anyway, some of the more popular
> > / visible ones like 1password and Yahoo Axis are hosted elsewhere. No matter
> > what we'll break them.
> 
> Do we have a way to communicate changes to the SDK to the authors of those
> add-ons, to ask them to do a manual repack?  AFAIK this is not the first
> time that we're preparing a repack, so we must have had to solve this
> problem before.

We pull an email list and send an email, in a workflow that involes IT. Yes, we've done it, it's a manual asynchronous process. Last time we had an uptake rate of about 20% IIRC.

> > I have created a timeline for landing the SDK in Firefox that addresses
> > re-packing:
> > 
> > https://wiki.mozilla.org/Jetpack/SDK_Apis_In_Firefox_21#Add-on_Re-packing
> > 
> > In my mind we would re-pack existing add-ons to *strip out* SDK apis once
> > Firefox 22 has *shipped*. We have a number of pieces to create, test and
> > validate before I can be confident that the re-packing that we can do is
> > reliable enough to go ahead. There are a number of moving pieces.
> 
> OK, this actually got me curious as to why I used to think that we can hope
> for a repack aligning with the release of Firefox 21, and it seems like over
> in bug 825471 comment 12, _you_ told me that.  ;-)  Now I understand that
> repacks are a painful process, all I'm asking for is to have an idea on
> where this repack is necessary.  Note that the add-on SDK is the only thing
> keeping the old service from being removed completely.

I decided in looking at that schedule that the best time to re-pack and rip out the SDK libraries was once Firefox 22 is launched and has uptake, so we have confidence that the SDK libraries are well-distributed as landed in Firefox.

I think what I wasn't clear enough about was that I hadn't considered the timeline for this. Given my personal preference, I'd like to wait on re-packs until after Firefox 22.

I think the real disconnect here is that when you say Firefox 21, you mean 'when 21 hits Aurora / Beta', and when I say it in reference to the APIs landing in Firefox, I mean when the product is shipped and has uptake, and there are ~12 weeks or so at least of drift between those ideas.

> > Help me understand better what the real trade-off is. What problems arise if
> > the dummy service is left as-is and producing deprecation warnings for a while?
> 
> Basically, anything that uses the private browsing service will be broken
> starting from Firefox 20.  If we hypothetically had a way to keep the old
> API and make it no-op or something, we would be masking those breakages
> completely (i.e., your add-on would continue to run without raising any
> exceptions and such and it would start to violate the users' privacy with
> absolutely no sign of anything going bad.)
> 
> The more of the API we keep for backwards compat, the higher the chances of
> the breakages would be.  I would like to be very explicit in what works and
> what doesn't in add-ons, and throwing as soon as the add-on makes any
> attempt to access the service is the only way to ensure that.
> 
> Also, note that the current nsIPrivateBrowsingService doesn't have any
> methods or attributes, and it will throw as soon as you attempt to access
> anything off of it.  There are no deprecation warnings being reported.  Now,
> that _could_ mask the breakage of an add-on if it just accesses the service
> and store a reference to it in a global variable and then attempt to
> actually access its attributes from a different code path that the author
> does not fully test, which means masking an actual breakage.
> 
> > I'm also open to an explanation as to why 6 weeks is a sufficient deprecation
> > period before removing the dummy service considering the known, vast amounts of
> > breakage removing it will cause. I want 24 weeks, minimum, unless you can
> > persuade me that the impact on Firefox users of leaving the dummy service in is
> > really that dire.
> 
> I hope the above is enough explanation on why the current situation is bad. 
> To make things more clear, we didn't deprecate nsIPrivateBrowsingService, we
> just hacked our way around the original bug in the SDK, which was the
> necessary evil to prevent all Jetpack based add-ons from breaking.  We
> generally do not have any deprecation policy before we remove an interface
> or a service, and sometimes we end up having to do drastic hacks like we've
> done in this case.

It is bad all the way around, but add-on authors have a few weeks before Firefox 20 is released to fix their code. I am sure Jorge is well aware of this issue and we can broadcast about it widely, but I will ping him all the same.

From my end there are a number of different tactics we can employ to mitigate this problem:

1. actively address the community of SDK add-on developers and get them to re-pack to avoid upcoming breakage.
2. produce a list of active add-ons on AMO with older versions and directly contact the authors
3. re-pack all possible add-ons to 1.13 using our current script and then directly contact the authors to get them to re-submit. This is what we did for 1.9.

I'll prioritize 1 & 2, and at least get an idea of what the size of the problem is. I'll see what amount of work can be done to do 3. I also want to establish a KPI so we can track the size of the issue as we head towards removing this. It really matters whether we're talking about 40 add-ons or 400.

Comment 8

[(no longer active)]

(In reply to comment #7)
> > Do we have a way to communicate changes to the SDK to the authors of those
> > add-ons, to ask them to do a manual repack?  AFAIK this is not the first
> > time that we're preparing a repack, so we must have had to solve this
> > problem before.
> 
> We pull an email list and send an email, in a workflow that involes IT. Yes,
> we've done it, it's a manual asynchronous process. Last time we had an uptake
> rate of about 20% IIRC.

Oh, that doesn't seem like an effective measure then.  :(

> I decided in looking at that schedule that the best time to re-pack and rip out
> the SDK libraries was once Firefox 22 is launched and has uptake, so we have
> confidence that the SDK libraries are well-distributed as landed in Firefox.
> 
> I think what I wasn't clear enough about was that I hadn't considered the
> timeline for this. Given my personal preference, I'd like to wait on re-packs
> until after Firefox 22.
> 
> I think the real disconnect here is that when you say Firefox 21, you mean
> 'when 21 hits Aurora / Beta', and when I say it in reference to the APIs
> landing in Firefox, I mean when the product is shipped and has uptake, and
> there are ~12 weeks or so at least of drift between those ideas.

Hmm, yeah I think that is the real disconnect...  But it's good that now I'm on the same page as you are!

So, given the above, we should keep the dummy service on mozilla-central until the beginning of July and then remove it and let the removal patch ride the trains, right?  :(

> > I hope the above is enough explanation on why the current situation is bad. 
> > To make things more clear, we didn't deprecate nsIPrivateBrowsingService, we
> > just hacked our way around the original bug in the SDK, which was the
> > necessary evil to prevent all Jetpack based add-ons from breaking.  We
> > generally do not have any deprecation policy before we remove an interface
> > or a service, and sometimes we end up having to do drastic hacks like we've
> > done in this case.
> 
> It is bad all the way around, but add-on authors have a few weeks before
> Firefox 20 is released to fix their code. I am sure Jorge is well aware of this
> issue and we can broadcast about it widely, but I will ping him all the same.

I'd appreciate that.  I hope bug 826079 would be an effective way for catching this problem on non-SDK AMO add-ons...

> From my end there are a number of different tactics we can employ to mitigate
> this problem:
> 
> 1. actively address the community of SDK add-on developers and get them to
> re-pack to avoid upcoming breakage.
> 2. produce a list of active add-ons on AMO with older versions and directly
> contact the authors
> 3. re-pack all possible add-ons to 1.13 using our current script and then
> directly contact the authors to get them to re-submit. This is what we did for
> 1.9.
> 
> I'll prioritize 1 & 2, and at least get an idea of what the size of the problem
> is. I'll see what amount of work can be done to do 3. I also want to establish
> a KPI so we can track the size of the issue as we head towards removing this.
> It really matters whether we're talking about 40 add-ons or 400.

OK, this all sounds good, and you're definitely a much better person to come up with the best plan forward than I am.  I do appreciate your help, and please feel free to morph this bug as needed.

And do let me know if you have any questions or concerns.

Thanks!

Comment 9

[John Nagle]

Today, we see the failure of the above plan.
- March 28th: SDK 1.14 released, but AMO's validator doesn't recognize it.
- March 29th: Validator recognizes SDK 1.14. 
- April 2nd: Firefox 20, which requires SDK 1.14, is released.

So add-on authors had FOUR DAYS to fix their code. Over the Easter weekend.  ONE working day.  

Stated Mozilla policy is that there should be 18 weeks.  (https://blog.mozilla.org/addons/2012/10/01/a-lifecycle-for-the-add-on-sdks-apis/)
(The API changed again in 1.14, so that restarts the clock. See "https://forums.mozilla.org/addons/viewtopic.php?p=28332&sid=2c967482e3abadea9624930e0671381b".)  

As usual, AMO is running behind, and there's a backlog of repacked add-ons that haven't been approved yet. 

FAIL.

Comment 10

[Jeff Griffiths (:canuckistani) (:⚡︎)]

(In reply to John Nagle from comment #9)
...
> FAIL.

I apologize for the problems, and I agree with you that we ended up breaking our own rules here, largely because we got dragged along by a larger platform change. As I said previously in this bug, "It is bad all the way around".

Add-ons using 1.13.2 should function fine with Firefox 20 as long as they do not use private-browsing. In your case, your add-on was one of ~ 40 or so that do use private browsing, and that we needed to either get re-packed or mark as incompatible with Firefox 20. I do not expect another platform change like this to happen again but, you know, no-one expects the Spanish Inquisition either.