776160 - Several themes have had their compatibility changed by the compatibility bump

Status: VERIFIED FIXED

(addons.mozilla.org Graveyard :: Compatibility Tools, defect, P1)

Description

[Jorge Villalobos [:jorgev] (he/him)]

Attachment: List of affected theme ids.

We just discovered that the last 2 compatibility bumps are being applied to themes. See this add-on feed for proof: https://addons.mozilla.org/en-US/developers/feed/320596

Themes do not qualify for compatibility bumps and shouldn't be considered. The next bump will happen in a couple of weeks, so we need this fixed ASAP.

We need 3 things to happen:
* Fix the compatibility validator so that themes aren't considered.
* Revert the compatibility information for all affected themes to their previous values. I'm attaching a list of theme ids that have their maxVersion set to 14.0, which should be the ones affected by this.
* Notify the developers of these themes about the reversal. I can write the contents of this message.

Comment 1

[Matt Basta [:basta]]

> Themes do not qualify for compatibility bumps and shouldn't be considered.

Why not?

Comment 2

[Tony Mechelynck [:tonymec]]

(In reply to Matt Basta [:basta] from comment #1)
> > Themes do not qualify for compatibility bumps and shouldn't be considered.
> 
> Why not?

The reason, of course, is that the Toolkit addons owner and peers have decided they shouldn't.

Now of course there must be some reason why they decided that. My guess is that (a) chrome features may be added and removed with much less advance warning than API features; (b) when a new chrome feature is *added* then any theme author should check that it is correctly displayed, while when a new API is added (as opposed to: removed) it usually doesn't affect existing extensions.

Comment 3

[Kris Maglione [:kmag]]

The reason is more that unlike extensions, themes need, without exception, to be updated to support each new version of Firefox. UI features are added and changed that need explicit support. For instance, in Firefox 14, the identity box changed significantly, and is broken in all themes that haven't been updated to account for it. The developer tools have also changed in nearly every recent release in ways that require updates in themes. We can't just look for things that we expect to cause breakage, because changes in Firefox require changes in themes in order to keep functioning.

Comment 4

[Kumar McMillan [:kumar]]

(In reply to Kris Maglione [:kmag] from comment #3)
> in Firefox 14, the
> identity box changed significantly, and is broken in all themes that haven't
> been updated to account for it. 

This suggests that themes *should* be part of the compatibility scanner, no?

Comment 5

[Kris Maglione [:kmag]]

The validator should (and does) do some compatibility checks for themes. Themes shouldn't have their compatibility automatically bumped, though. Sending out compatibility emails is probably useful regardless, if that's what you mean.

Comment 6

[Kumar McMillan [:kumar]]

ok, I see what you mean. Compatiblity scan + email is good but auto-upgrade is not.

Comment 7

[Jorge Villalobos [:jorgev] (he/him)]

No. The compatibility scan, email and auto-upgrade are exclusive to extensions. That's what needs to be fixed for the first point.

Comment 8

[Matt Basta [:basta]]

I don't understand why those features are exclusive to extensions, especially considering users will begin to be prompted with messages saying that their theme(s) are not compatible with the latest version of Firefox or Thunderbird.

I would be more inclined to think that we should work harder to detect incompatibilities and flag them than to prevent any sort of compatibility scan/email and have frustration get passed on to the user. Updates may be the single biggest pain point for users running Firefox these days and making decisions which make those pain points more painful is not a win for anybody.

Comment 9

[Kris Maglione [:kmag]]

Matt, this is not an issue of detecting incompatibilities. If themes aren't explicitly made compatible with a new version of Firefox, they're not compatible and no amount of validation will change that.

Comment 10

[Matt Basta [:basta]]

Certainly not every version of Firefox (or any other application) introduces breaking changes to the interface and surely every change to the interface doesn't break themes in a way that renders the application unusable. Of course it would be nice to only have fully-compatible themes in the wild but (if there's no better reason than the theme being potentially incompatible in certain ways) we need to weigh the cost of this to our user experience. Certainly there's a way of dealing with this that doesn't involve throwing our users and their themes under the bus.

Comment 11

[Kris Maglione [:kmag]]

Themes don't have an interface. They're basically all of the CSS, XBL, and graphics that style the browser, and yes, nearly every Firefox release requires them to change just to avoid UI breakage. If you don't believe me, look at the revision log for the default theme. If people want to override compatibility on their own and deal with partially broken themes, that's fine, but compatibility bumps, where we claim that they're explicitly and fully compatible are a different matter.

Comment 12

[Matt Basta [:basta]]

We should still be looking for the UI elements that changed, though, and engaging the theme developers to get those updated. We *can* detect conflicts or when themes are accessing elements that have been moved or changed with relatively small amount of work. I don't understand why we aren't affording our theme developers the same flags that we provide to our extension developers. Even if we don't bump the compatibility, there's no reason why we can't point out what specific problems theme developers are looking to face in their code.

Comment 13

[Kris Maglione [:kmag]]

What do you suggest we look for? If developers have made the required changes for any particular element, then they'll have marked their themes compatible with the Firefox version in the process. If they haven't, they won't. There's nothing to automate.

I'm not armchair philosophizing here. I review themes all the time, and when themes haven't been updated to support newer Firefox versions, things break. Very often they break badly. UI changes to things like the developer tools, the newtab page, things like app tabs, all need explicit support from themes and they break if they don't get it.

I *do* agree (as I've said) that we should be looking for known, specific compatibility problems and sending out emails, and if you want to argue for that, I'm with you.

Comment 14

[KLB]

Matt, I've been working with themes since FF2.0, I've been an AMO editor since June 2011 and I focus on reviewing themes. To the best of my memory every single major version bump has had UI changes that needed to be reflected in themes. There is also no way to automatically validate if a theme is compatible with the latest version because theme developers can exercise a great deal of artistic liberty with how UI changes are implemented and it requires a human eye to make judgement calls. As such themes really should NOT automatically have their version bumped.

Ideally, there would be an email that would go out about one week before a version bump to remind theme developers of the pending update so that they can make any necessary updates. This email should go out at the same time the max allowed version is bumped on AMO.

There is a theme changes tracking thread at MozillaZine (http://forums.mozillazine.org/viewtopic.php?f=18&t=2173163) that any theme developers use to stay on top of design changes in Firefox which tremendously helps developers update their themes.

Comment 15

[Matt Basta [:basta]]

(In reply to Kris Maglione [:kmag] from comment #13)
> I'm not armchair philosophizing here. I review themes all the time, and when
> themes haven't been updated to support newer Firefox versions, things break.
> Very often they break badly. UI changes to things like the developer tools,
> the newtab page, things like app tabs, all need explicit support from themes
> and they break if they don't get it.

Certainly this won't be the case forever, though, and each and every successive version of Firefox will not break themes. I understand that this isn't the case today, but I disagree that this belongs as hardcoded logic. Perhaps a constant in settings, instead.

> I *do* agree (as I've said) that we should be looking for known, specific
> compatibility problems and sending out emails, and if you want to argue for
> that, I'm with you.

This is exactly what I think we should be doing. Flat-out ignoring compatibility support for themes (as comment 7 suggests) is a destructive attitude towards our developer community.

Comment 16

[Andrew Williamson [:eviljeff]]

Can we just fix the issue reported and argue about a change of policy to include themes in compatibility bumps/reports in another bug please?  For the couple of confused/angry users I know of there are probably hundreds more who haven't stumbled on the forum.  *That* is destructive.

Comment 17

[Jorge Villalobos [:jorgev] (he/him)]

Please stop derailing this bug. This is not about what we could do for themes, it's about what we currently do and how it is broken. We need this fixed ASAP, so let's move this along.

Comment 18

[Wil Clouser [:clouserw]]

(In reply to Jorge Villalobos [:jorgev] from comment #17)
> Please stop derailing this bug. This is not about what we could do for
> themes, it's about what we currently do and how it is broken. We need this
> fixed ASAP, so let's move this along.

Did the compat bump just start analyzing themes or is this an old problem?

Comment 19

[Jorge Villalobos [:jorgev] (he/him)]

My observation on comment #0 is that this appears to have happened in the last 2 bumps (12->13 and 13->14).

Comment 20

[krupa raj[:krupa]]

Did we do any changes to the compat bump process recently? I haven't run this on -dev for a while since it was stable and d2c made the whole exercise (mostly) unnecessary.

Comment 21

[Kumar McMillan [:kumar]]

I made the change to exclude themes entirely from future compat bumps: https://github.com/mozilla/zamboni/commit/973fee9

Jorge, can you make a new bug to fix the historic themes? That will be a more time consuming dev project.

Comment 22

[Jorge Villalobos [:jorgev] (he/him)]

I did a test compat bump from 14 to 15 and checked many of the themes on the list. Everything looks right, none of them were in the upgrade log.

I filed bug 779346 to follow up with the rest of the work. I don't think it is that much. It could be done with an SQL update script, and I would just need the developer email address list to send a message to them.