1431473 - Consider adding tabs.setColorFilter to apply color transforms to documents.

Status: NEW

(WebExtensions :: General, enhancement, P5)

Description

[Eitan Isaacson [:eeejay]]

The main use case is accessibility, but this can also inspire other interesting extensions like 'night mode'. I wrote some background on the project here: https://wiki.mozilla.org/Accessibility/ColorFilters

It's a pretty straightforward API. Not sure if we should add extra permissions for it, I think 'tabs' is sufficient.

I'm thinking something along the lines of:

tabs.setColorFilter(tabID, matrix)
and..
tabs.unsetColorFilter(tabID)

A matrix would work the same way as an SVG <feColorMatrix> filter would in linear RGB color space. So this would be a color invert filter:
[-1, 0, 0, 0, 1,
 0, -1, 0, 0, 1,
 0, 0, -1, 0, 1,
 0, 0, 0, 1, 0]

I have a working prototype (it depends on changes to WebRender and docshell), and I wrote a sample extension that toggles a yellow-on-black filter:

https://github.com/eeejay/filter-it

Comment 1

[Caitlin Neiman [:caitmuenster]]

Hi Eitan, this has been added to agenda for the February 6, 2018 WebExtensions APIs triage. Would you be able to join us?

Here’s a quick overview of what to expect at the triage: 

* We normally spend 5 minutes per bug
* The more information in the bug, the better
* The goal of the triage is to give a general thumbs up or thumbs down on a proposal; we won't be going deep into implementation details

Relevant Links: 

* Wiki for the meeting: https://wiki.mozilla.org/WebExtensions/Triage#Next_Meeting
* Meeting agenda: https://docs.google.com/document/d/1731b2kkN1wndNzVvo--5gwUcagbOSKGNYv4769r68NM/edit#
* Vision doc for WebExtensions: https://wiki.mozilla.org/WebExtensions/Vision

Comment 2

[Eitan Isaacson [:eeejay]]

Hi Caitlin,

I'm going to be PTO until February 12. Any chance we can push this to the next meeting? Thanks!

Comment 3

[Caitlin Neiman [:caitmuenster]]

Hey Eitan! Sure, no problem. I'll add it to the agenda for the next meeting on February 13th. :)

Comment 4

[Caitlin Neiman [:caitmuenster]]

Hi Eitan, as a follow up, this has been added to the agenda for the February 13m 2018 WebExtensions APIs triage. 

The agenda is available here: https://docs.google.com/document/d/1lB8M4WhF2xRrf4A5DlUNkKKiiuxfpxfx6CfHLv-FoAM/edit#

Comment 5

[Eitan Isaacson [:eeejay]]

A few open questions that came up:
1. Should this be in the tabs.* namespace?
2. Should this need another permission besides 'tabs'?
3. Should there be a way to query the currently set filter on a tab?
4. Right now there is only WebRender support for this. If another compositor is used it is a no-op. Should we throw an error?

Comment 6

[Eitan Isaacson [:eeejay]]

Attachment: Add tabs color filter API. r?mixedpuppy

Comment 7

[Shane Caraveo (:mixedpuppy)]

I feel like r? is premature when comment 5 is not addressed.

Comment 8

[Shane Caraveo (:mixedpuppy)]

Comment on attachment 8950750 [details] [diff] [review]
Add tabs color filter API. r?mixedpuppy

Clearing review for now.  Generally looks good.  Needs tests.  Also need to address the open questions.

Comment 9

[Eitan Isaacson [:eeejay]]

Sounds good. I'm going to loop back to this when the dependent bugs are cleared up.

Comment 10

[Luca Greco [:rpl] [:luca] [:lgreco]]

I guess that this needinfo was actually meant to be assigned to Shane. I'm clearing the needinfo assigned to me (and I've nothing to add to the open questions already listed in comment 5).

Comment 11

[Eitan Isaacson [:eeejay]]

About to post a new revision with these questions addressed:

(In reply to Eitan Isaacson [:eeejay] from comment #5)
> A few open questions that came up:
> 1. Should this be in the tabs.* namespace?

Yes.

> 2. Should this need another permission besides 'tabs'?

No.

> 3. Should there be a way to query the currently set filter on a tab?

Yeah, adding tabs.getColorFilter().

> 4. Right now there is only WebRender support for this. If another compositor
> is used it is a no-op. Should we throw an error?

Yeah, throwing an error if feature is not supported. A dev can query for the feature by calling getColorFilter().

Comment 12

[Eitan Isaacson [:eeejay]]

Attachment: Add tabs color filter API. r?mixedpuppy

Comment 13

[Shane Caraveo (:mixedpuppy)]

(In reply to Eitan Isaacson [:eeejay] from comment #11)
> About to post a new revision with these questions addressed:
> 
> (In reply to Eitan Isaacson [:eeejay] from comment #5)
> > A few open questions that came up:
> > 1. Should this be in the tabs.* namespace?
> 
> Yes.

Why?  Chrome has an accessibilityFeatures API[1].  While this feature (color filter) specifically does not appear to be in Chrome, it seems to make sense in that namespace.  Is there a use case/reason to handle this per-tab, or was that the easiest path you saw?  What if you want to apply it to all tabs?  

ISTM that for accessibility, it would be good to have a way to globally set this for the browser and have all tabs affected, rather than depending on an extension for correct fine-grained management on a per-tab basis.  Fine grained control could be a second level.  It also seems like that would be better performing (to set globally) than having an extension set it each time for every tab.  docShell could probably pick it up directly from a pref.

So a straw man would be:

pref(accessibility.colorFilter, ...)

browser.accessibilityFeatures.colorFilter.set/get

I've also gone and read the background pages no color filter.  What I am not clear about is whether this feature is a long term commitment for Firefox.  There are no links to anything that would give me that indication.  What is the status on that?  It looks like this api is destined towards use for a shield study.  If that is the case, it should be an experimental api only.

[1]https://developer.chrome.com/extensions/accessibilityFeatures

Comment 14

[Eitan Isaacson [:eeejay]]

(In reply to Shane Caraveo (:mixedpuppy) from comment #13)
> (In reply to Eitan Isaacson [:eeejay] from comment #11)
> > About to post a new revision with these questions addressed:
> > 
> > (In reply to Eitan Isaacson [:eeejay] from comment #5)
> > > A few open questions that came up:
> > > 1. Should this be in the tabs.* namespace?
> > 
> > Yes.
> 
> Why?  Chrome has an accessibilityFeatures API[1].  While this feature (color
> filter) specifically does not appear to be in Chrome, it seems to make sense
> in that namespace.

The color filter feature goes beyond accessibility use cases, I don't think it would be wise to pigeonhole it in such a namespace. A similar argument could be made for the zoom API if it were proposed today. One example of filters being used outside of accessibility is "night mode" where the browser's contents would have a hue of red after a certain hour.

Chrome's 'accessibilityFeatures' namespace seems to be almost exclusively for managing ChromeOS's platform accessibility settings.

In general, I think calling certain things "accessibility features", is an anachronism that should be avoided. If a user needs to adjust zoom or colors because their eyes are tired after a long day, they wouldn't identify as vision impaired. You would also not need to be diagnosed with a vestibular disorder in order to find animated gifs nauseating. There are exceptions. When a user invests and sets up an assistive technology like a switch control or a screen reader, the features they need may very well be specialized in the "accessibility" category.

I'm not willing to die on that hill if you feel strongly about this :)

> Is there a use case/reason to handle this per-tab, or
> was that the easiest path you saw?  What if you want to apply it to all
> tabs?
> 
> ISTM that for accessibility, it would be good to have a way to globally set
> this for the browser and have all tabs affected, rather than depending on an
> extension for correct fine-grained management on a per-tab basis.  Fine
> grained control could be a second level.  It also seems like that would be
> better performing (to set globally) than having an extension set it each
> time for every tab.  docShell could probably pick it up directly from a pref.
> 
> So a straw man would be:
> 
> pref(accessibility.colorFilter, ...)
> 
> browser.accessibilityFeatures.colorFilter.set/get

A user may encounter content that they would like to enhance with a filter (for example, invert white-on-black text) and not apply globally to all tabs. My inspiration is Chrome's "High Contrast" addon.
https://chrome.google.com/webstore/detail/high-contrast/djcfdncoelnlbldjfhinnjlhdjlikmph?hl=en

I think we need a lightweight way to quickly toggle on and off different filters on each tab depending on their content as opposed to globally setting one filter.

> 
> I've also gone and read the background pages no color filter.  What I am not
> clear about is whether this feature is a long term commitment for Firefox. 
> There are no links to anything that would give me that indication.  What is
> the status on that?  It looks like this api is destined towards use for a
> shield study.  If that is the case, it should be an experimental api only.
> 

We are looking for a permanent replacement for our current high-contrast solution, as described in the background page. We were asked to form a shield study for the user experience around high contrast mode and color filters. The shield study is for a UI and will probably implemented in extension-form, and may depend on this API.

Aside from that, we want to see this as an API that others could use, innovate, and create alternative tools and filters that would answer diverse use-cases we have not thought up. That is the goal of this bug and this proposal.

Comment 15

[Kris Maglione [:kmag]]

(In reply to Eitan Isaacson [:eeejay] from comment #14)
> Aside from that, we want to see this as an API that others could use,
> innovate, and create alternative tools and filters that would answer diverse
> use-cases we have not thought up. That is the goal of this bug and this
> proposal.

I agree that this is a good idea. The benefit of doing it as a bundled experiment first, though, is that we have more chance to experiment with and adjust it internally before we expose it to third-party developers, after which changing it becomes difficult.

Comment 16

[Mike Conca [:mconca]]

I would also recommend we land this as a WebExtensions experiment first in order to support the shield study. The study may provide feedback on changes we'd like to see with the API, which is much easier to accommodate before it is landed as a full feature.

Comment 17

[Eitan Isaacson [:eeejay]]

Here is an addon I built with this API as an experiment. I just put this together so that it could be quickly installed and studied by the addons team. This is not for any kind of study.

https://github.com/eeejay/filter-it

https://github.com/eeejay/filter-it/releases/download/v1.1/filter_it-1.1-an.fx.xpi

Comment 18

[Boris Zbarsky [:bzbarsky]]

Eitan, are you still working on this? I'm trying to figure out whether I should fix the docshell setColorMatrix/getColorMatrix APIs to use nicer array logic or just remove them... They were added for this bug, right?

Comment 19

[Eitan Isaacson [:eeejay]]

Yeah, I've been meaning to pull that out. It was a stopgap implementation that would allow us to do this without accelerated SVG filters. I still think this extension API is worthwhile, but I don't think it needs to be set through the docshell, so it's dead code right now.

Comment 20

[Boris Zbarsky [:bzbarsky]]

Thank you, I filed bug 1557854.