1655053 - Fix color vision deficiency simulations

Status: RESOLVED FIXED

(DevTools :: Accessibility Tools, defect)

Description

[Matthew Petroff [:mpetroff]]

User Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0

Steps to reproduce:

Used color vision deficiency / color blindness simulations in accessibility tab of dev tools

Actual results:

The simulation is inaccurate, using color matrices with no scientific basis. While the source cited in devtools/server/actors/accessibility/constants.js does not list its source, the matrices appear to be the "colorjack.com matrices" [1] (and do not originate from the scientific literature). As far as I'm concerned, this is worse than not having such a simulation at all, since it gives people a false sense of accommodating individuals with color vision deficiencies.

Furthermore, deuteranomaly, protanomaly, and tritanomaly simulation options are presented as if they each describe a single deficiency even though they each actually describe a spectrum of deficiencies, which again gives a false impression.

[1] https://github.com/MaPePeR/jsColorblindSimulator#the-colormatrix-algorithm

Expected results:

The "-opia" simulations should be updated to be more accurate. An improved method with known provenance is to use the method of Machado et al. (2009) [2], which to the best of my knowledge is the state-of-the-art. The paper's supplementary information provides pre-computed matrices [3]; the "severity 1.0" matrices correspond to the "-opia" deficiencies.

The "-omaly" simulation options should be removed since they don't describe a single deficiency (see [4] for Chromium's extended reasoning for doing so).

I brought up these issues in a comment on the bug that tracked the addition of these simulations [5] but am submitting this new bug report, since said comment received no response. Chromium had the same issue, but it was promptly fixed once the issue was pointed out.

[2] https://doi.org/10.1109/TVCG.2009.113
[3] https://www.inf.ufrgs.br/~oliveira/pubs_files/CVD_Simulation/CVD_Simulation.html
[4] https://bugs.chromium.org/p/chromium/issues/detail?id=1003700#c31
[5] https://bugzilla.mozilla.org/show_bug.cgi?id=1564993#c5

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

Bugbug thinks this bug should belong to this component, but please revert this change in case of error.

Comment 2

[Matthew Petroff [:mpetroff]]

Attachment: Patch for updating CVD simulation matrices and removing "-omaly" variants

The attached patch updates the simulation matrices to use those from Machado et al. (2009) and removes the "-omaly" simulation variants. It's untested because I don't have a Firefox build environment set up, but it's pretty straightforward and removes all references to deuteranomaly, protanomaly, and tritanomaly from the code base.

Comment 3

[Eitan Isaacson [:eeejay]]

Yura, do you have any input on this issue? Anyone else need to be involved in this kind of decision?

Comment 4

[Yura Zenevich [:yzen] (please use "needinfo")]

I can help with that.

Matthew, thanks for the patch. I'd be happy to review so we could land the patch. Would you want me to use yours as is, or are you interested in submitting a patch via our review system - https://docs.firefox-dev.tools/contributing/code-reviews-setup.html ?

Comment 5

[Matthew Petroff [:mpetroff]]

(In reply to Yura Zenevich [:yzen] from comment #4)

Matthew, thanks for the patch. I'd be happy to review so we could land the patch. Would you want me to use yours as is, or are you interested in submitting a patch via our review system - https://docs.firefox-dev.tools/contributing/code-reviews-setup.html ?

Figuring out how to use Phabricator seems complicated, so I'd prefer if you could use the patch as-is, if it isn't too much trouble for you. Thanks.

Comment 6

[Yura Zenevich [:yzen] (please use "needinfo")]

(In reply to Matthew Petroff from comment #5)

(In reply to Yura Zenevich [:yzen] from comment #4)

Matthew, thanks for the patch. I'd be happy to review so we could land the patch. Would you want me to use yours as is, or are you interested in submitting a patch via our review system - https://docs.firefox-dev.tools/contributing/code-reviews-setup.html ?

Figuring out how to use Phabricator seems complicated, so I'd prefer if you could use the patch as-is, if it isn't too much trouble for you. Thanks.

sounds good.

Comment 7

[Yura Zenevich [:yzen] (please use "needinfo")]

Hi Matthew,
I'm about to post a patch and was wondering if you have a no-reply or other email address associated with your github, so it is associated with the commit. Also, please let me know if you don't want me to add that or if just the name would suffice.
Thanks

Comment 8

[Matthew Petroff [:mpetroff]]

(In reply to Yura Zenevich [:yzen] from comment #7)

I'm about to post a patch and was wondering if you have a no-reply or other email address associated with your github, so it is associated with the commit. Also, please let me know if you don't want me to add that or if just the name would suffice.

Using the email address on my Bugzilla account is fine. Thanks.

Comment 9

[Yura Zenevich [:yzen] (please use "needinfo")]

Attachment: Bug 1655053 - updating color vision deficiency simulation in accessibility panel. r=jdescottes

• removing simulations for protanomaly, deuteranomaly and tritanomaly
• updating (fixing) the color matrices for protanopia, deuteranopia and tritanopia
• adding a matrix to simulate achromatopsia

Co-authored-by: Matthew Petroff <[e-mail]>

Comment 10

[Yura Zenevich [:yzen] (please use "needinfo")]

(In reply to Matthew Petroff from comment #8)

(In reply to Yura Zenevich [:yzen] from comment #7)

I'm about to post a patch and was wondering if you have a no-reply or other email address associated with your github, so it is associated with the commit. Also, please let me know if you don't want me to add that or if just the name would suffice.

Using the email address on my Bugzilla account is fine. Thanks.

👍

Comment 11

[Yura Zenevich [:yzen] (please use "needinfo")]

And thanks for the patch!

Comment 12

[Julian Descottes [:jdescottes]]

Adding dev-doc-needed to update https://developer.mozilla.org/en-US/docs/Tools/Accessibility_inspector/Simulation

Comment 13

[Yura Zenevich [:yzen] (please use "needinfo")]

(In reply to Julian Descottes [:jdescottes] from comment #12)

Adding dev-doc-needed to update https://developer.mozilla.org/en-US/docs/Tools/Accessibility_inspector/Simulation

I'm guessing Janet will be looking at it, in summary we are removing protanomaly, deuteranomaly and tritanomaly and adding achromatopsia.
Perhaps it also makes sense referencing the research that is the source of the simulation transformation matrices: https://www.inf.ufrgs.br/~oliveira/pubs_files/CVD_Simulation/CVD_Simulation.html

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/1c63cdd41c08
updating color vision deficiency simulation in accessibility panel. r=jdescottes

Comment 15

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/1c63cdd41c08

Comment 16

[Hamish Willee]

I've just updated Color vision simulation docs in line with comment 13 above (all good).

One question. The docs say:

To enable this feature, you must have webrender enabled; it may be enabled by default, depending on your configuration of Firefox. In the Firefox Configuration Editor, make sure the gfx.webrender.all option is set to true.

However on my computer gfx.webrender.all is false but this still works. Does the feature no longer depend on having webrender enabled?

Comment 17

[Yura Zenevich [:yzen] (please use "needinfo")]

(In reply to Hamish Willee from comment #16)

I've just updated Color vision simulation docs in line with comment 13 above (all good).

One question. The docs say:

To enable this feature, you must have webrender enabled; it may be enabled by default, depending on your configuration of Firefox. In the Firefox Configuration Editor, make sure the gfx.webrender.all option is set to true.

However on my computer gfx.webrender.all is false but this still works. Does the feature no longer depend on having webrender enabled?

As the doc says, it might be enabled by default, especially now as it is being rolled out across different release channels. That pref is for when you might now have it on and would like to force enable webrender (experimental).

Comment 18

[Hamish Willee]

Hi Yura,

Sorry, but I'm still confused. I'm not sure what you mean by "it" in your response (i.e. "color vision simulation" or "webrender"?

Either way, what the docs say now is that you need gfx.webrender.all=true. What I think you may be saying is that webrender is being rolled out across versions and that it might be enabled irrespective of the value of gfx.webrender.all. Is that correct?

So essentially the docs should say:

For this feature to work webrender must be enabled. In the Firefox Configuration Editor, make sure the gfx.webrender.all option is set to true (note that webrender may be enabled irrespective of the value of this preference).

Correct?

Comment 19

[Yura Zenevich [:yzen] (please use "needinfo")]

Hi Hamish,
Sorry, yes, I meant webrender in place of "it". gfx.webrender.all=true will force enable web render, even if it is not enabled for a particular version/platform (it means that running web render is still experimental). I would say that the wording could change to indicate that setting gfx.webrender.all to true is only needed if the simulation feature is not enabled by default.

Comment 20

[Hamish Willee]

Hi Yura, crystal clear. Thank you!

Comment 21

[Nicolas Burrus]

Hi, I wanted to report that the current filter matrices are wrongly happening in the sRGB color space, while they should be happening in the linearRGB color space to be correct. A clear visual symptom is that pure red becomes way too dark with the protanopia simulation.

I reported the same issue to Chromium and it was due to the default value of the SVG color-interpolation-filters being sRGB instead of linearRGB. I don't know if the issue is the same here, but here is the link to the bug report for the full details: https://bugs.chromium.org/p/chromium/issues/detail?id=1003700#c47

Comment 22

[Matthew Petroff [:mpetroff]]

While I agree that applying the matrices to linear RGB makes more sense conceptually and that Firefox's implementation should probably be changed to use linear RGB, whether or not the current behavior is "wrong" is a bit more nuanced. For the figures in the Machado et al. (2009) paper, the authors applied the matrices to gamma-encoded RGB, not linear RGB, and they didn't actually specify what type of RGB should be used. In an email that was forwarded to me by someone who reached out to the authors about this, one of the authors stated that they used gamma-encoded RGB for the figures because they didn't see any significant difference between gamma-encoded RGB and linear RGB with their method. This is not unexpected, since the Farnsworth-Munsell 100 hue test used for the validation experiments only includes colors of medium lightness without extreme saturation and would thus not be affected much by gamma correction. This and the fact that only qualitative validation was performed highlights the limitations of current color vision deficiency simulation techniques and the need for more research in the area [the validation in Brettel et al. (1997) was even less rigorous, with qualitative validation with only a single subject for each type of color vision deficiency].

Comment 23

[Nicolas Burrus]

Thanks for the discussion Matthew! I actually heard about the original authors not applying sRGB decoding (in that github issue), but unfortunately did not get more input yet after reaching out to them directly. I think it's indeed likely that there was no very bright colors in the data they used, explaining why it did not matter as much. Because it definitely changes the output a lot for bright colors.

About the evaluation of these methods, I agree that sadly we don't have a lot of experimental data in our hands. This might become a bit out of scope, but even if Brettel et al. indeed had only two people in their evaluation, there are some counter arguments:

• In a way it has passed the test of time, especially with the implementation from Vischeck (now in GIMP display filters), cited in e.g. the reference Color Appearance Models book from Mark Fairchild.
• There are other more recent studies like Simon-Liedtke & Farup 2016 that also validate it. They also adapt Brettel et al. and Viénot et al. to properly handle an sRGB input.
• Maybe even more importantly, Machado et al. actually uses Brettel et al. as their reference to tweak their parameters for full dichromacy (see the discussion about the scaling factors for eq 15 and 16), and also conclude that the accuracy of their method was similar to Brettel et al. in their experiments for dichromacy. So I think that the main expected difference for Machado et al. is how their handle anomalous trichromacy, which is probably better than Brettel et al. with a trivial linear interpolation.

So I really don't see any reason that Brettel et al. would require a proper sRGB correction / CRT calibration and Machado et al. would not, besides the explanation that it does not matter as much for medium lightness.

Finally, this might be the first time where being a protan myself gives me an advantage, but I can clearly say that bright colors (especially reds) become way too dark with Machado et al. (and Brettel et al.) when not applied in the linearRGB space :-)

For what it's worth I've also put together an online simulator that compares these methods, and I'm hoping to collect feedback from more people with CVD to help determine the best method in practice, I'll followup if it becomes meaningful.

Comment 24

[Nicolas Burrus]

As a quick follow up, this Reddit thread did get some feedback. By no mean a large scale nor complete evaluation, but it tends to confirm that Brettel et al. 1997 and Viénot et al. 1999 are the most solid choices for full dichromacy, followed by Machado with proper sRGB decoding, which is the most similar to the first two. So personally I'd push to actually use the Viénot matrices for protanopia and deuteranopia, but even if we keep the Machado matrices I think it's quite important to at least make sure that everything happens in the linear RGB space.