Open Bug 916419 (mathmla11y) Opened 11 years ago Updated 2 years ago

Expose MathML as a hierarchical accessible tree

Categories

(Core :: Disability Access APIs, defect)

Product:
Core
Component:
Disability Access APIs
Type:
defect

Tracking

()

People

(Reporter: fredw, Unassigned)

References

(Depends on 2 open bugs,
URL
)

Details

(Keywords: helpwanted, parity-safari) 

It was mentioned on mozilla.dev.platform that MathML is currently not accessible in Gecko and that <math> nodes are only indicated to be "equations", which is very bad :-( Bug 427330 comment 5 mentions a bug entry for MathML accessibility support but I was not able to find it, so I'm opening a new one here (so that I can also reference it during the Mozilla Summit if people want to help). 

One proposed solution on mozilla.dev.platform was to replace MathML by a pure textual syntax and to just expose that raw text to accessibility tools, but that seems more a lazy solution than anything serious to help these third party tools (and we want to keep the MathML rendering anyway).

The <math> element has altimg and alttext attributes. There is also a <semantics> element to provide alternative syntax:

http://www.w3.org/TR/MathML3/chapter2.html#interf.toplevel.atts
http://www.w3.org/TR/MathML3/chapter5.html#mixing.semantic.annotations

Most TeX-to-MathML generators will make the TeX source available in the alttext or in an <annotation> element, so that might be a temporarily solution to expose at least something, but that won't work in general.

WebKit exposes a hierarchical accessible tree for MathML:

http://trac.webkit.org/browser/trunk/Source/WebCore/accessibility/AccessibilityRenderObject.cpp#L3406

I think this currently only works on Mac platform, but there are plans for other platforms like

https://bugs.webkit.org/show_bug.cgi?id=71738

Most other tools like ChromeVox (Chrome), MathPlayer (Internet Explorer) or a prototype MathJax extension by Benetech expose the MathML tree in some way to make mathematics accessible. I'm not an accessibility expert, but I think that's the solution we want here too and this is mentioned on an old wiki page from MDN:

https://developer.mozilla.org/en-US/docs/Accessibility/AT-APIs/Web_Specifications#MathML

I saw an interesting article on Planet Mozilla that highlights the difference between MathML tree exposed by DOM or by accessibility APIs:

http://www.marcozehe.de/2013/09/07/why-accessibility-apis-matter/

so it seems that more precisely we want to do things like in WebKit i.e. via APIs. From the blog post:

"Chromevox can now introduce experimental features, such as support for math expressions, ... anything else that improves the user experience."

so implementing accessible APIs for MathML would prove that we can do math support without cheating with DOM access methods.

BTW, one ChromeVox developer was also the author of the FireVox extension. But this extension as well as a more recent ML-Firefox fork just seems to be incompatible with recent versions of Firefox. I'm guessing the rapid release plan does not help to keep the add-on up-to-date. That's another reason to implement a durable solution in Core...
How much the mathml exposed by webkit is accessible by VoiceOver? I don't really see how they map all mathml diversity to OS X a11y API, it seems like they put all stuffs as a plain text (no extra roles, no relations, no handy methods). While it's cool for the start but ideally we should get some API that allows to extract math info in very friendly way.
I don't really know much about accessibility, but IIUC the cpp file I mentioned contains functions like isMathFraction(), isMathSquareRoot() corresponding to MathML elements and mathNumeratorObject(), mathDenominatorObject() to access their children. See also

https://bugs.webkit.org/show_bug.cgi?id=71742
https://bugs.webkit.org/show_bug.cgi?id=101263

(note that the second was fixed only last November, it's probably not in the WebKit Mac release yet)
(In reply to Frédéric Wang (:fredw) from comment #2)

> https://bugs.webkit.org/show_bug.cgi?id=101263

Ok, I see. At least we can copy it on mac and pretend to be accessible threre :)
What WebKit have done looks very interesting. It would certainly allow traversal of, events from and synchronisation with math nodes. On Mac, they are exposing sub-roles for the math node types and attributes with objects (same as relations in IA2) for root index, fraction denominator, fraction numerator, subscript, superscript, etc.

The only problem i see with doing something similar for IA2 is that we'd have to get a lot of new roles and relations into the spec. We could use extendedRole, but it still needs to be standardised anyway.

From an AT perspective, one issue is that most libraries for translating math (e.g. to speech or braille) take MathML anyway, as it's a simple, standardised way to pass math. I'm a big advocate for accessibility APIs, but it does seem wasteful to pull everything via accessibility APIs and then immediately convert it back into MathML. That said, having the accessibility objects does allow for events, synchronisation, etc.
> The only problem i see with doing something similar for IA2 is that we'd
> have to get a lot of new roles and relations into the spec. We could use
> extendedRole, but it still needs to be standardised anyway.

agreed, we could of course do what it looks like Apple has done and not spec it :p

> From an AT perspective, one issue is that most libraries for translating
> math (e.g. to speech or braille) take MathML anyway, as it's a simple,
> standardised way to pass math. I'm a big advocate for accessibility APIs,
> but it does seem wasteful to pull everything via accessibility APIs and then
> immediately convert it back into MathML. That said, having the accessibility
> objects does allow for events, synchronisation, etc.

I wonder what events you might want, and how big a deal dynamic changes actually are in the wild.

(In reply to Frédéric Wang (:fredw) from comment #0)
> One proposed solution on mozilla.dev.platform was to replace MathML by a
> pure textual syntax and to just expose that raw text to accessibility tools,
> but that seems more a lazy solution than anything serious to help these
> third party tools (and we want to keep the MathML rendering anyway).

I'm not really sure what rendering has to do with anything here other than needing to make sure a11y representation is in sync with it.

I think  a text representation have some significant practical advantages and would actually be useful even if slightly less theoretically pleasing than use a tree of accessibles with roles and such.

> Most other tools like ChromeVox (Chrome), MathPlayer (Internet Explorer) or
> a prototype MathJax extension by Benetech expose the MathML tree in some way

no, all of those go directly DOM -> speech / braille, The comparable thing for gecko to do would be to synthesize / braille stuff itself, or maybe generate ssml and hand that to the assistive technology or something...

> so it seems that more precisely we want to do things like in WebKit i.e. via
> APIs. From the blog post:

Well, I don't think anybody is proposing that we start doing speech synthesis or braille output in gecko, so trivially yes.
> (In reply to Frédéric Wang (:fredw) from comment #0)
> I'm not really sure what rendering has to do with anything here other than
> needing to make sure a11y representation is in sync with it.
> 
> I think  a text representation have some significant practical advantages
> and would actually be useful even if slightly less theoretically pleasing
> than use a tree of accessibles with roles and such.

This referred to the proposal on mozilla.dev.platform to just drop MathML support in Gecko at all and to ask people to always enter raw text for mathematical expressions instead. Then for accessibility people, exposing the math becomes trivial, but my point was simply that other people need MathML support in Gecko and so that's not an acceptable solution. Of course if you are just saying that Gecko could expose the alttext or <annotation> values of the MathML code, then I agree with you. If Gecko can rely more on the MathML tree to improve accessibility, then that's even better.

> > Most other tools like ChromeVox (Chrome), MathPlayer (Internet Explorer) or
> > a prototype MathJax extension by Benetech expose the MathML tree in some way
> 
> no, all of those go directly DOM -> speech / braille, The comparable thing
> for gecko to do would be to synthesize / braille stuff itself, or maybe
> generate ssml and hand that to the assistive technology or something...
> 
> > so it seems that more precisely we want to do things like in WebKit i.e. via
> > APIs. From the blog post:
> 
> Well, I don't think anybody is proposing that we start doing speech
> synthesis or braille output in gecko, so trivially yes.

From the discussions I've vaguely followed in the MathJax project, it seems to me that Design Science and Benetech are using (or planning to use) the MathML tree as well as ARIA roles on MathML nodes, so at least that part seems similar to what WebKit is doing, for a non-expert eye. Of course they are using the MathML DOM and they are converting to speech / braille at the end, but I never proposed to implement that part in Gecko. Sorry if it seemed that I suggested that or if I said things that are innacurate, I'm just trying to be helpful here.
(In reply to James Teh [:Jamie] from comment #4)

> The only problem i see with doing something similar for IA2 is that we'd
> have to get a lot of new roles and relations into the spec. 

that shouldn't be a problem, math is not restricted to the web, so openoffice could use it exposing latex or whatever is supports. Just IA2 is not extensible as OS X API is what makes us doubtful.

> From an AT perspective, one issue is that most libraries for translating
> math (e.g. to speech or braille) take MathML anyway, as it's a simple,
> standardised way to pass math. I'm a big advocate for accessibility APIs,
> but it does seem wasteful to pull everything via accessibility APIs and then
> immediately convert it back into MathML. That said, having the accessibility
> objects does allow for events, synchronisation, etc.

so an accessible, tag name and some DOM attributes is all we need.
(In reply to alexander :surkov from comment #7)
> that shouldn't be a problem, math is not restricted to the web, so
> openoffice could use it exposing latex or whatever is supports. Just IA2 is
> not extensible as OS X API is what makes us doubtful.
> 

openoffice uses ODF, so mathematics is encoded in MathML.
(In reply to Trevor Saunders (:tbsaunde) from comment #5)
> > That said, having the accessibility
> > objects does allow for events, synchronisation, etc.
> I wonder what events you might want, and how big a deal dynamic changes
> actually are in the wild.
Even something like scrolling to a particular point in an equation or getting the location of a given part on screen is a good case for accessible objects. If you add math editing to the mix, then events and objects start to matter even more.

> I think  a text representation have some significant practical advantages
> and would actually be useful even if slightly less theoretically pleasing
> than use a tree of accessibles with roles and such.
Except it doesn't work for braille at all. :(
(In reply to James Teh [:Jamie] from comment #9)
> (In reply to Trevor Saunders (:tbsaunde) from comment #5)
> > > That said, having the accessibility
> > > objects does allow for events, synchronisation, etc.
> > I wonder what events you might want, and how big a deal dynamic changes
> > actually are in the wild.
> Even something like scrolling to a particular point in an equation or
> getting the location of a given part on screen is a good case for accessible
> objects. If you add math editing to the mix, then events and objects start
> to matter even more.

hm, couldn't you get screen location with IAccessibleText::characterExtents()?  However if you support editing then this gets a little trickier, but I guess maybe you can do it with text change events.

> > I think  a text representation have some significant practical advantages
> > and would actually be useful even if slightly less theoretically pleasing
> > than use a tree of accessibles with roles and such.
> Except it doesn't work for braille at all. :(

Well, I think that really depends on the details ofthe text representation and what you want in the braille, if your text representation is just the mathml then you said earlier things already consume that.  If it is some more concise format then it may or may not be possible to convert that to braille as desired.  As an example I doubt reading frac(x, y)  literally converted char by char to braille with no interpretation is particularly nice, but I'd be shocked if its completely unreadible.
(In reply to alexander :surkov from comment #7)
> > The only problem i see with doing something similar for IA2 is that we'd
> > have to get a lot of new roles and relations into the spec. 
> that shouldn't be a problem, math is not restricted to the web
I know. It just takes time. :)
> Just IA2 is
> not extensible as OS X API is what makes us doubtful.
Aside from using strings for roles (and this could be done with IA2 extendedRole), there's nothing particularly more extensible about the Mac API than IA2 relevant to this case. It's just that the Mac API only has one consuming vendor (Apple), so they don't need to go through a standards group.
(In reply to Trevor Saunders (:tbsaunde) from comment #10)
> hm, couldn't you get screen location with
> IAccessibleText::characterExtents()?
I guess if the text representation knows how to handle that, sure.

> Well, I think that really depends on the details ofthe text representation
> and what you want in the braille, if your text representation is just the
> mathml then you said earlier things already consume that.
True. However, why expose a text representation for math when we don't do this for any other kind of content? What about synchronisation and navigation through an equation?
> As an example I doubt reading frac(x, y)  literally
> converted char by char to braille with no interpretation is particularly
> nice, but I'd be shocked if its completely unreadible.
Sure, but surely we want to work towards a proper, elegant solution here, not a "it's not completely unreadable so it's okay" solution.
(In reply to James Teh [:Jamie] from comment #12)

> > As an example I doubt reading frac(x, y)  literally
> > converted char by char to braille with no interpretation is particularly
> > nice, but I'd be shocked if its completely unreadible.
> Sure, but surely we want to work towards a proper, elegant solution here,
> not a "it's not completely unreadable so it's okay" solution.

two cents: mathematicians have a standard language, it'd be wrong to make the braille talk in another one
anyway, as a first step I suggest just to expose the accessible tree (tag name and important DOM attributes as a11y object attributes) (ATK, IA2), as a second step to implement OS X thing as webkit does. That might be all we need at this point.
(In reply to alexander :surkov from comment #13)
> (In reply to James Teh [:Jamie] from comment #12)
> 
> > > As an example I doubt reading frac(x, y)  literally
> > > converted char by char to braille with no interpretation is particularly
> > > nice, but I'd be shocked if its completely unreadible.
> > Sure, but surely we want to work towards a proper, elegant solution here,
> > not a "it's not completely unreadable so it's okay" solution.
> 
> two cents: mathematicians have a standard language, it'd be wrong to make
> the braille talk in another one

Also, the question is not whether the simplest formulas like "a+1/2" are acceptable to be read character by character (of course they are and one can even say you don't need MathML or Nemeth in these cases) but whether that will also work for more general expressions like

\begin{align} \Gamma(z) &= \lim_{n \to \infty} \frac{n! \; n^z}{z \; (z+1)\cdots(z+n)} = \frac{1}{z} \prod_{n=1}^\infty \frac{\left(1+\frac{1}{n}\right)^z}{1+\frac{z}{n} \\ \Gamma(z) &= \frac{e^{-\gamma z}}{z} \prod_{n=1}^\infty \left(1 + \frac{z}{n}\right)^{-1} e^{\frac{z}{n}} \end{align}

I think it's already hard to read the source code visually. That's probably even worse in braille. And I don't want to imagine what it would give character by character by a screen reader. And yes, I forgot one brace in the above expression.
> > As an example I doubt reading frac(x, y)  literally
> > converted char by char to braille with no interpretation is particularly
> > nice, but I'd be shocked if its completely unreadible.
> Sure, but surely we want to work towards a proper, elegant solution here,
> not a "it's not completely unreadable so it's okay" solution.

perhaps I should be more clear, at a minimum with no effort on the part of the AT you'd get something not great but readible, but if the AT chose to  process that text representation it could get something nice.

Thinking about this a little more it seems like this issomewhat similar to the question of embedded object chars and when they should be used instead of including text directly.
(In reply to Trevor Saunders (:tbsaunde) from comment #16)
> perhaps I should be more clear, at a minimum with no effort on the part of
> the AT you'd get something not great but readible, but if the AT chose to 
> process that text representation it could get something nice.
> 

In that case the fact that the text representation has "significant practical advantages" over a tree of accessibles is not obvious at all. It seems to me that the structure of mathematical expressions are sophisticated enough to require a tree representation in order to be processed adequately. For example to implement synchronization and navigation through an equation as mentioned in comment 12, "something nice" is not to highlight or navigate character by character but to have a visual and/or sound output that reflects the mathematical structures with fraction nodes, square root nodes etc I hardly see how the AT can do that without parsing the string, handling it at an abstract level, creating a two-dimensional visual representation and so forth. So strictly speaking you are sending a text representation (e.g. TeX or MathML) that can be parsed into a tree and just asking the AT to do all the work ; indeed I guess that will work well for existing sophisticated TeX/MathML screen readers. However, since Gecko already parses the MathML source, maintain an internal representation and is able to provide a dynamic mathematical rendering of the math expression, the accessibility API becomes more convenient here to save you a bunch of work (for example for someone who wants to write an AT to read Web pages or EPUB books on FirefoxOS where you don't have unlimited resource and can't easily install arbitrary binary programs).
(In reply to James Teh [:Jamie] from comment #4)
> one issue is that most libraries for translating
> math (e.g. to speech or braille) take MathML anyway
Actually, I realised that there's no open source library for doing MathML to speech that I can find. The one we're considering using for both speech and braille takes LaTeX, so we'd have to parse the MathML into a hierarchy ourselves anyway.
Can accessibility people clearly state if they agree that this bug is valid and is worth fixing?

If I understand correctly, James is interested in the idea of exposing the MathML as a hierarchical accessible tree for his NVDA tool. Alexander is OK with just mimicking WebKit's implementation. Trevor's point of view is less clear to me and I'm a bit confused... On the mozilla.dev.platform's thread he disagreed about the need of having a (MathML) tree of the mathematical formula in order to provide the best interaction with accessibility tools. He supported Benoit Jacob's idea to just drop MathML, to represent math by a text node only and to just expose that to accessibility tools. Or perhaps he just meant not to expose MathML via the DOM like the other tools I mentioned but then I'm not sure why he didn't mention potential usefulness of a MathML accessibility API until I opened this bug. Also, in bug 917598 he seems to be against exposing the serialized MathML source. So at that point I think he still prefers to expose a non-MathML string to screen reader from a pragmatic point of view (not sure how to get that string) ; he is fine with implementing a MathML accessibility API but just finds it "theoretically elegant". @trevor: Did I summarize your position correctly? Probably not, so could you please clarify?

If there is a consensus that this bug should be fixed and how to achieve that, then perhaps it can be converted into a "mentored bug". I'm happy to co-mentor it for the MathML part, but I think the help of someone who knows the accessibility code is necessary. Does anyone want to co-mentor this bug?
We should do what's helpful for our users. MathML is still very popular and it is part of Mozilla, NVDA is looking to add MathML support so all we should do is just help and let things go. Doing some experiments on this way grants us valuable experience imo. So I think this bug is actually beside the MathML future.
(In reply to Frédéric Wang (:fredw) from comment #19)
> Can accessibility people clearly state if they agree that this bug is valid
> and is worth fixing?

I don't object if someone wants to do the work.

> If I understand correctly, James is interested in the idea of exposing the
> MathML as a hierarchical accessible tree for his NVDA tool. Alexander is OK
> with just mimicking WebKit's implementation. Trevor's point of view is less
> clear to me and I'm a bit confused... On the mozilla.dev.platform's thread
> he disagreed about the need of having a (MathML) tree of the mathematical
> formula in order to provide the best interaction with accessibility tools.
> He supported Benoit Jacob's idea to just drop MathML, to represent math by a
> text node only and to just expose that to accessibility tools. Or perhaps he

for better or worse mathml is here to stay, so how is that relevent?

> just meant not to expose MathML via the DOM like the other tools I mentioned
> but then I'm not sure why he didn't mention potential usefulness of a MathML
> accessibility API until I opened this bug. Also, in bug 917598 he seems to
> be against exposing the serialized MathML source. So at that point I think

not really, I think it would be grosse, but acceptable to expose the source as a string in IA2, I mostly  just want to make life hard on people using ISimpleDOM.

> he still prefers to expose a non-MathML string to screen reader from a
> pragmatic point of view (not sure how to get that string) ; he is fine with
> implementing a MathML accessibility API but just finds it "theoretically
> elegant". @trevor: Did I summarize your position correctly? Probably not, so
> could you please clarify?

Well, I'm not sure I have a grand opinion ;) I think there's an argument to be made that exposing a tree will just lead to a bunch of machinary to deserialize and then reserialize mathml, in which case just exposing the source in the first place would have been better, but I also think just exposing the source is pretty grosse.

> If there is a consensus that this bug should be fixed and how to achieve
> that, then perhaps it can be converted into a "mentored bug". I'm happy to
> co-mentor it for the MathML part, but I think the help of someone who knows
> the accessibility code is necessary. Does anyone want to co-mentor this bug?

Well, to start someone would need to add the similar stuff to atk and ia2 as webkit uses with the NSAccessibilityProtocol thing.  Otherwise you could implement something in Gecko but it would be basically useless :)
Thanks for the clarification. So it seems to me that it could be possible to expose the MathML source for tools that directly take it as input (via ISimpleDOMNode::innerHTML or anything better in IA2) but also to expose an accessible tree for tools performing a finer interaction with Gecko like tree navigation or highlight of the formula.

(In reply to Trevor Saunders (:tbsaunde) from comment #21)
> Well, to start someone would need to add the similar stuff to atk and ia2 as
> webkit uses with the NSAccessibilityProtocol thing.  Otherwise you could
> implement something in Gecko but it would be basically useless :)

Is that needed for e.g. VoiceOver on Firefox Mac? What about James' NVDA tool?
Depends on: 920547
(In reply to Frédéric Wang (:fredw) from comment #22)
> > Well, to start someone would need to add the similar stuff to atk and ia2 as
> > webkit uses with the NSAccessibilityProtocol thing.  Otherwise you could
> > implement something in Gecko but it would be basically useless :)
> Is that needed for e.g. VoiceOver on Firefox Mac?
WebKit Mac already sets what is required for VoiceOver on Mac, so no.

> What about James' NVDA
> tool?
To do it properly/elegantly with IAccessible2, yes. I'm still not sure on the best way to expose this; the way WebKit does it has some drawbacks (e.g. can't tell if a child is a fraction's numerator or denominator without walking up to the parent and comparing).

Note that bug 920547 does allow IA2 to be used in a hacky way using the tag attribute (which is still better than using ISimpleDOM).
just in case, MathJax discussion about making it accessible: https://groups.google.com/forum/#!topic/mathjax-dev/UvFSEx8GOh8
(In reply to James Teh [:Jamie] from comment #18)
> (In reply to James Teh [:Jamie] from comment #4)
> > one issue is that most libraries for translating
> > math (e.g. to speech or braille) take MathML anyway
> Actually, I realised that there's no open source library for doing MathML to
> speech that I can find. The one we're considering using for both speech and
> braille takes LaTeX, so we'd have to parse the MathML into a hierarchy
> ourselves anyway.

James, I've written pyMathSpeak, an open-source library to convert from MathML to speech in accordance with the MathSpeak specification.  It is hosted at http://sourceforge.net/projects/pymathspeak/
I've just seen Jonathan Wei's presentation on AirMozilla, that's great to hear that things are finally moving forward here. I planned to mention (the lack of) MathML accessibility in Gecko in some of my work, so I'm happy to provide a link to Jonathan's work in progress instead. Some comments:

1) There are some elements like mglyph, mlabeledtr, maligngroup, malignmark, maction, mfenced whose importance is not really clear at this point and some of them are not implemented in the visual rendering and/or not much used in practice, so perhaps you don't need to add a role for them now. Similarly, I don't know if layout-only elements like mspace or mpadded are very important for accessibility.

2) Many default values for the mo attributes are determined by the operator form and the operator dictionary (http://www.w3.org/TR/MathML3/appendixc.html#oper-dict.entries-table). This is done in the layout/mathml/ code, but our dictionary currently only contains operator properties that have visual effect. See also bug 562460 and https://bugs.webkit.org/show_bug.cgi?id=124836.

3) It's important to note that <mfenced> is just a shorthand for the equivalent "expanded form" (that's why I list that element in 1), so I believe the accessible code should expose exactly the same tree as for this expanded form. You'll need to have the operator dictionary support to determine the fence/separator values. See http://www.w3.org/TR/MathML3/chapter3.html#presm.mfenced.

4) The MathJax Native MathML add-on might be useful to test MathML accessibility on Web sites using MathJax: https://addons.mozilla.org/en-US/firefox/addon/mathjax-native-mathml/
Depends on: 1001634
Depends on: 1001635
Depends on: 1001637
Depends on: 1001641
I'm currently adding support to NVDA for Design Science's new MathPlayer library, which is unfortunately still only available privately to a very limited group. In short, this allows for both intelligent speaking and brailling of math content, as well as interactive navigation. It takes MathML as input.

For Firefox, this means we just use ISimpleDOM to get the MathML. While just grabbing markup has obvious disadvantages as discussed in this bug, I've come to realise that there are also serious disadvantages to only exposing a tree. With text, because it is generally read in a linear fashion, it can be fetched and read in chunks. With math, you often need to process the entire expression to get a high level overview of what it is communicating. For braille, this is even more important because the way it is presented changes depending on the level of nesting of fractions, etc.

At least for now, NVDA will continue to just grab the markup and no other client is asking for proper exposure via the a11y tree. However, I suspect we'll get a better idea of what works and doesn't work as we get feedback on our first implementation, and other AT (e.g. Orca) will hopefully be able to provide input once they've done some requirements analysis. My feeling is that the ideal approach will be a combination of both markup and tree exposure with the ability to map between them; e.g. by providing identifiers for each node in the markup.

Regarding Jonathan's proposal, I have two concerns:
1. To work out whether a node is, for example, the numerator of a fraction or the index of a root, one has to either query for several relations or walk to the parent, check its role and then query the appropriate relations. I wonder if there's a more efficient way to handle this. Perhaps the operand arguments (e.g. numerator) could have a property/attribute specifying what they are.
2. Because the arguments of an operand occur in a specific order, I wonder whether relations are overly complicated. For example, in a  fraction, the numerator will always be the first child and the denominator will always be the second child, so the AT can just get the argument it wants by index.
(In reply to James Teh [:Jamie] from comment #27)

> I'm currently adding support to NVDA for Design Science's new MathPlayer
> library, which is unfortunately still only available privately to a very
> limited group. In short, this allows for both intelligent speaking and
> brailling of math content, as well as interactive navigation. It takes
> MathML as input.

Hi James. Thank you for the update, that sounds good news!

> However, I suspect
> we'll get a better idea of what works and doesn't work as we get feedback on
> our first implementation, and other AT (e.g. Orca) will hopefully be able to
> provide input once they've done some requirements analysis.

Do you know more about Orca's plans? Some bugs have been opened in WebKit (https://bugs.webkit.org/show_bug.cgi?id=71738 and https://bugs.webkit.org/show_bug.cgi?id=71742) for quite some time but I haven't seen much activity...
(In reply to Frédéric Wang (:fredw) from comment #28)
> Do you know more about Orca's plans?
My understanding is that there are no current plans, though support would obviously be desirable. Joanmarie Diggs (already copied) is the person to ask.
@Joanmarie: are you able to indicate the status/plans for Orca?
Flags: needinfo?(jdiggs)
I plan to support MathML. I've not yet had time to figure out any specifics as I'm currently re-writing all of Orca's support for Gecko. ;)

Having said that, I do think that accessible MathML using 100% FLOSS tools and libraries is extremely important. So could you ping me in a month or two so we can resume this then? (Sorry and thanks!)
Flags: needinfo?(jdiggs)
(In reply to James Teh [:Jamie] from comment #27)

[...]

> At least for now, NVDA will continue to just grab the markup and no other
> client is asking for proper exposure via the a11y tree.

I'm officially asking. ;) I've not yet reached any definitive conclusions about what exactly "proper exposure" looks like, but trying to piece things together from HTML tags living in object attributes seems far less than ideal.

> Regarding Jonathan's proposal, I have two concerns:

[...]

> 2. Because the arguments of an operand occur in a specific order, I wonder
> whether relations are overly complicated. For example, in a  fraction, the
> numerator will always be the first child and the denominator will always be
> the second child, so the AT can just get the argument it wants by index.

The child index approach *might* work (I'm still far too early in the process to have an opinion on that). But for the sake of argument, let's assume for the moment it works reliably for all the various and sundry expressions in HTML, what about LibreOffice and PDF viewers and calculator software? Having to rely upon an exact accessible hierarchy with children in a prescribed order seems like it may prove fragile.
(In reply to Joanmarie Diggs from comment #32)
> > At least for now, NVDA will continue to just grab the markup and no other
> > client is asking for proper exposure via the a11y tree.
> I'm officially asking. ;) I've not yet reached any definitive conclusions
> about what exactly "proper exposure" looks like, but trying to piece things
> together from HTML tags living in object attributes seems far less than
> ideal.
Note that what you get will be pretty close to the semantics of a MathML tree, just without the markup. Also note that if you decide to use existing libraries (e.g. liblouisutdml and pyMathSpeak) rather than writing your own implementation from scratch, most of them accept MathML, so you'll need to piece it back into markup for them anyway. These are just points of note, though; I'm not trying to suggest either way.

> The child index approach *might* work (I'm still far too early in the
> process to have an opinion on that). But...
> what about LibreOffice and PDF viewers
The current proposal for accessible math in PDF uses MathML, so it'd make sense for exposure there to be similar to (if not the same as) browsers.
> and calculator
> software? Having to rely upon an exact accessible hierarchy with children in
> a prescribed order seems like it may prove fragile.
Normally, I'd agree. However, with math, there are clearly defined arguments for an operation (for want of a better word) and I'm not sure where else they'd go if they weren't children. I guess you could reverse the order of the arguments (e.g. put the denominator before the numerator), but if you're going to expose relations, you might as well just expose the accessibels in the documented order.
(In reply to James Teh [:Jamie] from comment #33)
> (In reply to Joanmarie Diggs from comment #32)
> > > At least for now, NVDA will continue to just grab the markup and no other
> > > client is asking for proper exposure via the a11y tree.
> > I'm officially asking. ;) I've not yet reached any definitive conclusions
> > about what exactly "proper exposure" looks like, but trying to piece things
> > together from HTML tags living in object attributes seems far less than
> > ideal.
> Note that what you get will be pretty close to the semantics of a MathML
> tree, just without the markup. Also note that if you decide to use existing
> libraries (e.g. liblouisutdml and pyMathSpeak) rather than writing your own
> implementation from scratch, most of them accept MathML, so you'll need to
> piece it back into markup for them anyway. These are just points of note,
> though; I'm not trying to suggest either way.

afaik there's no current way with atk to get .innerHTML, so we'd need to add something if you want to avoid piecing something together into mathml.

> > The child index approach *might* work (I'm still far too early in the
> > process to have an opinion on that). But...
> > what about LibreOffice and PDF viewers
> The current proposal for accessible math in PDF uses MathML, so it'd make
> sense for exposure there to be similar to (if not the same as) browsers.
> > and calculator
> > software? Having to rely upon an exact accessible hierarchy with children in
> > a prescribed order seems like it may prove fragile.
> Normally, I'd agree. However, with math, there are clearly defined arguments
> for an operation (for want of a better word) and I'm not sure where else
> they'd go if they weren't children. I guess you could reverse the order of
> the arguments (e.g. put the denominator before the numerator), but if you're
> going to expose relations, you might as well just expose the accessibels in
> the documented order.

you might want to have accessibles for white space, or maybe one for the separator?
I'm not sure whether relations is a good thing for ATK/IA2 since I agree all relations can be figured from DOM order and relations mechanism on these platforms is rather heavy. But having those stuff named is a large plus since they are human readable and you don't need to think about mathml markup to figure where the stuff are. Ideally I would like to have more math-ish API than mathml-ish. Having said that I think internal Gecko relations is still a nice thing, both for AccessFu and OS X implementation.
(In reply to alexander :surkov from comment #35)
> I'm not sure whether relations is a good thing for ATK/IA2

Taking a look at all of the relations proposed, I'm thinking that for ATK at least, we won't have a need for those relations. Instead, I think a combination of roles, subroles + child position in the accessible tree will solve most of the issues. Furthermore, if you wanted ATK to support those new relations, we would need to add them to ATK and also AT-SPI2 before Orca could use them. Then I would need to add support for all of those new relations in Orca. And since I'm thinking that a role + subrole + child position approach will likely be best for Orca, the relation approach is not on my list of things to add to ATK, AT-SPI2, or Orca in the near future. Long way of saying, if this is mainly for ATK and Orca, please don't invest too much time on it before checking on what we wind up doing. 

Sorry and thanks!
Depends on: 1108378
Depends on: 1109022
Alias: mathmla11y
Depends on: 1128143
See Also: → 1163374
See Also: → 1163377
Removing parity-chrome / parity-ie flags as Trevor pointed out that ChromeVox / IE+MathPlayer are not using accessible tree at all, but their own ways to read formulas.
Whiteboard: [parity-webkit][parity-chrome][parity-ie] → [parity-webkit]
Depends on: 418685
Mass bug change to replace various 'parity' whiteboard flags with the new canonical keywords. (See bug 1443764 comment 13.)
Keywords: parity-safari
Whiteboard: [parity-webkit]
Severity: normal → S3
You need to log in before you can comment on or make changes to this bug.