Closed Bug 495912 Opened 16 years ago Closed 13 years ago

Expose alternative content in Canvas element to ATs

Categories

(Core :: Disability Access APIs, defect)

Product:
Core
Component:
Disability Access APIs
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

()

Status:
RESOLVED FIXED
Milestone:
mozilla13

People

(Reporter: MarcoZ, Assigned: davidb)

References

(Depends on 1 open bug, Blocks 1 open bug)

Details

(Keywords: access, Whiteboard: [parity-ie9])

Attachments

(3 files, 11 obsolete files)

testcase
2.95 KB, text/html
Details
testcase2 (from IBM)
4.59 KB, text/html
Details
Combined patch (v3)
13.78 KB, patch
Details | Diff | Splinter Review
Here's a description from Mick that pretty much sums it up: From the investigation I have done, it seems that Mozilla Gecko accessibility does not include any content with in a <canvas> element. In fact, if I look at the accessible hierarchy for the following HTML fragment: <html> <head> <title>Testing canvas</title> </head> <body> <p>Following is a canvas containing alternative content.</p> <canvas> <p> This is some alternative text for the canvas. </p> </canvas> </body> </html> I can see the first paragraph before the canvas, but where the canvas and in deed the contained paragraph should be, I just see a text node (MSAA object with state read-only and role of editableText), with no content inside it.
Sure because this content is not rendered, no frames are created and therefore no accessible objects.
No longer blocks: 483984
The best option would be to render hidden canvas content in memory. Advantages are 1. We'll expose any style information and generated content to AT (via text attributes) 2. We don't need urgen hacks in accessible module to create accessible objects by DOM (not by frame) and don't need to change the code to make it working without nsIFrame. Is there a way to render DOM tree in memory? Or how much hard to implement this?
I don't think there's a consensus that this is the right way to do accessibility for canvas. We should wait for a spec for this before we implement something.
What do you mean by "render in memory"? What are we actually trying to accomplish here? There are two modes of operation that are very simple to achieve: 1) Treat the canvas as a replaced element (like we do <img> or <object>). 2) Treat the canvas just based on its display (like we do <object> when it falls back, or <img> when it fails to load). It sounds like you want some other behavior. Can you describe the behavior you want in terms of both what it looks like on screen and what it looks like via the accessibility APIs? And yes, what roc said.
(In reply to comment #3) > I don't think there's a consensus that this is the right way to do > accessibility for canvas. We should wait for a spec for this before we > implement something. I think this approach is reasonable, allows to make canvas accessible easily and it should fit greatly to not very complex cavas usage. This approach uses existing technologies and nothing should be invented additionally. Any other approach is long term. We were asked to make kind of pilot version to show how it works. If we'll show it's great then I think we have good chances to get it in the spec. I don't insist it should be unique approach to make canvas accessible. But it should be very handy to make accessible 2d graphics by HTML and ARIA usage.
Attached file testcase —
this is Rich's example, I believe this approach should work well in this case.
(In reply to comment #4) > What do you mean by "render in memory"? What are we actually trying to > accomplish here? > > There are two modes of operation that are very simple to achieve: > > 1) Treat the canvas as a replaced element (like we do <img> or <object>). > 2) Treat the canvas just based on its display (like we do <object> when it > falls back, or <img> when it fails to load). > > It sounds like you want some other behavior. Can you describe the behavior you > want in terms of both what it looks like on screen and what it looks like via > the accessibility APIs? I mean canvas content shouldn't be visible for sighted users but it should be exposed to AT like normal content so that AT users can interact with it. Therefore we need to render this hidden for sighted users canvas's DOM subtree somewhere in memory or virtually. This allows our accessibility code to work without any changes.
Do you care whether canvas content is visible while AT is enabled?
(In reply to comment #8) > Do you care whether canvas content is visible while AT is enabled? If I understand your question right then I think it should be visible for AT users only, it shouldn't be visible for sighted users.
I don't think you understood correctly. What I'm proposing is that AT simply have a mode it can trigger in which we treat <canvas> the same way we treat <object> when it falls back: just render its children instead of rendering the bitmap. If that's not the behavior you want, then I want to see a clear description of the behavior you _do_ want. Consider the following example: Some text<canvas width="500" height="500" style="position: relative"> <div style="float: left">Some floated stuff</div> <iframe src="something"></iframe> <div style="position: absolute; width: 100%; height: 100%"> Some more stuff </div> </canvas>more text What do you expect to happen in this case?
I'm still not sure I do understand you or we understand each other. Ok. We build accessible tree based on frame and DOM trees which is used by AT, more precisely accessible tree consists from accessible objects implementing special interfaces which are used by AT. We need to build accessible tree for canvas children however children shouldn't be rendered for sighted users. So we need to treat <canvas> the same way we treat the <object> when it falls back but we should do this for AT users only. Concerning to your example, we should have the following accessible tree text accessible ("Some text") canvas accessible (html:canvas) section accessible (html:div) text accessible ("Some floated stuff") frame accessible (html:iframe) section accessible (html:div) text accessible ("Some more stuff") Since we create accessible objects for HTML content based on their nsIFrame objects then we should render canvas children somewhere and this somewhere shouldn't be a screen.
Why shouldn't it be a screen? That is, for AT users only, why can we not just pretend we don't support <canvas> (as far as the frame tree is concerned)? Do the accessible objects ever depend on the geometry of the frames?
Isn't it the case that some AT users are not completely blind and would actually like to be able to see the rendered canvas content and have accessibility at the same time?
Well, right. That's certainly part of my question.
(In reply to comment #13) > Isn't it the case that some AT users are not completely blind and would > actually like to be able to see the rendered canvas content and have > accessibility at the same time? Oh, I didn't think from this point of view. Originally we are interested to support canvas for completely blind people. But if not completely blind users uses screen readers then it's the same.
(In reply to comment #15) > (In reply to comment #13) > > Isn't it the case that some AT users are not completely blind and would > > actually like to be able to see the rendered canvas content and have > > accessibility at the same time? > > Oh, I didn't think from this point of view. Originally we are interested to > support canvas for completely blind people. But if not completely blind users > uses screen readers then it's the same. I don't know if it makes sense to show canvas bitmap and its child cotent tree the same time. Threfore I wonder how to render this content not on screen.
(In reply to comment #12) > Why shouldn't it be a screen? That is, for AT users only, why can we not just > pretend we don't support <canvas> (as far as the frame tree is concerned)? If we can support an ability to make canvas accessible for blind users then why we shouldn't do this. If web page author provides child tree for canvas to make it accessible then it would be nice to expose it. > Do the accessible objects ever depend on the geometry of the frames? I think yes. Accessible object reports x and y coordinates, width and height. There is an ability to find accessible from point. We expose visible/invisible state based on frame information.
If you only want to show one or the other, that's easy. I said that in comment 4. If you want to show canvas bitmap on screen but create a frame subtree for its kids also, you can do that. You need to either tell me that the geometry of those frames doesn't matter, or explain how it will behave.
If you just want to prototype this, you could have an extension that clones the DOM tree under the <canvas> and inserts it, say, just after the canvas in the DOM. Or you could add a context menu command to the browser for canvases that replaces them with their DOM contents. If you want to be fancier, you could add an ID to the first element child of the canvas, and then insert an SVG <use> element into the DOM which points to that child. Then you would get a rendered clone of the <canvas> children which would be live to DOM changes performed by page script. Mostly. All of those could be done by an extension without changing any core code.
(In reply to comment #13) > Isn't it the case that some AT users are not completely blind and would > actually like to be able to see the rendered canvas content and have > accessibility at the same time? Disclaimer: I'm totally blind, so my understanding of what's best for low vision users may be incorrect. Having said that, if the user can see the canvas element, then they probably don't need to see the specialised tree. They might want to zoom in, magnify, etc., but the alternative content isn't required. It gets trickier for low vision users that use a screen reader, but in that case, I'd argue they rely more heavily on the screen reader to provide content than the screen, so it's probably okay in that case too. Some users do use the mouse to have the screen reader announce text and that would certainly be a bit tricky, as the on-screen display and the content seen by the AT wouldn't be synchronised.
IIRC, people with dyslexia have been mentioned as one group who can see but who may need text and possibly other content announced to them.
(In reply to comment #19) > All of those could be done by an extension without changing any core code. Ok, it could be done by preference to show canvas child content always, it should be more preferable than an extension. At least it's more evident and screen reader users won't be forced to install any extension to read the page. (In reply to comment #18) > If you want to show canvas bitmap on screen but create a frame subtree for its > kids also, you can do that. You need to either tell me that the geometry of > those frames doesn't matter, or explain how it will behave. I think geometry is important however I don't know how it should behave. I think we can't render whole page for a11y (where canvas child content is "visible" for AT only), otherwise if we'll render canvas child content somehow separately from the page then it will overlap geometrically with normal underlying content.
In comment #5 you said "We were asked to make kind of pilot version to show how it works." An extension is just fine for that.
(In reply to comment #23) > In comment #5 you said "We were asked to make kind of pilot version to show how > it works." An extension is just fine for that. That would be easiest way for demonstration. However it can confirm paradigm in generally only. Since original approach assumes to "show" canvas child content to AT only then ARIA will be used to manage the focus, I mean by aria-activedesdendant attribute usage. This won't happen in extension approach because focus will be handed by usual way in this case.
(In reply to comment #18) > If you want to show canvas bitmap on screen but create a frame subtree for its > kids also, you can do that. You need to either tell me that the geometry of > those frames doesn't matter, or explain how it will behave. I think canvas child content should have geometry of canvas bitmap. If it's bigger than scolling should be available. Is it possible to draw canvas child content not on screen (somewhere in memory) so that document styles will be picked up and their frames will have position relative canvas element? (please correct me if I understand geometry term wrongly).
So floated or absolutely positioned kids of the canvas should still be constrained to the canvas area? I'm not sure why you keep talking about "drawing". We don't have to draw anything to just create frames....
(In reply to comment #26) > So floated or absolutely positioned kids of the canvas should still be > constrained to the canvas area? I think no because absolute or float position is used intentionally. Though I don't know what goal author could pursuit in this case. I think we shouldn't do any exception in styles processing. We just need to consider canvas element of the canvas bitmap size as a container of canvas child content I think. > I'm not sure why you keep talking about "drawing". We don't have to draw > anything to just create frames.... That's great. I really don't know much about mozilla layout. Therefore I might use terms incorrectly. Sorry for that. If we could create frame tree for canvas child content, we could get computed styles for them, their geometric properties then I think it's enough for us.
I suppose you could make the canvas frame claim to be a containing block but let floats and the like float out of it. This would be doable if we really have to. I'm not convinced it's a good idea.
As you know, canvas usage is not just about one-shot rendering; it can be interactive. Consider: canvas.canvas.addEventListener(... Accessibility 101 is keyboard accessibility. We need a 'point of regard' in canvas (AKA focus). A point of regard exists in the context of some structure. We need a structure. The structure can be defined by API (probably an Adapter). We should wait to see what comes out of the canvas accessibility discussions. Anyways, here are some possible directions: 1. make only non-interactive (AKA sane?) usage of canvas accessible using hooks. 2. make it possible for web developers to expose the information contained in canvas by supporting some API (like nsIAccessible and friends). 3. allow a DOM to be created based on fallback markup inside the canvas tags. 4. do nothing. 5. ? I'm strongly opposed to #4. Regarding #2, on the desktop it is possible for canvas-like applications (e.g. Paint) to be accessible by supporting API (e.g. MSAA server). It doesn't seem to actually happen often, but that it can happen is a good thing. Alexander, are you prototyping #3? (I thought Apple was doing that?) http://lists.w3.org/Archives/Public/public-canvas-api/2009OctDec/0019.html
The fact that on the desktop, developers haven't done a good job at #2 makes me wonder where we should focus our accessibility resources.
Prototyping the exposure of the DOM in between canvas tags doesn't sound too hard though.
Another approach is folding the AT (e.g. screen reader) into the canvas application. On this note, why is document.say("foxy!"); not standard functionality today? (just spun off bug 525444)
Ok, it sounds the approach to expose shadow DOM tree of canvas becoming avowed. It is supported by Rich Schwerdtfeger and If I get right Microsoft and Apple are going to follow this. David, what do you think?
I think we should let the TPAC dust settle, and then we can have a chat with Rich and others.
I support Alex's proposal to incorporate a shadow DOM which represents accessible objects and structure found in the canvas visual rendering. To avoid layout issues, focus within the shadow DOM should be managed through the use of aria-activedescendant on the canvas tag. The value of aria-activedescendant should be the id of an element in the shadow DOM and it's setting to a valid id in the shadow DOM should result in the browsers sending a focus change event to assistive technologies. Should the author wish to provide keyboard tabbing within the shadow DOM that will need to be managed by a keyboard handler on the canvas element which is already consistent with the way canvas must operate today. The shadow DOM should allow for standard HTML elements although these elements should not be rendered. All ARIA information within the shadow DOM should be exposed as is normal to an assistive technology as they would normally. I also recommend that the canvas element should support alternative views the default being the shadow DOM which would be bound by the developer to the canvas area. The reason for this is there will be scenarios where it will be easier and more end-user friendly to provide a different view to users. View selection would be controlled by the browser using a number of means. To do this we should follow accessibility metadata from the AccessForAll specification with the base assumption that <canvas> is a visual rendering. The format should be along the followiing lines: <canvas> <access mode="default"> /*Since canvas is visual this is the default moe*/ </default> <access mode="textual"> /*Here we could place a long description or an alternative aria-enabled widget*/ </access> <access mode="auditory"> </access> <access mode="tactile"> /*This is in access for all and we could lose it for now for obvious reasons*/ </access> <access mode="olfactory"> /*This is in access for all and we could lose it for now for obvious reasons*/ </access> <access mode="visual" type="signlanguage"> </access> </canvas> For each alternative we could include an adaptation type as listed here: audioDescription, caption, e-book, signLanguage, highContrast, transcript, alternativeText, longDescription, haptic} The modes and types are all taken out of Access For All V3 which we are about to publish as a working draft. Selection of the access mode would be based on user preferences in the browser or even accessible actions in the accessibility API supported by the browser. David, if you would I can set up a call so that we can all discuss. At the very least we should start with the shadow DOM.
It would be great to talk but I'm going to be heads down for a while, hopefully resurfacing mid next week. Let's coordinate over email.
We need to come up with a KISS solution that willing web devs can actually use. Regarding treatment of a DOM subtree to the canvas element, I've received some w3c actions and will be pinging Moz people for consult soon. For those with access: http://www.w3.org/WAI/PF/HTML/track/actions/4 http://www.w3.org/WAI/PF/HTML/track/actions/5 Aside: CC'ing Henri since he raises good points here: http://lists.w3.org/Archives/Public/public-html/2009Dec/0424.html Henri please un-cc yourself if you don't have the bandwidth, but your thoughts are much appreciated on this topic. For starters can you think of a good name for the canvas sub-dom?
"The canvas working group has reached consensus that we should provide a shadow DOM for <canvas> where the canvas UI is bond to accessible objects in the shadow DOM. Two things we need to agree on is being able to use standard input controls and contenteditable areas in the shadow DOM as will already be accessible in the browser. Since these interactive components are not visible and would be in the keyboard navigation sequence we need to know if this will be a problem for Firefox, Safari, or IE. Also, contenteditable provides a caret location, without which we would need to provide an API for the caret location to support magifiers and screen readers. We need to address these action items in the next 2 weeks." http://markmail.org/message/sxbenry2q75xl5bt (actions in comment 37)
Imo putting things in the keyboard navigation sequence without making them visible is a terrible idea. How is that supposed to work for users who are NOT using an AT?
(In reply to comment #39) > Imo putting things in the keyboard navigation sequence without making them > visible is a terrible idea. How is that supposed to work for users who are NOT > using an AT? My understanding is that the idea is that the web developer coordinates the syncing of the keyboard navigation (key events) in the non-visual DOM, with visual happenings in the canvas. Whatever the solution it will not be the case that visual keyboard users are left out.
If the focus and key handling is always on the canvas element, then it would need to drive virtual focus in the non-visual DOM via aria-activedescendant presumably (for AT).
TBH, from an evangelist standpoint, I'm not sure whether getting web devs to use a non-visual DOM will be easier than getting them to use alternatives to canvas for these kinds of interactive applications in the first place. That said, this discussion is about thinking through how a non-visual DOM solution might work.
(In reply to comment #40) > (In reply to comment #39) > > Imo putting things in the keyboard navigation sequence without making them > > visible is a terrible idea. How is that supposed to work for users who are NOT > > using an AT? > > My understanding is that the idea is that the web developer coordinates the > syncing of the keyboard navigation (key events) in the non-visual DOM, with > visual happenings in the canvas. > > Whatever the solution it will not be the case that visual keyboard users are > left out. Disadvantage of this if the canvas UI rendering is not in sync with shadow DOM then the web page will be broken for keyboard navigation. Possibly we should have kind of virtual focus and virtual caret for shadow content. For example, if canvas has couple of tab navigable controls, user press the tab key, canvas handles an event, prevents its default action (next control to canvas element doesn't get a focus) then next shadow DOM control gets virtual focus.
(In reply to comment #43) > (In reply to comment #40) > > (In reply to comment #39) > Possibly we should have kind of virtual focus and virtual caret for shadow > content. For example, if canvas has couple of tab navigable controls, user > press the tab key, canvas handles an event, prevents its default action (next > control to canvas element doesn't get a focus) then next shadow DOM control > gets virtual focus. Possibly. I'm really having my doubts about this non-visual ("shadow") DOM approach. In particular the engineering effort vs payoff (as compared to our other bugs) and the feeling of kludgey-ness I'm having. I wish I could offer an obviously better alternative.
(In reply to comment #44) > I'm really having my doubts about this non-visual ("shadow") DOM approach. In > particular the engineering effort vs payoff (as compared to our other bugs) and > the feeling of kludgey-ness I'm having. I wish I could offer an obviously > better alternative. I'd like to echo these thoughts with experience from the Bepsin project which makes significant use of the canvas widget. If we had the option to have a shadow DOM (either push or pull) we would probably _not_ use it to aid accessibility. In fact it could be a detriment to making Bespin type apps accessible. The fact is that we could have a similar effect trivially by creating a textarea element and just throwing our content in there. We don't want to do that because it's 'checkbox accessibility' - that is to say, following the letter of the law even though it's useless. We have plans to make Bespin accessible, but we want to do the job _usefully_ not just legally. I think the existence of such a shadow DOM could lead developers in our situation to do a useless legal minimum. This would be a terrible situation, allowing developers to be legal AND useless. So are we representative of any significant set of the uses of the canvas element? I think we're probably representative of anyone that is writing a complex app with canvas, although there are other uses of canvas that might be different. I would be interested to hear of a general use of the canvas element for which a parallel DOM was helpful.
Note I filed bug 548291 in case we the map approach gets more traction.
Update: IE9 will allow keyboard navigation into the content sub-tree inside the canvas element. IE9 info: http://www.paciellogroup.com/blog/?p=670 Current spec: http://mzl.la/dmFlUM
Whiteboard: [parity-ie9]
Note there is no attribute or anything to indicate that the sub-tree is intended for navigation and/or accessibility, it is just expected to be available by default, to all users. (In reply to comment #4) > There are two modes of operation that are very simple to achieve: > > 1) Treat the canvas as a replaced element (like we do <img> or <object>). > 2) Treat the canvas just based on its display (like we do <object> when it > falls back, or <img> when it fails to load). > > It sounds like you want some other behavior. Yes, we want a third behavior. I think we want an unpainted sub-tree bounded within the canvas geometry that otherwise works as a sub-tree should. Keyboard navigation should work, and our accessibility code will pick it up and expose it.
So tabbing to <canvas> subtree should work, and also accesskeys? IsAccessKeyTarget in nsEventStateManager.cpp could be tweaked a bit, and nsFocusManager::GetNextTabbableContent would need some more changes.
(In reply to comment #49) > So tabbing to <canvas> subtree should work, and also accesskeys? Yes. My guess is that accesskeys should work.
Boris, what about treating DOM content inside canvas tags (fallback) as if it is an @hidden container? The spec currently says: "Elements in a section hidden by the hidden attribute are still active, e.g. scripts and form controls in such sections still execute and submit respectively. Only their presentation to the user changes." It isn't clear to me whether active also means tab navigation works... ?
@hidden implies display:none. Hence no CSS boxes, hence no tab navigation.
OK thanks. Are display:none elements "active"?
On the sense quoted above, yes.
BZ does comment #40 seem reasonable? I'm just looking for your position, not suggesting this blocks FF4 by any means.
> BZ does comment #40 seem reasonable? I don't know. We can certainly create but not paint frames for canvas kids as needed, with some contortions.
OK I'd like to experiment with subdom exposure - ideally tonight. Boris, Henri, can you jump start me here? What code should I look at? I basically want to make sure we can expose the child DOM tree unrendered, but with regular events and focus management.
You probably want to talk to Olli and Neil Deakin, then.
(In reply to comment #57) > Boris, Henri, can you jump start me here? Sorry, I'm not familiar with focus management code.
(In reply to comment #56) > > BZ does comment #40 seem reasonable? > > I don't know. > > We can certainly create but not paint frames for canvas kids as needed, with > some contortions. I think we need to do this. It probably makes sense to make it a config option. I'm not sure I'm the right person, so help wanted.
Why do we need frames? Maintaining separate pref controlled code paths is a recipie for regressions. We need to be able to construct accessible objects for frame-less nodes anyway.
(In reply to comment #61) > Why do we need frames? Unfortunately, our a11y code still relies on frames for information in many cases. It would be nice to not do that, and I started trying to work around this but my patch started looking invasive. (I thought we had a bug open for getting rid of our reliance on frames but can't find it.) > Maintaining separate pref controlled code paths is a > recipe for regressions. Agreed. > We need to be able to construct accessible objects > for frame-less nodes anyway. Which case(s) are you thinking of here?
I thought part of the goal here, if we do this, was to expose all this stuff in the tab order, and you can't tab to things without frames.
> > We need to be able to construct accessible objects > > for frame-less nodes anyway. > > Which case(s) are you thinking of here? For things like aria-describedby.
(In reply to Boris Zbarsky (:bz) from comment #63) > I thought part of the goal here, if we do this, was to expose all this stuff > in the tab order, and you can't tab to things without frames. Okay, so tab order is tied to frames then? In that case we'll need frames for sure, unless we think severing this tie is something we want to do. (?)
The tab order is determined by the order in which the elements are displayed, which is best described by the order the frames are stored in.
Is getting the order the only reason that tabbing uses frames? Is there any situation when "frame order" != "document order". Where "document order" is depth-first tree traversal.
Frame order generally followed content order although there is some exceptions involving weirdness like fieldset legends and table captions.
And out of flows. And XBL.
Besides frame order, tab navigation checks some characteristics of elements that are available via the frame: - whether it is visible - whether in a visible view (visible tab or deck page) - check -moz-user-focus - for a element that is scrollable, check if scrollbars are visible - check for and constrain tab navigation in menupopup frames For this bug, if there is some special non-rendered content, it seems that only the second point would apply.
Attached patch WIP (obsolete) — — Splinter Review
Posting WIP 1. I'll be working on this bug periodically. This WIP results in a canvas subdom which is exposed to AT, and is keyboard navigable. As per an email from Boris, I've changed nsHTMLCanvasFrame to inherit from nsContainerFrame. I've made a somewhat naive change to the reflow implementation but I'm not sure if it is right.
Assignee: nobody → bolterbugz
The reflow change is almost certainly wrong. What you really want to do is walk your kids and reflow each one (via ReflowChild/FinishReflowChild) at whatever width you want and with unconstrained height.
Attached patch WIP (obsolete) — — Splinter Review
Updated WIP. Added some slapdash code to reflow children - probably doing it wrong.
Attachment #591775 - Attachment is obsolete: true
Attached file testcase2 (from IBM) —
Note the recent WIP barfs a little on this test case. Console output: WARNING: Unable to test style tree integrity -- no content node: file c:/Users/davidb/moz/inbound/ff-dbg/layout/base/../../../layout/base/nsCSSFrameConstructor.cpp, line 8078 ###!!! ASSERTION: Wrong line container hint: '!aForFrame || (aLineContainer == FindLineContainer(aForFrame) || (aLineContainer->GetType() == nsGkAtoms::letterFrame && aLineContainer->GetStyleDisplay() ->IsFloating()))', file c:/Users/davidb/moz/inbound/ff-dbg/layout/generic/../../../layout/generic/nsTextFrameThebes.cpp, line 1178 [...] ###!!! ASSERTION: leak: 'mSpansAllocated == mSpansFreed', file c:/Users/davidb/moz/inbound/ff-dbg/layout/generic/../../../layout/generic/nsLineLayout.cpp, line 272 ###!!! ASSERTION: leak: 'mFramesAllocated == mFramesFreed', file c:/Users/davidb/moz/inbound/ff-dbg/layout/generic/../../../layout/generic/nsLineLayout.cpp, line 273
Comment on attachment 594239 [details] [diff] [review] WIP Hi Boris can you take a quick look to see how off track I am? I'm getting assertions (Comment 74) and my puny brain hurts. I took a look at how this is rendering via tilt (the 3d view) and it seems the children render over top of the canvas (z order) which is probably not what I want. Also I wonder if I can force the children to render as visibility: hidden for good measure?
Attachment #594239 - Flags: feedback?(bzbarsky)
Oh, hmm. The canvas is reflowing inline content directly; that's not going to work. OK, this will take a bit more work on the frame construction side. What you probably want to do is to wrap the kids of a canvas inside an anonymous block. That would make the most sense behavior-wise, right? Doesn't tilt show the _dom_, not the rendering? There should be no rendering of the kids at all here with the attached code.
(In reply to Boris Zbarsky (:bz) from comment #76) > Oh, hmm. The canvas is reflowing inline content directly; that's not going > to work. OK, this will take a bit more work on the frame construction side. > What you probably want to do is to wrap the kids of a canvas inside an > anonymous block. That would make the most sense behavior-wise, right? I think so yeah. I'll try. > Doesn't tilt show the _dom_, not the rendering? Yes. > There should be no > rendering of the kids at all here with the attached code. True. Okay I'll assume there is nothing to worry about here.
> I think so yeah. I'll try. OK. If this gets hairy, let me know. We have some other places that do similar things (e.g. buttons), so some code refactoring might be in order to make it nice and clean no matter what. But for a prototype, just hacking it in somehow would be ok.
It worked :) I'll clean it up when I can and post another WIP.
Attached patch WIP 1.3 (obsolete) — — Splinter Review
Attachment #594239 - Attachment is obsolete: true
Attachment #594239 - Flags: feedback?(bzbarsky)
Boris what would you like to see added/changed in WIP 1.3 to make this production quality? The build has been tested and interested parties are quite liking it.
Comment on attachment 596772 [details] [diff] [review] WIP 1.3 Seeking feedback since I want to take this beyond prototype.
Attachment #596772 - Flags: feedback?(bzbarsky)
Well, not just blindly copy/pasting the button code. ;) If this works reasonably, then I guess one question is whether :before and :after should work on canvas. I assume they should?
Comment on attachment 596772 [details] [diff] [review] WIP 1.3 (In reply to Boris Zbarsky (:bz) from comment #83) > Well, not just blindly copy/pasting the button code. ;) Yeah I had a code comment about refactoring that :) > If this works reasonably, then I guess one question is whether :before and > :after should work on canvas. I assume they should? Sounds like yes as per IRC discussion. Thanks! (removing feedback request)
Attachment #596772 - Flags: feedback?(bzbarsky)
One note on that last patch, offhand: you need to implement GetContentInsertionFrame on the canvas and point it to the block inside.
Pardon the potentially slightly unrelated question, but how do the nodes within the alternative content get their location info? How does this synchronise with the visual points in the canvas? If there's no way to do this, this will break mouse tracking, touch screen access, clicking of certain elements and possibly other things. Any links appreciated if this question is answered somewhere in spec.
Depends on: 728111
(In reply to James Teh [:Jamie] from comment #86) > Pardon the potentially slightly unrelated question, but how do the nodes > within the alternative content get their location info? How does this > synchronise with the visual points in the canvas? If I get right (Boris and David please correct me if I'm wrong) then elements inside canvas are "rendered" (not visible on screen), i.e. frame objects are created, so they have coordinates and size. If their size exceeds the size of canvas then it should be equivalent as they were inside of scrollable area.
Ah, David, please make sure to provide testing for accessible tree, boundaries and hittesting.
(In reply to alexander :surkov from comment #87) > > how do the nodes > > within the alternative content get their location info? How does this > > synchronise with the visual points in the canvas? > If I get right (Boris and David please correct me if I'm wrong) then > elements inside canvas are "rendered" (not visible on screen), i.e. frame > objects are created, so they have coordinates and size. Tricky. So if the canvas is drawing text, the alternative content has to make sure every text node has the same location as the corresponding piece of text in the canvas. Is that even possible? It's very possible that the canvas text will be rendered with different sized characters and spacing, so it'd be virtually impossible for the alternative content to synchronise with this. Think pdf.js for a really complex example.
The only thing I can come up with is to represent every single character or word in its own div and use css in weird and wonderful ways to make sure the location info ends up as required.
(In reply to James Teh [:Jamie] from comment #89) > (In reply to alexander :surkov from comment #87) > > > how do the nodes > > > within the alternative content get their location info? How does this > > > synchronise with the visual points in the canvas? > > If I get right (Boris and David please correct me if I'm wrong) then > > elements inside canvas are "rendered" (not visible on screen), i.e. frame > > objects are created, so they have coordinates and size. > Tricky. So if the canvas is drawing text, the alternative content has to > make sure every text node has the same location as the corresponding piece > of text in the canvas. Is that even possible? That must be a nightmare of this approach for canvas authors since they need to ensure rendering image in canvas is exact equivalent of underlying DOM. That was one of my points when I argued why I don't like this approach. Maybe I miss something though.
(In reply to James Teh [:Jamie] from comment #90) > The only thing I can come up with is to represent every single character or > word in its own div and use css in weird and wonderful ways to make sure the > location info ends up as required. right, that's what they should do. If they draw a text then presumably they know its position and size so they do css to update position and size of corresponding DOM.
(In reply to alexander :surkov from comment #91) > That must be a nightmare of this approach for canvas authors since they need > to ensure rendering image in canvas is exact equivalent of underlying DOM. > That was one of my points when I argued why I don't like this approach. Agreed; it's one of several reasons I don't like the approach, but that's probably beyond the scope of this bug. (In reply to alexander :surkov from comment #92) > If they draw a text then presumably they > know its position and size so they do css to update position and size of > corresponding DOM. For full access in something complex like pdf.js, this will probably have to be done for every individual character or word, which means a lot of nodes. Also, scrolling is going to present a real challenge where there is more than a screen of text. I suspect there will have to be complex JavaScript to watch when an element in the fallback is scrolled so that the canvas can update accordingly and vice versa.
(In reply to Boris Zbarsky (:bz) from comment #85) > One note on that last patch, offhand: you need to implement > GetContentInsertionFrame on the canvas and point it to the block inside. OK thanks. (In reply to alexander :surkov from comment #88) > Ah, David, please make sure to provide testing for accessible tree, > boundaries and hittesting. Yeah I anticipate lots of tests. I don't want the common use cases to suffer extended delays though. There are additional potential phases to canvas accessibility implementation, this bug is phase one. Alexander, Jamie, I understand the temptation to jump to the complex cases. It is certainly useful when designing solutions. The canvas discussions have been happening for 2? years and it is implementation time. This bug is about getting at least parity to the IE9 approach. It is okay if this approach fails. We won't always win. Dominic is implementing something similar for chrome. If we want further debate, let's take it off bug please.
(In reply to David Bolter [:davidb] from comment #94) > (In reply to Boris Zbarsky (:bz) from comment #85) > > One note on that last patch, offhand: you need to implement > > GetContentInsertionFrame on the canvas and point it to the block inside. > > OK thanks. > > (In reply to alexander :surkov from comment #88) > > Ah, David, please make sure to provide testing for accessible tree, > > boundaries and hittesting. > > Yeah I anticipate lots of tests. I don't want the common use cases to suffer > extended delays though. There are additional potential phases to canvas > accessibility implementation, this bug is phase one. I hope the phase one includes automated testing, otherwise we can't be even sure that it continues working. > Alexander, Jamie, I understand the temptation to jump to the complex cases. > It is certainly useful when designing solutions. The canvas discussions have > been happening for 2? years and it is implementation time. This bug is about > getting at least parity to the IE9 approach. > > It is okay if this approach fails. We won't always win. Dominic is > implementing something similar for chrome. If we want further debate, let's > take it off bug please. That's better than nothing. I'm just flooding, take it easy :)
(In reply to alexander :surkov from comment #95) > (In reply to David Bolter [:davidb] from comment #94) > > (In reply to Boris Zbarsky (:bz) from comment #85) > I hope the phase one includes automated testing, otherwise we can't be even > sure that it continues working. Yes definitely tests will be included :)
Attached patch WIP 1.4 (obsolete) — — Splinter Review
Attachment #596772 - Attachment is obsolete: true
But 728516 adds the infrastructure to make the frame construction part of this easy.
Depends on: 728516
This should probably get review from roc or Timothy
Comment on attachment 598277 [details] [diff] [review] WIP 1.4 Marking the prototype code obsolete. Boris has the solid approach covered via a cool new patch here and bug 728516 (sweet foundation work).
Attachment #598277 - Attachment is obsolete: true
Note that that just handles the frame construction bits. The actual canvas frame changes still need to happen.
Attachment #598277 - Attachment is obsolete: false
Attachment #598277 - Attachment is obsolete: true
Attached patch HTML Canvas frame changes (obsolete) — — Splinter Review
Removed the canvas frame construction prototyping code. Combining this with BZ's work -- initial tests look good. Pushing a try build to hand out for extra testing. Please feel free to comment on this patch.
Comment on attachment 600070 [details] [diff] [review] HTML Canvas frame changes For the reflow code, you should be able to assert that you only have one child and then just reflow the one child, right? Other than that, looks reasonable to me though I'd like it if roc or dbaron reviewed too.
(In reply to Boris Zbarsky (:bz) from comment #103) > Comment on attachment 600070 [details] [diff] [review] > HTML Canvas frame changes > > For the reflow code, you should be able to assert that you only have one > child and then just reflow the one child, right? Yeah how silly of me. I'll fix that.
Note I'll write some accessibility mochitests separately.
Attachment #600070 - Attachment is obsolete: true
Attachment #600113 - Flags: review?(bzbarsky)
Comment on attachment 600113 [details] [diff] [review] HTML Canvas frame changes - now reflow only child. r=me Note that this needs to land in the same changeset as the frame construction changes, so you should just fold those in here.
Attachment #600113 - Flags: review?(bzbarsky) → review+
Attached patch a11y mochitests (unfinished) (obsolete) — — Splinter Review
Alexander I began piggybacking on some existing tests. Let me know what you think of this approach. TODO: event tests.
Attachment #600486 - Flags: feedback?(surkov.alexander)
Comment on attachment 600486 [details] [diff] [review] a11y mochitests (unfinished) Review of attachment 600486 [details] [diff] [review]: ----------------------------------------------------------------- ::: accessible/tests/mochitest/states/test_visibility.html @@ +38,5 @@ > + document.getElementById("canvas_div").style.visibility = "hidden"; > + document.getElementById("canvas_div_off").style.visibility = "hidden"; > + document.getElementById("canvas_div_abschild").style.visibility = "hidden"; > + document.body.clientWidth; // flush layout > + testAccessibleTree("canvas_outer_div", { children: [] }); I'd say second part of the block should be located under treeupdate stuffs, anyway you don't test states here. Ah, I see, you just copy the block above which was introduced in bug 591363 where you've got temporary freedom from my oppression :) actually it seems all you need is to add tests under treeupdate to make sure we expose and update the tree for canvas subdom. ::: accessible/tests/mochitest/treeupdate/test_gencontent.html @@ +132,5 @@ > + gQueue.push(new addGenContent("container2", "container2_child")); > + > + // Test that this works inside a canvas subdom. > + gQueue.push(new insertNodeHavingGenContent("canvas_container1")); > + gQueue.push(new addGenContent("canvas_container2", "canvas_container2_child")); I wonder if bz talked about generated content for canvas itself rather than to its content ::: accessible/tests/mochitest/treeupdate/test_recreation.html @@ +138,5 @@ > + gQueue.push(new removeRole("canvas_span")); > + gQueue.push(new changeRole("canvas_div1")); > + gQueue.push(new changeOnclick("canvas_div2")); > + gQueue.push(new changeHref("canvas_anchor")); > + gQueue.push(new changeMultiselectable("canvas_div3")); if canvas expose a tree which gets updated then I'd say you don't need to copy these special tests for canvas content
Attachment #600486 - Flags: feedback?(surkov.alexander)
(In reply to alexander :surkov from comment #108) > actually it seems all you need is to add tests under treeupdate to make sure > we expose and update the tree for canvas subdom. of course invisible and offscreen states test is good to have and additionally I'd like to see boundaries and childAtPoint testing. Anything else?
> I wonder if bz talked about generated content for canvas itself Yes. I was asking about "canvas::before { content: 'Hello'; }". Generated content on kids of the canvas will obviously just work "normally".
Comment on attachment 598488 [details] [diff] [review] Frame construction part of this bug after refactoring BZ's part of the canvas a11y work.
Attachment #598488 - Flags: review?(roc)
(In reply to Boris Zbarsky (:bz) from comment #110) > > I wonder if bz talked about generated content for canvas itself > > Yes. I was asking about "canvas::before { content: 'Hello'; }". Generated > content on kids of the canvas will obviously just work "normally". Yes I understood :) My tests here may be overkill.
Attached patch Combined patch (obsolete) — — Splinter Review
Attachment #601034 - Flags: review?(surkov.alexander)
Attachment #601034 - Attachment is obsolete: true
Attachment #601034 - Flags: review?(surkov.alexander)
Attached patch Combined patch (obsolete) — — Splinter Review
Alexander, here are the tests as per our IRC. I didn't add newlines or whitespace to the canvas tree test because there are some \n frames that need to be optimized away (in layout) in a follow up (as per IRC with bz). (Note: I left the extra frame construction patch to make it easier for Robert's review; it is included in this combined patch though)
Attachment #600113 - Attachment is obsolete: true
Attachment #600486 - Attachment is obsolete: true
Attachment #601045 - Flags: review?(surkov.alexander)
Comment on attachment 601045 [details] [diff] [review] Combined patch Review of attachment 601045 [details] [diff] [review]: ----------------------------------------------------------------- ::: accessible/tests/mochitest/states/test_visibility.html @@ +33,5 @@ > > + // Test that this works inside a canvas subdom > + testStates("canvas_div", 0, 0, STATE_INVISIBLE); > + testStates("canvas_div_off", STATE_OFFSCREEN, 0, STATE_INVISIBLE); > + testStates("canvas_div_abschild", 0, 0, STATE_INVISIBLE); isn't offscreen/invisilbe a part of second stage of canvas work (element path definition)? (under that assumption I agreed to not test boundaries and childAtPoint) @@ +38,5 @@ > + document.getElementById("canvas_div").style.visibility = "hidden"; > + document.getElementById("canvas_div_off").style.visibility = "hidden"; > + document.getElementById("canvas_div_abschild").style.visibility = "hidden"; > + document.body.clientWidth; // flush layout > + testAccessibleTree("canvas_outer_div", { children: [] }); didn't we agreed to move this block under treeupdate tests? ::: accessible/tests/mochitest/tree/test_canvas.html @@ +26,2 @@ > ] > + }; wrong indentation, I prefer short style for tree testing (see treeupdate/test_gencontent.html for a hint) @@ +43,5 @@ > <div id="content" style="display: none"></div> > <pre id="test"> > </pre> > > + <canvas id="canvas" tabindex="0"><input type="checkbox" /><input /></canvas> btw, does keyboard navigation work for canvas content or is it part of some other canvas work item?
Attachment #601045 - Flags: review?(surkov.alexander)
(In reply to alexander :surkov from comment #115) > @@ +43,5 @@ > > <div id="content" style="display: none"></div> > > <pre id="test"> > > </pre> > > > > + <canvas id="canvas" tabindex="0"><input type="checkbox" /><input /></canvas> > > btw, does keyboard navigation work for canvas content or is it part of some > other canvas work item? Yes it works because we have frames :)
(In reply to David Bolter [:davidb] from comment #116) > > btw, does keyboard navigation work for canvas content or is it part of some > > other canvas work item? > > Yes it works because we have frames :) cover by a test?
Comment on attachment 598488 [details] [diff] [review] Frame construction part of this bug after refactoring Thanks. This is integrated into the combined patch. Reviews carried forward.
Attachment #598488 - Attachment is obsolete: true
Attachment #601045 - Attachment is obsolete: true
Attachment #601614 - Flags: review?(surkov.alexander)
Comment on attachment 601614 [details] [diff] [review] Combined patch (new test changes) Review of attachment 601614 [details] [diff] [review]: ----------------------------------------------------------------- r=me for a11y mochitests ::: accessible/tests/mochitest/events/test_focus_canvas.html @@ +23,5 @@ > + > + var gQueue = null; > + function doTests() > + { > + gQueue = new eventQueue(EVENT_FOCUS); no need to pass EVENT_FOCUS as argument @@ +49,5 @@ > + <pre id="test"> > + </pre> > + > + <canvas> > + <input id="button" type="button" /> no need to use xhtml syntax, just '>' ::: accessible/tests/mochitest/tree/test_canvas.html @@ +19,5 @@ > function doTest() > { > + var accTree = { > + role: ROLE_CANVAS, > + children: [ wrong indentation @@ +26,2 @@ > ] > + }; as I said earlier I prefer compact style: var accTree = { CANVAS: [ { CHECKBUTTON: [] }, { ENTRY: [] } ] }; @@ -26,2 @@ > testAccessibleTree("canvas", accTree); > - I woudn't remove empty line here @@ +43,5 @@ > <div id="content" style="display: none"></div> > <pre id="test"> > </pre> > > + <canvas id="canvas" tabindex="0"><input type="checkbox" /><input /></canvas> same, just '>', not '/>' ::: accessible/tests/mochitest/treeupdate/test_canvas.html @@ +35,5 @@ > + getNode("dialog").style.display = "block"; > + getNode("table").style.visibility = "visible"; > + getNode("a").textContent = "link"; > + getNode("input").value = "hello"; > + getNode("input").focus(); that's a mind hurricane especially the focus, you don't test these things here. I'd use simple test like display:none changing to display:block. @@ +64,5 @@ > + > + function doTest() > + { > + // ensure we start with no subtree > + testAccessibleTree("canvas", { CANVAS: [] }); I prefer to keep it as part of invoker (inside invoke() function)
Attachment #601614 - Flags: review?(surkov.alexander) → review+
Beat me to it! Thanks. I can only assume gremlins did something to my tests. Will fix and re-land (tonight/tomorrow).
Attachment #601614 - Attachment is obsolete: true
https://hg.mozilla.org/mozilla-central/rev/0d48a869291f
Status: NEW → RESOLVED
Closed: 13 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla13
Attachment #594255 - Attachment is patch: false
Attachment #594255 - Attachment mime type: text/plain → text/html
Depends on: 733755
Depends on: 852803
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: