Closed Bug 364373 Opened 18 years ago Closed 17 years ago

Message thread status is not communicated through accessibility API

Categories

(Thunderbird :: Mail Window Front End, defect)

Product:
Thunderbird
Component:
Mail Window Front End
Platform:
All
Windows XP
Type:
defect
Priority:
Not set
Severity:
major

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: vtsaran, Assigned: davidb)

References

(Blocks 1 open bug)

Details

(Keywords: access)

Attachments

(1 file)

pic of conversation (thread) starter email states
63.44 KB, image/png
Details 
User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1) Gecko/20061010 Firefox/2.0
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1) Gecko/20061010 Firefox/2.0

Currently, if arrowing through the message list, the status of the message thread, i.e. whether the thread (conversation) is in focus or just an E-mail message, is not communicated to the accessibility API. Thus, the user is not aware whether their cursor is on the message thread which needs to be expanded/collapsed or on a single message.
In addition, the state of the message is not communicated as well, e.g. whether the thread under the cursor is expanded or collapsed.

Reproducible: Always

Steps to Reproduce:
1. Launch a screen reader of your choice (unfortunately, this is probably the easiest). You can also use one of the Microsoft tools, like AccExplorer or Object Inspector to look at the properties of an object in focus.
2. Sort your messages by thread and possibly collapse all the threads.
3. Start arrowing up and down the list and listen to what your screen reader says.
Actual Results:  
Only information about the messages is spoken.

Expected Results:  
Wherever the message thread is encountered, the word "conversation" or "message thread" should be announced, along with the state of the thread, e.g. expanded or collapsed.

Contact me if more information is needed and/or David Bolter" <david.bolter@utoronto.ca>.
How does this compare to the Bookmark manager in Firefox?  Same problem there with collapsed branches, or is this specific to the mail thread pane?
Keywords: access
Victor, does the parent message for the thread expose either STATE_COLLAPSED or STATE_EXPANDED as one of its states?

I've got a broken at-poke on Linux ATM so I can't check myself.
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
mscott, sorry I accidentally assigned this to you this morning. Reassigning to me (I hope thats OK).
Assignee: mscott → david.bolter
Status: ASSIGNED → NEW
Status: NEW → ASSIGNED
Will, I suspect this might be a windows screen reader bug. Does orca recognize a thread root email (conversation starter email)?  Perhaps using the expandable state?
Hi David, the expanded state would be a really nice way to detectthisinformation.  I've seen this used in other email applications.  As thunderbird uses a tree table for it's message list anyway this behavior would be quite natural for the user.  
Thanks Mike.

Here is a screen shot for a thread starter message showing states EXPANDABLE and EXPANDED, as well as an action named Collapse. This is already there in tbird build version 3 alpha 1 (20070208)

For collapsed threads, the state will include EXPANDABLE, and have an action named Expand.

This was tested with messages viewed as threaded.

I think I can add one nicety here which would be to have the threaded cell for the thread starter message to have a nicer accessible name. Perhaps "conversation"?  Or "thread starter"?

What do you guys think?
Marking 368351 as a dependency as the work related to comment #6 will be done there. Once that work is complete, this bug shouldn't be a problem/issue anymore.
Depends on: 368351
My system and Firefox are crashing frequently today, so I'll try to make this brief before I get another crash.  My thought is that a threaded message list is just basically a hierarchical structure that we run into all the time in GTK:

1) In a table that acts as a tree, the parent nodes should support the EXPANDABLE state, and it should be present regardless if the parent is expanded or collapsed.  Thus, the thread starter supports EXPANDABLE.

2) Each child should support the NODE_CHILD_OF relation that points back to the parent.  Thus, each message has a NODE_CHILD_OF relation that points back to the message starter (or however fine grained you want to do the threading).

Given these two things, a screen reader should be able to infer and present the hierarchical form of a threaded message list in the same way it presents any other tree-like structure.

Thanks Will,

Thanks very much for getting your comment out. I have another question :)

As you know, each row in a message list represents a message, and each row has many cells. One cell is the cell belonging to the thread status column. The child-of relationship appears to be already implemented for this cell only. Is this enough, to satisfy #2 or would you like the relation to spread across every cell in the row?

FWIW it seems to be that the thread status column is the first column.
Good question, and I'm not sure I know the answer.  When I use at-poke to look at the "Tree Store" demo under the "Tree View" node in gtk-demo, it looks like they apply it to just the first cell.  But then again, the accessible hierarchy of it looks whacked enough that I cannot seem to see all the cells in the table to be sure.  I'm trying to recall another place where we come across a similar beast, but my mind is not helping me.

From a user standpoint, we want to be able to present the column header as the user arrows from column to column, and I think we want to be able to present any changes in node depth of the tree as they move up and down, regardless of which column the user happens to be in.  Our goals should be to support that model via the architecture.  :-)

For me, I think the simplest means to do that would be if the relation was spread across every cell in the row.  But, we can always script to discover the cell in a row that has a child-of relationship. I'd rather not have to do that if it is easier for you to change it at the source, though.  If you can do this, I'm guessing the question then becomes "which cell is the parent?"  I don't really know.  I'm not sure it makes a difference in Orca right now, since I think we just use child-of for a depth count.  
Hi Dave,
What do you mean by spreading relationship across all the cells in a row? If you mean, inserting the textual indication of the row's relation to other rows, then I think it may be impractical as AT will speak that information when reading across the row. If I understand the issue correctly, the first cell should be sufficient.
Hi Victor,
Thanks very much for your comment. I mean't simply exposing an accessible relation type (such as child-of) as opposed to actually inserting text. 

So for example the message subject would still have the regular subject text, but an AT could query that cell for any relationships and act as it sees fit (e.g. perhaps based on a key combination it could speak the author of the message pointed to by the child-of relation, i.e. the author for parent message in the thread). Right now the only relation (AFAIK) is for the thread cell and the other cells don't "know" about any child-of relation.
In that case, I am not sure if we need any relationships at all because it would be difficult to decide what criteria are to be used for establishing the connection between the cells. In my view, establishing relationships should be a responsibility of a particular assistive technology tool. It seems that we just need to make sure that the information is there.

I think we need to think more about what the user experience should be here.  

Victor has outlined a useful feature, which is to let you know if you are on a message that is a thread starter.  It seems as though we can accomplish this by looking for the EXPANDABLE state on some cell on the line for a message (I'm not sure we can be guaranteed it will always be the first cell since the columns can be reordered).  The EXPANDED and COLLAPSED states can tell us the state.

When arrowing up and down the messages in a thread, I think it may also be useful to let you know when you've entered or left a thread.  Some people (e.g., myself) will reword a subject line in a thread to make it more in line with the content of the message. I've seen some applications include the message with the reworded subject in the same thread as the messages with the original subject line.  I believe Thunderbird does this.  

As such, it becomes difficult for an assistive technology to determine just when a user has moved from one thread to another.  If we can keep the NODE_CHILD_OF relation for at least the thread status column, it could give the assistive technology information to allow it to better infer that the user has moved from one thread to another.



Will, thanks for your comments. I agree, this is an important issue to get right and I'm also just now noticing your comment #10 -- sorry about that! I haven't looked into implementation yet but the issue of what cell would be the parent is a  poser.

The issue of detecting navigation out of, into, and across threads seems important too. We'll definitely keep the NODE_CHILD_OF unless we replace it with something better. I suspect Thunderbird is getting this "for free" based on core xul accessibility. I've got some higher priority items, but hope to poke around and get into this soon.
Taking MS Outlook as an example...
1. If the thread is collapsed, the accessible value contains thread state and the word "conversation" or "thread" along with the subject of that particular thread. Also, it communicates how many items are in that particular thread, e.g. "3 (2 unread)".
2. If thread is expanded, accessible value contains the same information as before, but this time the word "closed" (or as in MS's case "groupped by"), is substituted with the word "expanded".
Once the user moves into thread with down arrow, the messages should read as usual.
As soon as the user crosses the boundaries of one thread and crosses into another, the new thread will be marked as per 1 and 2 above. This will indicate that the user is now in a new thread.
How does this sound?
Hi Victor. I think this sounds reasonable. Essentially the special case message (indicating a new thread) is the thread starter message and with arrowing navigation that should never be skipped over or missed (as it might be with pointer use).

Will, if I can do anything specific to help with that behavior (for orca or otherwise) on this side, let me know. With the patch that went in today the column cell name will be either "Expanded" or "Collapsed" for thread starter/root messages.
This model seems pretty good to me.  I just need to be clear on where we are right now to see if we have enough information from the AT-SPI perspective to implement the proposed user model.  

When presenting things to users, we definitely like to use what is in the interface (e.g., the "Expanded" or "Collapsed" strings).  So, that's good.

When doing internal logic, such as determining the number of items in a thread and how many are read/unread, however, we really prefer to key off of programmatic things such as role names, states, and relations.  The reason for this is that we want the logic to work under any number of locales and don't want it to be driven off of user interface strings.  

My understanding of where we are with respect to that is:

1) If we have a message starter thread, the 'thread' cell will:
     a) Provide an EXPANDABLE state.
     b) Provide the EXPANDED state if the thread is expanded.  
     c) Provide the COLLAPSED state if it is not expanded.  
     d) Provide a localized name that we can present to the user.

2) If we have a message that is not a message starter thread, but part of a thread, the 'thread' cell will provide a NODE_CHILD_OF relation that points to the 'thread' cell of the message start thread.

3) The depth of the message threads is no greater than 1 (i.e., all threads are visually represented as direct children of the starter thread -- Thunderbird won't do multiple nested levels of threading as we see in some web-based discussion archives).

4) The 'thread' cell for all messages that are not part of a thread will not provide any of the information from #1 and #2 above.

Is that correct?  If so, we probably could work with this as long as we have a guarantee about the order of the children of the tree table holding the cells in the message list.  Are they in row-major order and will the tree table be populated with all the cells (i.e., no manages-descendants type stuff)?

Hi Will,

I completely understand the preference to key off roles states and relations -- I've been there :).  That is all there, this is just a little "extra"... i.e. providing a localized name. Let me address the points for clarity sake (and I have a question at the end):

1. Correct.
2. Correct.
3. Not so -- at least in tbird v3. (See bug 109744)
4. correct.

For the row-major ordering and manages-descendants type stuff I'll have to dig a little and get back to you.

BTW tbird has an optional column that displays the total messages in the thread (this cell is empty for non thread starter messages). There is a button thingy as the rightmost column header cell (above the scrollbar) which lets you turn columns on and off. I don't know if that button is keyboard accessible or has alternate GUI (can't find any). Will (or anyone) do you happen to know?  Might need to file another bug.
Hi David:

I think the little "extra" is nice because it means we don't have to generate it.  :-)  Thanks!  The similar stuff for the image-only column headers is also equally useful.

Given that the depth of the messages can be greater than one, I think we probably still could work with it as long as the thread cell for each submessage has a child-of relationship to the appropriate parent (i.e., the parent of the sub-thread).

The reason the row-major ordering stuff is useful is that it might us a way to more quickly run through the children of the table, searching for the thread cells in a logical and linear fashion.  However, as long as the accessible table API works (in particular: getAccessibleAt(row, column)), we may not need to rely on row-major ordering.  See bug 360184 (https://bugzilla.mozilla.org/show_bug.cgi?id=360184) for why I might be concerned about this, though.

The extra columns are actually pretty interesting.  I think they might be something we could use rather than requiring extra logic in Orca to do the same thing.  I have no clue how to use the keyboard to enable them, though.  :-(

Thanks Will, I've opened bug 370437 regarding access to the column picker.

Thanks everyone for the discussion! Interesting bug.

Closing this as fixed via dependancy work on 368351. Please reopen in the future if necessary. (We'll need to keep tabs on the related work but I've got many blips on my radar ATM)
Status: ASSIGNED → RESOLVED
Closed: 17 years ago
Resolution: --- → FIXED
Blocks: orca
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: