Closed Bug 395484 Opened 17 years ago Closed 3 years ago

It is not possible to use Vista's speech recognition application to dictate into a text field on a webpage

Categories

(Core :: DOM: Editor, defect)

x86
All
defect
Not set
major

Tracking

()

RESOLVED INCOMPLETE

People

(Reporter: David, Unassigned)

References

Details

(Keywords: access)

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6

It is not possible to use Vista's speech recognition application to dictate into a text field on a webpage.  Attempting to do so causes the dictation correction panel to open and display the dictated text.  If the appropriate item is then selected text may be transferred into the webpage field.

There is no problem dictating directly into a text field in Internet Explorer.

Reproducible: Always

Steps to Reproduce:
1.Open any web page with a text field 
2.Place cursor in field 
3.Dictate 
Actual Results:  
Dictation correction panel opens displaying one or more options 

Expected Results:  
Dictated words should appear in webpage text field
Component: Form Manager → Editor
Product: Firefox → Core
QA Contact: form.manager → editor
Vista is shortly to become the only Windows operating system supported by Microsoft.  It is surely a serious shortcoming of both applications if neither Firefox or Thunderbird will work correctly under this 0S.  Does Mozilla have any plans to address the problem?
Severity: minor → major
Keywords: access
Vista speech technologies use MSAA so it may be accessibility bug (see http://msdn.microsoft.com/en-us/library/bb875962.aspx):

Windows Vista automatically provides basic speech capabilities to any application that is designed to work with two Windows accessibility technologies: Microsoft Active Accessibility (MSAA) and Microsoft Windows UI Automation (WUIA). At runtime, when speech is used to open or switch to an application, the speech engine queries that application to determine which accessibility features it supports, then works through those.
From this I presume that neither Firefox nor Thunderbird will work with MSAA or WUIA?  Are there any plans to address this shortcoming?

(In reply to comment #5)
> From this I presume that neither Firefox nor Thunderbird will work with MSAA or
> WUIA?  Are there any plans to address this shortcoming?

Firefox has been supporting MSAA since version 1.5. The question is what calls Vista's speech recognition makes that Firefox doesn't react to, or reacts improperly to.
I would assume we should implement IAccessibleEx interface to add UI Automation. I'll try to look more deeply. Marco, can you confirm the bug since I haven't Vista?
I am not sure I can confirm the bug. I am not certain I can actually use speech recognition since its controls may be visual (training it etc.).
Vista speech recognition will work, albeit with poor recognition accuracy, "out of the box", without any training.

I don't know how useful this is but I have just tried dictating into Open Office and a text editor called EditPad, (http://www.editpadpro.com/).  Neither will work.

Unsurprisingly it is possible to dictate into any Microsoft application which will accept text.
(In reply to comment #9)
> I don't know how useful this is but I have just tried dictating into Open
> Office and a text editor called EditPad, (http://www.editpadpro.com/).  Neither
> will work.

That doesn't surprise me: OpenOffice.org doesn't implement MSAA at all yet, and EditPad Pro uses a custom widget as its text editor that implements all the standard APIs for a Windows Edit control, but the speech recognition software doesn't know that a window with class TJgSoftEditorControl is behaving like a normal edit.

> Unsurprisingly it is possible to dictate into any Microsoft application which
> will accept text.

Right. This probably uses the text edit API which even NotePad implements, or the RichText API that WordPad implements, and probably has special rules for MS Word.

As for Firefox, even though we expose textboxes as MSAA type Edit, we probably don't implement something else the speech recognition software needs, but which isn't clearly spelled out. Screen readers for the blind have no problem identifying the textboxes, either in HTML or XUL, as such, so I would assume the implementation is generally correct.
If I get correctly from articles http://msdn.microsoft.com/en-us/accessibility/cc307845.aspx then we have two choices a) implement UI automation or b) implement IAccessibleEx (some kind of partial implementation of UI automation), sure if we want to get firefox more accessible on windows vista (I guess speech is only one issue we meet).

Marco, iirc there was some discussion about UI Automation implementation on a11y.org but I don't remember details. Do you?
If the industry can come together than we might be able to merge IAccessibleEx with IAccessible2 somewhat, which would make this much easier. That's dependent on how things go at http://www.accessinteropalliance.org/
Status: UNCONFIRMED → NEW
Ever confirmed: true
BTW, like IAccessible2, IAccessibleEx is also built on top of MSAA. That means we support part of IAccessibleEx. I would have though it's enough for basic control, because for simple things like buttons, links, menus, that's all you really need.

We probably need to ask Microsoft why the basics aren't working.
Aaron, if you keep someone from Microsoft then can you write them a letter and cc me because I'm actually not sure IAccessibleEx will help here since it provides UA Automation partially only.
So, microsoft guys said we need TSF (Text Services Framework, http://msdn.microsoft.com/en-us/library/ms629032(VS.85).aspx). The irony is we have been asked to be mentors for TSF implementation in Gecko (http://groups.google.com/group/mozilla.dev.accessibility/browse_thread/thread/76b3fdb5a807a2f6/058b4ba3e0361978?lnk=gst&q=TSF#058b4ba3e0361978) in the google summer project bounds.
Depends on: 88831
So TSF is not going to make it into 3.1? When will I be able to use Speech Recognition with Fx? To me this is not an necessarily an accessibility feature. Wouldn't we all prefer to speak to our computers instead of having to type and use the mouse? Speech Recognition in Vista is vastly improved over XP. Other aspects of Speech Recognition work with Fx. However, the most useful part is being able to dictate to a text box in Fx for forums. Plus, I don't use Thunderbird but if I did I would want it to accept Speech Recognition like Outlook Express does.
I am the guy that first reported this bug over a year ago now - see the beginning of this thread.

I couldn't agree more with the previous posting.  Vista's speech recognition application is outstanding but it can't be used with Mozilla products.

How difficult is it to incorporate TSF?

Hopefully someone will find time to do it in the next 12 months…

David
From comment #80 in bug https://bugzilla.mozilla.org/show_bug.cgi?id=88831 TSF missed the feature freeze for Fx 3.1 "by a lot" and thus it cannot be pushed into 3.1. But it should land on the trunk in November.  So, if you like to use the nightlies then not too much longer to wait.  If you read the referenced bug you will see that it started in 2001 and just now is Fx about to get TSF. At one point, they were sure it would make it into 3.1 so it's coming soon.
it looks like we're looking at yet another year of waiting for this issue to be resolved.  A shame really, considering, as you mentioned above, just the vast amount of potential Vista's speech recognition has been showing since its inception.
Have all attempts to fix this bug been abandoned?  It has been present since Vista was launched and now Vista is about to this superseded by Windows 7.  The problem also exists in Windows 7 beta.  Maybe this will be fixed just in time for the launch of Windows 8?
David, Bug 88831, which introduced support for the Text Services Framework, was fixed about a month ago on Firefox trunk AKA 3.6a1pre. So if you're willing, you can test out how well Vista's speech recognition fairs with that development version.
I'd love to test the development version but I am only a user not a programmer and have no idea how to get the development version from the trunk even if I knew what the trunk was...

Would I be right in thinking that the same fix would fix the same bug which also exists in Thunderbird?  How could I get somebody to do something about this?

As I said previously the same problem exists with Windows 7 beta.  Will the fix that fixed the problem in Vista also work for Windows 7?  I'd be happy to try that as well if only I knew how to get the development version from the trunk.
Marco kindly e-mailed me instructions for downloading the development version.  This installed Minefield version 3.2a1pre.

Sadly I have to report that it is still not possible to dictate into text fields.  I tried several different websites with text fields all of which I could dictate into when they were opened in Internet Explorer.  None of them would accept dictation when opened in the development version of Firefox.
You have to enable TSF first by going to about:config and changing the value of intl.enable_tsf_support from false to true.
Yes Aiko is right, at least until bug 478029 is fixed (then it should be true by default).
Could somebody tell me how to find "about:config"?
Sure. Just type or copy & paste about:config into the address bar and press Return. Please read the warning before you click the button. Now, type intl.ena into the filter; intl.enable_tsf_support should be the only option still available, double click it to change its value.

I'm not sure whether this option is just checked once at startup so you'd better restart the browser at this point.

Please make a backup of your current profile or create a new one before you start any lengthy experiments, this combination of a nightly build and a somewhat experimental feature is not likely to be very stable.
firefox-3.6a1pre.en-US.win32.installer.exe allows text to be dictated from Vista's speech recognition application into a text field such as the "Additional Comments" field on this thread but only if dictation is continuous.

To see this effect dictate a short phrase into the field.  The phrase is printed.

Pause for a few seconds.

Dictate again.  The new dictation is printed.

Pause for a few seconds

Dictate again.  Nothing happens

The field has apparently lost the focus of the speech recognition application and it can only be regained by manually deleting all the text in the field.

The effect can also be seen by pasting some text into an empty field and then trying to dictate into it.

Typing into an empty field and then dictating into it allows this initial dictation to be printed but the field then loses focus.
See bug #556952:  not even the address bar works with Speech Recognition.

Steps to reproduce:

1. Open Speech Recognition
2. Open Firefox.
3. Say "go to address."  Doesn't work.
4. Say "press control-L."
5. Dictate a URL.
Jonathan, I originally reported this bug  back in the days of Vista but the problems are exactly the same under Windows 7.

As you can see from this thread fixing the bug depends upon the implementation of TSF.  This must be extremely difficult because it has not proved possible for nearly three years.
Nearly a year has elapsed since my last comment and still nothing has been done and apparently nobody is working on it.

Vista came and went and it was never possible to dictate into either Firefox or Thunderbird .

Windows 7 has been around now for a couple of years and still it isn't possible to dictate into either Firefox or Thunderbird.

If no date can be put upon the implementation of TSF would it be possible to put a date on when work might start on attempting the implementation?
If you enable TSF does this work as expected? ('intl.enable_tsf_support' = true)
No.  Please see Comment 30.
(In reply to comment #38)
> No.  Please see Comment 30.

Ok, so with TSF enabled, it works, except for the focus problem you mentioned. The focus problem should be filed as a separate bug and marked as blocking bug 478029. This bug can then be resolved as a dupe of bug 478029. (Or alternatively, morph this into the bug about the focus problem and mark it blocking bug 478029.)

I would suggest not mentioning "Vista" in bug titles, a better description would be something along the lines of "focus problem in TSF support with Windows speech recognition".
[as far as your original question goes about default support - I'd suggest asking 
Masayuki in bug 478029 for a status update. I'd be curious about that myself, since tablet / touch input support is going to be *real* important with Win8.]
I am surprised that you can use Speech Recognition. When I switch it on I have problems when scrolling (stops often) and problems to enter text - has delays too.

This all under Win7 and every version of Firefox (3.6 to 4). No problems with other browsers.

You don't have these problems?
I didn't say I can use speech recognition in either Firefox or Thunderbird.  The problems I described under Comment 30, (which are essentially the same as you are experiencing, make it completely unusable.

As requested by Jim Mathies I have filed a separate bug report, 650833, on the loss of focus. (Incidentally I haven't figured out how to put a hyperlink on a bug reference).
OS: Windows Vista → All
I didn't say I can use speech recognition in either Firefox or Thunderbird.  The problems I described under Comment 30, (which are essentially the same as you are experiencing, make it completely unusable.

As requested by Jim Mathies I have filed a separate bug report, 650833, on the loss of focus. (Incidentally I haven't figured out how to put a hyperlink on a bug reference).
It is coming up to the 4th the anniversary of my filing of this bug report and I still cannot dictate into  Firefox text fields.  Even more importantly than this I can't dictate into Thunderbird.  Do you think TSF will ever be implementedin these application?
Vista is no longer supported in newest versions.  So, is this bug applicable to newer versions of Firefox?

(In reply to Jim Mathies [:jimm] from comment #40)
> [as far as your original question goes about default support - I'd suggest
> asking Masayuki in bug 478029 for a status update. I'd be curious about that
> myself, since tablet / touch input support is going to be *real* important
> with Win8.]

That was fixed in version 41.
Flags: needinfo?(masayuki)
Currently, TSF mode works really well with keyboard. However, at least I've never tested with other input devices. Even with keyboard, we struggle with a lot of compatibility issues like bug 1462257. So, I guess that there may be a lot of compatibility issues with each TIP with non-keyboard devices.

I'd like to suggest that we close this bug since this is filed before we enable TSF mode by default and multi-process mode by default, and there are a lot of too old comments. If somebody still has problem, they should file new bugs for *each* symptom with following information:
1. Version of Windows and Firefox.
2. TIP (Text Input Processor) information, e.g., name, version, etc.
3. Attach a minimum log file with launching Firefox with the following environment
  MOZ_LOG=nsTextStoreWidgets:5,sync
  MOZ_LOG_FILE=c:\fx-log.txt
   and do simplest steps to reproduce a symptom, then, close the Firefox.
   (Note that the log file include all text input, so, DO NOT type password or something your privacy.)
4. It works well whether on Edge and/or Chrome.
Flags: needinfo?(masayuki)

Based in comment 46, I'll mark this as Resolved > Incomplete since the last activity on this issue was 3 years ago and it might not be relevant anymore. If this is still reproducible on your end in the latest FF versions, feel free to file a new bug with the details requested in comment 46.
Thanks.

Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → INCOMPLETE
You need to log in before you can comment on or make changes to this bug.