1161324 - Aural feedback with VoiceOver needed when loading a webpage

Status: RESOLVED FIXED

(Firefox for iOS :: Browser, defect)

Description

[Boris Dušek]

When a user triggers loading of a webpage, VoiceOver provides no feedback (even if VoiceOver cursor is put onto the progress bar). Safari does provide (non-spoken) aural feedback on page loading.

A solution is to do what is needed "manually" - e.g. have an NSTimer which during webpage loading periodically triggers playing of a sound. This would be of course restricted only to happen when VoiceOver is running (for which there is an API).

See also http://lists.apple.com/archives/accessibility-dev/2015/Apr/msg00010.html (shamefully the answer of Apple developer there is clobbered by the mailman system, but the answer basically was "no automatic VoiceOver support now, you have to do it manually e.g. as you suggested").

Comment 1

[Boris Dušek]

Attachment: Pull Request

Comment 2

[Boris Dušek]

Attachment: Recording - example of the aural progress bar

Marco, I am attaching a recorded example of the aural progress bar in action. Would you mind listening to it and give your impression of it? Note that different webpages have different loading progress patterns, this recorded one is well "presentable" as it has many "progress" ticks (the ones with rising pitch). Some have less, some have many in the end with nearly the same pitch.

Comment 3

[Boris Dušek]

One more note - just like I am not graphic designer but can try my best, I also am no sound designer but tried my best. :-) The sounds are a little robotic, just a plain sine wave, no wider frequency spectrum, no attack-release (i.e. not "cool" but works). Pitches, volumes, durations, periods and stereo pans of both sounds (ticks) are easily configurable in the code (in the accompanying UX struct).

Comment 4

[Marco Zehe (:MarcoZ)]

Comment on attachment 8604404 [details]
Recording - example of the aural progress bar

Thanks for doing this! Very interesting approach, and it sort of mimics the behavior the Windows screen reader NVDA has when indicating progress bars. It has a single ascending beep similar to yours, and no low frequency "heartbeat" tone. But the concept is the same, and it's one of my favorite NVDA features. :) So yes, I am very much for implementing it in this way.

perhaps we could even look at the exact code the frequencies are being calculated by. NVDA is open-source.

Comment 5

[Boris Dušek]

Comment on attachment 8604401 [details]
Pull Request

Oops, wrong URL. Will redo.

Comment 6

[Boris Dušek]

Attachment: Pull Request

This is the correct URL for the pull request.

Comment 7

[Boris Dušek]

Hello Marco,

(In reply to Marco Zehe (:MarcoZ) from comment #4)
> perhaps we could even look at the exact code the frequencies are being
> calculated by. NVDA is open-source.

I did not mention it here, but did in the pull request, so I will do it here and add to it, it provides some insight/information about how the aural progress bar is designed:

The aural progress bar was inspired in part by Safari's "ticking" with VoiceOver (tickPlayer in code, i.e. the "heartbeat"), and in part by NVDA's "rising" progress (progressPlayer). We do both approaches as they serve different purposes:

- the periodic fixed-pitch ticking (i.e. what you called heartbeat, great term) communicates that the page is being loaded, that something is in progress at all, that the progress "lives", is active, even when no new data from webpage is coming as some point (i.e. when the progress temporarily stalls for a few seconds)
- the non-periodic, rising-pitch progress ticking communicates current value of progress and that the progress is actually progressing; it sounds exactly when actual data comes from webpage, i.e. when the progress increases

Progress 0% has the same pitch as the periodic ticking (heartbeat), progress 100% is one octave higher. This makes one tell by the difference of pitches of the two sounds (heartbeat and progress) how far the loading progress is. I.e. the heartbeat provides pitchwise a "reference pitch", the 0% progress. The fact that you want the progress to get to 100% so that you can work with the page here means that you want the progress tick to get to being exactly one octave higher than the heartbeat. I.e. I made some (hopefully sensible) explicit choice for the frequencies for our current situation, so I don't think we need to mimic NVDA frequency-wise, i.e. no need to look at NVDA's source code.

The fixed-pitch ticking's pitch is chosen to make a "tune" (cadence) with VoiceOver's "screen changed" sound. The "screen changed" is two tones, I made the heartbeat sound to be pitchwise exactly one tone below the first tone of the "screen changed" sound. This was to create a cadence of heartbeat being subdominant, first tone of "screen changed" being dominant and the second (last) tone of "screen changed" to be tonic.

There are three things I think could be improved (but that is maybe for some later version, more important work now):

1. The tuning relative of the heartbeat to VoiceOver's "screen changed" first tone would work best without the progress ticking as that is by definition out of tune, violates the harmony. I tried to remedy it at least to some extent by making the progress tick less loud, this also makes it easier to listen as the heartbeat is volume-wise the prominent sound, and the progress ticks are less loud additions.

2. When a webpage is long loading without any new progress, the progress ticks are not playing so the user could forget after some time the relative pitch difference between heartbeat and progress, i.e. how far the progress is.

3. The sounds are not properly designed, just plain sine wave of required frequency. Maybe add some "spirit" to them by attack-sustain-release (I tried a little bit and it made the sounds too "diluted"), or record some real sounds.

I also thought about an enhancement that I could make the progress tick change not only pitchwise, but also in stereo pan. Currently heartbeat is in the stereo slightly to the left, progress ticks slightly to the right. The progress tick could then be not fixed in the stereo pan, but also travel, 0% progress being same left as the heartbeat, 100% being same as it is currently (i.e. mirrored to the right from the heartbeat).

But these are all things that I think should not block this first version to ship.

Comment 8

[Brian Nicholson (:bnicholson)]

Comment on attachment 8604489 [details] [review]
Pull Request

Very neat! As usual, thanks for putting so much effort into this. I left a few (minor) comments in the PR.

Comment 9

[Boris Dušek]

Comment on attachment 8604489 [details] [review]
Pull Request

I rebased on latest master and addressed all your comments (except for the last one, for which I provided comment on GitHub) - please see the PR.

Comment 10

[Brian Nicholson (:bnicholson)]

Merged. Thanks again for the contribution!