532492 - Expose read/write access for left and right channel sound spectrum data in the <audio> tag.

Status: RESOLVED DUPLICATE of bug 490705

(Core :: Audio/Video, enhancement)

Description

[Thomas Saunders (:thomassaunders )]

User-Agent:       Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1.5) Gecko/20091102 Firefox/3.5.5
Build Identifier: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1.5) 

The bug ( more accurately, the missing feature ) is that the Firefox browser does not offer the ability to analyze the spectrum data contained in an audio tag.

I currently have this ability to analyze sound data with Flash, but I don't want to use flash.  This holds me back from doing the work that I love to do in Javascript and HTML5, and until this feature is implemented I'll have to use flash.

Reproducible: Always

Steps to Reproduce:
1. No Audio API
2. 
3.
Actual Results:  
No Audio API

Expected Results:  
Audio API

Proposed Javascript API:

A proposed API could probably work similar to Actionscript 3's native SoundMixer API ( more information here http://livedocs.adobe.com/flash/9.0/ActionScriptLangRefV3/flash/media/SoundMixer.html )

The main method of the SoundMixer is computeSpectrum(), which returns a (byte) array of the values of both the left and right channels, with or without a Fourier Transform.  

In most uses, this method is called every frame ( 30 fps ) so performance is a consideration to keep in mind.  Usually this data is enough to create an engaging sound and visual experience.  

Another part of the API would be the ability to programmatically set the values of the sound channels, or manipulate them.  This would enable developers not only to remix other sounds on-the-fly, but it also could have accessibility implications, such as JavaScript text-to-speak technology.

Comment 1

[David Humphrey (:humph)]

NOTE: I am finding that the demo (http://modern-carpentry.com/workshop/html5/jsmusic/) only works if I have no tabs with other audio, even a paused flash video.  Very cool demo!

Comment 2

[Alistair MacDonald]

We would also need a way to change the buffer-size at which we captured the audio data. for slower machines.

Which leads us to a deeper problem...

I really think the whole way Timers work in the UA right now is TERRIBLE! And will need fixing sooner rather than later. John Resig had an interesting post about this:
http://ejohn.org/blog/accuracy-of-javascript-time/
http://ejohn.org/blog/how-javascript-timers-work/

JavaScript intervals are simply unreliable, so attempting to play multimedia or control user interaction through the passage of time, will remain a case of throwing mud at the wall -to see what sticks.

You know when you notice it most? Painfully... you notice is most with user-interface devices. *USER INTERACTION* That's what suffers the most, especially when you start seeing the delays between your touch-screen device and your canvas output. YUK!!

Why is the browser blocked by movements of the mouse in Linux? For example, this demonstration shows you that moving your mouse can stall your JavaScript interval (try that on your laptop): http://hyper-metrix.com/git/canvas-lab/performance/stall-sensor.html

Everyone knows that Canvas stuff looks great in Chrome, it renders quickly, appearing as if it can drop frames (a foundational function of any high-end visual software ). Should we start advising people that our next generation of web apps. are "best viewed in Chrome"? Is that really what it's all of Mozilla's work is going to come down to? Bad timers in JavaScript.

HTML-Media in general is in danger of going off the rails due timers of it's ECMA controller (and probably of the whole C implementation), for Firefox and the rest of the Web have not needed to change things since the 80s (or perhaps since 2k).

Yet, people are starting to use this technology in professional studios. So why are browsers not going to employ the use of professional timer resolutions that are used in software like Propellerheads Reason, without-which... HTML is going feel behind, still.

I don't want to start a talks about future contracts with prospective clients saying... "here are you limitations..."

Maybe the web is destined to be behind because it's a giant messy open source project with everyone pulling in the opposite direction, but that's just a sad excuse.

There really is no good reason for the merge of Desktop and Web to be so unbalanced. And to be so late in the game with HTML5 at "last-calls", it feels like the development community have become content with the Web still being "less than professional" in terms of multimedia.

Are we really going to put this stuff off until HTML6?

I hope not.

Al

Comment 3

[cajbir (:cajbir)]

Regarding programmatic setting of the sound data, see bug 490705.

Comment 4

[David Humphrey (:humph)]

Based on comment 3, let's move the discussion over to the other bug, but keep this going.