92110 - Allow simple audio format (wav, aiff) sound files to be played internally

Status: RESOLVED FIXED

(Core :: Audio/Video, enhancement)

Description

[Steve Chapel]

When I download an audio file of type audio/aiff or audio/wav, I'm able to open
the audio file with a helper app and play the sound. However, this method of
playing the sound is quite distracting because the File Download dialog box
flashes momentarily, then the window of the helper app comes to the front.

I would prefer an option to have Mozilla play the audio file internally so that
the only result of clicking on a link to an AIFF or WAV audio file would be that
the audio plays. This option would be a welcome relief to the users of a web
tool I developed that allows users to review audio and graphics files.

Comment 1

[Boris Zbarsky [:bzbarsky]]

Over to XP miscellany for starters

Comment 2

[r0polach]

It seems to be related to (more general) 92073

Comment 3

[Andrew Thompson]

Setting depends on rather than duplicate as I don't know for sure that Mozilla
can handle AIF and WAV internally. In fact, in general one would need to know
for each format whether Mozilla can do it, as 4.X (bugilly) did. What about AU
for example?

Comment 4

[Michael Hearn]

I'm sure Mozilla can play sound using the nsSound component, I'd have to check
out what formats it supports though. Anyway, that's how the mail component does
the New Mail sound for instance.

thanks -mike

Comment 5

[timeless]

.

Comment 6

[Blake Ross]

Reassigning obsolete bugs to their respective Seamonkey owners (i.e. nobody). 
If you want this fixed for Firefox, change the Product and Component accordingly
and reassign back to me.

Comment 7

[Steve Chapel]

Attachment: testcase

Comment 8

[Steve Chapel]

See http://www.phon.ucl.ac.uk/home/mark/audio/play.htm (Example #1) for a
description of this technique for playing a sound.

Comment 9

[Elmar Ludwig]

*** Bug 293410 has been marked as a duplicate of this bug. ***

Comment 10

[Hixie (not reading bugmail)]

See:
   http://whatwg.org/specs/web-apps/current-work/#sound

Comment 11

[Darin Fisher]

Mozilla Application Suite must not be the right product for this bug.

Over to core...

Comment 12

[kcoleman]

I am working on a product that will have a lot of usage and could really take
advantage of the change proposed here. Would be great to see this in 1.1.

Comment 13

[Robin Monks]

*** Bug 113560 has been marked as a duplicate of this bug. ***

Comment 14

[Mike Connor [:mconnor]]

not on the radar for 1.8, unless a solid and low-risk patch appears

Comment 15

[Peter Kasting]

Taking for preliminary investigation.

Comment 16

[jonathan chetwynd]

Peter, 

should you choose to take this mission on, would you please consider SVG.
I've been asking for sound for almost a decade :-)

if you want a test page http://www.peepo.co.uk isn't quite right, and I'll be happy to provide test cases

cheers

Comment 17

[Peter Kasting]

My knowledge of SVG is basically 0.  I think the best course, if I can do something on this bug, would be to get an initial patch that fixes the HTML cases, then see what more would need to be done for SVG at that point.

Unless I'm misunderstanding your request.

Comment 18

[Mike Shaver (:shaver -- probably not reading bugmail closely)]

The spec anchor has changed since Hixie originally linked it:

http://whatwg.org/specs/web-apps/current-work/#scs-sound

Comment 19

[jonathan chetwynd]

re #16 https://bugzilla.mozilla.org/show_bug.cgi?id=334920

Comment 20

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

If this becomes about implementing the WHATWG Audio object (and I think it should be -- this would be an awesome thing to have), the bug description should probably morph into that.  I had some comments regarding Audio on the whatwg mailing list -- see the thread starting at http://listserver.dreamhost.com/pipermail/whatwg-whatwg.org/2006-April/006260.html .

I do think that the whatwg spec needs some more input here before being implemented; I know the Opera folks have gone ahead and implemented what Hixie specced, but I think it's premature.  Some of the questions got raised in the thread linked above.

I would ignore SVG for now (audio has no business being in a 2d vector graphics format, grumble); whatever internal interfaces end up being created to handle audio (some more advanced version of nsISound) can be reused for whatever svg wants to expose.

Comment 21

[Steve Chapel]

(In reply to comment #20)
> If this becomes about implementing the WHATWG Audio object...

Could you file a new bug report about that, please? I think the original request is just as valid as ever.

Comment 22

[Peter Kasting]

(1) Let's definitely leave the WHATWG Audio stuff to a separate bug.  It's already unclear to me what this bug is really about, I don't want to confuse the issue further.
(2) So, speaking of: what _IS_ this bug about?  I have some reluctance to do what the original requester seems to want, because we eliminate the ability for helper apps or plugins to handle audio at all -- and if we're going to do that, our in-browser implementation better be quality, i.e. have a set of transport controls etc.  As far as I can tell, IE doesn't do this -- it lets WMP (or another plugin) handle audio.  And frankly, for clicked-on audio links, that seems sufficient to me.

So what we're left with is things like <bgsound>, <object>, and <embed>.  Perhaps we should implement support for bgsound natively -- that seems doable and I don't think it will interfere with anything or require us to draw play controls.  As for <object> and <embed>, IE seems to be passing those off to plugins, why shouldn't we?

The point I'm trying to make is that taking over audio playback from cases where plugins currently support it is both risky and not as simple to get right as it sounds.  I don't see much upside there.

I'm open to others telling me why I'm stupid and wrong.  Until then, I'm inclined to consider this bug a request to implement <bgsound>, and I believe we should implement the WHATWG Audio spec (or a future version), tracked under a separate bug.

Comment 23

[Steve Chapel]

(In reply to comment #22)
> I have some reluctance to do
> what the original requester seems to want, because we eliminate the ability
> for helper apps or plugins to handle audio at all...

No, as the original reporter, eliminating ability is not what this bug is about at all. Currently, when the browser plays an audio file, you *must* pass that functionality off to a helper app. This bug requests that *in addition* to that ability, you also have the option of having the browser play the sound itself. Having a helper app come up every time you click on an audio link can be very annoying. Ideally, every way of playing a sound described at <http://www.phon.ucl.ac.uk/home/mark/audio/play.htm> except those that use a plugin should be able to be handled by the browser itself, at least for the simplest audio formats such as wav and aiff and possibly au.

> consider this bug a request to implement <bgsound>

That's not what this bug is about either. Requests to support <bgsound> have been correctly marked WONTFIX.

Comment 24

[Peter Kasting]

First, on IE, every one of those methods uses a plugin, as far as I can tell.  IE does not seem to play .WAV or .AIFF or anything else internally in the case where you click a link to such a file, and I still don't understand why we should either.  What's wrong with letting the user's plugin kick off to handle it?  --Because I don't want to be in the business of adding a full-fledged audio player with transport controls to the core, and that's what you'd need to do to be able to play such a link internally properly, as far as I can tell.

Second, from what I can tell on bug 92073, we currently mandate that internally-handlable types get handled internally, and don't allow them to be passed to plugins.  Hence my comments about eliminating ability.  To do what you want, 92073 must be fixed first, as far as I can tell.  Yes?

So, if we implement 92073, and then implement this, we gain the ability for users to play linked .wav files internally.

The effort to fix each of these properly (UI, specifically) versus the benefit over letting Quicktime or some other plugin handle the linked .wav file like it already will on every other, more common type of audio playback, doesn't seem compelling to me.  It ESPECIALLY doesn't seem compelling if we implement the WHATWG Audio spec so there IS a way for pages to play sounds when they need to -- and implementing that is going to be less work than implementing both these other bugs.

Finally, in passing, why do you say it's "correct" for requests to implement <bgsound> to have been marked WONTFIX?  If there are particular bug numbers you're thinking of feel free to link me.

Comment 25

[jdperlow]

(In reply to comment #24)
> First, on IE, every one of those methods uses a plugin, as far as I can tell. 
> IE does not seem to play .WAV or .AIFF or anything else internally in the case
> where you click a link to such a file, and I still don't understand why we
> should either.  What's wrong with letting the user's plugin kick off to handle
> it?  --Because I don't want to be in the business of adding a full-fledged
> audio player with transport controls to the core, and that's what you'd need to
> do to be able to play such a link internally properly, as far as I can tell.
> 

When we implemented Gmail chat, we experimented with adding audio notifications, but the pluggin model in Firefox caused us endless problems. Each pluggin had a different implementation of the API and different UI. For example, Real Player and sometimes QuickTime would often popup external UI and make the user accept some licensing agreement or make them download some other component. This inconsistent experience resulted in us having to abandon this approach to playing sound. Developers need a consistent and deterministic API. We couldn't add this feature since it degraded the Gmail experience. In IE, we actually had a much more consistent experience for whatever reason.

Comment 26

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

(In reply to comment #25)
> This inconsistent experience resulted in us having to abandon this
> approach to playing sound. Developers need a consistent and deterministic API.

I think this is exactly one of the use cases that the WHATWG audio spec tries to solve; filed bug 336164 regarding implementing it.

Comment 27

[Peter Kasting]

(In reply to comment #25)
> This inconsistent experience resulted in us having to abandon this
> approach to playing sound. Developers need a consistent and deterministic API.

I agree with vlad; for this case the WHATWG stuff is the way to go.

I wonder if we control any API or UI stuff like what you described, where we could fix some of those issues (though I doubt there's anything we can do here).

As for whatever else should be done, I suspect we could (and perhaps should) implement it in terms of the work done for the WHATWG stuff.  I'm going to make this bug depend on that one.

Comment 28

[jonathan chetwynd]

It might be worth contemplating that MS has recently lost a ~$600M court case regarding the way that plugins are handled: see Eolas Technologies patents.

Comment 29

[jdperlow]

Just to clarify my previous comment. I didn't mean to suggest an implementation - I was just trying to explain a pain point we had and why the current implementation wasn't working for us. I'm hoping the future design will be better.

Comment 30

[Mike Shaver (:shaver -- probably not reading bugmail closely)]

jdpederlow: can you elaborate (here or in a wiki page) about the approaches you tried, and the differences in behaviour?  It'd be good to capture the practical use-cases here to make sure that they work well with whatever combination of internal-play and <audio> we end up implementing.

(Welcome, pkasting!)

Comment 31

[Steve Chapel]

(In reply to comment #24)
> What's wrong with letting the user's plugin kick off to handle
> it?  --Because I don't want to be in the business of adding a full-fledged
> audio player with transport controls to the core, and that's what you'd need to
> do to be able to play such a link internally properly, as far as I can tell.

Many plugins have a horrible UI for playing quick audio snippets, which is usually what aiff and wav files are used for. I've seen a user with dozens of helper app windows open, because each click on an audio file link opened up a new window. That horror is what led me to file this bug report, I believe. Opera already plays audio internally, and there should already be an audio player built in to Firefox and SeaMonkey to play UI sounds such as when find-as-you-type fails to find the text. It seems like it would be easy to do, and would benefit the user so much that at least one other browser has implemented this feature already.

> Second, from what I can tell on bug 92073, we currently mandate that
> internally-handlable types get handled internally, and don't allow them to be
> passed to plugins.  Hence my comments about eliminating ability.  To do what
> you want, 92073 must be fixed first, as far as I can tell.  Yes?

It looks that way, yes. In a way, this bug is merely an extension of bug 92073 specifically for the ability to play audio internally. My guess is that bug not being fixed yet is the main reason this bug is still open.

> Finally, in passing, why do you say it's "correct" for requests to implement
> <bgsound> to have been marked WONTFIX?  If there are particular bug numbers
> you're thinking of feel free to link me.

Sorry, I should have said INVALID. I was thinking of bug 226635. Surely a bug requesting <bgsound> functionality would be quickly closed as a dupe of that bug, whether this bug morphs into that request or we open a new bug report.

Comment 32

[Peter Kasting]

(In reply to comment #31)
> Opera already plays audio internally, and there should already be an audio
> player built in to Firefox and SeaMonkey to play UI sounds such as when
> find-as-you-type fails to find the text. It seems like it would be easy to do,
> and would benefit the user so much that at least one other browser has
> implemented this feature already.

Actually playing the audio, without any user-visible controls or any way to pass the file off to a plugin, wouldn't be too bad.  The audio interface you're thinking of is NSISound, which is indeed what we use for the find bar "fart noise" and similar audio events.  I'm not sure this would be sufficient to implement what you want, but certainly an implementation of bug 336164 would be.

No, what I'm much more concerned about is the effort to "do this right" such that it's not an annoyance to users who want transport controls or the ability to let their existing plugins handle things.  It's certainly doable, but it seems low-priority and high effort given the amount of end-user payoff.  If both dependencies of this bug are fixed, this bug should be possible to implement; creating and hooking up the transport controls will be the most difficult thing.  Maybe we could just punt on that aspect to some degree and provide only play and stop buttons.

> > Finally, in passing, why do you say it's "correct" for requests to implement
> > <bgsound> to have been marked WONTFIX?  If there are particular bug numbers
> > you're thinking of feel free to link me.
> 
> Sorry, I should have said INVALID. I was thinking of bug 226635. Surely a bug
> requesting <bgsound> functionality would be quickly closed as a dupe of that
> bug, whether this bug morphs into that request or we open a new bug report.

All right, thanks very much for the bug number.

Comment 33

[Justin Kerk]

I don't know if WhatWG is necessarily the answer to this because there's piles upon piles of legacy pages that use audio that won't be using whatever interface WhatWG comes up with because it doesn't exist yet. And there's also plenty of sites with regular old non-embedded links to .wav files.

The answer of "just use a plugin" would be more satisfactory if the plugins available weren't so terrible as far as bloat, etc. QuickTime seems to be the de facto standard, but QuickTime has issues like long gaps when playing looping sounds before the looping starts working properly. Plus, most people don't have any Firefox plugin at all set up to handle audio so whenever I send someone a link to a page with audio embedded I get a chorus of "it says additional plugins are required, I can't be arsed, why doesn't it just work like in IE". If there were a small, free, non-full-of-commercial-crap audio playing plugin that I could just point people to, that would take care of some of the problems (vlc has one but it's hard to install and has no control UI at all) but of course I'd still have to evangelize it.

Netscape used to come bundled with a small audio playing plugin called LiveAudio, a straight resurrection of that with MP3 thrown in would be perfect IMO.

Comment 34

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

There's a bug on file that I can't find right now about hooking up internal handlers for some content types; if that was fixed, this could be implemented as a simple audio-playing plugin that had some native components for the necessary interfaces.

Comment 35

[Peter Kasting]

Thought: perhaps there is a way to better autodetect and integrate with WMP on initial install so we can work more like IE on Windows (which uses plugins for those sorts of things, but integrates nicely out-of-the-box so it provides a good user experience).

Comment #33 and comment #34 about shipping a plugin to do audio might make sense, and would address the problems people have with <object> and <embed>, as well as handling the "link to a .wav" case that this bug is currently about.

Vlad, can you clarify more about "internal handlers" and "plugin" and such?  Maybe I'm getting my terminology mixed up, but I thought "internal handler" was something like what we do for JPEGs where we just decode it internally.  What bugs would therefore need to be fixed to make this work?  I don't understand how we need an "internal handler" if we are shipping a "plugin".  Also would this mean bug 92073 doesn't need fixing if we go this route?

Comment 36

[Steve Chapel]

Does this bug also depend on bug 258012, which requests that Firefox be allowed to be picked as the helper application that views the file?

Comment 37

[Peter Kasting]

I don't think it does; although I suppose that would be an alternate way of solving the problem in bug 92073.  I don't see a lot of use cases where it would be good to have for this bug, though, because I think you either want .wav links to open in your helper app or in Firefox, and if you want them to open in Firefox, having to explicitly pick that every time gets old fast.

Comment 38

[Mike Shaver (:shaver -- probably not reading bugmail closely)]

In the download/helper/etc. dialog, the

  [X] Do this every time

would help here, but I'm not entirely convinced that we want to do this for all simple audio format files.  For a long WAV, not having pause/etc. controls might well be quite frustrating.

It'd be interesting to get jpederlow's notes on what IE does, at the least, to start to think through those cases.  (This bug should probably just be about the functionality for internal play, and the UI twiddles to happen in a Firefox-product bug.)

Comment 39

[Peter Kasting]

I'm not going to get to this one anytime in the foreseeable future; dumping back to nobody.

Comment 40

[Robert O'Callahan (:roc) (email my personal email if necessary)]

Fixed, now that HTML5 <audio> is implemented and supports WAV (and Ogg Vorbis).

Comment 41

[Eric Shepherd [:sheppy]]

I'm not convinced this is fixed until <audio> also supports AIFF.