963565 - [RTSP][V1.4] RTSP audio-only streaming cannot be played by video app

Status: VERIFIED FIXED

(Firefox OS Graveyard :: Gaia::Video, defect)

Description

[Ethan Tseng [:ethan]]

In FFOS v1.3, the video app can play an RTSP audio-only streaming, such as:
rtsp://m.streaming.imusica.com.br/vod/mp4:takemeout?=3529931==web.local=1262=0

However, in FFOS v1.4, when we click the same RTSP link, the video app would reject to play the audio and alert an error message similar as "The video format is unsupported".
The root cause is the fix of bug 938033 changes the video app's behavior. Now the video app would check "dom.player.videoWidth/videoHeight". If the value of either of them is zero, the app would treat the media as an audio and return an error.

"Shall the video app be allowed to play an audio-only streaming?" could be an open issue.
This bug is filed to indicate we have a problem of RTSP audio feature and need to resolve it in v1.4.

Comment 1

[John Hu [:johnhu][:johu][:醬糊小弟]]

Ethan,

It's nice to see this bug. We had met lots related bug before, including video track wrapped in a music container, an audio file wrapped in a video container (bug 938033).

In video app, we don't show the video file without "video track" because it is viewed as an audio file in normal case. We had introduced this kind of checking at bug 800235 since October 2012. But the view activity of video doesn't do the same check until the bug 938033. In the case of bug 938033, user may have a all black screen when audio file is played with view activity of video app because the video controls automatically fade out after 2 seconds. So, I feel the patch of bug 938033 is correct to sync the behavior between video app and view activity of video app.

We should ask UX, rob, for the correct behavior on this kind of case:
1. Do we view "audio" stream with video container in video app?
2. If yes, what should we display on the screen after video controls fade out?

Comment 2

[Jason Smith [:jsmith]]

Something that doesn't make sense to me is why we are even deciding to render RTSP streams in a media application in Gaia. We already have had one content provider complain about a past design decision when we rendered YouTube videos in the video app because the content provider wanted the videos to render always in their own app's context, not in the context of another app. This also doesn't align with the UX design of playback of media codecs when you visit a link to a media file in FxOS, FxDesktop, & FxAndroid - in those cases, we open a browser page that shows media controls to allow playback of the media file. Why can't we just follow the same approach we've done with media files for RTSP streams?

Comment 3

[Ethan Tseng [:ethan]]

Hi Jason, 
In my understanding, I suppose you are saying why don't we just render RTSP streaming in the builtin media player within the browser?
Actually we had already discussed the behaviors under different circumstances in bug 945603.
The current conclusion is: (Status means the implementation status)
+---------------------------|-----------------------------------------|--------+
| Case                      | Expected Behavior                       | Status |
|---------------------------|-----------------------------------------|--------|
| <video>                   | Display by the builtin media player     | Done   |
|---------------------------|-----------------------------------------|--------|
| <a href="rtsp://...">     | Launch video app                        | Done   |
|---------------------------|-----------------------------------------|--------|
| URL bar                   | Display by the builtin media player     | N/A    |
|---------------------------|-----------------------------------------|--------|
| <iframe src="rtsp://..."> | Display by the builtin media player     | N/A    |
+---------------------------|-----------------------------------------|--------+

We only launch a Gaia app for the case of HTML <a> tag. For other cases, we render the RTSP media within the browser.

Besides, as I know, on both FxDesktop and FxAndroid, visiting an RTSP link no matter from URL bar or <a> tag, the behavior is to launch an external application. They don't play RTSP streaming within the Firefox browser.

Comment 4

[Jason Smith [:jsmith]]

(In reply to Ethan Tseng [:ethan] from comment #3)
> Besides, as I know, on both FxDesktop and FxAndroid, visiting an RTSP link
> no matter from URL bar or <a> tag, the behavior is to launch an external
> application. They don't play RTSP streaming within the Firefox browser.

Well right, RTSP isn't even supported on FxAndroid & FxDesktop. However, the other media consumption points such as playback of media files renders those files in a built-in media player on a page solely for that media player.

Example of What I'm Talking About - http://mozilla.github.io/qa-testcase-data/Video/multi2.ogg

Comment 5

[Jason Smith [:jsmith]]

(In reply to Jason Smith [:jsmith] from comment #4)
> (In reply to Ethan Tseng [:ethan] from comment #3)
> > Besides, as I know, on both FxDesktop and FxAndroid, visiting an RTSP link
> > no matter from URL bar or <a> tag, the behavior is to launch an external
> > application. They don't play RTSP streaming within the Firefox browser.
> 
> Well right, RTSP isn't even supported on FxAndroid & FxDesktop. However, the
> other media consumption points such as playback of media files renders those
> files in a built-in media player on a page solely for that media player.
> 
> Example of What I'm Talking About -
> http://mozilla.github.io/qa-testcase-data/Video/multi2.ogg

The basis of point of then is arguing - why not open the built-in media player when clicking a <a href="rtsp://..."> link, since other media files do that already? That ensures parity with other media consumption points & keeps complexity to a minimum.

Comment 6

[John Hu [:johnhu][:johu][:醬糊小弟]]

IMHO, I agree with Jason's point. Why not to play with the built-in media player? Although they may still have a chance to play the video inside of the video element, like comment 3, I feel we need to define the term "built-in media player" is. If it is video app, we need to handle audio only file/streaming in open/view activity. If not, we should just use the video/audio element to play it, like normal browser.


For some people who don't know the YouTube thing:

In the case of YouTube, we use the mime type of YouTube for Android to trap the YouTube link and parse them inside the video app. With the protected content, like music video, we need to decode the secret signature to get the correct URL. That's definitely a hack on YouTube. And that's not allowed with the developer agreement from YouTube. During that time, we had faced a YouTube "secret signature change" event. That's very hard for us to keep track of the change of that. After few weeks later, YouTube rewrite their mobile web to have a inline video element when they found the device is Firefox OS.

Comment 7

[John Hu [:johnhu][:johu][:醬糊小弟]]

(In reply to John Hu [:johnhu][:johu][:醬糊小弟] from comment #6)
> Although they may still have a chance to play the video inside of
> the video element, like comment 3, I feel we need to define the term
> "built-in media player" is. If it is video app, we need to handle audio only
> file/streaming in open/view activity. If not, we should just use the
> video/audio element to play it, like normal browser.

Oh, sorry. I found that in comment 3, Ethan had mentioned that the built-in media play is a media element inside browser itself. I totally agree with Jason's point. I will make a offline discussion with Ethan.

Comment 8

[Ken Chang[:kenkai]]

blocking 1.4 RTSP feature.

Comment 9

[Ken Chang[:kenkai]]

It blocks 1.4 feature.

Comment 10

[Ethan Tseng [:ethan]]

It seems everyone wants to render RTSP streaming within the built-in media player inside the browser.
If it's true, we have to back out bug 884702, which launches video app for <a href="rtsp://xxx">.
In the meantime, my work on bug 945603 finally could support rendering RTSP streaming in all cases, i.e., URL bar, <a> tag, <video> tag, ..., etc.

Marvin, could you please help to confirm this decision?
Are we going to support all cases of RTSP inside the browser without invoking any external apps?

Comment 11

[Ethan Tseng [:ethan]]

> If it's true, we have to back out bug 884702, which launches video app for
> <a href="rtsp://xxx">.
BTW, since v1.3 still needs bug 884702. We should not "back out" that bug.
We could file a new bug to correct the behavior in v1.4.

Comment 12

[Rob MacDonald [:robmac]]

Re-assigning to Jacqueline. Thanks!

Comment 13

[Marvin Khoo [:Marvin_Khoo]]

Hi Ethan,
as discussed with Fabrice and team, this should not be blocker for 1.4. <a> tag RTSP audio will be playback thru Video app.

for 1.5, please ensure all video, audio, a tag, URL, iframe whatsoever RTSP streaming shall be playback thru build-in player at system browser.

Comment 14

[Ethan Tseng [:ethan]]

(In reply to Marvin Khoo [:Marvin_Khoo] from comment #13)
Hi Marvin,
It's great to hear that.
Thanks for your coordination!

Comment 15

[Jason Smith [:jsmith]]

(In reply to Marvin Khoo [:Marvin_Khoo] from comment #13)
> Hi Ethan,
> as discussed with Fabrice and team, this should not be blocker for 1.4. <a>
> tag RTSP audio will be playback thru Video app.

This needs to be decided by those that own the contract with YouTube. If they don't sign off on this, then we've got no choice here, as YT will fail us in certification in 1.4.

Vishy - Can you find out if YT is going to be okay or not with us allowing RTSP video from YouTube to render in the video app?

Comment 16

[Jason Smith [:jsmith]]

(In reply to Jason Smith [:jsmith] from comment #15)
> (In reply to Marvin Khoo [:Marvin_Khoo] from comment #13)
> > Hi Ethan,
> > as discussed with Fabrice and team, this should not be blocker for 1.4. <a>
> > tag RTSP audio will be playback thru Video app.
> 
> This needs to be decided by those that own the contract with YouTube. If
> they don't sign off on this, then we've got no choice here, as YT will fail
> us in certification in 1.4.
> 
> Vishy - Can you find out if YT is going to be okay or not with us allowing
> RTSP video from YouTube to render in the video app?

Vishy indicates over email that we should be okay here, but he's going to double check to be sure.

Comment 17

[Jacqueline Savory [:jsavory] UX]

In response to comment 1 regarding the video container. 

I think using the video container will possibly confuse the user as they might expect a video to begin to play and assume it is an error when it doesn't. The audio player is optimized for audio only situations and if we are certain that there is no video, if possible, it should use the audio player instead.

Comment 18

[Jason Smith [:jsmith]]

(In reply to jsavory from comment #17)
> In response to comment 1 regarding the video container. 
> 
> I think using the video container will possibly confuse the user as they
> might expect a video to begin to play and assume it is an error when it
> doesn't. The audio player is optimized for audio only situations and if we
> are certain that there is no video, if possible, it should use the audio
> player instead.

I'm not sure if that's actually possible - if I recall, I thought we couldn't detect that a RTSP URL was video or audio directly on URL selection - we only know when we start to try to play it back. So we won't know which UI (audio or video) to use until playback starts.

Comment 19

[Marvin Khoo [:Marvin_Khoo]]

Hi Jacqueline,
Jason is right, I'd say there is no sufficient info indicates that whether the URL link is video / audio until it being rendered.

so most system handle URL / <a> tag streaming cases like below,

1. When user enter RTSP URL link or <a> tag RTSP link,
2. System prompt UI for user to choose all registered stream services, it could be Music, Video app or Media streaming app (mostly video app, it show fancy firework for audio case) etc.
3. User has to select proper stream service in order to playback, e.g: Audio stream service playback a Video streaming may cause error, vise versa. except Media streaming app, it works for both audio and video case.

with above scenario, Music has works to do, given the among of time for 1.4 and Roadmap, We do not suggest to do this since we will move all RTSP playback to system-built in player on 1.5.


I guess we should close this issue, and continue to let audio steam to playback in video app up til 1.5. can we do that?

Comment 20

[Jason Smith [:jsmith]]

(In reply to Marvin Khoo [:Marvin_Khoo] from comment #19)
> I guess we should close this issue, and continue to let audio steam to
> playback in video app up til 1.5. can we do that?

No. The problem right now is that video app currently does not allow an audio stream to play in the video app in 1.4 (it's a regression). That's what this bug is tracking. The side tangent that happened above was the fact of questioning if there was even value to consider bringing the video app back into the picture to handle RTSP streams, which apparently there's agreement on, but we'll do it in 1.5. Based on the conclusion above, it sounds like we need to re-enable the video app support in 1.4 in this bug. In 1.5, we'll move in a different direction.

Comment 21

[Jason Smith [:jsmith]]

(In reply to Jason Smith [:jsmith] from comment #20)
> (In reply to Marvin Khoo [:Marvin_Khoo] from comment #19)
> > I guess we should close this issue, and continue to let audio steam to
> > playback in video app up til 1.5. can we do that?
> 
> No. The problem right now is that video app currently does not allow an
> audio stream to play in the video app in 1.4 (it's a regression). That's
> what this bug is tracking. The side tangent that happened above was the fact
> of questioning if there was even value to consider bringing the video app
> back into the picture to handle RTSP streams, which apparently there's
> agreement on, but we'll do it in 1.5. Based on the conclusion above, it
> sounds like we need to re-enable the video app support in 1.4 in this bug.
> In 1.5, we'll move in a different direction.

When I say agreement here - I mean agreement that we should move to the system player, rather the video app.

Comment 22

[Marvin Khoo [:Marvin_Khoo]]

(In reply to Jason Smith [:jsmith] from comment #21)
> (In reply to Jason Smith [:jsmith] from comment #20)
> > (In reply to Marvin Khoo [:Marvin_Khoo] from comment #19)
> > > I guess we should close this issue, and continue to let audio steam to
> > > playback in video app up til 1.5. can we do that?
> > 
> > No. The problem right now is that video app currently does not allow an
> > audio stream to play in the video app in 1.4 (it's a regression). That's
> > what this bug is tracking. The side tangent that happened above was the fact
> > of questioning if there was even value to consider bringing the video app
> > back into the picture to handle RTSP streams, which apparently there's
> > agreement on, but we'll do it in 1.5. Based on the conclusion above, it
> > sounds like we need to re-enable the video app support in 1.4 in this bug.
> > In 1.5, we'll move in a different direction.
> 
> When I say agreement here - I mean agreement that we should move to the
> system player, rather the video app.

Sounds fair, 1.4 enable video app to stream audio + video, 1.5 we move all to system-build in player (browser).

Comment 23

[Ethan Tseng [:ethan]]

Hi John Hu,
According the conclusion above, this bug should be fixed by the video app.
Could you help to assign this bug to the right owner? :)

Comment 24

[John Hu [:johnhu][:johu][:醬糊小弟]]

Hi Russ,

May you help to with this bug?

The key is at this line[1]. In this case, we may just let mime type with 'video/rtsp' and url starting with 'rtsp://' pass to play.

Once you finish it, I can help to review the patch. Thanks.

[1] https://github.com/mozilla-b2g/gaia/blob/e0a0faca582c5d4ea79f23a7da50b82418faacda/apps/video/js/view.js#L269

Comment 25

[Russ Nicoletti [:russn]]

Hi John,

Yes, I will take this. I may not get to it until next week due to other work in progress. Is that ok?

Comment 26

[John Hu [:johnhu][:johu][:醬糊小弟]]

Russ,

This is 1.4+. The deadline will be next week. If you can provide the patch at the early of next week, I think it is fine.

Comment 27

[John Hu [:johnhu][:johu][:醬糊小弟]]

This is caused by the bug 938033. And the bug 956142 is also caused by the same bug. We may just fix it by backing out the patch of bug 938033.

Comment 28

[Jason Smith [:jsmith]]

(In reply to John Hu [:johnhu][:johu][:醬糊小弟] from comment #27)
> This is caused by the bug 938033. And the bug 956142 is also caused by the
> same bug. We may just fix it by backing out the patch of bug 938033.

Looks like we just did. Fixed via backout of bug 938033.

Comment 29

[Dominic Kuo [:dkuo]]

Thanks, Jason, I was reading the comments to see if I have to close this bug as well, and you did it for us!

Comment 30

[William Hsu [:whsu]]

Verified it.

By the way, now, we use built-in media player (Browser) to play RTSP streaming.
Attach the screenshot. (2014-05-05-15-50-40.png)

* Build information:
 - Gaia      f55fc5c507312c7aac51ec9bb73061fd4ed5c5fb
 - Gecko     https://hg.mozilla.org/mozilla-central/rev/3285e030d9c0
 - BuildID   20140504160202
 - Version   32.0a1

Comment 31

[William Hsu [:whsu]]

Attachment: 2014-05-05-15-50-40.png

Comment 32

[Ethan Tseng [:ethan]]

Thanks for verification. :)
As comment 30 said, we don't render RTSP in Gaia video app anymore, I will remove related code in bug 1000340.