Closed Bug 1119681 Opened 6 years ago Closed 6 years ago

[RTSP] add time stamps to RTSP live streams


(Firefox OS Graveyard :: RTSP, defect)

Gonk (Firefox OS)
Not set


(blocking-b2g:2.2+, firefox37 wontfix, firefox38 fixed, b2g-v2.2 fixed, b2g-master fixed)

2.2 S4 (23jan)
blocking-b2g 2.2+
Tracking Status
firefox37 --- wontfix
firefox38 --- fixed
b2g-v2.2 --- fixed
b2g-master --- fixed


(Reporter: jhao, Assigned: jhao)



(Whiteboard: [caf priority: p2][CR 811805])


(2 files, 7 obsolete files)

No description provided.
At first, we weren't sure if RTP timestamps of real time streams are reliable. However, it turns out doing well and significantly improves the performance. The difference is very obvious on the following stream rtsp://

There's already a wip patch in bug 1080467. But it still needs some fixes to make playout delay work. Also, we should take care of the discontinuity when live streams are paused then play again.
Attached patch Adding timestamps (obsolete) — Splinter Review
As Benjamin said in if we have valid timestamps, we should pretend it's not a live stream to make MediaDecoderStateMachine apply playout delay.

However, we are not sure if we may encounter live streams with invalid RTP timestamps, so we keep two private boolean values now:
1. mHasTimeStamps - which is currently always true. This value will be returned by public method IsRealTime()
2. mIsLiveStream - which will be used to determine if the stream has end time, or if the stream is seekable.

I haven't dealt with the problem with pausing and seeking. I'm just asking Benjamin's if the above logic and naming is OK, and if the comments are enough.
Attachment #8546445 - Flags: feedback?(bechen)
Attachment #8546445 - Attachment is patch: true
Comment on attachment 8546445 [details] [diff] [review]
Adding timestamps

Review of attachment 8546445 [details] [diff] [review]:

::: dom/media/RtspMediaResource.h
@@ +251,5 @@
>    bool mIsConnected;
>    // live stream
> +  bool mIsLiveStream;
> +  // has rtp timestamps
> +  bool mHasTimestamps;

Each RTP packet has only one timestamp, right?
So we should name it timestamp instead timestamps?
Attachment #8546445 - Flags: feedback?(bechen) → feedback+
After testing, it seems that some live streams server doesn't accept PAUSE request. In order not to keep receiving data after pausing, we have to close the entire session. I will try to put the logic in RTSPSource or RTSPConnectionHandler to avoid drastic changes to DecoderStateMachine.
Attached patch Send teardown if paused (obsolete) — Splinter Review
In this patch, if user press pause, RTSPSource will disconnect with the server, and then restart a new session. In media decoder, I dispatch a playbackended() to make it go to PLAY_STATE_ENDED.

However, there's a timing issue. If the user press pause and immediately plays again, the new connection may not be done yet, and the decoder starts decoding prematurely.

Perhaps making it go to ENDED state is not a good idea, instead we could make it go to the beginning or the DORMANT state.
Attached patch Adding timestamps (obsolete) — Splinter Review
This patch uses RTP-time as timestamps for live streams. Changes are mostly in RtspMediaResource as I asked Benjamin for feedbacks. Since there're timestamps now, MediaDecoderStateMachine can treat live streams just as non-live streams, so play-out delay can be done.

Besides, there's the problem that live streams server can't handle PAUSE requests. My solution is that in RTSPSource, if |pause()| is called, |stop()| is performed instead, and a flag mDisconnectedToPauseLiveStream is set. With this flag on, we don't report disconnection to the controller, thus letting Decoder go to PAUSED state. Later when RTSPSource::play() is called, a new session is started. Due to timing issues, a play() is added at the end of RTSPSource::onConnected(), to ensure it starts playing after successfully connected to the server.

I deliberately avoid touching MediaDecoder and MediaDecoderStateMachine so that I don't break its logic.
Attachment #8546445 - Attachment is obsolete: true
Attachment #8551047 - Attachment is obsolete: true
Attachment #8551186 - Flags: review?(ettseng)
Attachment #8551186 - Flags: review?(bechen)
With this patch, the performance will be great. But if user paused and play again, the play-out delay will be gone. The performance will be bad for a few seconds, but will recover afterwards.
Comment on attachment 8551186 [details] [diff] [review]
Adding timestamps

Review of attachment 8551186 [details] [diff] [review]:

Jonathan, thanks for writing this patch!

I have some comments and questions inline.
It seems this patch is based on some trial patch in your local mercurial queue.
(ETHAN_LOG() is included in the previous version of patch difference!).

Please clean up your MQ and rebase the patch.

::: dom/media/RtspMediaResource.cpp
@@ +509,5 @@
>      nsIChannel* aChannel, nsIURI* aURI, const nsACString& aContentType)
>    : BaseMediaResource(aDecoder, aChannel, aURI, aContentType)
>    , mIsConnected(false)
> +  , mIsLiveStream(false)
> +  , mHasTimestamps(true)

Is mHasTimestamps always true?
I don't see any logic to determine the value of this flag.

::: netwerk/protocol/rtsp/rtsp/RTSPSource.cpp
@@ +121,5 @@
>  {
>      LOGI("RTSPSource::pause()");
>      ETHAN_LOG("RTSPSource::pause()");
> +
> +    // live streams can't be paused, we have to disconnect now

Please capitalize the first letter and add a period in the end.

@@ +126,5 @@
> +    if (isLiveStream()) {
> +        mDisconnectedToPauseLiveStream = true;
> +        stop();
> +        return;
> +    }

Once paused for a live stream, mDisconnectedToPauseLiveStream becomes true
What would happen if we re-play the paused live stream and then a "real"
disconnection occurs (either stopped by decoder or caused by network error)?
Should we set mDisconnectedToPauseLiveStream as false in RTSPSource::play()?

@@ +665,5 @@
>      ETHAN_LOG("RTSPSource::onConnected()");
>      mState = CONNECTED;
> +
> +    play();

Is this play added for the case of re-playing after pausing a live stream?
Please add comments to make it clear.
Furthermore, this could break the logic of auto-play setting of video element.
Maybe we should add some checks before executing play here.

@@ +685,5 @@
>      if (mDisconnectReplyID != 0) {
>          finishDisconnectIfPossible();
>      }
> +    // if the disconnection is caused by pausing live stream
> +    // do not report back to the controller

Capitalize the first letter and add comma and period.
// If the disconnection is caused by pausing live stream,
// do not report back to the controller.

@@ +764,3 @@
>      while (dequeueAccessUnit(info->mIsAudio, &accessUnit) == OK) {
>          ETHAN_LOG("dequeue");
>      }

This while() loop doesn't exist in current repository.
You have to rebase your patch by cleaning your local mercurial queue.
Attachment #8551186 - Flags: review?(ettseng) → review-
Attached patch Adding timestamps (obsolete) — Splinter Review
I rebased the patch, and revised the comments. Also, I set mDisconnectedToPauseLiveStream to false at the beginning of RTSPSource::start().

As for HasTimeStamps always being true, it was Benjamin's concerns that maybe someday we'll find a live stream server sending us packets with invalid rtp-times, so we should preserve this flag. For now, it's always true.
Attachment #8551186 - Attachment is obsolete: true
Attachment #8551186 - Flags: review?(bechen)
Attachment #8551551 - Flags: review?(ettseng)
Attachment #8551551 - Flags: review?(bechen)
Attachment #8551551 - Flags: review?(bechen) → review+
Comment on attachment 8551551 [details] [diff] [review]
Adding timestamps

Review of attachment 8551551 [details] [diff] [review]:

I notice you removed the call to play() in RTSPSource::onConnected() in your previous patch.
Please make sure this is what you want.

The patch looks fine to me. Thanks.
Attachment #8551551 - Flags: review?(ettseng) → review+
Attached patch Adding timestamps (obsolete) — Splinter Review
Hi Ethan,

Sorry, I overlooked your concern about the immediate |play()| after connected. It doesn't break the logic of autoplay because an HTML video tag with autoplay attribute false doesn't even setup the connection. However, Ethan told me offline that there's also a "preload" option. Although RTSP streams doesn't have cache yet, I'll set another flag |mPlayOnConnected| to determine if a connection is triggered by a play request from above.

Please check one more time. Thanks a lot!
Attachment #8551551 - Attachment is obsolete: true
Attachment #8552948 - Flags: review?(ettseng)
Comment on attachment 8552948 [details] [diff] [review]
Adding timestamps

Review of attachment 8552948 [details] [diff] [review]:

Excellent! Adding mPlayOnConnected is much more clear. r=me

::: netwerk/protocol/rtsp/rtsp/RTSPSource.h
@@ +158,5 @@
> +    // in order to pretend pausing a live stream.
> +    bool mDisconnectedToPauseLiveStream;
> +
> +    bool mPlayOnConnected;
> +

Although this variable name is somewhat self-explained, I suggest it's still
better to add comment to explain its purpose.

Here's my suggestion:
// While performing a play operation, if the current state of RTSP connection
// is disconnected, we will start over establishing connection to the server.
// In this case (mPlayOnConnected = true), we have to perform play again when 
// onConnected, to ensure we complete the play operation.
Attachment #8552948 - Flags: review?(ettseng) → review+
Attached patch Adding timestamps (obsolete) — Splinter Review
I added the suggested comment of Ethan, and modified the commit message.
Attachment #8552948 - Attachment is obsolete: true
Attachment #8553446 - Flags: review+
Attached patch Adding timestamps (obsolete) — Splinter Review
Modify comments again.
Attachment #8553446 - Attachment is obsolete: true
Attachment #8553451 - Flags: review+
Mistakenly uploaded an empty patch. This one is right.
Attachment #8553451 - Attachment is obsolete: true
Attachment #8553452 - Flags: review+
Closed: 6 years ago
Resolution: --- → FIXED
Target Milestone: --- → 2.2 S4 (23jan)
Comment on attachment 8587823 [details] [diff] [review]
Rebasing to 2.2 for possible uplift

Hi Bhavana,
This uplift is for bug 1147537, which is blocking-b2g: 2.2+
Could you approve it?

NOTE: Please see to better understand the B2G approval process and landings.

[Approval Request Comment]
Bug caused by (feature/regressing bug #):
bug 1147537

User impact if declined:
RTSP streams wouldn't autoplay, and performance would be bad

Testing completed:
Although there are five builds failed, Edgar said that the reason is probably not my patch. He suggested me to push an empty patch to try server
and as he predicted, those five builds still failed. The reason may be that 
try server used master branch to test my 2.2 gecko code, and there's some mismatch between build systems.

Risk to taking this patch (and alternatives if risky): 
Not risky.
This patch only affects RTSP live streams.

String or UUID changes made by this patch:
Flags: needinfo?(bbajaj)
Attachment #8587823 - Flags: review+
Attachment #8587823 - Flags: approval-mozilla-b2g37?
move this bug to 2.2+, the patch can solve bug 1149616
blocking-b2g: --- → 2.2+
Flags: needinfo?(bbajaj)
Attachment #8587823 - Flags: approval-mozilla-b2g37? → approval-mozilla-b2g37+
Duplicate of this bug: 1147537
Whiteboard: [CR 811805]
Whiteboard: [CR 811805] → [caf priority: p2][CR 811805]
You need to log in before you can comment on or make changes to this bug.