951175 - [RTSP][V1.3] Disabling network on the hotspot will cause a system hang while FxOS is playing RTSP streaming

Status: VERIFIED FIXED

(Firefox OS Graveyard :: General, defect)

Description

[William Hsu [:whsu]]

* Description:
  This problem happened on v1.3 build.
  While FxOS connects to a hotspot and playing a RTSP streaming, disabling the hotspot that will cause a system crash.
  Attaching the video.
  - https://dc1.safesync.com/LMsJysyJ/FxOS_Crash.mp4?a=35DYUraJZG0

* Reproduction steps:
  1. Connect to a hotspot
  2. Launch the following page via browser
     - http://goo.gl/lE2eE3
  2. Tap the "Video test page"
  3. After FxOS launches the video to play RTSP stream, please disable the hotspot

* Expected result:
  FxOS keeps loading

* Actual result:
  A crash happened.

* Reproduction build: V1.3 (Buri)
 - Gaia:     dca0a3dcf062ce3e422a9c56d141c14543c816fb
 - Gecko:    http://hg.mozilla.org/releases/mozilla-aurora/rev/1f7db4cc788e
 - BuildID   20131217004001
 - Version   28.0a2

Comment 1

[Jason Smith [:jsmith]]

Can you include a crash report URL?

Comment 2

[William Hsu [:whsu]]

Thanks for the reminder!

Attached:

https://crash-stats.mozilla.com/report/index/573ce236-144e-42a4-8094-311892131217

Comment 3

[Ethan Tseng [:ethan]]

Hi William,
Did you try the case of disabling Wifi while playing RTSP streaming?
If yes, what is the behavior of video app in that case?

Comment 4

[William Hsu [:whsu]]

Do you mean the WIFI of FxOS device?
My test result is the RTSP streaming was stopped and no system crash happened.
Thanks!

Comment 5

[Ethan Tseng [:ethan]]

William, I want to make sure this is a system crash, not a content process crash.
We located the root cause of bug 951117 and uploaded a fix patch. When I come back to this bug, I found your attached crash report shows the same crash point as we found in bug 951117. However, in that case, the crash happened in content process, not b2g process.

Comment 6

[William Hsu [:whsu]]

Hi, Ethan,

Sorry. I attached the incorrect crash report.

I double checked this bug. This is a system crash.
If the issue happens, the FxOS device will hang/crash there and no crash report was generated.

Thanks!

Comment 7

[Ethan Tseng [:ethan]]

*** Work Note ***
There are 3 cases I tested for this bug.
Phone A could be any smart phone that is able to share hotspot.
Phone B is FFOS phone.

Case 1: Normal condition.
  Scenario: [Internet] <--- 3G ---> [Phone A] <--- Hotspot ---> [Phone B]
  Result  : RTSP streaming is played normally.

Case 2: Disable hotspot on phone A.
  Scenario: [Internet] <--- 3G ---> [Phone A] <--- X ---> [Phone B]
  Result  : The media loading stops. But phone B could still work. We can press the back button to quit the video app.

Case 3: Disable 3G network on phone A.
  Scenario: [Internet] <--- X ---> [Phone A] <--- Hotspot ---> [Phone B]
  Result  : This is the scenario of this bug. Phone B hangs there.
            Actually the system does not crash. The b2g process is blocked somewhere and does not response anything to the user input.

We already identified the root cause in the stagefright library. RTSPSource object calls postAndAwaitResponse() to send a disconnect message but never get the according response.

Comment 8

[Ethan Tseng [:ethan]]

Attachment: bug-951175-fix.patch

This patch fixes a possible hanging problem using rtsp feature.

RTSPSource::stop() calls postAndAwaitResponse() to send a kWhatDisconnect message, which should be replied by RTSPSource::finishDisconnectIfPossible(). Originally this function would check its internal state, if it is not DISCONNECTED, it will call handler's disconnect function and return without replying the message.
This bug is caused by shutting down the networking on the phone that shares hotspot. The hotspot remains connected so our phone doesn't aware the network state is DISCONNECTED.

Actually we think the original return is unnecessary. RTSPSource::finishDisconnectIfPossible() could/should reply the message no matter what the state is.

Comment 9

[Ethan Tseng [:ethan]]

Attachment: bug_951175_gdb.log

This is the gdb backtrace output.
It shows the main thread is blocked in __pthread_cond_timedwait().

Comment 10

[Ethan Tseng [:ethan]]

The result of try server:
https://tbpl.mozilla.org/?tree=Try&rev=501f1d458390

Comment 11

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/integration/b2g-inbound/rev/92f05bd3d5da

Comment 12

[Ethan Tseng [:ethan]]

Hi, William, I just changed the title of this bug to reflect the real scenario I fixed.
1. "Disable network on the hotspot" instead of "disable the hotspot".
2. The system "hangs" instead of "crashes".

I think we had discussed this privately. I changed the title in order to prevent some misdirection in the feature.
In my tests, no crash or hanging happens when I turn off the hotspot on the phone that shares hotspot.
If you still encounter a crash or hanging when you turn off the hotspot, could you please file another bug?

Comment 13

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/92f05bd3d5da

Comment 14

[William Hsu [:whsu]]

(In reply to Ethan Tseng [:ethan] from comment #12)
> Hi, William, I just changed the title of this bug to reflect the real
> scenario I fixed.
> 1. "Disable network on the hotspot" instead of "disable the hotspot".
> 2. The system "hangs" instead of "crashes".
[William] You got that right! My previous title is easy to cause misunderstanding.

> 
> I think we had discussed this privately. I changed the title in order to
> prevent some misdirection in the feature.
> In my tests, no crash or hanging happens when I turn off the hotspot on the
> phone that shares hotspot.
> If you still encounter a crash or hanging when you turn off the hotspot,
> could you please file another bug?
[William] Sure! I will file a bug if I see the related issue. Thanks! :)

Comment 15

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/2353b531ccc6

Comment 16

[William Hsu [:whsu]]

@Ethan,

After I tested this patch, no crash happened but the slider indicator stop here without popping up warning message.
Attaching the demo video.(WP_20140220_003.mp4)

Could I know how long will it time out?
I had waited here a few minutes but nothing happened.
Thanks.

Comment 17

[William Hsu [:whsu]]

Attachment: WP_20140220_003.mp4

Comment 18

[Ethan Tseng [:ethan]]

(In reply to William Hsu [:whsu] from comment #16)
> After I tested this patch, no crash happened but the slider indicator stop
> here without popping up warning message.
> Could I know how long will it time out?
> I had waited here a few minutes but nothing happened.
Oh, We didn't add a timeout mechanism for this situation. What I resolved in this bug is just to avoid a scenario of "hanging the entire system".
If a timeout and error message is expected in this case, we need to do more research to see what we can do.

BTW, bug 951188 provides timeout and error message for at least 4 cases. See:
https://bugzilla.mozilla.org/show_bug.cgi?id=951188#c18
https://bugzilla.mozilla.org/show_bug.cgi?id=951188#c26

Maybe we should fire a new bug to track your proposed issue. :)

Comment 19

[William Hsu [:whsu]]

Sure! Thanks for the help!
Let's discuss remaining work on the other bug.