880601 - [A/V] Youtube cannot be played well on 3G connection or tethering via other device

Status: RESOLVED FIXED

(Firefox OS Graveyard :: General, defect)

Description

[leo.bugzilla.gecko]

Precondition : connect via 3G connection or other device's tethering

Almost every youtube files cannot be played at the end of contents.

While playing, there are so many jitterings.
And sometimes, video frame freeze.

I will upload video file which is recorded for this problem.

Comment 1

[Jason Smith [:jsmith]]

Was this tested with the patches included in bug 877461?

Comment 2

[Jason Smith [:jsmith]]

(In reply to Jason Smith [:jsmith] from comment #1)
> Was this tested with the patches included in bug 877461?

Actually, disregard. I just confirmed the same behavior playing 3GP media content on unagi. I'm seeing intermittent freezes while watching the video.

Comment 3

[leo.bugzilla.gecko]

(In reply to Jason Smith [:jsmith] from comment #2)
> Actually, disregard. I just confirmed the same behavior playing 3GP media
> content on unagi. I'm seeing intermittent freezes while watching the video.

I cannot upload recorded file because of security issue.
Could you please confirm that this problem is reproducible?

If not, I will find another way to inform you.

Comment 4

[Jason Smith [:jsmith]]

(In reply to leo.bugzilla.gecko from comment #3)
> (In reply to Jason Smith [:jsmith] from comment #2)
> > Actually, disregard. I just confirmed the same behavior playing 3GP media
> > content on unagi. I'm seeing intermittent freezes while watching the video.
> 
> I cannot upload recorded file because of security issue.
> Could you please confirm that this problem is reproducible?
> 
> If not, I will find another way to inform you.

I'd be curious to see if we're talking about the same problem. If you can't upload the recorded file here, then feel free to email me directly at [e-mail] with the recording.

For testing I did, I saw the video doing intermittent freezes a few times in a row watching videos ranging from 3 - 40 minutes. Note that the freezes I saw weren't permanent - if I waited a few seconds, I would see the video continue to play. But the freezes were prominent enough that user experience watching videos was rough (i.e. a bunch of small freezes watching a 3 minute video).

Comment 5

[Robert O'Callahan (:roc) (email my personal email if necessary)]

Attachment: some logs (gzipped)

This is a log from where this (or something very much like this is happening).

The video does recover and play through, but there are periods where the video is not playing and/or the video controls are in the "waiting" state. Note that the media state machine is always in the DECODING state, never reaching the BUFFERING state.

I collected this log with my dev phone connected to the Wifi of another phone acting as a hotspot and itself using 3G to get to the Internet. Yet, maybe I'm wrong, but it looks like we don't actually stall waiting for data during decoding.

Comment 6

[Robert O'Callahan (:roc) (email my personal email if necessary)]

Another interesting thing about this log is the rapid cycling between HAVE_CURRENT_DATA and HAVE_ENOUGH_DATA. I'm not sure why this is happening yet.

Comment 7

[Robert O'Callahan (:roc) (email my personal email if necessary)]

Unfortunately right now I can't reproduce problems on any video, even with my tethered-to-3G setup. Maybe because I'm at home or because it's night. I'll try again tomorrow.

Comment 8

[Robert O'Callahan (:roc) (email my personal email if necessary)]

(Do we have any tools for restricting the bandwidth of the network artificially?)

Comment 9

[Jason Smith [:jsmith]]

(In reply to Robert O'Callahan (:roc) (Mozilla Corporation) from comment #8)
> (Do we have any tools for restricting the bandwidth of the network
> artificially?)

Good question. Jason - Do you know?

Comment 10

[Mike Habicher [:mikeh] (high bugzilla latency)]

(In reply to Robert O'Callahan (:roc) (Mozilla Corporation) from comment #8)
>
> (Do we have any tools for restricting the bandwidth of the network
> artificially?)

It's a bit hackish, but to reproduce bug 870564 consistently, I wrapped my AP in aluminium foil and dropped it into a stock pot. It was good for -2 bars of WiFi at 1-metre distance.

Also, I haven't tried this, but I understand iptables can be used to limit bandwidth: http://www.oocities.org/youssef116/writing/ratelim.html

Oh, and if you're using an access point for your data link, some models will let you specify what 802.11 speeds to enable.

Comment 11

[Jason Duell]

Nick, can you fill in roc on any tools we used for Stone Ridge that would be handy here? See comment 8.

Roc: do you need just bandwidth limits, or are you also looking to reproduce packets getting dropped?

Comment 12

[u408661]

(In reply to Jason Duell (:jduell) from comment #11)
> Nick, can you fill in roc on any tools we used for Stone Ridge that would be
> handy here? See comment 8.
> 
> Roc: do you need just bandwidth limits, or are you also looking to reproduce
> packets getting dropped?

Netem is probably the way to go, here. For bandwidth limiting, you can take a look at https://github.com/mozilla/stoneridge/blob/master/linux/init/server/srinterface#L24 (lines 24 and 25 set the bandwidth limit as well as latency) in conjunction with https://github.com/mozilla/stoneridge/blob/master/linux/init/stoneridge#L22 (lines 22 through 57, in particular look at the example values for UMTS and/or GSM to simulate a 3G/tethered connection).

Netem can also do packet loss, though not in any deterministic fashion (which is why we don't model packet loss in stone ridge). Some examples for packet loss are at http://www.linuxfoundation.org/collaborate/workgroups/networking/netem#Packet_loss (you will probably just need to add the loss parameters to the command line on line 25 of the first link above).

Comment 13

[Robert O'Callahan (:roc) (email my personal email if necessary)]

I've been struggling with this bug for days now. Issues seem to come and go. I filed bug 882027 for comment 6, with a patch, but there are still issues.

I slurped the Troy video locally and put it on a local Web server, using 'tc' to rate-limit the download as Nick suggested. At 500Kbit, sometimes everything works fine, other times things work very poorly. Still trying to work out why.

Comment 14

[Robert O'Callahan (:roc) (email my personal email if necessary)]

For future reference, the magic tc command line is
sudo tc qdisc add dev wlan0 root handle 1:0 tbf rate 500Kbit maxburst 1600 limit 4800

Comment 15

[Robert O'Callahan (:roc) (email my personal email if necessary)]

Attachment: (partial?) fix

This patch seems to help a lot on my tests. Lengthy explanation in the patch.

Comment 16

[Robert O'Callahan (:roc) (email my personal email if necessary)]

The video I'm playing is 9564390 bytes and has a duration of 4 minutes 58 seconds, so that's a nominal bitrate of a little of 250Kb/s. If I throttle bandwidth to 300Kb/s I can repeatedly play the video fine with this patch, but I couldn't before.

Throttling to 200Kb/s, there's jerkiness and buffering, as you'd expect, but audio and video stay in sync and we can play periodically.

Comment 17

[Robert O'Callahan (:roc) (email my personal email if necessary)]

At times I still seem to get into a bad state. In this state, calls into libstagefright codec 'read' calls are often slow (50-100ms or more, for audio or video). The data read calls are not blocking, they only read cached data. With the above patch we tolerate this as best we can and play back jerkily, but it's pretty bad. I'm not sure what's going on in this case, but it seems like its own bug.

Comment 18

[Chris Pearce [:cpearce (Not reading bugmail)]]

Comment on attachment 761414 [details] [diff] [review]
(partial?) fix

Review of attachment 761414 [details] [diff] [review]:
-----------------------------------------------------------------

I think that you should put part of your comment here in the code.

Comment 19

[Robert O'Callahan (:roc) (email my personal email if necessary)]

Comment on attachment 761414 [details] [diff] [review]
(partial?) fix

NOTE: Please see https://wiki.mozilla.org/Release_Management/B2G_Landing to better understand the B2G approval process and landings.

[Approval Request Comment]
Bug caused by (feature/regressing bug #): none
User impact if declined: streaming video won't play reliably on mediocre network connections
Testing completed: only local tests
Risk to taking this patch (and alternatives if risky): reasonably low risk; a small change that mimics behavior we have on other platforms, but in delicate code. We kinda have to have this, or something like it, for Youtube to work well on slow connections, though.
String or UUID changes made by this patch: none

Comment 20

[Robert O'Callahan (:roc) (email my personal email if necessary)]

[e-mail], can you try this patch and see if it makes things better?

Comment 21

[leo.bugzilla.gecko]

(In reply to Robert O'Callahan (:roc) (Mozilla Corporation) from comment #20)
> [e-mail], can you try this patch aYnd see if it makes
> things better?

Yes, it's much more smoother than before.
But there're many jittering yet and sometimes a/v is suddenly stopped.

Some patches from Bug 878343 is not landed yet on LG side.
So i'll update more information after landing it.

Comment 22

[Robert O'Callahan (:roc) (email my personal email if necessary)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/b1686ef2af5d

Comment 23

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/b1686ef2af5d

Comment 24

[Robert O'Callahan (:roc) (email my personal email if necessary)]

https://hg.mozilla.org/releases/mozilla-b2g18/rev/f1b84d841ced

Comment 25

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-b2g18_v1_1_0_hd/rev/f1b84d841ced

Comment 26

[Darren]

Created 2 test cases:
https://moztrap.mozilla.org/manage/cases/?filter-id=8985
https://moztrap.mozilla.org/manage/cases/?filter-id=8984