1386759 - Disable network.http.throttle.enable in FF55 release

Status: VERIFIED FIXED

(Core :: Networking: HTTP, defect)

Description

[Ben Kelly [:bkelly, not reviewing]]

A user found an issue with our network throttling code in FF55 beta13 in bug 1386323.  Its unclear if a fix will be ready in time given FF55 ships in less than a week.  We may just want to disable network.http.throttle.enable in release until then.

Comment 1

[Shih-Chiang Chien [:schien] (UTC+8) (use ni? plz)]

@mayhemer is this on your plate?

Comment 2

[Honza Bambas (:mayhemer)]

Yup, going to whip up a patch.

Comment 3

[Honza Bambas (:mayhemer)]

Attachment: v1 (flip the pref on m-r)

This completely disables any throttling of any transaction.  With this patch we behave the same way as before this was implemented.

Comment 4

[Ben Kelly [:bkelly, not reviewing]]

Note, we may need a shield pref flip for the first 55.0.  The patch is still good to get it off by default in 55.0.1.  This is what RyanVM suggested to me on IRC last night.

Comment 5

[Honza Bambas (:mayhemer)]

(In reply to Ben Kelly [:bkelly] from comment #4)
> Note, we may need a shield pref flip for the first 55.0.  The patch is still
> good to get it off by default in 55.0.1.  This is what RyanVM suggested to
> me on IRC last night.

I'm not familiar with how sheild prefs work.  If you want me to do something more, I'll need some docs about it.  Thanks.

Comment 6

[Ben Kelly [:bkelly, not reviewing]]

I've never done it either.  Ryan, can you help us understand what a "shielf pref" entails?

Comment 7

[Ryan VanderMeulen [:RyanVM]]

302 Osmose.

Comment 8

[Julien Cristau [:jcristau]]

Are we ready/confident to use shield for pref flips on release now?  Previously we would have pushed a go faster system add-on for this, but I guess shield is lower overhead?

Comment 9

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

We are considering shipping an rc3 with this pref disabled. tracked for 55.

Comment 10

[Matt Grimes [:Matt_G]]

Pref rollout (100% pref change with 1 value) is not currently implemented. If this is a temporary situation while we wait for the patch to land in 55.0.1 we could use the pref experiment action. The downside to this is that it is not as fault tolerant as the real pref rollout action would be. I'll wait for osmose to give us his take on any risks in doing this.

Comment 11

[Sylvestre Ledru [:Sylvestre]]

Could you land this in m-r please?
We are considering a rc3
Thanks

Comment 12

[Honza Bambas (:mayhemer)]

Should I request release approval?

Comment 13

[Sylvestre Ledru [:Sylvestre]]

Yeah, please. So that we have a clear trace

Comment 14

[Osmose [:osmose, :mkelly]]

(In reply to Matt Grimes [:Matt_G] from comment #10)
> Pref rollout (100% pref change with 1 value) is not currently implemented.
> If this is a temporary situation while we wait for the patch to land in
> 55.0.1 we could use the pref experiment action. The downside to this is that
> it is not as fault tolerant as the real pref rollout action would be. I'll
> wait for osmose to give us his take on any risks in doing this.

Downsides to this approach vs a system add-on:

1. We don't have solid uptake numbers for preference experiments yet, while rhelmer's already done the work verifying system add-on uptake.
2. 100% of release will absolutely overload the experiment annotation infra in Telemetry. I got tentative approval from sunah for running something at 100% of Beta, and even then she was concerned.

If we need a hotfix, I think we should stick to a system add-on hotfix for this particular instance. We're working on resolving both of those issues but aren't there yet.

Comment 15

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-release/rev/9e30e915f132

Comment 16

[Bogdan Maris, Desktop Test Engineering [OUT until 12 Jan]]

I successfully verified that 'network.http.throttle.enable' was 'false' in Firefox RC 55.0 build 3 using Windows 10 x64, Ubuntu 16.04 x64 and macOS 10.12. 
I also tried to reproduce the issue from bug 1386323 that lead to this, but unfortunately I couldn't reproduce the bug. So the only thing I managed to do was verify the pref.

Comment 17

[Honza Bambas (:mayhemer)]

(In reply to Bogdan Maris, QA [:bogdan_maris] from comment #16)
> I also tried to reproduce the issue from bug 1386323 that lead to this, but
> unfortunately I couldn't reproduce the bug.

I think you need to open a chat window on facebook to reproduce (chat opens a long standing request that throttles the downloads).

Comment 18

[Honza Bambas (:mayhemer)]

(In reply to Honza Bambas (:mayhemer) from comment #17)
> (In reply to Bogdan Maris, QA [:bogdan_maris] from comment #16)
> > I also tried to reproduce the issue from bug 1386323 that lead to this, but
> > unfortunately I couldn't reproduce the bug.
> 
> I think you need to open a chat window on facebook to reproduce (chat opens
> a long standing request that throttles the downloads).

Or:
- download (by simply going to) http://speed.hetzner.de/1GB.bin
- then go to https://www.janbambas.cz/moz/bug1386759/ and observe if the download slows down (this page just waits 2 minutes for an image to load, which is enough to reproduce the problem)

Comment 19

[Bogdan Maris, Desktop Test Engineering [OUT until 12 Jan]]

(In reply to Honza Bambas (:mayhemer) from comment #18)
> (In reply to Honza Bambas (:mayhemer) from comment #17)
> > (In reply to Bogdan Maris, QA [:bogdan_maris] from comment #16)
> > > I also tried to reproduce the issue from bug 1386323 that lead to this, but
> > > unfortunately I couldn't reproduce the bug.
> > 
> > I think you need to open a chat window on facebook to reproduce (chat opens
> > a long standing request that throttles the downloads).
> 
> Or:
> - download (by simply going to) http://speed.hetzner.de/1GB.bin
> - then go to https://www.janbambas.cz/moz/bug1386759/ and observe if the
> download slows down (this page just waits 2 minutes for an image to load,
> which is enough to reproduce the problem)

Thank you for the additional comments, those helped. The problem was that I was using this http://speedtest.tele2.net/ site as a measuring tool (it was suggested in the bug 1386323), but I think it might be defective because it didn't show any difference between the two methods of downloading.

When I opened the downloads doorhanger, the difference was huge. In the end, I managed to reproduce the bug on Beta 55.0b13 using macOs 10.12 ('network.http.throttle.enable' is 'false') and confirmed it's fixed on 55.0 build 3 with 'network.http.throttle.enable' is 'false'.