1414091 - Massive leak while loading psarips.com

Status: RESOLVED WORKSFORME

(Core :: General, defect)

Description

[Marco Castelluccio [:marco]]

The page never finishes loading and Firefox keeps allocating memory indefinitely. Most of the memory is never freed and is under "heap unclassified":
4,107.98 MB (100.0%) -- explicit
├──3,424.01 MB (83.35%) ── heap-unclassified

Profile while the page is loading: https://perfht.ml/2ipc7pL.

mozregression is pointing me to bug 1406212, but I'm not sure it's a legit regression range.

I can't reproduce this in a clean profile.

Comment 1

[Marco Castelluccio [:marco]]

I've run mozregression again with the exact same result, so I guess it is correct: https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=a9b6e1e83c7125d60bd92000da195f6115b50a37&tochange=38758d31bd62eb0a1d8c1374aaaa61787018d4da.

Comment 2

[Marco Castelluccio [:marco]]

I can only reproduce with HTTPS Everywhere enabled.

Comment 3

[Blake Kaplan (:mrbkap) (inactive)]

This is probably a bug in HTTPS everywhere. If bug 1406212 is truly at fault, then that would seem to indicate a bug in their e10s compatibility code. Looking at the extension's page on AMO [1], it seems like they released a WebExtension on October 30. Do you know what version you were testing with in comment 0?

[1] https://addons.mozilla.org/en-US/firefox/addon/https-everywhere/

Comment 4

[Blake Kaplan (:mrbkap) (inactive)]

Oh, psarips.com also redirects https back to http, so that seems like a case where HTTPS everywhere might get confused.

Comment 5

[Marco Castelluccio [:marco]]

(In reply to Blake Kaplan (:mrbkap) from comment #3)
> This is probably a bug in HTTPS everywhere. If bug 1406212 is truly at
> fault, then that would seem to indicate a bug in their e10s compatibility
> code. Looking at the extension's page on AMO [1], it seems like they
> released a WebExtension on October 30. Do you know what version you were
> testing with in comment 0?
> 
> [1] https://addons.mozilla.org/en-US/firefox/addon/https-everywhere/

Notice I'm testing with mozregression and my profile, so this can't be a regression in HTTPS Everywhere (the exact same version of HTTPS Everywhere is working before bug 1406212 and is not working after bug 1406212).
e10s is enabled for me both before and after bug 1406212.

Comment 6

[Blake Kaplan (:mrbkap) (inactive)]

(In reply to Marco Castelluccio [:marco] from comment #5)
> Notice I'm testing with mozregression and my profile, so this can't be a
> regression in HTTPS Everywhere (the exact same version of HTTPS Everywhere
> is working before bug 1406212 and is not working after bug 1406212).
> e10s is enabled for me both before and after bug 1406212.

This could have been a latent bug in HTTPS everywhere that was exposed by the removal of the e10srollout addon.

I've tried reproducing this on Linux (debug+opt) and OSX (nightly) without success today. Installing HTTPS Everywhere and navigating to psarips.com finishes loading and doesn't appear to use more memory than any of my other tabs. What versions of HTTPS Everywhere and Firefox are you testing with?

Comment 7

[Marco Castelluccio [:marco]]

(In reply to Blake Kaplan (:mrbkap) from comment #6)
> (In reply to Marco Castelluccio [:marco] from comment #5)
> > Notice I'm testing with mozregression and my profile, so this can't be a
> > regression in HTTPS Everywhere (the exact same version of HTTPS Everywhere
> > is working before bug 1406212 and is not working after bug 1406212).
> > e10s is enabled for me both before and after bug 1406212.
> 
> This could have been a latent bug in HTTPS everywhere that was exposed by
> the removal of the e10srollout addon.
> 
> I've tried reproducing this on Linux (debug+opt) and OSX (nightly) without
> success today. Installing HTTPS Everywhere and navigating to psarips.com
> finishes loading and doesn't appear to use more memory than any of my other
> tabs. What versions of HTTPS Everywhere and Firefox are you testing with?

Latest nightly, HTTPS Everywhere 2017.10.4.1337 (it looks like it's the latest version, I can't update it).

Comment 8

[Trevor Rowbotham [:rowbot]]

(In reply to Marco Castelluccio [:marco] from comment #7)
> (In reply to Blake Kaplan (:mrbkap) from comment #6)
> > (In reply to Marco Castelluccio [:marco] from comment #5)
> > > Notice I'm testing with mozregression and my profile, so this can't be a
> > > regression in HTTPS Everywhere (the exact same version of HTTPS Everywhere
> > > is working before bug 1406212 and is not working after bug 1406212).
> > > e10s is enabled for me both before and after bug 1406212.
> > 
> > This could have been a latent bug in HTTPS everywhere that was exposed by
> > the removal of the e10srollout addon.
> > 
> > I've tried reproducing this on Linux (debug+opt) and OSX (nightly) without
> > success today. Installing HTTPS Everywhere and navigating to psarips.com
> > finishes loading and doesn't appear to use more memory than any of my other
> > tabs. What versions of HTTPS Everywhere and Firefox are you testing with?
> 
> Latest nightly, HTTPS Everywhere 2017.10.4.1337 (it looks like it's the
> latest version, I can't update it).

The latest version of HTTPS Everywhere is 2017.10.30, which is available through AMO[1]. I assume you have an earlier experimental version of HTTPS Everywhere installed, which likely came from [2], based on the version number.

[1] https://addons.mozilla.org/en-US/firefox/addon/https-everywhere/
[2] https://www.eff.org/files/https-everywhere-test/index.html.

Comment 9

[Marco Castelluccio [:marco]]

(In reply to Trevor Rowbotham from comment #8)
> (In reply to Marco Castelluccio [:marco] from comment #7)
> > (In reply to Blake Kaplan (:mrbkap) from comment #6)
> > > (In reply to Marco Castelluccio [:marco] from comment #5)
> > > > Notice I'm testing with mozregression and my profile, so this can't be a
> > > > regression in HTTPS Everywhere (the exact same version of HTTPS Everywhere
> > > > is working before bug 1406212 and is not working after bug 1406212).
> > > > e10s is enabled for me both before and after bug 1406212.
> > > 
> > > This could have been a latent bug in HTTPS everywhere that was exposed by
> > > the removal of the e10srollout addon.
> > > 
> > > I've tried reproducing this on Linux (debug+opt) and OSX (nightly) without
> > > success today. Installing HTTPS Everywhere and navigating to psarips.com
> > > finishes loading and doesn't appear to use more memory than any of my other
> > > tabs. What versions of HTTPS Everywhere and Firefox are you testing with?
> > 
> > Latest nightly, HTTPS Everywhere 2017.10.4.1337 (it looks like it's the
> > latest version, I can't update it).
> 
> The latest version of HTTPS Everywhere is 2017.10.30, which is available
> through AMO[1]. I assume you have an earlier experimental version of HTTPS
> Everywhere installed, which likely came from [2], based on the version
> number.
> 
> [1] https://addons.mozilla.org/en-US/firefox/addon/https-everywhere/
> [2] https://www.eff.org/files/https-everywhere-test/index.html.

Thanks! I've installed 2017.10.30. I can still reproduce this bug.

Comment 10

[Blake Kaplan (:mrbkap) (inactive)]

(In reply to Marco Castelluccio [:marco] from comment #9)
> Thanks! I've installed 2017.10.30. I can still reproduce this bug.

Can you attach your about:support here? In theory we should now be using the exact same configuration/version. Also, is navigating to "psarips.com" sufficient or are there more STR?

Comment 11

[Marco Castelluccio [:marco]]

Attachment: aboutsupport.tar.xz

(In reply to Blake Kaplan (:mrbkap) from comment #10)
> (In reply to Marco Castelluccio [:marco] from comment #9)
> > Thanks! I've installed 2017.10.30. I can still reproduce this bug.
> 
> Can you attach your about:support here?

Attached!

> In theory we should now be using the
> exact same configuration/version. Also, is navigating to "psarips.com"
> sufficient or are there more STR?

Just navigating to "psarips.com" is sufficient.
There's one thing that comes to mind that might be different in our profiles. This site used to be https a while ago, so I have https://psarips.com/ in my history.

Comment 12

[Andrew Overholt [:overholt]]

Did you get a chance to dig anymore here, Blake?

Comment 13

[Blake Kaplan (:mrbkap) (inactive)]

I've tried reproducing this again without any luck. One thing I noticed in Marco's about:support is that he's disallowing third-party cookies (network.cookie.cookieBehavior=1), which might affect things here.

Two more questions: Does having HTTPS Everywhere installed but disabled fix the problem here? If you clear all cookies for psarips.com (but have HTTPS Everywhere installed and enabled) does that change anything?

Comment 14

[Marco Castelluccio [:marco]]

(In reply to Blake Kaplan (:mrbkap) from comment #13)
> I've tried reproducing this again without any luck. One thing I noticed in
> Marco's about:support is that he's disallowing third-party cookies
> (network.cookie.cookieBehavior=1), which might affect things here.
> 
> Two more questions: Does having HTTPS Everywhere installed but disabled fix
> the problem here? If you clear all cookies for psarips.com (but have HTTPS
> Everywhere installed and enabled) does that change anything?

I can reproduce after enabling third-party cookies too and after removing psarips.com cookies and after removing psarips.com from history.

I can't reproduce if I disable HTTPS Everywhere (either from about:addons or from the HTTPS Everywhere popup).

Let me know if you need more info.

Comment 15

[Marco Castelluccio [:marco]]

Attachment: dmd_reports.tar.xz

Here are DMD reports, maybe understanding where the heap-unclassified memory is coming from can help.

Comment 16

[Blake Kaplan (:mrbkap) (inactive)]

Looking at the DMD logs shows us creating lots of objects to support HTTP redirects, so there's no smoking gun, but we're almost certainly redirecting thanks to the extension.

This seems like it's going to be a bug in HTTPS everywhere. I don't know if we have a contact over there to help us debug this. One thing to try would be to turn on HTTPS Everywhere logging [1] to see if that gives any clues as to what's happening.

I do see that HTTPS everywhere has a rule for psarips.com, so another possibility would be if their redirect loop detection is somehow broken in Firefox but not Chrome. Maybe our requestIds in onBeforeRequest change for redirects?

I have no idea why your setup would be different from mine :-/

[1] https://github.com/EFForg/https-everywhere/blob/f09fb11a607316333b1a298e8f34bcca3ee27f9d/chromium/util.js#L13

Comment 17

[Ethan Tseng [:ethan]]

Hey Tom,
It seems the root cause is in the add-on HTTPS Everywhere itself.
Do you know Bill at EFF has a Bugzilla account that we can needinfo him?  Or anyone else?

Comment 18

[William Budington]

Thank you, I see this and I'll look into it.  If I can reproduce it, I hope a fix to be incorporated into our Wednesday release of the extension.

I've mirrored this issue on our own tracker: https://github.com/EFForg/https-everywhere/issues/13876

Comment 19

[William Budington]

I can not reproduce this error on Linux x64 with either Firefox unbranded stable (57.0) or beta (58.0b3) in a new profile.  Marco, do you have other extensions enabled?

Comment 20

[Marco Castelluccio [:marco]]

(In reply to William Budington from comment #19)
> I can not reproduce this error on Linux x64 with either Firefox unbranded
> stable (57.0) or beta (58.0b3) in a new profile.  Marco, do you have other
> extensions enabled?

I can reproduce with only HTTPS Everywhere enabled.

Comment 21

[William Budington]

Marco, what platform are you using when you encounter this bug?

Comment 22

[Marco Castelluccio [:marco]]

(In reply to William Budington from comment #21)
> Marco, what platform are you using when you encounter this bug?

Linux 64-bit (in particular, Ubuntu 17.10).
It's possible that this bug is due to particular conditions of my profile, so it might be very hard to reproduce for you.
I can help test debugging builds with additional logging, if you have any.

Comment 23

[William Budington]

Even when flipping all the same prefs you have in `about:support` and installing all the active addons, I still can't reproduce this.

I'm unsure how to parse the DMD reports.  What tools are useful here?

I've removed this individual ruleset in https://github.com/EFForg/https-everywhere/commit/cf1c66c56923c31872aa5ef5b23f909fc6f4265f.  We're going to release a new version tomorrow, which will deliver updated rulesets with this change.  This should fix the immediate problem, if this is caused by the HTTPS downgrade redirect.  However, the extension should account for this - not sure why we're seeing problems in your particular instance.

Comment 24

[Marco Castelluccio [:marco]]

(In reply to William Budington from comment #23)
> Even when flipping all the same prefs you have in `about:support` and
> installing all the active addons, I still can't reproduce this.
> 
> I'm unsure how to parse the DMD reports.  What tools are useful here?
> 
> I've removed this individual ruleset in
> https://github.com/EFForg/https-everywhere/commit/
> cf1c66c56923c31872aa5ef5b23f909fc6f4265f.  We're going to release a new
> version tomorrow, which will deliver updated rulesets with this change. 
> This should fix the immediate problem, if this is caused by the HTTPS
> downgrade redirect.  However, the extension should account for this - not
> sure why we're seeing problems in your particular instance.

There's some documentation at https://developer.mozilla.org/en-US/docs/Mozilla/Performance/DMD under the '"Dark matter" mode output' paragraph.

Maybe njn has more suggestions.

I suppose the rule update is going to fix this problem, but I guess the underlying problem of the infinite allocations is still there.

Comment 25

[Nicholas Nethercote [inactive]]

Those DMD reports are output intended for human consumption. The link Marco gave is the right one for the docs.

dmd-1511477771-19268.txt is the file for the process that has by far the highest memory usage, so focus on that.

Heap block record 1 is lacking stacks. DMD doesn't record stacks for all allocations by default to run faster, but normally the amount of memory covered without stack is less. Marco, you could re-run with --stacks=full to get full coverage.

Heap block record 2 involves URL cloning of some kind. There are 748,281 allocations made with that stack trace, all of which are 256 bytes in size.

Heap block records 4 and 5 involve some kind of strings being created under HttpChannelParent::DoAsyncOpen(). They are all 144 bytes.

Other heap blocks records show similar sorts of things.

To properly understand what this output is telling you will likely require someone who knows the networking/URL code.

Comment 26

[Trevor Rowbotham [:rowbot]]

Possible relation to Bug 1418815? Marco, what happens if you disable the content sandbox?

Comment 27

[Marco Castelluccio [:marco]]

(In reply to Trevor Rowbotham from comment #26)
> Possible relation to Bug 1418815? Marco, what happens if you disable the
> content sandbox?

I can't reproduce the bug anymore with the latest version of the addon, probably because the rule was removed.
Is there a way to manually add the rule? Or can you point me to the old version of the addon so I can install it?

Comment 28

[Trevor Rowbotham [:rowbot]]

You can find a list of older versions here: https://addons.mozilla.org/en-US/firefox/addon/https-everywhere/versions/

Comment 29

[William Budington]

Marco, are you still seeing this?

Comment 30

[Mike Taylor [:miketaylr]]

ni? Marco for Comment #29.

Comment 31

[Marco Castelluccio [:marco]]

I can still reproduce with version 2017.11.21, no matter the value of security.sandbox.content.level.

Comment 32

[Andrew Overholt [:overholt]]

mccr8 suggested the Necko team take a look. Jason, can you or someone on your team take a look, please?

Comment 33

[Jason Duell]

Valentin, can you take a look at this? It looks like HTTPS Everywhere is interacting with redirects to create a memory leak, and it's at least in part related to URL code (see comment 25)

Comment 34

[Valentin Gosu [:valentin] (he/him)]

Marco, I can no longer reproduce the bug, in either beta or nightly (with ver 2017.11.21 of the addon)
It might be that the website has changed. Can you check to confirm? Or do you know of any other website where we can reproduce this bug?

Comment 35

[Marco Castelluccio [:marco]]

I've replied on IRC, but I'll reply here too.

(In reply to Valentin Gosu [:valentin] from comment #34)
> Marco, I can no longer reproduce the bug, in either beta or nightly (with
> ver 2017.11.21 of the addon)
> It might be that the website has changed. Can you check to confirm? Or do
> you know of any other website where we can reproduce this bug?

It sounds like this isn't easy to reproduce, Blake wasn't able to reproduce either.
I can still reproduce though and I can help you should you need more information.
I can only reproduce on psarips.com.

If I open the Network panel in the developer tools, I see Firefox is making an infinite number of requests to https://psarips.com (which replies with 301).

Comment 36

[Valentin Gosu [:valentin] (he/him)]

As Marco said on IRC, the reason he can reproduce it is probably that he visited the website before they switched from HTTPS to HTTP. It would be interesting if that is so. I'll try to create a testcase for that, but probably will not get to it this week.

Comment 37

[Mike Taylor [:miketaylr]]

(marking as fix-optional for 58, given the general difficulty in reproducing here, and time left in the 58 window)

Comment 38

[Liz Henry (:lizzard) (relman/hg->git project)]

Too late to fix in 59, but we can still take a patch in Nightly if the site is affected in 61.

Comment 39

[Andrei Purice]

Marking this as Resolved > Worksforme since it cannot be reproduced anymore on the latest versions of Firefox Nightly 96.0a1 (2021-11-29), beta 95.0b12 or release 94.0.2 on Windows 10. Tested with https://psarips.top/ since http://psarips.com is not available anymore.
If anyone can still reproduce this issue either re-open this ticket or file a new one.