Open Bug 1414091 Opened 4 years ago Updated 4 years ago

Massive leak while loading psarips.com

Categories

(Core :: General, defect)

defect
Not set
major

Tracking

()

Tracking Status
firefox-esr52 --- unaffected
firefox57 --- unaffected
firefox58 --- fix-optional
firefox59 --- wontfix
firefox60 --- ?
firefox61 --- ?

People

(Reporter: marco, Unassigned, NeedInfo)

References

()

Details

(Keywords: regression, Whiteboard: [MemShrink:P2])

Attachments

(2 files)

The page never finishes loading and Firefox keeps allocating memory indefinitely. Most of the memory is never freed and is under "heap unclassified":
4,107.98 MB (100.0%) -- explicit
├──3,424.01 MB (83.35%) ── heap-unclassified

Profile while the page is loading: https://perfht.ml/2ipc7pL.

mozregression is pointing me to bug 1406212, but I'm not sure it's a legit regression range.

I can't reproduce this in a clean profile.
Keywords: regression
I can only reproduce with HTTPS Everywhere enabled.
Flags: needinfo?(mrbkap)
This is probably a bug in HTTPS everywhere. If bug 1406212 is truly at fault, then that would seem to indicate a bug in their e10s compatibility code. Looking at the extension's page on AMO [1], it seems like they released a WebExtension on October 30. Do you know what version you were testing with in comment 0?

[1] https://addons.mozilla.org/en-US/firefox/addon/https-everywhere/
Flags: needinfo?(mrbkap) → needinfo?(mcastelluccio)
Oh, psarips.com also redirects https back to http, so that seems like a case where HTTPS everywhere might get confused.
(In reply to Blake Kaplan (:mrbkap) from comment #3)
> This is probably a bug in HTTPS everywhere. If bug 1406212 is truly at
> fault, then that would seem to indicate a bug in their e10s compatibility
> code. Looking at the extension's page on AMO [1], it seems like they
> released a WebExtension on October 30. Do you know what version you were
> testing with in comment 0?
> 
> [1] https://addons.mozilla.org/en-US/firefox/addon/https-everywhere/

Notice I'm testing with mozregression and my profile, so this can't be a regression in HTTPS Everywhere (the exact same version of HTTPS Everywhere is working before bug 1406212 and is not working after bug 1406212).
e10s is enabled for me both before and after bug 1406212.
Flags: needinfo?(mcastelluccio)
(In reply to Marco Castelluccio [:marco] from comment #5)
> Notice I'm testing with mozregression and my profile, so this can't be a
> regression in HTTPS Everywhere (the exact same version of HTTPS Everywhere
> is working before bug 1406212 and is not working after bug 1406212).
> e10s is enabled for me both before and after bug 1406212.

This could have been a latent bug in HTTPS everywhere that was exposed by the removal of the e10srollout addon.

I've tried reproducing this on Linux (debug+opt) and OSX (nightly) without success today. Installing HTTPS Everywhere and navigating to psarips.com finishes loading and doesn't appear to use more memory than any of my other tabs. What versions of HTTPS Everywhere and Firefox are you testing with?
Flags: needinfo?(mcastelluccio)
(In reply to Blake Kaplan (:mrbkap) from comment #6)
> (In reply to Marco Castelluccio [:marco] from comment #5)
> > Notice I'm testing with mozregression and my profile, so this can't be a
> > regression in HTTPS Everywhere (the exact same version of HTTPS Everywhere
> > is working before bug 1406212 and is not working after bug 1406212).
> > e10s is enabled for me both before and after bug 1406212.
> 
> This could have been a latent bug in HTTPS everywhere that was exposed by
> the removal of the e10srollout addon.
> 
> I've tried reproducing this on Linux (debug+opt) and OSX (nightly) without
> success today. Installing HTTPS Everywhere and navigating to psarips.com
> finishes loading and doesn't appear to use more memory than any of my other
> tabs. What versions of HTTPS Everywhere and Firefox are you testing with?

Latest nightly, HTTPS Everywhere 2017.10.4.1337 (it looks like it's the latest version, I can't update it).
Flags: needinfo?(mcastelluccio)
(In reply to Marco Castelluccio [:marco] from comment #7)
> (In reply to Blake Kaplan (:mrbkap) from comment #6)
> > (In reply to Marco Castelluccio [:marco] from comment #5)
> > > Notice I'm testing with mozregression and my profile, so this can't be a
> > > regression in HTTPS Everywhere (the exact same version of HTTPS Everywhere
> > > is working before bug 1406212 and is not working after bug 1406212).
> > > e10s is enabled for me both before and after bug 1406212.
> > 
> > This could have been a latent bug in HTTPS everywhere that was exposed by
> > the removal of the e10srollout addon.
> > 
> > I've tried reproducing this on Linux (debug+opt) and OSX (nightly) without
> > success today. Installing HTTPS Everywhere and navigating to psarips.com
> > finishes loading and doesn't appear to use more memory than any of my other
> > tabs. What versions of HTTPS Everywhere and Firefox are you testing with?
> 
> Latest nightly, HTTPS Everywhere 2017.10.4.1337 (it looks like it's the
> latest version, I can't update it).

The latest version of HTTPS Everywhere is 2017.10.30, which is available through AMO[1]. I assume you have an earlier experimental version of HTTPS Everywhere installed, which likely came from [2], based on the version number.

[1] https://addons.mozilla.org/en-US/firefox/addon/https-everywhere/
[2] https://www.eff.org/files/https-everywhere-test/index.html.
(In reply to Trevor Rowbotham from comment #8)
> (In reply to Marco Castelluccio [:marco] from comment #7)
> > (In reply to Blake Kaplan (:mrbkap) from comment #6)
> > > (In reply to Marco Castelluccio [:marco] from comment #5)
> > > > Notice I'm testing with mozregression and my profile, so this can't be a
> > > > regression in HTTPS Everywhere (the exact same version of HTTPS Everywhere
> > > > is working before bug 1406212 and is not working after bug 1406212).
> > > > e10s is enabled for me both before and after bug 1406212.
> > > 
> > > This could have been a latent bug in HTTPS everywhere that was exposed by
> > > the removal of the e10srollout addon.
> > > 
> > > I've tried reproducing this on Linux (debug+opt) and OSX (nightly) without
> > > success today. Installing HTTPS Everywhere and navigating to psarips.com
> > > finishes loading and doesn't appear to use more memory than any of my other
> > > tabs. What versions of HTTPS Everywhere and Firefox are you testing with?
> > 
> > Latest nightly, HTTPS Everywhere 2017.10.4.1337 (it looks like it's the
> > latest version, I can't update it).
> 
> The latest version of HTTPS Everywhere is 2017.10.30, which is available
> through AMO[1]. I assume you have an earlier experimental version of HTTPS
> Everywhere installed, which likely came from [2], based on the version
> number.
> 
> [1] https://addons.mozilla.org/en-US/firefox/addon/https-everywhere/
> [2] https://www.eff.org/files/https-everywhere-test/index.html.

Thanks! I've installed 2017.10.30. I can still reproduce this bug.
(In reply to Marco Castelluccio [:marco] from comment #9)
> Thanks! I've installed 2017.10.30. I can still reproduce this bug.

Can you attach your about:support here? In theory we should now be using the exact same configuration/version. Also, is navigating to "psarips.com" sufficient or are there more STR?
Flags: needinfo?(mcastelluccio)
Attached file aboutsupport.tar.xz
(In reply to Blake Kaplan (:mrbkap) from comment #10)
> (In reply to Marco Castelluccio [:marco] from comment #9)
> > Thanks! I've installed 2017.10.30. I can still reproduce this bug.
> 
> Can you attach your about:support here?

Attached!

> In theory we should now be using the
> exact same configuration/version. Also, is navigating to "psarips.com"
> sufficient or are there more STR?

Just navigating to "psarips.com" is sufficient.
There's one thing that comes to mind that might be different in our profiles. This site used to be https a while ago, so I have https://psarips.com/ in my history.
Flags: needinfo?(mcastelluccio)
Did you get a chance to dig anymore here, Blake?
Flags: needinfo?(mrbkap)
I've tried reproducing this again without any luck. One thing I noticed in Marco's about:support is that he's disallowing third-party cookies (network.cookie.cookieBehavior=1), which might affect things here.

Two more questions: Does having HTTPS Everywhere installed but disabled fix the problem here? If you clear all cookies for psarips.com (but have HTTPS Everywhere installed and enabled) does that change anything?
Flags: needinfo?(mrbkap) → needinfo?(mcastelluccio)
(In reply to Blake Kaplan (:mrbkap) from comment #13)
> I've tried reproducing this again without any luck. One thing I noticed in
> Marco's about:support is that he's disallowing third-party cookies
> (network.cookie.cookieBehavior=1), which might affect things here.
> 
> Two more questions: Does having HTTPS Everywhere installed but disabled fix
> the problem here? If you clear all cookies for psarips.com (but have HTTPS
> Everywhere installed and enabled) does that change anything?

I can reproduce after enabling third-party cookies too and after removing psarips.com cookies and after removing psarips.com from history.

I can't reproduce if I disable HTTPS Everywhere (either from about:addons or from the HTTPS Everywhere popup).

Let me know if you need more info.
Flags: needinfo?(mcastelluccio)
Attached file dmd_reports.tar.xz
Here are DMD reports, maybe understanding where the heap-unclassified memory is coming from can help.
Looking at the DMD logs shows us creating lots of objects to support HTTP redirects, so there's no smoking gun, but we're almost certainly redirecting thanks to the extension.

This seems like it's going to be a bug in HTTPS everywhere. I don't know if we have a contact over there to help us debug this. One thing to try would be to turn on HTTPS Everywhere logging [1] to see if that gives any clues as to what's happening.

I do see that HTTPS everywhere has a rule for psarips.com, so another possibility would be if their redirect loop detection is somehow broken in Firefox but not Chrome. Maybe our requestIds in onBeforeRequest change for redirects?

I have no idea why your setup would be different from mine :-/

[1] https://github.com/EFForg/https-everywhere/blob/f09fb11a607316333b1a298e8f34bcca3ee27f9d/chromium/util.js#L13
Hey Tom,
It seems the root cause is in the add-on HTTPS Everywhere itself.
Do you know Bill at EFF has a Bugzilla account that we can needinfo him?  Or anyone else?
Flags: needinfo?(tom)
Flags: needinfo?(tom) → needinfo?(bill)
Thank you, I see this and I'll look into it.  If I can reproduce it, I hope a fix to be incorporated into our Wednesday release of the extension.

I've mirrored this issue on our own tracker: https://github.com/EFForg/https-everywhere/issues/13876
Flags: needinfo?(bill)
I can not reproduce this error on Linux x64 with either Firefox unbranded stable (57.0) or beta (58.0b3) in a new profile.  Marco, do you have other extensions enabled?
(In reply to William Budington from comment #19)
> I can not reproduce this error on Linux x64 with either Firefox unbranded
> stable (57.0) or beta (58.0b3) in a new profile.  Marco, do you have other
> extensions enabled?

I can reproduce with only HTTPS Everywhere enabled.
Marco, what platform are you using when you encounter this bug?
(In reply to William Budington from comment #21)
> Marco, what platform are you using when you encounter this bug?

Linux 64-bit (in particular, Ubuntu 17.10).
It's possible that this bug is due to particular conditions of my profile, so it might be very hard to reproduce for you.
I can help test debugging builds with additional logging, if you have any.
Even when flipping all the same prefs you have in `about:support` and installing all the active addons, I still can't reproduce this.

I'm unsure how to parse the DMD reports.  What tools are useful here?

I've removed this individual ruleset in https://github.com/EFForg/https-everywhere/commit/cf1c66c56923c31872aa5ef5b23f909fc6f4265f.  We're going to release a new version tomorrow, which will deliver updated rulesets with this change.  This should fix the immediate problem, if this is caused by the HTTPS downgrade redirect.  However, the extension should account for this - not sure why we're seeing problems in your particular instance.
(In reply to William Budington from comment #23)
> Even when flipping all the same prefs you have in `about:support` and
> installing all the active addons, I still can't reproduce this.
> 
> I'm unsure how to parse the DMD reports.  What tools are useful here?
> 
> I've removed this individual ruleset in
> https://github.com/EFForg/https-everywhere/commit/
> cf1c66c56923c31872aa5ef5b23f909fc6f4265f.  We're going to release a new
> version tomorrow, which will deliver updated rulesets with this change. 
> This should fix the immediate problem, if this is caused by the HTTPS
> downgrade redirect.  However, the extension should account for this - not
> sure why we're seeing problems in your particular instance.

There's some documentation at https://developer.mozilla.org/en-US/docs/Mozilla/Performance/DMD under the '"Dark matter" mode output' paragraph.

Maybe njn has more suggestions.

I suppose the rule update is going to fix this problem, but I guess the underlying problem of the infinite allocations is still there.
Flags: needinfo?(n.nethercote)
Those DMD reports are output intended for human consumption. The link Marco gave is the right one for the docs.

dmd-1511477771-19268.txt is the file for the process that has by far the highest memory usage, so focus on that.

Heap block record 1 is lacking stacks. DMD doesn't record stacks for all allocations by default to run faster, but normally the amount of memory covered without stack is less. Marco, you could re-run with --stacks=full to get full coverage.

Heap block record 2 involves URL cloning of some kind. There are 748,281 allocations made with that stack trace, all of which are 256 bytes in size.

Heap block records 4 and 5 involve some kind of strings being created under HttpChannelParent::DoAsyncOpen(). They are all 144 bytes.

Other heap blocks records show similar sorts of things.

To properly understand what this output is telling you will likely require someone who knows the networking/URL code.
Flags: needinfo?(n.nethercote)
Possible relation to Bug 1418815? Marco, what happens if you disable the content sandbox?
(In reply to Trevor Rowbotham from comment #26)
> Possible relation to Bug 1418815? Marco, what happens if you disable the
> content sandbox?

I can't reproduce the bug anymore with the latest version of the addon, probably because the rule was removed.
Is there a way to manually add the rule? Or can you point me to the old version of the addon so I can install it?
Marco, are you still seeing this?
ni? Marco for Comment #29.
Flags: needinfo?(mcastelluccio)
I can still reproduce with version 2017.11.21, no matter the value of security.sandbox.content.level.
Flags: needinfo?(mcastelluccio)
Whiteboard: [MemShrink]
mccr8 suggested the Necko team take a look. Jason, can you or someone on your team take a look, please?
Flags: needinfo?(jduell.mcbugs)
Severity: normal → major
Valentin, can you take a look at this? It looks like HTTPS Everywhere is interacting with redirects to create a memory leak, and it's at least in part related to URL code (see comment 25)
Flags: needinfo?(jduell.mcbugs) → needinfo?(valentin.gosu)
Marco, I can no longer reproduce the bug, in either beta or nightly (with ver 2017.11.21 of the addon)
It might be that the website has changed. Can you check to confirm? Or do you know of any other website where we can reproduce this bug?
Flags: needinfo?(valentin.gosu) → needinfo?(mcastelluccio)
I've replied on IRC, but I'll reply here too.

(In reply to Valentin Gosu [:valentin] from comment #34)
> Marco, I can no longer reproduce the bug, in either beta or nightly (with
> ver 2017.11.21 of the addon)
> It might be that the website has changed. Can you check to confirm? Or do
> you know of any other website where we can reproduce this bug?

It sounds like this isn't easy to reproduce, Blake wasn't able to reproduce either.
I can still reproduce though and I can help you should you need more information.
I can only reproduce on psarips.com.

If I open the Network panel in the developer tools, I see Firefox is making an infinite number of requests to https://psarips.com (which replies with 301).
Flags: needinfo?(mcastelluccio)
Whiteboard: [MemShrink] → [MemShrink:P2]
As Marco said on IRC, the reason he can reproduce it is probably that he visited the website before they switched from HTTPS to HTTP. It would be interesting if that is so. I'll try to create a testcase for that, but probably will not get to it this week.
Flags: needinfo?(valentin.gosu)
(marking as fix-optional for 58, given the general difficulty in reproducing here, and time left in the 58 window)
Too late to fix in 59, but we can still take a patch in Nightly if the site is affected in 61.
You need to log in before you can comment on or make changes to this bug.