625480 - memory leak when using PAC file (regex ?)

Status: RESOLVED FIXED

(Core :: JavaScript Engine, defect)

Description

[Jo Hermans]

Since bug 605662 (enable per-compartment GC) is checked in, I noticed a serious memory leak in the nightly builds when I'm using a PAC file. Since a PAC file uses JavaScript, I'm thinking that its compartment is never gc'ed or something similar.

If I'm at home, I don't have access to the proxy-server, and I'm using direct connections. My memory usage (win32/workingset is about:memory) is more or less stable around 200-240 MB, and I can see memory being released all the time.

But when I'm at work, I have to use a proxy server thru the PAC-file (which is pretty large).  My memory usage is growing rapidly, and can reach 500 MB after 20 minutes or so. It never goes down. The malloc/allocated counter doesn't change much, it's mostly the win32/* and js/* counters.

I know it's pretty vague, but I tried to eliminate all other possibilities. I've run in safe mode, I've created a new profile, etc ...  I'm using the exact same browser and profile (and hardware of course) in both environments. The only difference is that I'm using a PAC file at the office. Note that I'm not changing my preferences at all, at home I'm not able to load the PAC-file, and thus I'm falling back to direct connections.

Since there is no way currently to see better memory reports detailed per compartment or tab (I've seen the bugs), what do you suggest that I should use to debug this further?

Comment 1

[Andreas Gal :gal]

We should figure out whats going on here.

Comment 2

[Jo Hermans]

I'll ask some help from the Nightly-testers mailinglist, to see if someone else can reproduce. I'll also attach the PAC file in a moment.

Comment 3

[Jo Hermans]

Attachment: PAC file

Comment 4

[David Mandelin [:dmandelin]]

Not sure exactly what those functions do, but it looks like this could be creating a zillion regexps and compiling them to native code. It could be a leak, but maybe we just need a discard algorithm like we have for methodjit code.

Comment 5

[David Mandelin [:dmandelin]]

(In reply to comment #4)
> Not sure exactly what those functions do, but it looks like this could be
> creating a zillion regexps and compiling them to native code. It could be a
> leak, but maybe we just need a discard algorithm like we have for methodjit
> code.

I suppose if it started only after compartment GC landed, then it more likely is a leak.

Comment 6

[Gregor Wagner [:gwagner]]

Does it help if you set javascript.options.mem.gc_per_compartment in about:config to false?

Comment 7

[Jo Hermans]

No.

I'm testing with a group of 15 URLS that I load as a group.

               gc_per_compartment  gc_per_compartment
               == true             == false
PAC file       427.515.904         464.564.22
without PAC    277.647.36          307.945.47

As you can see that PAC-file uses a lot more memory, but changing javascript.options.mem.gc_per_compartment had not much influence (interesting, setting it to true seems to use less memory).

Comment 8

[Jo Hermans]

(In reply to comment #5)
> (In reply to comment #4)
> > Not sure exactly what those functions do, but it looks like this could be
> > creating a zillion regexps and compiling them to native code. It could be a
> > leak, but maybe we just need a discard algorithm like we have for methodjit
> > code.
> 
> I suppose if it started only after compartment GC landed, then it more likely
> is a leak.

maybe bug 587288 ?

Comment 9

[David Mandelin [:dmandelin]]

(In reply to comment #8)
> (In reply to comment #5)
> > I suppose if it started only after compartment GC landed, then it more likely
> > is a leak.
> 
> maybe bug 587288 ?

That one landed to TM, so you could try the latest nightly and see if the problem is still there (unless you're already on it).

Comment 10

[Jo Hermans]

(In reply to comment #9)
> That one landed to TM, so you could try the latest nightly and see if the
> problem is still there (unless you're already on it).

I am running the latest nightlies. I was thinking that the problem was *caused* by bug 587288, because it also appeared in the same build as I first saw it. And it mentions both regex and compartments. But that is just a wild guess. I'm starting to think it's relates to one of the YARR/regex changes. Apparently, my PAC file uses a lot of regex (I didn't wrote that monstrosity), which make the problem easy to detect.

I'll try to repeat the problem without a PAC file, just a manually configured proxy server, but it will take some time, as this setup doesn't work through the VPN at home. I'll have to go to the office.

Comment 11

[Jo Hermans]

Bingo. When  I configure a manual proxy (which is the same one as the bottom rule of attachment 503618 [details], and all test URLS were using that one anyway), I fall back to the same memory usage as without a proxy server. It seems indeed that it's caused by the regular expressions.

Chris, do you have any idea ?

               gc_per_compartment  gc_per_compartment
               == true             == false
PAC file       436,064,256         445,788,160
without PAC    274,132,99          269,496,320
manual proxy   272,568,320         272,441,344


Note that I repeated the tests several times and took an average of 3 requests requests, because the results can vary a lot (+- 30 MB easily), due to the large amount of simultaneous requests. 

PS : I forgot to mention that I didn't see this behavior in earlier nightlies, or in Firefox 3.6.*. It is a regression.

Comment 12

[Jo Hermans]

The list of test urls is this one :

http://slashdot.org/
http://www.theregister.co.uk/
http://arstechnica.com/
http://www.deredactie.be/cm/vrtnieuws
http://www.gva.be/
http://www.demorgen.be/
http://news.google.be/news?ned=nl_be
http://news.google.be/news?ned=es_co
http://news.google.be/news?hl=en&ned=us&q=alcatel&btnG=Search+News
http://news.google.be/news?hl=en&ned=us&ie=UTF-8&q=colombia&btnG=Search+News
http://www.lapatria.com/
http://www.cambio.com.co/archivo/buscar?producto=cambio&orden=reciente
http://twitter.com/piedadcordoba
http://www.buienradar.nl/h.aspx?jaar=-3&soort=loop1uur1x1
http://www.google.com/finance?q=EPA:ALU

Comment 13

[Jo Hermans]

The memory leak seems to be gone, now that bug 626118 is fixed.

Running with a PAC file still consumes a lot more memory than without one (due to the complexity of attachment 503618 [details]). But during normal surfing (not loading all at once), it doesn't hurt anymore. And the memory is reclaimed, which was this bug.

Comment 14

[h6.msangel]

This is available in 43-44 version of firefox. Without regexps for some load browser consume 500M, with pac-file with 400 regexps with same load it consume 1.6G

Comment 15

[h6.msangel]

Veryfy/reopen please