1453735 - Firefox has hundreds of IPC shared memory files open (as file descriptors, not memory mappings)

Status: RESOLVED WORKSFORME

(Core :: IPC, defect, P3)

Description

[Germano Massullo]

Attachment: lsof. against Firefox

User Agent: Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0
Build ID: 20180327091359

Steps to reproduce:

OS: Fedora 27
Firefox 59.0.2

While investigating on bugreports concerning graphical artifacts
https://bugzilla.mozilla.org/show_bug.cgi?id=1421353
https://bugs.freedesktop.org/show_bug.cgi?id=104216
I found out that Firefox opens hundreds of /dev/shm/org.chromium.* files
that is one of the causes that lead the system to exceed max ulimit -n

See attachment

Comment 1

[Jed Davis [:jld] ⟨⏰|UTC-7⟩ ⟦he/him⟧]

So here's something interesting: normally, I don't see Firefox having *any* IPC shared memory files open as actual file descriptors.  ipc::Shmem maps the file in both processes (it's assocated with an IPDL actor) and then closes the file, and mappings don't count against RLIMIT_NOFILE.  (The ipc::Shmem object itself is “sent” to control which process is allowed to access the memory region, using mprotect or similar to trap data races.)  Note that lsof lists memory mappings as well; they can be distinguished by the lack of a number in the "FD" column.

However, the CrossProcess{Mutex,Semaphore} classes appear to leave the file open, and other code that uses SharedMemoryBasic (or base::SharedMemory) directly might also.  I can cause a few fds to appear by flipping the pref layers.progressive-paint (I don't actually know what that does; I found it by code inspection), but the attachment shows hundreds of fds.

Maybe there's some other graphics feature doing this?  And maybe it's a simple matter of putting a SharedMemoryBasic::Close(false) in the right place to not use so many fds?

Something that might help debug this: "stat -L /proc/PID/fd/FD" (where PID and FD are numbers) will print info about the actual file, even if it's unlinked.  If the size is something like 40 or 48, that's probably CrossProcessSemaphore or CrossProcessMutex; if it's larger, it's probably something else.

Cc'ing :jesup, who's also mentioned encountering problems with the 1k fd limit.

Comment 2

[Germano Massullo]

Attachment: lsof _p and stat _L

(In reply to Jed Davis [:jld] (⏰UTC-6) from comment #1)
> Something that might help debug this: "stat -L /proc/PID/fd/FD" (where PID
> and FD are numbers)

See new attachment :-)

Comment 3

[Jed Davis [:jld] ⟨⏰|UTC-7⟩ ⟦he/him⟧]

Not that I ought to be looking at work on the weekend, but: those are probably semaphores (glibc sem_t on 64-bit has size 32 & alignment 8; CrossProcessSemaphore adds an int32_t refcount; 36 aligned up to 8 is 40).  And they should be cleaning up their file descriptors courtesy of bug 1416726[1][2] — the only user of CrossProcessSemaphore known to searchfox is the TextureClient locking — but somehow that's not working.

(Also, I just noticed the sizes were already in the lsof output under the SIZE/OFF column; sorry about that.)

[1] https://searchfox.org/mozilla-central/rev/fca4426325624fecbd493c31389721513fc49fef/gfx/layers/client/TextureClient.cpp#1759
[2] https://searchfox.org/mozilla-central/rev/fca4426325624fecbd493c31389721513fc49fef/ipc/glue/CrossProcessSemaphore_posix.cpp#79

Comment 4

[Hani Yacoub, Desktop QA]

Couldn't reproduce it. I'll set this bug to "IPC" component.

Comment 5

[Germano Massullo]

If you need me to run any kind of tests, please let me know

Comment 6

[orlien.76]

Hi, it seems that firefox isn't releasing properly shm memory

Current firefox main process is still holding thousands of invalid handles like this:
>cat /proc/2453/maps | grep org.chromium | grep deleted
>[..]
>7f27f8c03000-7f27f8c04000 r--s 00000000 00:17 3544                       /dev/shm/org.chromium.CDuRkQ (deleted)
>7f27fbe00000-7f27fbe01000 r--p 00000000 00:17 12                         /dev/shm/org.chromium.8e9BQ5 (deleted)
>7f27fbe02000-7f27fbe03000 r--s 00000000 00:17 3941                       /dev/shm/org.chromium.JuAsDI (deleted)
>7f27fbe03000-7f27fbe04000 r--s 00000000 00:17 3450                       /dev/shm/org.chromium.a8krjl (deleted)
>7f27fbe04000-7f27fbe05000 r--s 00000000 00:17 3972                       /dev/shm/org.chromium.4oscoV (deleted)
>7f28170d9000-7f28170da000 r--p 00000000 00:17 3                          /dev/shm/org.chromium.iuXKrq (deleted)

>cat /proc/2453/maps | grep org.chromium | grep deleted | wc -l
>18499

>ls -la /dev/shm
>total 0
>drwxrwxrwt  2 root root   40 Jul 19 14:22 .
>drwxr-xr-x 20 root root 4.5K Jul 19 10:30 ..

>cat /proc/2453/status
>[..]
>VmPeak:	 9019164 kB
>VmSize:	 8822280 kB
>VmLck:	       0 kB
>VmPin:	       0 kB
>VmHWM:	 5666828 kB
>VmRSS:	 5535504 kB
>RssAnon:	  673964 kB
>RssFile:	  189676 kB
>RssShmem:	 4671864 kB
>[..]

It is not a cause of crash, but keeping so much deleted memory referenced seems strange (and is displaying high memory usage in tasks manager)

User agent: Mozilla/5.0 (X11; Linux x86_64; rv:63.0) Gecko/20100101 Firefox/63.0
Firefox version: 63.01a (2018-07-02) (64-bit) (using webrender)
OS: Ubuntu 18.04

Please let me know if you need more informations

Comment 7

[Jed Davis [:jld] ⟨⏰|UTC-7⟩ ⟦he/him⟧]

(In reply to orlien.76 from comment #6)
> Current firefox main process is still holding thousands of invalid handles

They're perfectly valid, just not referenced from the filesystem.  Opening and unlinking files is a *very* old Unix feature, for getting a temporary file that's deleted from storage when it's closed.

> >cat /proc/2453/maps | grep org.chromium | grep deleted | wc -l
> >18499

But that number is a lot higher than it should be…

> >cat /proc/2453/status
> >[..]
> >VmPeak:	 9019164 kB
> >VmSize:	 8822280 kB
> >VmLck:	       0 kB
> >VmPin:	       0 kB
> >VmHWM:	 5666828 kB
> >VmRSS:	 5535504 kB
> >RssAnon:	  673964 kB
> >RssFile:	  189676 kB
> >RssShmem:	 4671864 kB

And that looks like a memory leak.  Could you file a separate bug for this?

Comment 8

[orlien.76]

(In reply to Jed Davis [:jld] (⏰UTC-6) from comment #7)
> And that looks like a memory leak.  Could you file a separate bug for this?

Sure, there it is: https://bugzilla.mozilla.org/show_bug.cgi?id=1477186

Comment 9

[Jim Mathies [:jimm]]

Germano, are you still seeing this?

Comment 10

[Germano Massullo]

Attachment: oct_2018_firefox_test.txt

Comment 11

[Germano Massullo]

(In reply to Jim Mathies [:jimm] from comment #9)
> Germano, are you still seeing this?

Well, consider that actually the amount of tabs I am using if a lot less than in past.
Anyway I runned again the lsof -p and stat -L and attached them here

Comment 12

[Jed Davis [:jld] ⟨⏰|UTC-7⟩ ⟦he/him⟧]

I didn't see evidence of fd leaks in attachment 9016966 [details].

Comment 13

[Germano Massullo]

Attachment: nov_2018.txt

Today I started experiencing GUI glitches, so I immediately have taken some infos by running

# lsof -p 18193 && lsof -p 8916 && lsof -p 8212 && lsof -p 8074 && lsof -p 7994 && lsof -p 7873 
# stat -L /proc/18193/fd/* && stat -L /proc/8916/fd/* && stat -L /proc/8212/fd/* && stat -L /proc/8074/fd/* && stat -L /proc/7994/fd/* && stat -L /proc/7873/fd/*

$ ulimit
returns "unlimited"

Could you please check if everything is okay? In any case what should I check in order to do that on myself next time, so I don't have to ask you everytime?
Thank you for your time