1245239 - Shared memory allocations can SIGBUS if there's not enough space in /dev/shm

Status: RESOLVED FIXED

(Core :: IPC, defect, P3)

Description

[Jeff Muizelaar [:jrmuizel]]

We should try to fail early here instead of getting SIGBUSs when we try to access the pages.

Comment 1

[Jeff Muizelaar [:jrmuizel]]

Attachment: Fail to allocate shared memory if there's not enough space.

Comment 2

[Jed Davis [:jld] ⟨⏰|UTC-7⟩ ⟦he/him⟧]

Comment on attachment 8715007 [details] [diff] [review]
Fail to allocate shared memory if there's not enough space.

Review of attachment 8715007 [details] [diff] [review]:
-----------------------------------------------------------------

::: ipc/chromium/src/base/shared_memory_posix.cc
@@ +221,3 @@
>        if (ftruncate(fileno(fp), size) != 0)
>          return false;
>        if (fseeko(fp, size, SEEK_SET) != 0)

It looks like nothing cares about where the file offset winds up after this, so it shouldn't matter that the old code would seek to the end and the new code doesn't.

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

I don't know if landing this was blocked on fixing the shm size in TaskCluster -- but that's fixed now, so I'd love to see this land.

Comment 4

[Jeff Muizelaar [:jrmuizel]]

It looks like this causes a disaster on try:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=e9f94762ef36&selectedJob=16295556

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

Indeed, that's pretty bad.  At a guess, the fallocate is claiming more RAM than firefox would otherwise have used, and causing immediate failure in low-memory situations?

Comment 6

[Jed Davis [:jld] ⟨⏰|UTC-7⟩ ⟦he/him⟧]

Comment on attachment 8715007 [details] [diff] [review]
Fail to allocate shared memory if there's not enough space.

Review of attachment 8715007 [details] [diff] [review]:
-----------------------------------------------------------------

I found something I should have noticed, but didn't, when I reviewed this the first time.

Also, as I mentioned in bug 1338771 comment #10, callers probably aren't expecting this to be a fallible allocation.  It seems better to just crash here with NS_ABORT_OOM (or similar) and let callers that can meaningfully use a fallible version (if any) opt in mozilla::fallible_t, assuming we can use that in this part of the codebase.

::: ipc/chromium/src/base/shared_memory_posix.cc
@@ +214,5 @@
> +      // Using posix_fallocate will ensure that there's
> +      // actually space for this file. Otherwise we end
> +      // up with a sparse file that can give SIGBUS if
> +      // we run out of space while writing to it.
> +      if (posix_fallocate(fileno(fp), 0, size) != 0)

This needs to handle EINTR; that might be why this caused failures on Try.

Comment 7

[Jed Davis [:jld] ⟨⏰|UTC-7⟩ ⟦he/him⟧]

Attachment: bug1245239-sigbus-wip0.diff

I added some HANDLE_EINTR to :jrmuizel's patch and pushed it to Try, but it causes some “Assertion failure: mBuffer” crashes that look related:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=c11a0b6b5c504f0fdaf0442108a6dea5c1aec2ae

These tests aren't normally crashing with SIGBUS, so something odd is going here.

(I decided not to get into fallibility here, because I looked through the callers and they're mostly not ignoring the failure case, and trying to change this API everywhere would be nontrivial.)

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+1]]

We got a bug filed against Marionette which I just duped. Given that the reporter added some more interesting details about this particular crash, I wanted to reference those here:

(In reply to gil from bug 1567168 comment #9)

For future reference:

The problem was the shm-size. The problem persisted because my local test used docker-compose and did not enable shm-size=2g. Furthermore, the shm-size solution couldn't work in production, because we use Kubernetes to deploy, and Kubernetes doesn't yet have shm-size support (https://github.com/kubernetes/kubernetes/issues/28272).

Why did this happen in the move from FF 67 to FF 68? Because I did disable SHM usage in FF by disabling E10s using user_pref("browser.tabs.remote.autostart", false);, but as this bug (https://bugzilla.mozilla.org/show_bug.cgi?id=1548941) says, support for this flag was removed in FF 68.

The workaround (given that K8s doesn't support changing the shm-size)? There's still an environment variable (MOZ_FORCE_DISABLE_E10S=true) that disables E10s. Not sure when that support will die, but by then maybe K8s will support hanging the shm-size.

Comment 10

[bugzilla.mozilla.org]

This is mostly FYI, along with 'usage case issues' and resolution.

• Debian Buster
• 68.2.0esr
• 4G ram

Prior to the recent buster upgrade from 60.x-ESR, I noticed endless tab crashes. By 'endless', I mean all of the time. Every minute or so when browsing, and with some tabs consistently crashing repeatedly. Yet these crashes made no sense, as they were not consistent, and all of my debugging resulted in somewhat random sigs such as:

• libyuv::ARGBSetRow_X86
• sse2::memset32
• __new_sem_init
• __memcpy_sse2_unaligned_erms

And others... so at first I thought 'memory', yet out of 4G free? I had a mere 1.4G in use along with buff/cache, and had more than 2G of free ram. After a bit of googling, I looked at /dev/shm, yet it often showed 30% usage, a few hundred M out of 800M (20% of system ram).

Yet the crashes persisted. I even did watch -n.1 df -H, just to see what was happening in /dev/shm with more fidelity.

So 100% believing that the issue was not ram/shm (would you think it was ram with the above stats?), I moved on to other issues. Disabling E10 with an env var. Disabling webgl. Examining all settings which had changed between 60 and 68, the list goes on.

After about 5 hours of debugging, and googling some gdb / strace output, I ran across this:

https://bugzilla.mozilla.org/show_bug.cgi?id=1432333

Loading the Google Books page crashed my tab with a crash sig much like the ones above, so again .. memory? Weird! OK...

I changed my ram config from 4G, to 12G. Bam. All problems vanished.

I reverted to 4G of RAM, changed my /dev/shm to 80% of system ram, from 20%.. and all of my issues vanished completely.

I'm guessing that /dev/shm is being used and reaped very quickly, and that even with .1s I am not catching the full extent of its use. Because when I show so little RAM and shm usage, but out-of-ram/memory crashes are persistent (constant, every few minutes)... and 80% resolves everything? What else could be the case.. (current stats):

tmpfs 3.4G 660M 2.8G 20% /dev/shm
total used free shared buff/cache available
Mem: 4041456 1888412 969652 713860 1183392 1222192

I've been running a while with the above 80% config. Stable. Rock solid stable.

After this change, at one point I managed to catch it balloon > 800M, more than a G, prior to whatever reaping happens, which makes sense with the prior issues at 20%/800M /dev/shm.

So a few things;

• I'm guessing that there's no easy way to know what space requirements are prior to usage, and this is why there is no check before using /dev/shm? And even if there was, there's no fallback, no other tmp location for fallback.. so why check regardless?

• Without any real, meaningful way to catch issues due to /dev/shm exhaustion, a LOT of people are going to spend a lot of time debugging this... and potentially end up on the wrong track.

• I believe "something is wrong", if the default config over a large number of linux distros, is going to result in this condition.

• With only 4G of system ram, yet my modification to 80%, the problem is clearly not a ram issue, merely a config issue.

I have no easy answers here, just the above info/usage case.

My personal thoughts as a takeaway? Using a system resource like /dev/shm blindly, and not having a catch when it fails (no fallback to other tmp locations, no check prior to use, and if failure occurs, a simple crash, and no debugging info) seems... errant?

Comment 11

[Gian-Carlo Pascutto [:gcp]]

Using a system resource like /dev/shm blindly, and not having a catch when it fails (no fallback to other tmp locations, no check prior to use, and if failure occurs, a simple crash, and no debugging info) seems... errant?

1. Firefox just uses shm_open(), it's that call inside glibc that ends up using /dev/shm, because that's where Linux Says Shmem Should Be. For the same reason, there's no fallback - performance would be terrible anyway if it was a real disk.
2. Reporting errors when you have run out of memory is very problematic because...you can't allocate any memory while attempting to do so. Dying while setting a bit in the crash reporter is typically about the best we can do, but for shmem exhaustion we can't even communicate with the crash reporter!
3. Guessing how much memory is required isn't possible in advance, because we don't know what's on the webpages we are requested to load.
4. The "my distro allocated too few shmem and now random multiprocessed software doesn't work" isn't unique to Firefox. Aside from a lot of server stuff, there's https://bugs.chromium.org/p/chromium/issues/detail?id=715363 and https://bugs.chromium.org/p/chromium/issues/detail?id=736452. Chrome seems to have a command line option to do the switch to /tmp manually. That doesn't solve your use case, though, because once you figure the problem you can fix the config.

On modern Linux kernels, the problem can be worked around with memfd_create: https://bugzilla.mozilla.org/show_bug.cgi?id=1440203 which bypasses the filesystem entirely. We have experimental patches for that but they need some more work.

Comment 12

[Gian-Carlo Pascutto [:gcp]]

I wonder if we could add a check for /dev/shm size being < 2G and print at the very least a clear console warning on startup. That should be a trivial patch and might make it a bit easier to find out what's going on...

Falling back to /tmp (if bigger) automatically might be feasible. The problem is that shm_open uses /dev/shm and that's in glibc. We moved away from manually opening files towards shm_open in bug 1447867. But the fallback codepaths still appear to be there, I'm just unsure of the security properties.

Or maybe someone could just fix docker instead of messing with every single Linux application that needs shared memory.

Comment 13

[bugzilla.mozilla.org]

To be clear, my report was simply about a Debian desktop. I know this is a docker report, but the symptoms are shared.

Linux desktop, buster (aka, current stable), 4G ram. FYI, I'm not using systemd, so I'm not sure what it defaults to as a /dev/shm sized mount.

By default sysvinit sources /etc/default/tmpfs, and its default in buster/stretch/etc is 20% of overall ram on boot. Google says that systemd tends to default at 50% of shared ram, but I had another desktop @2G which had issues via this bug, until I upped /dev/shm to 80% (1.6G).

What's happening here, is that on systems with less ram -- /dev/shm is a MASSIVE limiter, not RAM size itself.

Why these old systems, you ask? Older lab computers, computers hooked up to scanners for scanning only, there are loads of legacy systems I have with only 4G of ram. They work fine, but my point here is there are loads of systems in countries where purchasing power isn't all that ... powerful.

I get why firefox is handling things this way -- speed.

However, that other bug with Google Books + needing 32G of ram to render, is also significantly hampered by /dev/shm more than RAM constraints.

Comment 14

[Robert Mader [:rmader]]

Just wanted to confirm that Fedora 31 (systemd) defaults to 50% of ram. So systems with systemd are probably much less likely to run into this problem - 4GB * 0.5 = 2GB while 12GB * 0.2 = 2.4GB

I believe "something is wrong", if the default config over a large number of linux distros, is going to result in this condition.

To me it rather sounds like it would be in their (the affected distributions) own interest to change these default values, as most major distribution use systemd today.

Comment 15

[BugBot [:suhaib / :marco/ :calixte]]

The severity field for this bug is relatively low, S3. However, the bug has 7 See Also bugs.
:jld, could you consider increasing the bug severity?

For more information, please visit auto_nag documentation.

Comment 16

[BugBot (nomail) [:suhaib / :marco/ :calixte]]

The last needinfo from me was triggered in error by recent activity on the bug. I'm clearing the needinfo since this is a very old bug and I don't know if it's still relevant.

Comment 17

[Jed Davis [:jld] ⟨⏰|UTC-7⟩ ⟦he/him⟧]

I think this is fixed in most cases by bug 1440203 (to use memfd if available), and the /dev/shm fallback case should have been improved by bug 1582954 (we'll preallocate the space and fail in shared memory creation, instead of faulting later when it's accessed).