1928237 - Assertion failure: !stats.opt_randomize_small (Content process should not randomize small allocations), at /home/user/code/mozilla-unified/dom/ipc/ContentChild.cpp:812

Status: RESOLVED FIXED

(Core :: DOM: Content Processes, defect, P2)

Description

[Tyson Smith [:tsmith] (PTO -> Sept/8)]

This is triggered on launch. Local build m-c ae91dcae58b1c803d2c5bbb11bf9ff034bc35741 (20241030141433).

mozconfig file:

mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/objdir-ff-debug
mk_add_options MOZ_MAKE_FLAGS=-j8
ac_add_options --enable-undefined-sanitizer
ac_add_options --enable-debug
ac_add_options --disable-crashreporter
ac_add_options --disable-install-strip
ac_add_options --enable-optimize="-O1 -g"
export MOZ_DEBUG_SYMBOLS=1
ac_add_options --enable-debug-symbols

Assertion failure: !stats.opt_randomize_small (Content process should not randomize small allocations), at /home/user/code/mozilla-unified/dom/ipc/ContentChild.cpp:812

#0 0x72dde882455d in mozilla::dom::ContentChild::Init(mozilla::ipc::UntypedEndpoint&&, char const*, bool) /home/user/code/mozilla-unified/dom/ipc/ContentChild.cpp:811:3
#1 0x72dde88a827d in mozilla::dom::ContentProcess::InfallibleInit(int, char**) /home/user/code/mozilla-unified/dom/ipc/ContentProcess.cpp:143:12
#2 0x72dde88a7ed8 in mozilla::dom::ContentProcess::Init(int, char**) /home/user/code/mozilla-unified/dom/ipc/ContentProcess.cpp:70:3
#3 0x72ddea4139b5 in XRE_InitChildProcess(int, char**, XREChildData const*) /home/user/code/mozilla-unified/toolkit/xre/nsEmbedFunctions.cpp:557:21
#4 0x5787ddd9bc32 in main /home/user/code/mozilla-unified/browser/app/nsBrowserApp.cpp:397:22

Comment 1

[BugBot [:suhaib / :marco]]

Set release status flags based on info from the regressing bug 1917844

:gerard-majax, since you are the author of the regressor, bug 1917844, could you take a look? Also, could you set the severity field?

For more information, please visit BugBot documentation.

Comment 2

[Tyson Smith [:tsmith] (PTO -> Sept/8)]

A Pernosco session is available here: https://pernos.co/debug/uFSeCgaaDsIkPAYCszhhNA/index.html

Comment 3

[:gerard-majax]

The condition applies on MOZ_MEMORY && DEBUG defined and the reset is performed by https://searchfox.org/mozilla-central/rev/53e8dfd81c32f1ab275516406ec06a68136aaef0/memory/build/mozjemalloc.cpp#5191-5224 which is called in https://searchfox.org/mozilla-central/rev/53e8dfd81c32f1ab275516406ec06a68136aaef0/ipc/glue/ForkServer.cpp#335-336

Comment 4

[:gerard-majax]

(In reply to Tyson Smith [:tsmith] from comment #2)

A Pernosco session is available here: https://pernos.co/debug/uFSeCgaaDsIkPAYCszhhNA/index.html

There is no forkserver there, so the code that sets the allocator options is the one we have not changed: https://searchfox.org/mozilla-central/rev/53e8dfd81c32f1ab275516406ec06a68136aaef0/ipc/glue/GeckoChildProcessHost.cpp#1110-1111

Comment 5

[:gerard-majax]

BaseProcessLauncher::DoSetup() properly sets the env but in the content process launched during malloc_init_hard() there seems to be no MALLOC_OPTIONS: https://pernos.co/debug/uFSeCgaaDsIkPAYCszhhNA/index.html#f{m[u94,Frg_,t[uw,A1cK_,f{e[u94,Fng_,s{aVgMf9pAA,bBQ,uA9xe,oBosv___/

Comment 6

[:gerard-majax]

According to https://pernos.co/debug/uFSeCgaaDsIkPAYCszhhNA/index.html#f{m[uNc,CAG4_,t[Fw,A1ZM_,f{e[uNc,WCs_,s{adKio4AAA,bAaQ,uBQHK5w,oBQQkqA___,v[{wiX8,v[{f'list',q'stdouterr',p{_,xTiA_,{f'container',q'stack',p{_,xTiA_,{f'list',q'breakpoint',p{'url''https:3A%2F%2Fhg.mozilla.org%2Fmozilla-unified%2Fraw-file%2Fae91dcae58b1c803d2c5bbb11bf9ff034bc35741%2Fipc%2Fchromium%2Fsrc%2Fbase%2Fprocess_util_linux.cc','points'[{lQw,cAQ,oByU,'o8'ByU__,'print'':24pid'_,xLYI_,{f'list',q'breakpoint',p{'url''https:3A%2F%2Fhg.mozilla.org%2Fmozilla-unified%2Fraw-file%2Fae91dcae58b1c803d2c5bbb11bf9ff034bc35741%2Fipc%2Fchromium%2Fsrc%2Fbase%2Fprocess_util_posix.cc','points'[{lAag,cAQ,oNfg,'o8'Nfg___,xLbM_,{f'list',q'breakpoint',p{'url''https:3A%2F%2Fhg.mozilla.org%2Fmozilla-unified%2Fraw-file%2Fae91dcae58b1c803d2c5bbb11bf9ff034bc35741%2Fipc%2Fchromium%2Fsrc%2Fbase%2Fprocess_util_posix.cc','points'[{lAaU,cAQ,oNcU,'o8'NcU__,'print''entry'_,xjyo___,{w/SA,v[{f'source',q'source',p{_,x7So_,{f'list',q'execution',p{'symbol''mozilla:3A%3ASandboxLaunch%3A%3AFork'_,xNd0_,{f'list',q'execution',p{'symbol''base:3A%3ABuildEnvironmentArray'_,xY5c____/ we properly build the environment, if im' not mistaken we see MALLOC_OPTIONS=r when we are taking care of the crashing content process

Comment 7

[:gerard-majax]

From https://pernos.co/debug/uFSeCgaaDsIkPAYCszhhNA/index.html#f{m[u94,FsM_,t[uw,A1cK_,f{e[u94,Fro_,s{aVgMf9pAA,bBQ,uDTSd,oDWej___,v[{wiLg,v[{f'list',q'stdouterr',p{_,xAYag_,{f'container',q'stack',p{_,xAYag_,{f'list',q'alerts',p{_,xAYag___,{w/eg,v[{f'source',q'source',p{_,xAYag_,{f'list',q'execution',p{'symbol''getenv','condition'':24pid%20%3D%3D%20218890'_,xAYag____/ we see that getenv("MALLOC_OPTIONS") somehow returned garbage (all getenv ?)

Comment 8

[:gerard-majax]

And I cannot find any execution of setproctitle() in the pernosco trace, so I doubt it's due to https://phabricator.services.mozilla.com/D221021

Comment 9

[Tyson Smith [:tsmith] (PTO -> Sept/8)]

(In reply to :gerard-majax from comment #3)

The condition applies on MOZ_MEMORY && DEBUG defined and the reset is performed by https://searchfox.org/mozilla-central/rev/53e8dfd81c32f1ab275516406ec06a68136aaef0/memory/build/mozjemalloc.cpp#5191-5224 which is called in https://searchfox.org/mozilla-central/rev/53e8dfd81c32f1ab275516406ec06a68136aaef0/ipc/glue/ForkServer.cpp#335-336

Is there any other info I can provide to help get this fixed?

Comment 10

[:gerard-majax]

(In reply to Tyson Smith [:tsmith] from comment #9)

(In reply to :gerard-majax from comment #3)

The condition applies on MOZ_MEMORY && DEBUG defined and the reset is performed by https://searchfox.org/mozilla-central/rev/53e8dfd81c32f1ab275516406ec06a68136aaef0/memory/build/mozjemalloc.cpp#5191-5224 which is called in https://searchfox.org/mozilla-central/rev/53e8dfd81c32f1ab275516406ec06a68136aaef0/ipc/glue/ForkServer.cpp#335-336

Is there any other info I can provide to help get this fixed?

I think i shared enough already? Can you confirm what i checked? I think the only relationship is we added the assert.

Comment 11

[Tyson Smith [:tsmith] (PTO -> Sept/8)]

I don't understand why I can repro with a local builds but not with a fuzzing debug build from TC.

I also hit the assertion if I run standalone vs with Grizzly/Site-scout/FFPuppet.

Comment 12

[:gerard-majax]

(In reply to Tyson Smith [:tsmith] from comment #11)

I don't understand why I can repro with a local builds but not with a fuzzing debug build from TC.

I also hit the assertion if I run standalone vs with Grizzly/Site-scout/FFPuppet.

https://pernos.co/debug/uFSeCgaaDsIkPAYCszhhNA/index.html#f{m[u94,FvQ_,t[uw,A1cK_,f{e[u94,FvQ_,s{aVgMf9pAA,bBQ,uDTSd,oDWfa___,v[{wiLg,v[{f'list',q'stdouterr',p{_,xAYag_,{f'container',q'stack',p{_,xAYag_,{f'list',q'alerts',p{_,xAYag___,{w/eg,v[{f'source',q'source',p{_,xAYag____/

environ is nullptr

Comment 13

[:gerard-majax]

(In reply to :gerard-majax from comment #12)

(In reply to Tyson Smith [:tsmith] from comment #11)

I don't understand why I can repro with a local builds but not with a fuzzing debug build from TC.

I also hit the assertion if I run standalone vs with Grizzly/Site-scout/FFPuppet.

https://pernos.co/debug/uFSeCgaaDsIkPAYCszhhNA/index.html#f{m[u94,FvQ_,t[uw,A1cK_,f{e[u94,FvQ_,s{aVgMf9pAA,bBQ,uDTSd,oDWfa___,v[{wiLg,v[{f'list',q'stdouterr',p{_,xAYag_,{f'container',q'stack',p{_,xAYag_,{f'list',q'alerts',p{_,xAYag___,{w/eg,v[{f'source',q'source',p{_,xAYag____/

environ is nullptr

But not at the time of the assert.

Comment 14

[:gerard-majax]

jed on matrix noticed this:

(pernosco) bt 
#0  getenv (name=0x56031ff9dc40 "MALLOC_LOG") at /home/user/code/mozilla-unified/mozglue/interposers/env_interposer.cpp:36
#1  0x0000560320029544 in logalloc_init (aTable=0x7ffd6cae8ab8, aBridge=0x560320aa94f8 <gReplaceMallocBridge>) at /home/user/code/mozilla-unified/memory/replace/logalloc/LogAlloc.cpp:139
#2  0x000056032001205f in init () at /home/user/code/mozilla-unified/memory/build/mozjemalloc.cpp:5452
#3  0x0000560320012256 in ReplaceMalloc::malloc (arg1=98) at /home/user/code/mozilla-unified/memory/build/malloc_decls.h:51
#4  malloc (arg1=98) at /home/user/code/mozilla-unified/memory/build/malloc_decls.h:51
#5  0x000077657cf00cca in malloc (size=98) at ../include/rtld-malloc.h:56
#6  __GI__dl_exception_create_format (exception=exception@entry=0x7ffd6cae8cf0, objname=0x7ffd6cae993d "/home/user/code/mozilla-unified/objdir-ff-debug/dist/bin/firefox", fmt=fmt@entry=0x77657cf29eb4 "undefined symbol: %s%s%s") at ./elf/dl-exception.c:157
#7  0x000077657cf083dc in _dl_lookup_symbol_x (undef_name=0x56031ff95d31 "swift_demangle", undef_map=0x77657cf372e0, ref=0x7ffd6cae8d68, symbol_scope=<optimized out>, version=0x0, type_class=0, flags=3, skip_map=0x0) at ./elf/dl-lookup.c:877
#8  0x000077657c975954 in do_sym (flags=2, vers=0x0, who=0x56031fffff94 <LateInitialize()+20>, name=0x56031ff95d31 "swift_demangle", handle=<optimized out>) at ./elf/dl-sym.c:146
#9  _dl_sym (handle=<optimized out>, name=0x56031ff95d31 "swift_demangle", who=0x56031fffff94 <LateInitialize()+20>) at ./elf/dl-sym.c:195
#10 0x000077657c890738 in dlsym_doit (a=a@entry=0x7ffd6cae8fe0) at ./dlfcn/dlsym.c:40
#11 0x000077657c974a98 in __GI__dl_catch_exception (exception=exception@entry=0x7ffd6cae8f30, operate=<optimized out>, args=<optimized out>) at ./elf/dl-error-skeleton.c:208
#12 0x000077657c974b63 in __GI__dl_catch_error (objname=0x7ffd6cae8f88, errstring=0x7ffd6cae8f90, mallocedp=0x7ffd6cae8f87, operate=<optimized out>, args=<optimized out>) at ./elf/dl-error-skeleton.c:227
#13 0x000077657c89012e in _dlerror_run (operate=operate@entry=0x77657c890720 <dlsym_doit>, args=args@entry=0x7ffd6cae8fe0) at ./dlfcn/dlerror.c:138
#14 0x000077657c8907b4 in dlsym_implementation (dl_caller=<optimized out>, name=0x56031ff95d31 "swift_demangle", handle=<optimized out>) at ./dlfcn/dlsym.c:54
#15 ___dlsym (handle=<optimized out>, name=0x56031ff95d31 "swift_demangle") at ./dlfcn/dlsym.c:68
#16 0x000056031fffff94 in InitializeSwiftDemangler () at /builds/worker/fetches/llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_symbolizer_posix_libcdep.cpp:75
#17 LateInitialize () at /builds/worker/fetches/llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_symbolizer_posix_libcdep.cpp:510
#18 0x0000560320002cb4 in CommonStandaloneInit () at /builds/worker/fetches/llvm-project/compiler-rt/lib/ubsan/ubsan_init.cpp:55
#19 InitAsStandalone () at /builds/worker/fetches/llvm-project/compiler-rt/lib/ubsan/ubsan_init.cpp:61
#20 0x0000560320007a76 in __ubsan::PreInitAsStandalone() () at /builds/worker/fetches/llvm-project/compiler-rt/lib/ubsan/ubsan_init_standalone_preinit.cpp:27
#21 0x000077657cf025be in _dl_init (main_map=0x77657cf372e0, argc=27, argv=0x7ffd6cae90a8, env=0x7ffd6cae9188) at ./elf/dl-init.c:102
#22 0x000077657cf1c2ca in _dl_start_user () at /sysroot/lib64/ld-linux-x86-64.so.2

It would look like we get into malloc because of a failing load InitializeSwiftDemangler and we're wondering if this is happening too early for environ to be set yet. No environ, no malloc options, hence assertion

Comment 15

[Jens Stutte [:jstutte]]

I am assuming that this has no (catastrophic) consequences in non-DEBUG builds. I assume we still will pass through jemalloc_reset_small_alloc_randomization and this would stop us from randomizing? Or would we continue to randomize in content processes?

Comment 16

[:gerard-majax]

(In reply to Jens Stutte [:jstutte] from comment #15)

I am assuming that this has no (catastrophic) consequences in non-DEBUG builds. I assume we still will pass through jemalloc_reset_small_alloc_randomization and this would stop us from randomizing? Or would we continue to randomize in content processes?

I'm more worried that such early malloc initialization cannot parse the correct MALLOC_OPTIONS that were expected. The assertion is just a nice by-product here.

Comment 17

[:gerard-majax]

(In reply to Jens Stutte [:jstutte] from comment #15)

I am assuming that this has no (catastrophic) consequences in non-DEBUG builds. I assume we still will pass through jemalloc_reset_small_alloc_randomization and this would stop us from randomizing? Or would we continue to randomize in content processes?

The shared pernosco session was not making use of forkserver, likely because of UBSan usage: https://searchfox.org/mozilla-central/rev/fcf53e1685bfb990b5abc7312ac1daf617f0991f/modules/libpref/init/StaticPrefList.yaml#3016-3018 so the specific case of malloc randomization would be handled via MALLOC_OPTIONS. In this case it's not really a problem that the randomization is enabled when it should not, the only impact is perf.

However inability to parse MALLOC_OPTIONS from environment may be more problematic

Comment 18

[Yannis Juglaret [:yannis]]

(In reply to Tyson Smith [:tsmith] from comment #11)

I don't understand why I can repro with a local builds but not with a fuzzing debug build from TC.

While discussing this bug, :pbone noted that your mozconfig does not feature --disable-jemalloc and that this is unusual when using sanitizers. The fuzzing debug build from TC probably has --disable-jemalloc, which would explain why it doesn't run into the issue explicited by comment 17. It seems we at least had an intention to enforce --disable-jemalloc when using sanitizers in bug 1435148, even for ubsan.

Comment 19

[:gerard-majax]

Paul, can you help assert how bad the situation here? The fact we cannot parse MALLOC_OPTIONS

Comment 20

[Paul Bone [:pbone]]

I think it's fine. This is an unsupported build configuration (I think) but it should be handled more gracefully.

I can check forther.

Comment 21

[:gerard-majax]

(In reply to Paul Bone [:pbone] from comment #20)

I think it's fine. This is an unsupported build configuration (I think) but it should be handled more gracefully.

I can check forther.

We can mitigate by #ifdef the assert on MOZ_UBSAN but only if you assert it's okay to not assert there :)

Comment 22

[Paul Bone [:pbone]]

(In reply to :gerard-majax from comment #21)

(In reply to Paul Bone [:pbone] from comment #20)

I think it's fine. This is an unsupported build configuration (I think) but it should be handled more gracefully.

I can check forther.

We can mitigate by #ifdef the assert on MOZ_UBSAN but only if you assert it's okay to not assert there :)

Yes. I assert that we can assert this ;-)

There are multiple problems with sanitisers and mozjemalloc so I'd also like to update our build system to disable mozjemalloc when sanitisers are enabled.

Comment 23

[Paul Bone [:pbone]]

Attachment: Bug 1928237 - Disable mozjemalloc in asan/tsan/msan builds r=glandium

This change will disable mozjemalloc in these builds, making the default
behaviour better and avoding developer confusion. However it won't forbid
building this configuration, which may or may not be desirable.

Comment 24

[Paul Bone [:pbone]]

Attachment: Bug 1928237 - Don't assert mozjemalloc read the env in ubsan builds r=gerard-majax

Comment 25

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a6c200cc140a
Disable mozjemalloc in asan/tsan/msan builds r=gerard-majax,firefox-build-system-reviewers,glandium
https://hg.mozilla.org/integration/autoland/rev/3e3a44b81ed5
Don't assert mozjemalloc read the env in ubsan builds r=gerard-majax

Comment 26

[amarc]

https://hg.mozilla.org/mozilla-central/rev/a6c200cc140a
https://hg.mozilla.org/mozilla-central/rev/3e3a44b81ed5

Comment 27

[BugBot [:suhaib / :marco]]

The patch landed in nightly and beta is affected.
:pbone, is this bug important enough to require an uplift?

• If yes, please nominate the patch for beta approval.
• If no, please set status-firefox134 to wontfix.

For more information, please visit BugBot documentation.