1884996 - libfaketime does not work any more since Firefox 120, due to re-entrant malloc via clock() function

Status: RESOLVED FIXED

(Core :: Memory Allocator, defect)

Description

[Rob Wu [:robwu]]

libfaketime (and its faketime program) is a tool to simulate time. An example of a use case is to test issues related to certificate expiration.

Since Firefox 120, bisected to bug 1854550, starting Firefox with faketime results in crashes. Test case:

faketime 2024-03-02 firefox --no-remote -profile /tmp/profdir

The crash happen because there is re-entrancy in malloc: malloc depends on clock() as a source of "randomness" in PHC.cpp. The clock function is handled by libfaketime, which may trigger allocations. I tried to patch libfaketime to avoid malloc, but there is only so much that can be done because many of these allocations are done indirectly through libc functions. Here are examples of attempts to get libfaketime+Firefox working again: https://github.com/wolfcw/libfaketime/issues/130#issuecomment-1989773770

A comment in PHC.cpp claims that clock() is used instead of RandomUint64() because the latter has an implementation on macOS that allocates. Is that still accurate? Can RandomUint64() be used, with a fallback to something else (whether clock or something else) on macOS?

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1854550

:pbone, since you are the author of the regressor, bug 1854550, could you take a look? Also, could you set the severity field?

For more information, please visit BugBot documentation.

Comment 2

[Rob Wu [:robwu]]

Here is the stack of a crash/stack overflow, using faketime v0.9.10 and Firefox (with debug symbols, pulled from autoland in Treeherder). The time at the top of the stack shows that it gets into libfaketime, which ends up calling malloc again as shown in https://github.com/wolfcw/libfaketime/issues/130#issuecomment-1989773770

#119019 0x00007f9e612008f5 in time () from /usr/lib/faketime/libfaketime.so.1
#119020 0x0000559cbfe681b0 in GMut::GMut() ()
#119021 0x0000559cbfe67ef6 in GMut* InfallibleAllocPolicy::new_<GMut>() ()
#119022 0x0000559cbfe67d7a in maybe_init()::{lambda()#1}::operator()() const ()
#119023 0x0000559cbfe63143 in MaybePageAlloc(mozilla::Maybe<unsigned long> const&, unsigned long, unsigned long, bool) ()
#119024 0x0000559cbfe61c13 in MozJemallocPHC::malloc(unsigned long) ()
#119025 0x00007f9e60eac4fb in (anonymous namespace)::pool::pool (this=0x7f9e61076280 <(anonymous namespace)::emergency_pool>) at /usr/src/debug/gcc/gcc/libstdc++-v3/libsupc++/eh_alloc.cc:235
#119026 __static_initialization_and_destruction_0 () at /usr/src/debug/gcc/gcc/libstdc++-v3/libsupc++/eh_alloc.cc:373
#119027 _GLOBAL__sub_I_eh_alloc.cc(void) () at /usr/src/debug/gcc/gcc/libstdc++-v3/libsupc++/eh_alloc.cc:456
#119028 0x00007f9e612151be in call_init (env=0x7ffce7e6af80, argv=0x7ffce7e6af58, argc=4, l=<optimized out>) at dl-init.c:74
#119029 call_init (l=<optimized out>, argc=4, argv=0x7ffce7e6af58, env=0x7ffce7e6af80) at dl-init.c:26
#119030 0x00007f9e612152ac in _dl_init (main_map=0x7f9e612462e0, argc=4, argv=0x7ffce7e6af58, env=0x7ffce7e6af80) at dl-init.c:121
#119031 0x00007f9e6122bbc0 in _dl_start_user () from /lib64/ld-linux-x86-64.so.2

Comment 3

[Paul Bone [:pbone]]

Thanks Rob,

I think what I'll do is change when PHC uses time() so that it's not reenterant.

Comment 4

[Paul Bone [:pbone]]

Attachment: Bug 1884996 - Avoid reenterant malloc in PHC's RNG r=glandium

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/96bb1ab83924
Avoid reenterant malloc in PHC's RNG r=glandium

Comment 6

[pstanciu]

https://hg.mozilla.org/mozilla-central/rev/96bb1ab83924

Comment 7

[BugBot [:suhaib / :marco/ :calixte]]

The patch landed in nightly and beta is affected.
:pbone, is this bug important enough to require an uplift?

• If yes, please nominate the patch for beta approval.
• If no, please set status-firefox124 to wontfix.

For more information, please visit BugBot documentation.

Comment 8

[Paul Bone [:pbone]]

Hi Rob Wu,

Is this important enough to request an uplift? 124 is already shipping so it is only if there's a point release.

Comment 9

[Rob Wu [:robwu]]

Nice to have, no big deal.

For people who do want to test Firefox 120 until 124 with libfaketime, follow the instructions at https://github.com/wolfcw/libfaketime/issues/130#issuecomment-1989773770 and hope for the best. At present that requires compiling libfaketime yourself, but that is pretty straightforward.

Until Firefox 119 and starting from Firefox 125, faketime + Firefox works out of the box.