1993285 - [meta] Avoid undefined behavior the compiler can exploit in C++

Status: NEW

(Core :: XPCOM, task)

Description

[Andrew McCreight [:mccr8]]

While the big categories of memory safe problems like use-after-frees, buffer overflows and races are all undefined behavior, another interesting category is undefined behavior that the compiler can detect and use to delete things like bounds checks, rendering code unsafe.

One typical and somewhat surprising example is null derefs. Chromium had a problem with their WeakPtr implementation. The basic pattern works like this: a getter method is defined that uses a safety check to prevent something unsafe, and it returns null if the safety check fails:

Foo* Get() const {
  if (!BlahIsSafeToAccess() {
    return nullptr;
  }
  return mBlah;
}

A valid call site would look like this:

  if (!something->Get()) {
    return NS_ERROR_FAILURE;
  }
  something->Get()->DoSomething();

But what if you skip the null check on Get()? The compiler can tell that that first if in Get() will always return null, and we'll always dereference null in that case, but dereferencing null is undefined behavior, so it can delete the check and you end up always dereferencing mBlah directly without any check, which can be unsafe and lead to memory safety problems.

Another issue that was mentioned in the CPPNow 2025 talk "C++ Memory Safety in WebKit" was equivalent to something like this:

if (i < array.Length()) {
  return NS_ERROR_FAILURE;
}
array[i]->Whatever();
while (1) {};

The infinite loop was unintentional and not that obvious, but the compiler was still able to figure out that there was an infinite loop. An infinite loop is apparently currently undefined behavior in C++, so the compiler is allowed to do whatever it wants on the entire execution path involving undefined behavior (though it sounds like there's a proposal to only allow that after the undefined behavior) so it just deleted the bounds check, leading to a memory safety problem if you ever ended up in this code, instead of just an infinite loop. This specific case is likely not common but it is still something to keep in mind.