Closed Bug 1516374 Opened Last year Closed 11 months ago

Upgrade clang to 7.0.1


(Firefox Build System :: Toolchains, enhancement)

Not set


(firefox66 fixed)

Tracking Status
firefox66 --- fixed


(Reporter: glandium, Assigned: glandium)




(1 file)

It was released a few days ago.

I finally figured out what goes wrong, but I don't know yet exactly what the root cause is.

What goes wrong is that llvm-symbolizer requires more memory to do its work in some cases. In the order of 1GB instead of a few dozens MB. When Firefox exits, every single process that has been launched terminates, and for each of them that had leaks (most), a llvm-symbolizer process is launched. In many cases, that's 5 processes. Each sucking 1GB of memory. All at the same time. So there's an instant rush to have 5GB of memory available, on machines that only have 7.5GB physical memory and no swap, running an Ubuntu desktop environment, and Firefox with ASAN... so there isn't enough memory available. Subsequently, because of the rush of memory demand, and the lack of available memory, the kernel goes on a process killing spree, and ends up killing the worker, leaving us with no logs as to what just happened (bug 1516575 was filed to try to avoid the situation).

Now, and this is where things get fun: I've actually managed to have the problem on try with the current, non-upgraded clang 7. And for some reason it doesn't happen on integration branches. And here's another fun fact: it doesn't happen when sccache is disabled. But it still happens when sccache is enabled and clang is compiled such that sccache only gets cache misses. It also doesn't happen when sccache is patched to include the input file name in the hash it uses for the cache key. That's something I noticed in the failed builds with a different clang executable: some c++ files still had cache hits while most others didn't, and most of those with cache hits are files that once preprocessed are essentially empty (but have different sources).

At this point, my assumption is that the presence of those files makes a subtle difference in the debug info that makes llvm-symbolizer use more memory.

Flags: needinfo?(mh+mozilla)
Depends on: 1518726

It appears a try on current central doesn't create a with a size multiple of 4KiB, so this would presumably not fail when landing again, independently of bug 1518726. Also, considering it's a libxul size problem that's independent of the clang version, it could just as well happen with the current version, so let's go ahead with a relanding.

No longer depends on: 1518726
Pushed by
Upgrade clang to 7.0.1. r=dmajor
Closed: 11 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla66

The code coverage mozconfigs use hardcoded paths to 7.0.0 libs:

Maybe these could benefit from the "" trick:

Can we just fix things so that configure automatically puts that directory into LDFLAGS when we --enable-coverage?

I'll go with the no-brainer, and will file a bug for comment 10.

Flags: needinfo?(mh+mozilla)
Pushed by
Upgrade clang to 7.0.1. r=dmajor

(In reply to Mike Hommey [:glandium] from comment #11)

I'll go with the no-brainer, and will file a bug for comment 10.

That bug already exists: bug 1476339

Closed: 11 months ago11 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla66
See Also: → 1536790
You need to log in before you can comment on or make changes to this bug.