1850940 - "LLVM Profile Error: Failed to write file" when doing clang+PGO build in 117.0

Status: ASSIGNED

(Firefox Build System :: Toolchains, defect, P3)

Description

[Joonas Niilola]

Steps to reproduce:

Build Firefox-117.0 with MOZ_PGO=1 and clang+llvm+lld.

You can read a full build.log from:
https://reproducible.archlinux.org/api/v0/builds/500286/log
(ctrl+f for "Profile Error:")

Build recipe is available in:
https://gitlab.archlinux.org/archlinux/packaging/packages/firefox/-/blob/main/PKGBUILD?ref_type=heads

This also happens in Gentoo when building with clang. Downstream bug:
https://bugs.gentoo.org/909724

Actual results:

During the profiling (I guess), there are messages such as:

JavaScript error: resource://gre/modules/XULStore.sys.mjs, line 60: Error: Can't find profile directory.
LLVM Profile Error: Failed to write file "default_29091_random_1582966729283276413_0.profraw": Broken pipe
LLVM Profile Error: Failed to write file "default_29091_random_8577073746504265386_0.profraw": Broken pipe
LLVM Profile Error: Failed to write file "default_29091_random_7269814057032389890_0.profraw": Broken pipe
LLVM Profile Error: Failed to write file "default_29091_random_7386420086373065007_0.profraw": Broken pipe
LLVM Profile Error: Failed to write file "default_29091_random_1971493760237642638_0.profraw": Broken pipe
LLVM Profile Error: Failed to write file "default_29091_random_3426472508432559282_0.profraw": Broken pipe
LLVM Profile Error: Failed to write file "default_29091_random_16745329148469669040_0.profraw": Broken pipe
LLVM Profile Error: Failed to write file "default_29091_random_15115056063447035087_0.profraw": Broken pipe
LLVM Profile Error: Failed to write file "default_29091_random_5691425745200705770_0.profraw": Broken pipe

The build will continue and looks like profiling is successful nevertheless.

Expected results:

It shouldn't print errors at least.

When building with GCC+pgo there aren't any "errors" like this.

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'Firefox Build System::Toolchains' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Comment 2

[Joonas Niilola]

profdata* files are created normally,

# ls /var/tmp/portage/www-client/firefox-117.0/work/firefox_build/instrumented/*.profraw | wc -l
125

file /var/tmp/portage/www-client/firefox-117.0/work/firefox_build/instrumented/merged.profdata 
/var/tmp/portage/www-client/firefox-117.0/work/firefox_build/instrumented/merged.profdata: LLVM indexed profile data, version 9

ls -lh /var/tmp/portage/www-client/firefox-117.0/work/firefox_build/instrumented/merged.profdata 
-rw-r--r-- 1 portage portage 67M Sep  1 06:50 /var/tmp/portage/www-client/firefox-117.0/work/firefox_build/instrumented/merged.profdata

and -fprofile-use correctly points to the specified merged.profdata file. So I believe in the end PGO is successful, but maybe some small subset is crashing and not included?

Comment 3

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:glandium, could you have a look please?

For more information, please visit BugBot documentation.

Comment 4

[[:sergesanspaille]]

compiler-rt patch submitted upstream: https://github.com/llvm/llvm-project/pull/68622
This seems to fix the issue on my setup.

Comment 5

[[:sergesanspaille]]

The patch has been merged upstream \o/ I'll backport it.

Comment 6

[[:sergesanspaille]]

Attachment: Bug 1850940 - backport profile flushing improvement patch r=glandium

Upstream patch: 97b989b27496d5aeadb3b90cbb9305ddcd9e35d2
Upstream review: https://github.com/llvm/llvm-project/pull/68622

Comment 7

[Joonas Niilola]

Awesome, thank you :)

I may be able to test it with firefox-118.0.2, if the patch applies cleanly on top of compiler-rt-17.0.

Comment 8

[[:sergesanspaille]]

It does apply cleanly, I'll be happy to read the result of your tests.

Comment 9

[Joonas Niilola]

Attachment: firefox-118.0.2:20231012-080901.log.xz

Hmm, I still got it. I'll attach my build log as compressed due to its size, you can find plenty of LLVM Profile Error: Failed to write file "default_19939_random_11267494169240745218_0.profraw": Broken pipe errors there again.

My compiler-rt-17.0.2 is built with your patch included,

 * Applying user patches from /etc/portage/patches ...
 * Applying 8c6535baa41d8af62190af61336196bc3623655d.patch ...                                         [ ok ]
 * User patches applied.

(From https://github.com/llvm/llvm-project/pull/68622/commits/8c6535baa41d8af62190af61336196bc3623655d)