Closed Bug 1680306 Opened 4 years ago Closed 4 years ago

PGO build failing on SUSE: "Could not find profraw files in the current directory"

Categories

(Firefox Build System :: General, defect)

Product:
Firefox Build System
Component:
General
Type:
defect

Tracking

(firefox85 fixed)

Status:
RESOLVED FIXED
Milestone:
85 Branch
Tracking Flags:
Tracking Status
firefox85 --- fixed

People

(Reporter: mhentges, Assigned: mhentges)

References

Details

Attachments

(3 files)

Build log
487.71 KB, application/zstd
Details
1680306-prep-instr-build.patch
570 bytes, patch
Details | Diff | Splinter Review
Bug 1680306: Don't set LLVM_PROFDATA for non-clang PGO builds
47 bytes, text/x-phabricator-request
Details | Review

Originally reported over here. The issue seems different enough from the original bug (TypeError ...) to warrant a new ticket.

Log provided by the reporter, copied here:

[ 2584s] 42:29.55 gmake[6]: Leaving directory '/home/abuild/rpmbuild/BUILD/obj/instrumented/browser/branding/official/locales'
[ 2584s] 42:29.56 gmake[6]: Entering directory '/home/abuild/rpmbuild/BUILD/obj/instrumented/browser/branding/official/locales'
[ 2584s] 42:29.56 gmake[6]: Nothing to be done for 'libs'.
[ 2584s] 42:29.56 gmake[6]: Leaving directory '/home/abuild/rpmbuild/BUILD/obj/instrumented/browser/branding/official/locales'
[ 2584s] 42:29.56 gmake[6]: Entering directory '/home/abuild/rpmbuild/BUILD/obj/instrumented/browser/branding/official/locales'
[ 2584s] 42:29.56 gmake[6]: Nothing to be done for 'tools'.
[ 2584s] 42:29.56 gmake[6]: Leaving directory '/home/abuild/rpmbuild/BUILD/obj/instrumented/browser/branding/official/locales'
[ 2584s] 42:29.56 gmake[5]: Leaving directory '/home/abuild/rpmbuild/BUILD/obj/instrumented/browser/branding/official/locales'
[ 2584s] 42:29.56 gmake[4]: Leaving directory '/home/abuild/rpmbuild/BUILD/obj/instrumented/browser/locales'
[ 2584s] 42:29.56 gmake[4]: Entering directory '/home/abuild/rpmbuild/BUILD/obj/instrumented/browser/locales'
[ 2584s] 42:29.57 ../../config/nsinstall -D ../../dist/linux-x86_64/xpi/
[ 2584s] 42:29.57 /home/abuild/rpmbuild/BUILD/obj/instrumented/_virtualenvs/init_py3/bin/python -m mozbuild.action.langpack_manifest --locales en-US --min-app-ver 83.0 --max-app-ver 83.* --app-name 'Firefox' --l10n-basedir '/home/abuild/rpmbuild/BUILD/l10n' --defines /home/abuild/rpmbuild/BUILD/firefox-83.0/toolkit/locales/en-US/defines.inc /home/abuild/rpmbuild/BUILD/firefox-83.0/browser/locales/en-US/defines.inc  --langpack-eid 'langpack-en-US@firefox.mozilla.org' --input ../../dist/xpi-stage/locale-en-US
[ 2585s] 42:29.75 /home/abuild/rpmbuild/BUILD/obj/instrumented/_virtualenvs/init_py3/bin/python -m mozbuild.action.zip -C ../../dist/xpi-stage/locale-en-US -x **/*.manifest -x **/*.js -x **/*.ini /home/abuild/rpmbuild/BUILD/obj/instrumented/dist/linux-x86_64/xpi/firefox-83.0.en-US.langpack.xpi chrome localization browser manifest.json
[ 2585s] 42:29.92 gmake[4]: Leaving directory '/home/abuild/rpmbuild/BUILD/obj/instrumented/browser/locales'
[ 2585s] 42:29.92 gmake[3]: Leaving directory '/home/abuild/rpmbuild/BUILD/obj/instrumented/browser/locales'
[ 2585s] 42:29.92 gmake[2]: Leaving directory '/home/abuild/rpmbuild/BUILD/obj/instrumented/browser/installer'
[ 2585s] 42:29.92 gmake[2]: Entering directory '/home/abuild/rpmbuild/BUILD/obj/instrumented/browser/installer'
[ 2585s] 42:29.92 gmake[2]: Nothing to be done for 'tools'.
[ 2585s] 42:29.92 gmake[2]: Leaving directory '/home/abuild/rpmbuild/BUILD/obj/instrumented/browser/installer'
[ 2585s] 42:29.92 gmake[1]: Leaving directory '/home/abuild/rpmbuild/BUILD/obj/instrumented/browser/installer'
[ 2585s] 42:29.92 gmake: Leaving directory '/home/abuild/rpmbuild/BUILD/obj/instrumented'
[ 2589s] console.warn: SearchSettings: "get: No settings file exists, new profile?" (new Error("", "(unknown module)"))
[ 2597s] JavaScript error: resource://gre/modules/XULStore.jsm, line 66: Error: Can't find profile directory.
[ 2597s] JavaScript error: resource://gre/modules/XULStore.jsm, line 66: Error: Can't find profile directory.
[ 2597s] JavaScript error: resource://gre/modules/XULStore.jsm, line 66: Error: Can't find profile directory.
[ 2597s] JavaScript error: resource://gre/modules/XULStore.jsm, line 66: Error: Can't find profile directory.
[ 2597s] JavaScript error: resource://gre/modules/XULStore.jsm, line 66: Error: Can't find profile directory.
[ 2597s] JavaScript error: resource://gre/modules/XULStore.jsm, line 66: Error: Can't find profile directory.
[ 2598s] 
[ 2598s] (/home/abuild/rpmbuild/BUILD/obj/instrumented/dist/firefox/firefox:31167): Gtk-WARNING **: 15:22:54.859: Could not load a pixbuf from /org/gtk/libgtk/theme/Adwaita/assets/bullet-symbolic.svg.
[ 2598s] This may indicate that pixbuf loaders or the mime database could not be found.
[ 2976s] jarlog: /home/abuild/rpmbuild/BUILD/obj/jarlog/en-US.log
[ 2976s] Could not find profraw files in the current directory: /home/abuild/rpmbuild/BUILD/obj/instrumented
[ 2976s]  Config object not found by mach.
[ 2976s] Configure complete!
[ 2976s] Be sure to run |mach build| to pick up any changes
[ 2976s] To view resource usage of the build, run |mach resource-usage|.
[ 2976s] To take your build for a test drive, run: |mach run|
[ 2976s] For more information on what to do now, see https://firefox-source-docs.mozilla.org/setup/contributing_code.html
[ 2976s] Error running mach:
[ 2976s] 
[ 2976s]     ['build', '-v']
[ 2976s] 
[ 2976s] The error occurred in code that was called by the mach command. This is either
[ 2976s] a bug in the called code itself or in the way that mach is calling it.
[ 2976s] You can invoke |./mach busted| to check if this issue is already on file. If it
[ 2976s] isn't, please use |./mach busted file build| to report it. If |./mach busted| is
[ 2976s] misbehaving, you can also inspect the dependencies of bug 1543241.
[ 2976s] 
[ 2976s] If filing a bug, please include the full output of mach, including this error
[ 2976s] message.
[ 2976s] 
[ 2976s] The details of the failure are as follows:
[ 2976s] 
[ 2976s] subprocess.CalledProcessError: Command '['/home/abuild/rpmbuild/BUILD/obj/instrumented/_virtualenvs/init_py3/bin/python', '/home/abuild/rpmbuild/BUILD/firefox-83.0/build/pgo/profileserver.py']' returned non-zero exit status 1.
[ 2976s] 
[ 2976s]   File "/home/abuild/rpmbuild/BUILD/firefox-83.0/python/mozbuild/mozbuild/build_commands.py", line 118, in build
[ 2976s]     subprocess.check_call(pgo_cmd, cwd=instr.topobjdir,
[ 2976s]   File "/usr/lib64/python3.8/subprocess.py", line 364, in check_call
[ 2976s]     raise CalledProcessError(retcode, cmd)
[ 2976s] error: Bad exit status from /var/tmp/rpm-tmp.bhB33z (%build)
[ 2976s] 
[ 2976s] 
[ 2976s] RPM build errors:
[ 2976s]     Bad exit status from /var/tmp/rpm-tmp.bhB33z (%build)
[ 2976s] 
[ 2976s] marxinbox.suse.cz failed "build MozillaFirefox.spec" at Wed Dec  2 15:29:12 UTC 2020.
[ 2976s]

Martin, if you don't mind answering some questions for me:

  • Can you paste the contents of your mozconfig file?
  • What command are you running? Just a simple ./mach build -v?
  • Can you reproduce the error with a clean checkout of the tip of central (hg pull && hg up central)?
Flags: needinfo?(mliska)

I build the package via OBS (Open Build Service), that's why I can't try the current tip.
Command run: xvfb-run '--server-args=-screen 0 1920x1080x24' ./mach build -v

Mozilla config:

$ cat /var/tmp/build-root/openSUSE_Factory-x86_64/home/abuild/rpmbuild/BUILD/mozconfig
mk_add_options MOZILLA_OFFICIAL=1
mk_add_options BUILD_OFFICIAL=1
mk_add_options MOZ_MAKE_FLAGS=-j16
mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/../obj
. $topsrcdir/browser/config/mozconfig
ac_add_options --prefix=/usr
ac_add_options --libdir=/usr/lib64
ac_add_options --includedir=/usr/include
ac_add_options --enable-release
ac_add_options --enable-default-toolkit=cairo-gtk3-wayland
# bmo#1441155 - Disable the generation of Rust debug symbols on Linux32
ac_add_options --enable-debug-symbols
# building with elf-hack started to fail everywhere with FF73
#%if 01550 > 1549
ac_add_options --disable-elf-hack
#%endif
ac_add_options --with-system-nspr
ac_add_options --with-system-nss
ac_add_options --with-ccache
ac_add_options --with-l10n-base=/home/abuild/rpmbuild/BUILD/l10n
#ac_add_options --with-system-jpeg    # libjpeg-turbo is used internally
#ac_add_options --with-system-png     # doesn't work because of missing APNG support
ac_add_options --with-system-zlib
ac_add_options --disable-updater
ac_add_options --disable-tests
ac_add_options --enable-alsa
ac_add_options --disable-debug
#ac_add_options --enable-chrome-format=jar
ac_add_options --enable-update-channel=release
ac_add_options --with-mozilla-api-keyfile=/home/abuild/rpmbuild/SOURCES/mozilla-api-key
# Google-service currently not available for free anymore
#ac_add_options --with-google-location-service-api-keyfile=/home/abuild/rpmbuild/SOURCES/google-api-key
ac_add_options --with-google-safebrowsing-api-keyfile=/home/abuild/rpmbuild/SOURCES/google-api-key
ac_add_options --with-unsigned-addon-scopes=app
ac_add_options --allow-addon-sideload
ac_add_options --enable-official-branding
ac_add_options --enable-libproxy
# mitigation/workaround for bmo#1512162
# LTO needs newer toolchain stack only (at least GCC 8.2.1 (r268506)
# TW's gcc is currently also broken with LTO https://gcc.gnu.org/bugzilla/show_bug.cgi?id=93951
ac_add_options MOZ_PGO=1
Flags: needinfo?(mliska)
Attached file Build log —

Very helpful, thanks Martin!
Heh, we're not very clear in our log what specifically is causing the failure - just a dump of warnings, followed by a failed build.
Let me dig in a little bit here :)

See Also: → 1680584

Here's what I've found: when using PGO, the ./mach build call will do three things (as documented here):

  1. [subprocess, pipe output] Perform a Firefox build that will generate performance stats
  2. [subprocess, check_call (no output)] Run a bunch of tests with the built Firefox to get performance stats
  3. [subprocess, pipe output] Perform a Firefox build using the generated performance stats

I think that in your case, step 2 is failing but we aren't seeing the output, so we can't debug why. I've reported a bug to handle this hidden-output issue.

In the meantime, so you and I can debug this, can you do the following?

  1. Apply the attached patch (hg import --no-commit patch)
  2. ./mach build -v (note: this will exit early once the instrument-able build is done due to the above patch)
  3. cd /home/abuild/rpmbuild/BUILD/obj/instrumented && _virtualenvs/init_py3/bin/python /home/abuild/rpmbuild/BUILD/firefox-83.0/build/pgo/profileserver.py

This should split out step 2 so that we can see the output.
Send me the new log and we should be able to diagnose this!

Assignee: nobody → mhentges
Flags: needinfo?(mhentges)
Flags: needinfo?(mhentges) → needinfo?(mliska)

Hm, when I run it manually it works fine:

abuild@marxinbox:~/rpmbuild/BUILD/obj/instrumented> xvfb-run '--server-args=-screen 0 1920x1080x24' /home/abuild/rpmbuild/BUILD/obj/instrumented/_virtualenvs/init_py3/bin/python /home/abuild/rpmbuild/BUILD/firefox-83.0/build/pgo/profileserver.py
console.warn: SearchSettings: "get: No settings file exists, new profile?" (new Error("", "(unknown module)"))

###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost


###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost

JavaScript error: resource://gre/modules/XULStore.jsm, line 66: Error: Can't find profile directory.
JavaScript error: resource://gre/modules/XULStore.jsm, line 66: Error: Can't find profile directory.
JavaScript error: resource://gre/modules/XULStore.jsm, line 66: Error: Can't find profile directory.
JavaScript error: resource://gre/modules/XULStore.jsm, line 66: Error: Can't find profile directory.
JavaScript error: resource://gre/modules/XULStore.jsm, line 66: Error: Can't find profile directory.
JavaScript error: resource://gre/modules/XULStore.jsm, line 66: Error: Can't find profile directory.

(/home/abuild/rpmbuild/BUILD/obj/instrumented/dist/firefox/firefox:13732): Gtk-WARNING **: 13:07:47.911: Could not load a pixbuf from /org/gtk/libgtk/theme/Adwaita/assets/bullet-symbolic.svg.
This may indicate that pixbuf loaders or the mime database could not be found.

###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost

abuild@marxinbox:~/rpmbuild/BUILD/obj/instrumented> echo $?
0
abuild@marxinbox:~/rpmbuild/BUILD/obj/instrumented> xvfb-run '--server-args=-screen 0 1920x1080x24' /home/abuild/rpmbuild/BUILD/obj/instrumented/_virtualenvs/init_py3/bin/python /home/abuild/rpmbuild/BUILD/firefox-83.0/build/pgo/profileserver.py
console.warn: SearchSettings: "get: No settings file exists, new profile?" (new Error("", "(unknown module)"))

###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost

JavaScript error: resource://gre/modules/XULStore.jsm, line 66: Error: Can't find profile directory.
JavaScript error: resource://gre/modules/XULStore.jsm, line 66: Error: Can't find profile directory.
JavaScript error: resource://gre/modules/XULStore.jsm, line 66: Error: Can't find profile directory.
JavaScript error: resource://gre/modules/XULStore.jsm, line 66: Error: Can't find profile directory.
JavaScript error: resource://gre/modules/XULStore.jsm, line 66: Error: Can't find profile directory.
JavaScript error: resource://gre/modules/XULStore.jsm, line 66: Error: Can't find profile directory.

(/home/abuild/rpmbuild/BUILD/obj/instrumented/dist/firefox/firefox:22714): Gtk-WARNING **: 13:16:36.055: Could not load a pixbuf from /org/gtk/libgtk/theme/Adwaita/assets/bullet-symbolic.svg.
This may indicate that pixbuf loaders or the mime database could not be found.
abuild@marxinbox:~/rpmbuild/BUILD/obj/instrumented> echo $?
0
Flags: needinfo?(mliska)

Comparing the log warnings with a successful run vs the log warnings in the failed, run the one that stands out is the "Could not find profraw files" error.

And, behold, that error is associated with a sys.exit(1).

Hm, when I run it manually it works fine:

That's probably because it only looks for profraw files when LLVM_PROFDATA is set. It's set here when profileserver.py is run through the ./mach build command.

If you look inside your obj/instrumented directory, do you see any .profraw files? For example, in my local build, I'm seeing roughly fifty files named like default_482759_random_10565891533688250919_0.profraw.
I wonder if one of your MOZCONFIG options is suppressing the generation of these profraw files, I'll take a closer look.

EDIT: profraw files aren't generated during the build or package steps, but rather are generated when profileserver.py runs. I wonder why it isn't generating the files for you 🤔
EDIT 2: My theory on profileserver.py output being ignored was incorrect, it was being output, I had just mistaken it for earlier build output.

Okay Martin, I've done some more research on my end.
My understanding of how these profraw files are generated are as so:

  1. When LLVM is building Firefox, it's instructed to embed profiling logic.
  2. Then, just by running Firefox and clicking the close button, a bunch of *.profraw files are created. Note that CTRL-C-ing Firefox will prevent the *.profraw files from being created.
  3. Since you're using xvfb, we can't just click the close button. So, we'll need to continue testing with profileserver.py, since it injects scripts to gracefully close Firefox. Note: for your own testing, *.profraw files should be created within ~5 seconds of running profileserver.py.

When you ran profileserver.py there and it returned code 0, how long did it take to fully complete? It should take a minute or two when it's running correctly.
It seems like your firefox binary isn't having that profiling logic embedded, so it's not creating our profraw files.

Can you try removing lines from your mozconfig as much as possible to see if you can triangulate if a single option is causing it?
Perhaps start by seeing if the following mozconfig still doesn't have profraw files created:

mk_add_options MOZ_MAKE_FLAGS=-j16
mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/../obj
ac_add_options --prefix=/usr
ac_add_options --libdir=/usr/lib64
ac_add_options --includedir=/usr/include
ac_add_options --with-system-nspr
ac_add_options --with-system-nss
ac_add_options --with-l10n-base=/home/abuild/rpmbuild/BUILD/l10n
ac_add_options --with-system-zlib
ac_add_options MOZ_PGO=1

Then, if you can remove the --with-system, --with-l10n-base and prefix/lib/include options as well and still reproduce, let me know.

Flags: needinfo?(mliska)

Thank you for your help. I forgot to mention that I do NOT build Firefox with Clang, but with GCC.
After the build failure I can't see any *.profraw file as GCC emits a different set of profile files.

$ time xvfb-run '--server-args=-screen 0 1920x1080x24' /home/abuild/rpmbuild/BUILD/obj/instrumented/_virtualenvs/init_py3/bin/python /home/abuild/rpmbuild/BUILD/firefox-83.0/build/pgo/profileserver.py
...
real	6m34.158s
user	5m20.353s
sys	0m12.435s

It generates a bunch of GCC profile files:

find . -name '*gcda' | wc -l
2510

So as you mentioned, the problem must be that LLVM_PROF is expecting the file.

Flags: needinfo?(mliska)

And I can confirm 'LLVM_PROFDATA': '/usr/bin/llvm-profdata' is set at build_commands.py:112:

            pgo_env = os.environ.copy()
            pgo_env['LLVM_PROFDATA'] = instr.config_environment.substs.get('LLVM_PROFDATA')
            pgo_env['JARLOG_FILE'] = mozpath.join(orig_topobjdir, 'jarlog/en-US.log')
            pgo_cmd = [
                instr.virtualenv_manager.python_path,
                mozpath.join(self.topsrcdir, 'build/pgo/profileserver.py'),
            ]

I forgot to mention that I do NOT build Firefox with Clang, but with GCC.

You should consider to change the compiler.
We are focusing clang currently for all these options!

Heh, yes, that's the issue right there - PGO builds aren't supported with gcc.
Are you able to build with clang instead?

Flags: needinfo?(mhentges)
Flags: needinfo?(mhentges) → needinfo?(mliska)

Sorry, but we as SUSE use (and actively develop GCC compiler), that's why we don't want to use Clang.
The PGO build works for me with the following patch:

--- ./build/pgo/profileserver.py	2020-12-02 15:34:46.453053287 +0100
+++ ./build/pgo/profileserver.py	2020-12-02 15:34:57.988970433 +0100
@@ -203,7 +203,8 @@
             sys.exit(1)
 
         llvm_profdata = env.get('LLVM_PROFDATA')
-        if llvm_profdata:
+        cc = env.get('CC')
+        if llvm_profdata and not 'gcc' in cc:
             profraw_files = glob.glob('*.profraw')
             if not profraw_files:
                 print('Could not find profraw files in the current directory: %s' % os.getcwd())

Can you please upstream the patch?

Flags: needinfo?(mliska) → needinfo?(mhentges)

Hey, I'm glad it's working!
Rather than ignoring the LLVM_PROFDATA variable, what if we instead don't create it for non-clang builds?
Can you try:

--- a/python/mozbuild/mozbuild/build_commands.py
+++ b/python/mozbuild/mozbuild/build_commands.py
@@ -126,9 +126,10 @@ class Build(MachCommandBase):
                 return status
 
             pgo_env = os.environ.copy()
-            pgo_env["LLVM_PROFDATA"] = instr.config_environment.substs.get(
-                "LLVM_PROFDATA"
-            )
+            if instr.config_environment.substs.get("CC_TYPE") == "clang":
+                pgo_env["LLVM_PROFDATA"] = instr.config_environment.substs.get(
+                    "LLVM_PROFDATA"
+                )
             pgo_env["JARLOG_FILE"] = mozpath.join(orig_topobjdir, "jarlog/en-US.log")
             pgo_cmd = [
                 instr.virtualenv_manager.python_path,
Flags: needinfo?(mhentges)

LLVM_PROFDATA is only relevant when LLVM is being used.

Flags: needinfo?(mliska)

Sorry, but we as SUSE use (and actively develop GCC compiler), that's why we don't want to use Clang.

Sure, as a Debian Developer, I can relate to that too :)
Don't hesitate to contact me if you ever want to chat about it!

(In reply to Mitchell Hentges [:mhentges] 🦀 from comment #14)

Created attachment 9191987 [details]
Bug 1680306: Don't set LLVM_PROFDATA for non-clang PGO builds

LLVM_PROFDATA is only relevant when LLVM is being used.

Thank you for the patch, it will work for me.

Flags: needinfo?(mliska)

Note, I don't think the code actually handles GCC PGO, so while the fix would allow the build to go through, the final step of the build is probably not going to use the profile data from the profile step because the files will be in the instrumented build directory, and the final build doesn't know to look there.

That's a good point. I'm vaguely optimistic since we have gcc-specific flags being set to generate and use PGO, so maybe this will behave as expected?
The specific part of the PGO build that was failing was a step where all the profraw files were combined into a single file (or failing if no profraw files could be found). Hopefully the build itself will still slurp the gcda files.

Pushed by mhentges@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/bdb1fd5fd91e Don't set LLVM_PROFDATA for non-clang PGO builds r=firefox-build-system-reviewers,dmajor

PGO with GCC doesn't work by aggregating profile data into one file, and none of the flags we give GCC tell it where it can find the profile data corresponding to each source file, and there is no way for it to know they are in the instrumentation objdir (where I think they are) automatically.

https://hg.mozilla.org/mozilla-central/rev/bdb1fd5fd91e

Status: NEW → RESOLVED
Closed: 4 years ago
status-firefox85: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → 85 Branch

none of the flags we give GCC tell it where it can find the profile data corresponding to each source file,

Hmm, ok. I've created a gcc + PGO bug for future reference.
Martin, it sounds like despite the build not failing, it probably isn't being optimized.

Hmm, ok. I've created a gcc + PGO bug for future reference.

Please do not create one another bug (so far we have 3 PRs that are pretty about the same).
So yes, right now I see many -Wmissing-profile warnings and I've just tested the following patch:

--- ./build/pgo/profileserver.py	2020-12-02 15:34:46.453053287 +0100
+++ ./build/pgo/profileserver.py	2020-12-02 15:34:57.988970433 +0100
@@ -9,6 +9,7 @@
 import sys
 import glob
 import subprocess
+from pathlib import Path
 
 import mozcrash
 from mozbuild.base import MozbuildObject, BinaryNotFoundException
@@ -214,3 +215,12 @@
                 print('INFRA-ERROR: Failed to merge profile data. Corrupt profile?')
                 # exit with TBPL_RETRY
                 sys.exit(4)
+        else:
+            # symlink all *.gcda files in instrumented folder to parent folder
+            gcda_files = glob.glob('**/*.gcda', recursive=True)
+            for gcda in gcda_files:
+                gcda = Path(gcda).resolve()
+                target = Path(str(gcda).replace('instrumented/', ''))
+                target.parent.mkdir(parents=True, exist_ok=True)
+                target.symlink_to(gcda)
+            print(f'Symlinks were created for {len(gcda_files)} gcda files.')

which symlinks all .gcda files from the training phase.
Is it feasible?

Please do not create one another bug (so far we have 3 PRs that are pretty about the same).

These bugs all represent different goals.

  • 1642410 was a bug around py2-to-py3 conversion and wasn't related to gcc
  • this bug was about diagnosing your specific (unknown at report-time) compilation failure
  • 1681536 is to capture the intent to implement GCC support for PGO

So yes, right now I see many -Wmissing-profile warnings and I've just tested the following patch:

Are you seeing performance improvements after doing a build with that patch?

Are you seeing performance improvements after doing a build with that patch?

Yes, it helps as documented in a series of blog posts:
http://hubicka.blogspot.com/2018/12/firefox-64-built-with-gcc-and-clang.html

So right now the profile is accepted, but I see quite some mismatches:

[ 2310s] 36:52.66 /home/abuild/rpmbuild/BUILD/firefox-83.0/xpcom/base/nsWeakReference.cpp: In member function ‘mozilla::MozPromise<ProcessInfo, nsresult, false>::ThenValue<nsSystemInfo::GetProcessInfo(JSContext*, mozilla::dom::Promise**)::{lambda(ProcessInfo const&)#2}, nsSystemInfo::GetProcessInfo(JSContext*, mozilla::dom::Promise**)::{lambda(nsresult)#3}>::DoResolveOrRejectInternal(mozilla::MozPromise<ProcessInfo, nsresult, false>::ResolveOrRejectValue&)’:
[ 2310s] 36:52.66 /home/abuild/rpmbuild/BUILD/obj/dist/include/mozilla/MozPromise.h:762:10: warning: profile for function ‘mozilla::MozPromise<ProcessInfo, nsresult, false>::ThenValue<nsSystemInfo::GetProcessInfo(JSContext*, mozilla::dom::Promise**)::{lambda(ProcessInfo const&)#2}, nsSystemInfo::GetProcessInfo(JSContext*, mozilla::dom::Promise**)::{lambda(nsresult)#3}>::DoResolveOrRejectInternal(mozilla::MozPromise<ProcessInfo, nsresult, false>::ResolveOrRejectValue&)’ not found in profile data [-Wmissing-profile]
[ 2310s] 36:52.66   762 |     void DoResolveOrRejectInternal(ResolveOrRejectValue& aValue) override {
[ 2310s] 36:52.66       |          ^~~~~~~~~~~~~~~~~~~~~~~~~
[ 2310s] 36:52.66 /home/abuild/rpmbuild/BUILD/firefox-83.0/xpcom/base/nsWeakReference.cpp: In member function ‘mozilla::MozPromise<ProcessInfo, nsresult, false>::ThenValue<nsSystemInfo::GetProcessInfo(JSContext*, mozilla::dom::Promise**)::{lambda(ProcessInfo const&)#2}, nsSystemInfo::GetProcessInfo(JSContext*, mozilla::dom::Promise**)::{lambda(nsresult)#3}>::CompletionPromise() const’:
[ 2310s] 36:52.66 /home/abuild/rpmbuild/BUILD/obj/dist/include/mozilla/MozPromise.h:758:21: warning: profile for function ‘mozilla::MozPromise<ProcessInfo, nsresult, false>::ThenValue<nsSystemInfo::GetProcessInfo(JSContext*, mozilla::dom::Promise**)::{lambda(ProcessInfo const&)#2}, nsSystemInfo::GetProcessInfo(JSContext*, mozilla::dom::Promise**)::{lambda(nsresult)#3}>::CompletionPromise() const’ not found in profile data [-Wmissing-profile]
[ 2310s] 36:52.66   758 |     MozPromiseBase* CompletionPromise() const override {
[ 2310s] 36:52.66       |                     ^~~~~~~~~~~~~~~~~
[ 2310s] 36:52.66 /home/abuild/rpmbuild/BUILD/firefox-83.0/xpcom/base/nsWeakReference.cpp: In member function ‘mozilla::MozPromise<ProcessInfo, nsresult, false>::ThenValue<nsSystemInfo::GetProcessInfo(JSContext*, mozilla::dom::Promise**)::{lambda(ProcessInfo const&)#2}, nsSystemInfo::GetProcessInfo(JSContext*, mozilla::dom::Promise**)::{lambda(nsresult)#3}>::Disconnect()’:
[ 2310s] 36:52.66 /home/abuild/rpmbuild/BUILD/obj/dist/include/mozilla/MozPromise.h:746:10: warning: profile for function ‘mozilla::MozPromise<ProcessInfo, nsresult, false>::ThenValue<nsSystemInfo::GetProcessInfo(JSContext*, mozilla::dom::Promise**)::{lambda(ProcessInfo const&)#2}, nsSystemInfo::GetProcessInfo(JSContext*, mozilla::dom::Promise**)::{lambda(nsresult)#3}>::Disconnect()’ not found in profile data [-Wmissing-profile]

[ 2310s] 36:52.66 /home/abuild/rpmbuild/BUILD/obj/dist/include/nsExpirationTracker.h:514:8: warning: source locations for function ‘nsExpirationTracker<mozilla::layers::ActiveResource, 3u>::NotifyHandlerEndLocked(detail::PlaceholderAutoLock const&)’ have changed, the profile data may be out of date [-Wcoverage-mismatch]
[ 2310s] 36:52.66 /home/abuild/rpmbuild/BUILD/firefox-83.0/xpcom/base/nsWeakReference.cpp: In member function ‘nsExpirationTracker<mozilla::layers::ActiveResource, 3u>::NotifyExpiredLocked(mozilla::layers::ActiveResource*, detail::PlaceholderAutoLock const&)’:
[ 2310s] 36:52.66 /home/abuild/rpmbuild/BUILD/obj/dist/include/nsExpirationTracker.h:505:8: warning: source locations for function ‘nsExpirationTracker<mozilla::layers::ActiveResource, 3u>::NotifyExpiredLocked(mozilla::layers::ActiveResource*, detail::PlaceholderAutoLock const&)’ have changed, the profile data may be out of date [-Wcoverage-mismatch]
[ 2310s] 36:52.66   505 |   void NotifyExpiredLocked(T* aObject, const AutoLock&) override {
[ 2310s] 36:52.66       |        ^~~~~~~~~~~~~~~~~~~
[ 2310s] 36:52.66 /home/abuild/rpmbuild/BUILD/obj/dist/include/nsExpirationTracker.h:505:8: warning: source locations for function ‘nsExpirationTracker<mozilla::layers::ActiveResource, 3u>::NotifyExpiredLocked(mozilla::layers::ActiveResource*, detail::PlaceholderAutoLock const&)’ have changed, the profile data may be out of date [-Wcoverage-mismatch]
[ 2310s] 36:52.66 /home/abuild/rpmbuild/BUILD/obj/dist/include/nsExpirationTracker.h:505:8: warning: source locations for function ‘nsExpirationTracker<mozilla::layers::ActiveResource, 3u>::NotifyExpiredLocked(mozilla::layers::ActiveResource*, detail::PlaceholderAutoLock const&)’ have changed, the profile data may be out of date [-Wcoverage-mismatch]
[ 2310s] 36:52.66 In file included from /home/abuild/rpmbuild/BUILD/obj/dist/include/mozilla/ipc/MessageChannel.h:16,
Summary: PGO build failing on SUSE → PGO build failing on SUSE: "Could not find profraw files in the current directory"

So right now the profile is accepted, but I see quite some mismatches ...

My knowledge in how the compiler is invoked and parameterized in the Firefox build is not very strong, so I think the toolchain team will be better at diagnosing this than me. I've sent the gcc + PGO bug to them accordingly.
However, our teams have been squeezed pretty tight recently, so I'm not sure how much they'll be able to prioritize the work.

See Also: → 1681536
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: