1553823 - figure out how to remove the default.profraw hardcoded filename

Status: NEW

(Firefox Build System :: General, task, P3)

Description

[Nathan Froyd [:froydnj]]

Gijs noticed that the patches in bug 1542746 break our 3-stage PGO builds, because we no longer end up with a single default.profraw file in the topobjdir for scripts to poke at: instead, we wind up with several .profraw files. The "normal" PGO build doesn't particularly care what the files are called; it just globs a bunch of things together:

https://searchfox.org/mozilla-central/source/Makefile.in#191
https://searchfox.org/mozilla-central/source/Makefile.in#279-281

But for multi-stage builds, we care very much:

https://searchfox.org/mozilla-central/search?q=default.profraw&path=

I think the mozconfigs are easy to deal with, and the run-profileserver.sh instance is easy to deal with (we could run llvm-profdata merge in run-profileserver.sh, I think?). But I don't know what to do about the android side of things. We're polling the device:

https://searchfox.org/mozilla-central/source/testing/mozharness/scripts/android_emulator_pgo.py#250-268

to pull the file, but under the new PGO scheme in bug 1542746, we wouldn't actually know what the files are named to poll for them. I guess we could poll ls on the device, but then we'd be hardcoding a count of the .profraw files that the profiling run would deposit, which doesn't seem very robust.

mshal, do you have ideas on what we could do here?

Comment 1

[Nathan Froyd [:froydnj]]

Apparently on Android this is not a problem, because we set LLVM_PROFILE_FILE=mumble/default.profraw. So we just have to solve this on Linux, which is an easier task. But some future Android build that's e10s enabled will still run into this problem...punting things down the road is OK?

Comment 2

[Michael Shal [:mshal]]

(In reply to Nathan Froyd [:froydnj] from comment #0)

I think the mozconfigs are easy to deal with, and the run-profileserver.sh instance is easy to deal with (we could run llvm-profdata merge in run-profileserver.sh, I think?). But I don't know what to do about the android side of things. We're polling the device:

We don't (currently) have a clang toolchain in the 'run' task, so llvm-profdata isn't available. (In the build task, we obviously have clang and then setup the path to llvm-profdata in the mozconfig). We could probably add that if it makes things simpler, or maybe just tar up *.profraw and have all the files in profdata.tar.xz, with the merge still happening in the 3rd task.

https://searchfox.org/mozilla-central/source/testing/mozharness/scripts/android_emulator_pgo.py#250-268

to pull the file, but under the new PGO scheme in bug 1542746, we wouldn't actually know what the files are named to poll for them. I guess we could poll ls on the device, but then we'd be hardcoding a count of the .profraw files that the profiling run would deposit, which doesn't seem very robust.

Android also differs slightly in that we explicitly call llvm_profile_dump() since the atexit handlers aren't called:

https://searchfox.org/mozilla-central/rev/662de518b1686c4769320d6b8825ce4864c4eda0/mozglue/android/APKOpen.cpp#385
https://searchfox.org/mozilla-central/rev/662de518b1686c4769320d6b8825ce4864c4eda0/toolkit/xre/Bootstrap.cpp#89

That may impact how the files are written out.

Comment 3

[BugBot [:suhaib / :marco/ :calixte]]

Bugbug thinks this bug is a task, but please change it back in case of error.