1563053 - Investigate setting LLVM_PROFILE_FILE to include pid for Android

Status: RESOLVED FIXED

(Firefox Build System :: General, task)

Description

[Michael Shal [:mshal]]

As of bug 1196094, desktop PGO writes out separate profraw files for each PID. Android however still writes out to a single "default.profraw" file. We should investigate whether it helps to use the same mechanism in Android and merge them all.

This is slightly complicated by the fact that we have to pull the profraw files from the emulator, and we don't necessarily know when they finish getting written out. We currently poll based on default.profraw's existence, but perhaps that should be updated to just check if the process is still running.

Comment 1

[Michael Shal [:mshal]]

I took a look at this - even with '%p' in LLVM_PROFILE_FILE, we still only generate a single profraw file for Android. However, adding support for it doesn't look too difficult so I'll submit the patches anyway.

The code that polls for 'default.profraw' needed to be changed since the mozharness script won't know the full filename ahead of time. It seems like it should be more stable in any case to check for the Android process completing rather than looking at files that may not be fully written out (perhaps this is an alternate solution to bug 1560755, but we can keep that as a failsafe).

Comment 2

[Michael Shal [:mshal]]

Attachment: Bug 1563053 - Poll Android PGO based on process existence, not file existence; r?snorp

After execute_script() returns, the Android process may not be fully
shutdown, so the profile data may not exist yet. The initial
implementation waited for this by checking if the profraw file existed
and was non-zero length, but this may be contributing to issues with
getting partial profraw files.

Instead we can wait for the whole process to exit by using
process_exist(), so the profraw file should be completely written out
by then. In the event that the profraw file is truncated for some
reason, the merge step will fail and the process will be retried (per
bug 1560755).

Comment 3

[Michael Shal [:mshal]]

Attachment: Bug 1563053 - Put all Android PGO outputs in a separate directory; r?#firefox-build-system-reviewers

If we end up generating multiple profraw files in the future, it will
help to have them all in a separate directory so that we can just 'adb
pull' the whole directory at once.

Depends on D36839

Comment 4

[Michael Shal [:mshal]]

Attachment: Bug 1563053 - Wildcard profraw files for Android PGO; r?#firefox-build-system-reviewers

This adds support for the Android process writing out multiple profraw
files and pulling them all from the device. We currently only generate a
single profraw file, but if that changes in the future we should be able
to get a PGO build using the full set of files now.

Depends on D36840

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/8d69fa280da3
Poll Android PGO based on process existence, not file existence; r=snorp
https://hg.mozilla.org/integration/autoland/rev/3b86e1396fd2
Put all Android PGO outputs in a separate directory; r=firefox-build-system-reviewers,chmanchester
https://hg.mozilla.org/integration/autoland/rev/2ad7b1da6d22
Wildcard profraw files for Android PGO; r=firefox-build-system-reviewers,chmanchester

Comment 6

[Arthur Iakab [arthur_iakab]]

https://hg.mozilla.org/mozilla-central/rev/8d69fa280da3
https://hg.mozilla.org/mozilla-central/rev/3b86e1396fd2
https://hg.mozilla.org/mozilla-central/rev/2ad7b1da6d22