935637 - Publish PGO profiles

Status: RESOLVED WORKSFORME

(Firefox Build System :: General, defect)

Description

[Gregory Szorc [:gps]]

To achieve verifiable and reproducible Firefox builds, Mozilla will need to publish the intermediate PGO profiles obtained from profiling and used to produce the final PGO-optimized binaries.

I'm not sure what the overhead of these are. We should check with release engineering that the extra storage is acceptable before we enable this.

I believe this bug is fully contained within build config since we can upload arbitrary files as part of the build.

Comment 1

[(not currently active) Ted Mielczarek]

I think this will require tarring up all .gcda files (one per source file) in the object directory, and preserving the relative directory structure. Shouldn't be incredibly complex, something like
find $objdir -name "*.gcda" | xargs tar

would suffice, but we could write it in Python to make it less shell-y if you wanted.

Comment 2

[Trevor Saunders (:tbsaunde)]

(In reply to Gregory Szorc [:gps] from comment #0)
> To achieve verifiable and reproducible Firefox builds, Mozilla will need to
> publish the intermediate PGO profiles obtained from profiling and used to
> produce the final PGO-optimized binaries.

So, arguably this data is a product of the build not an input, and so your basically screwed if you want a buildthat you can reproduce and uses pgo.  Using this data would basically be saying "I only want to reproduce the final build, not the first build or the profile run".  Which I guess is kind of fine if you assume for all possible sets of profile data the compiler will produce a correct if not optimal binary, but I'm kind of dubious that's the case in practice.

> I'm not sure what the overhead of these are. We should check with release
> engineering that the extra storage is acceptable before we enable this.

Presumably people would only find this useful for releasish builds like nightlies betas and such, not random periodic pgo done for testing, so its probably ok? I'm not sure how big a random .gcda file tends to be.
presumably people would only want this for things that are atleast vaguely

Comment 3

[Gregory Szorc [:gps]]

AFAICT deterministic/idempotent PGO builds are not possible if the profile generation step is performed because the profile generation step collects very granular timings and things like I/O variance and context switching can impact timings. Therefore the best we can achieve is verifiable builds. i.e. an external entity can verify that the bits Mozilla ships are what we say we ship. They'll have to plug in our published PGO profile, but it's possible.

Yes, we likely only care about verifiability for releasish builds, not "random" PGO builds in inbound, etc. This may impact our implementation for uploading PGO profile data.

Does release automation even tell us when we're doing an "official"/"shipped" build as opposed to a regular build? I thought the configs were mostly the same and the only difference was in release automation land.

Comment 4

[Trevor Saunders (:tbsaunde)]

(In reply to Gregory Szorc [:gps] from comment #3)
> AFAICT deterministic/idempotent PGO builds are not possible if the profile
> generation step is performed because the profile generation step collects
> very granular timings and things like I/O variance and context switching can

yes, that's what I meant by being screwed.

> impact timings. Therefore the best we can achieve is verifiable builds. i.e.
> an external entity can verify that the bits Mozilla ships are what we say we
> ship. They'll have to plug in our published PGO profile, but it's possible.

At which point your just trusting our signature of the pgo data instead of the final binary, but maybe that's kind of useful I guess.

> Yes, we likely only care about verifiability for releasish builds, not
> "random" PGO builds in inbound, etc. This may impact our implementation for
> uploading PGO profile data.
> 
> Does release automation even tell us when we're doing an
> "official"/"shipped" build as opposed to a regular build? I thought the
> configs were mostly the same and the only difference was in release
> automation land.

yes its available see MOZ_UPDATE_CHANNEL and friends, we use it already for things like disabling stuff on aurora and later.

Comment 5

[(not currently active) Ted Mielczarek]

(In reply to Trevor Saunders (:tbsaunde) from comment #4)
> At which point your just trusting our signature of the pgo data instead of
> the final binary, but maybe that's kind of useful I guess.

The PGO data is a lot simpler than the binary. Being able to prove that a specified build environment + a set of PGO profile data results in the same binary is useful.

Comment 6

[Dan]

Hi! 

So, I would like to work on this bug. Would it be OK to be assigned to me? 

Thanks,
Dan

Comment 7

[Gregory Szorc [:gps]]

There has been no movement on this bug in 3 years: feel free to start working on it.

I will caution you: I don't think this bug is a good first bug. PGO takes a long time to run and it is somewhat hackily integrated into the build system. But it isn't so bad once you know where to look for things. Consider joining #build on irc.mozilla.org and ask for help. ted is probably the person who can help you the most.

Since comment #1 was written, we now have a binaries.json file in the object directory. This file can likely be used to find locations of .gcda files. I'd strongly prefer to not use `find` or `os.walk()` as part of this solution (for performance reasons).

Comment 8

[Dan]

OK, I will take your advice into mind. Thanks for the heads up :) 

Also, I will begin working on this bug ASAP, so this means that
I will take on the challenge to fix it.

Comment 9

[Dan]

Hi Ted! This is Dan. 

I was wondering if you recommend any good incision points
for starting to work on this bug ( where should I begin looking? Any resources you 
recommend for me to start?) 

Thanks a ton, and if you can, can you please assign me to this bug?

Dan

Comment 10

[(not currently active) Ted Mielczarek]

Hi Dan,

I apologize for the radio silence! If you're still interested in taking this bug, I'd recommend you try something simple to start. the `stage-package` target in packager.mk is the thing that does most of the work during packaging:
https://dxr.mozilla.org/mozilla-central/rev/f0f15b7c6aa77a0c5750918aa0a1cb3dc82185bc/toolkit/mozapps/installer/packager.mk#40

It looks like recently someone added support for packaging up the .gcno files from code coverage builds, which is shockingly similar to what you'd need to do here:
https://dxr.mozilla.org/mozilla-central/rev/f0f15b7c6aa77a0c5750918aa0a1cb3dc82185bc/toolkit/mozapps/installer/packager.mk#70

The Python script that is invoked from there is here:
https://dxr.mozilla.org/mozilla-central/source/python/mozbuild/mozbuild/codecoverage/packager.py

You could probably reuse that, just changing .gcno to .gcda and get what you need. Maybe add an option to that script to specify the file extension it's looking for?

You'd need to have your new invocation inside `ifdef MOZ_PGO` so it's only used during PGO builds. Actually, we call the packager in the middle of the PGO build so we can run Firefox in a configuration that matches what we ship:
https://dxr.mozilla.org/mozilla-central/rev/f0f15b7c6aa77a0c5750918aa0a1cb3dc82185bc/client.mk#233

...so we might want to make the new invocation stricter such that we're not doing extra work in the middle of the build. The `make package` call in client.mk during the PGO build sets `MOZ_PGO_INSTRUMENTED=1`, so you can probably do:
ifeq (1_,$(MOZ_PGO)_$(MOZ_PGO_INSTRUMENTED))
... invoke the script here
endif

...sorry, make's conditional syntax is terrible.

Let me know if that's enough to get you started. If you get stuck, feel free to ping me via email or on irc (ted).

Comment 11

[(Away)]

This was achieved as a side effect of bug 1507334.