676252 - PGO has very few profile data on incremental builds

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Mike Hommey [:glandium]]

We don't rebuild the objects that are already in the objdir, and they are final objects, not ones with instrumentation... (doh)

So we end up running the profile without instrumentation in most object files.

The easy solution would be to clobber before the profile generate pass, but that would be a huge waste of time. I think an easy way to avoid this problem, while keeping fast(ish) builds is to change the object file extension in one of the passes so that we effectively never overwrite object files from the profile generate pass in the profile use pass. That would double disk usage for object files, though.

On the other hand, we could probably skip debugging symbols in the profile generate pass, which would make the object files smaller. (though they might be useful in the case running firefox for the profile crashes)

Comment 1

[Justin Lebar (not reading bugmail)]

> The easy solution would be to clobber before the profile generate pass, but 
> that would be a huge waste of time.

Catlee has been working on patches to ccache to support -fprofile-generate and -fprofile-use.  If used that, clobbering wouldn't be so bad.

But of course what you're proposing sounds more clever...

Comment 2

[Chris AtLee [:catlee]]

Is this specific to Linux, or would this impact Windows as well?

Comment 3

[Mike Hommey [:glandium]]

This is specific to linux.

Comment 4

[Mike Hommey [:glandium]]

Attachment: Use a different object file extension for the first PGO build pass with GCC

Comment 5

[Mike Hommey [:glandium]]

Attachment: Use a different object file extension for the first PGO build pass with GCC

Comment 6

[(not currently active) Ted Mielczarek]

Comment on attachment 551432 [details] [diff] [review]
Use a different object file extension for the first PGO build pass with GCC

Review of attachment 551432 [details] [diff] [review]:
-----------------------------------------------------------------

::: config/config.mk
@@ +784,5 @@
>  endif
> +
> +ifdef MOZ_PROFILE_GENERATE
> +ifdef GNU_CC
> +OBJ_SUFFIX := i_o

This could stand a short comment explaining why it's here.

::: config/rules.mk
@@ +916,5 @@
> +ifdef GNU_CC
> +# Force rebuilding libraries and programs in both passes because each
> +# pass uses different object files.
> +$(PROGRAM) $(SHARED_LIBRARY) $(LIBRARY): pgo.relink
> +.PHONY: pgo.relink

If this is phony, just use FORCE like we do elsewhere.

Comment 7

[Mike Hommey [:glandium]]

Attachment: Use a different object file extension for the first PGO build pass with GCC

As landed

Comment 8

[Mike Hommey [:glandium]]

http://hg.mozilla.org/projects/build-system/rev/dfe63797f67a

Comment 9

[Mike Hommey [:glandium]]

I don't know why I thought NO_PROFILE_GUIDED_OPTIMIZE would work in build/unix/elfhack. Fixup:
http://hg.mozilla.org/projects/build-system/rev/66b336cc59b1

Comment 10

[Mike Hommey [:glandium]]

Backed out. It looks like Linux x86 doesn't like the i_o extension and fails with:
 /tools/gcc-4.5/lib/gcc/i686-pc-linux-gnu/4.5.2/include/emmintrin.h:32:3: error: #error "SSE2 instruction set not enabled"

http://hg.mozilla.org/projects/build-system/rev/15efac661d6a

:(

Comment 11

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

http://hg.mozilla.org/mozilla-central/rev/dfe63797f67a

Comment 12

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

Ah, this bounced http://hg.mozilla.org/mozilla-central/rev/15efac661d6a

Comment 13

[Mike Hommey [:glandium]]

Attachment: part 1 - Don't use $(OBJ_SUFFIX) before including config.mk

Comment 14

[Mike Hommey [:glandium]]

Attachment: part 2 - Use a different object file extension for the first PGO build pass with GCC

Comment 15

[Mike Hommey [:glandium]]

http://hg.mozilla.org/projects/build-system/rev/6bedd46abb82
http://hg.mozilla.org/projects/build-system/rev/480f913edf3f

Comment 16

[Mike Hommey [:glandium]]

Backed out part 2.
http://hg.mozilla.org/projects/build-system/rev/ac2115d5bf39

Comment 17

[Mike Hommey [:glandium]]

Relanded unchanged (red was due to bug 682794).
http://hg.mozilla.org/projects/build-system/rev/66fb17949bd6

Comment 18

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

http://hg.mozilla.org/mozilla-central/rev/6bedd46abb82
http://hg.mozilla.org/mozilla-central/rev/480f913edf3f
Backout of second cset:
http://hg.mozilla.org/mozilla-central/rev/ac2115d5bf39
Relanding:
http://hg.mozilla.org/mozilla-central/rev/66fb17949bd6