Last Comment Bug 676252 - PGO has very few profile data on incremental builds
: PGO has very few profile data on incremental builds
Status: RESOLVED FIXED
fixed-in-bs
:
Product: Core
Classification: Components
Component: Build Config (show other bugs)
: Trunk
: All Linux
: -- normal (vote)
: mozilla9
Assigned To: Mike Hommey [:glandium]
:
Mentors:
Depends on: 682794
Blocks: 653961 677191
  Show dependency treegraph
 
Reported: 2011-08-03 07:39 PDT by Mike Hommey [:glandium]
Modified: 2011-08-31 08:05 PDT (History)
6 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---


Attachments
Use a different object file extension for the first PGO build pass with GCC (10.10 KB, patch)
2011-08-08 05:43 PDT, Mike Hommey [:glandium]
no flags Details | Diff | Review
Use a different object file extension for the first PGO build pass with GCC (10.21 KB, patch)
2011-08-08 05:50 PDT, Mike Hommey [:glandium]
ted: review+
Details | Diff | Review
Use a different object file extension for the first PGO build pass with GCC (10.93 KB, patch)
2011-08-16 00:42 PDT, Mike Hommey [:glandium]
no flags Details | Diff | Review
part 1 - Don't use $(OBJ_SUFFIX) before including config.mk (10.61 KB, patch)
2011-08-18 22:56 PDT, Mike Hommey [:glandium]
ted: review+
Details | Diff | Review
part 2 - Use a different object file extension for the first PGO build pass with GCC (12.07 KB, patch)
2011-08-18 22:57 PDT, Mike Hommey [:glandium]
ted: review+
Details | Diff | Review

Description Mike Hommey [:glandium] 2011-08-03 07:39:57 PDT
We don't rebuild the objects that are already in the objdir, and they are final objects, not ones with instrumentation... (doh)

So we end up running the profile without instrumentation in most object files.

The easy solution would be to clobber before the profile generate pass, but that would be a huge waste of time. I think an easy way to avoid this problem, while keeping fast(ish) builds is to change the object file extension in one of the passes so that we effectively never overwrite object files from the profile generate pass in the profile use pass. That would double disk usage for object files, though.

On the other hand, we could probably skip debugging symbols in the profile generate pass, which would make the object files smaller. (though they might be useful in the case running firefox for the profile crashes)
Comment 1 Justin Lebar (not reading bugmail) 2011-08-03 08:07:18 PDT
> The easy solution would be to clobber before the profile generate pass, but 
> that would be a huge waste of time.

Catlee has been working on patches to ccache to support -fprofile-generate and -fprofile-use.  If used that, clobbering wouldn't be so bad.

But of course what you're proposing sounds more clever...
Comment 2 Chris AtLee [:catlee] 2011-08-03 09:00:18 PDT
Is this specific to Linux, or would this impact Windows as well?
Comment 3 Mike Hommey [:glandium] 2011-08-03 09:07:50 PDT
This is specific to linux.
Comment 4 Mike Hommey [:glandium] 2011-08-08 05:43:41 PDT
Created attachment 551430 [details] [diff] [review]
Use a different object file extension for the first PGO build pass with GCC
Comment 5 Mike Hommey [:glandium] 2011-08-08 05:50:40 PDT
Created attachment 551432 [details] [diff] [review]
Use a different object file extension for the first PGO build pass with GCC
Comment 6 Ted Mielczarek [:ted.mielczarek] 2011-08-12 09:02:22 PDT
Comment on attachment 551432 [details] [diff] [review]
Use a different object file extension for the first PGO build pass with GCC

Review of attachment 551432 [details] [diff] [review]:
-----------------------------------------------------------------

::: config/config.mk
@@ +784,5 @@
>  endif
> +
> +ifdef MOZ_PROFILE_GENERATE
> +ifdef GNU_CC
> +OBJ_SUFFIX := i_o

This could stand a short comment explaining why it's here.

::: config/rules.mk
@@ +916,5 @@
> +ifdef GNU_CC
> +# Force rebuilding libraries and programs in both passes because each
> +# pass uses different object files.
> +$(PROGRAM) $(SHARED_LIBRARY) $(LIBRARY): pgo.relink
> +.PHONY: pgo.relink

If this is phony, just use FORCE like we do elsewhere.
Comment 7 Mike Hommey [:glandium] 2011-08-16 00:42:07 PDT
Created attachment 553397 [details] [diff] [review]
Use a different object file extension for the first PGO build pass with GCC

As landed
Comment 8 Mike Hommey [:glandium] 2011-08-16 00:44:39 PDT
http://hg.mozilla.org/projects/build-system/rev/dfe63797f67a
Comment 9 Mike Hommey [:glandium] 2011-08-16 10:29:48 PDT
I don't know why I thought NO_PROFILE_GUIDED_OPTIMIZE would work in build/unix/elfhack. Fixup:
http://hg.mozilla.org/projects/build-system/rev/66b336cc59b1
Comment 10 Mike Hommey [:glandium] 2011-08-16 23:32:12 PDT
Backed out. It looks like Linux x86 doesn't like the i_o extension and fails with:
 /tools/gcc-4.5/lib/gcc/i686-pc-linux-gnu/4.5.2/include/emmintrin.h:32:3: error: #error "SSE2 instruction set not enabled"

http://hg.mozilla.org/projects/build-system/rev/15efac661d6a

:(
Comment 11 Kyle Huey [:khuey] (khuey@mozilla.com) 2011-08-18 03:51:01 PDT
http://hg.mozilla.org/mozilla-central/rev/dfe63797f67a
Comment 12 Kyle Huey [:khuey] (khuey@mozilla.com) 2011-08-18 03:51:33 PDT
Ah, this bounced http://hg.mozilla.org/mozilla-central/rev/15efac661d6a
Comment 13 Mike Hommey [:glandium] 2011-08-18 22:56:18 PDT
Created attachment 554313 [details] [diff] [review]
part 1 - Don't use $(OBJ_SUFFIX) before including config.mk
Comment 14 Mike Hommey [:glandium] 2011-08-18 22:57:18 PDT
Created attachment 554314 [details] [diff] [review]
part 2 - Use a different object file extension for the first PGO build pass with GCC
Comment 16 Mike Hommey [:glandium] 2011-08-28 23:33:06 PDT
Backed out part 2.
http://hg.mozilla.org/projects/build-system/rev/ac2115d5bf39
Comment 17 Mike Hommey [:glandium] 2011-08-29 04:53:19 PDT
Relanded unchanged (red was due to bug 682794).
http://hg.mozilla.org/projects/build-system/rev/66fb17949bd6

Note You need to log in before you can comment on or make changes to this bug.