874139 - Windows PGO builds sometimes appear to build without PGO

Status: RESOLVED WORKSFORME

(Firefox Build System :: General, defect)

Description

[Matt Brubeck (:mbrubeck)]

A small but consistent portion of Windows PGO builds apparently fail to actually use profile-guided optimization.  These builds show up as outliers on our Windows PGO performance graphs, both on the cost side (they do not use the extra linker memory that PGO builds use) and the benefit side (their performance does not benefit from PGO).  This has been happening for at least several months.

Among other things, this breaks automated regression detection on PGO branches because these outliers add a huge amount of variance to the data.

Ehsan mentioned that a previous investigation of these builds always showed changes to configure.in or some other forced reconfig.  Ted thought this bug might be related to our use of "pgomgr -clear" to reset .pgd files.

Comment 1

[(not currently active) Ted Mielczarek]

So, we initially implemented this in bug 659942 as "rm whatever.pgd", and that broke for the fairly obvious reason that if we did a PGO depend build and we didn't have to re-link certain DLLs the PGD file didn't get recreated during the first build pass, so the second build pass would fail looking for it.

bug 679334 fixed this by using 'pgomgr -clear', but it's possible that that's not really doing what we want. We can try going back to rm'ing the pgd file, I think we could make that work if we inserted the rm into the library link rule, like:
http://mxr.mozilla.org/mozilla-central/source/config/rules.mk#957

ifdef MOZ_PROFILE_GENERATE
  $(RM) *.pgd
endif

This should be okay because it'd only get invoked when we're re-linking the binary in the instrumentation phase. We can push that to try and spin a few extra PGO builds to see if it actually has any effect.

Comment 2

[Matt Brubeck (:mbrubeck)]

It looks like this has not happened since around the time bug 871712 landed.