1045133 - Windows PGO build times increased by ~50% on Friday

Status: RESOLVED WORKSFORME

(Firefox Build System :: General, defect)

Description

[Ryan VanderMeulen [:RyanVM]]

Routinely pushing (and often hitting) the 5.5hr timeout limit now:
https://tbpl.mozilla.org/php/getParsedLog.php?id=44724884&tree=Mozilla-Inbound

Regression range:
https://tbpl.mozilla.org/?jobname=WINNT%205.2%20mozilla-central%20pgo-build&fromchange=6c0971104909&tochange=e07264876182

Lots of build system changes in that inbound mega-merge. Will try to narrow it down on inbound next.

Comment 1

[Ryan VanderMeulen [:RyanVM]]

Tentative inbound regression range:
https://tbpl.mozilla.org/?tree=Mozilla-Inbound&jobname=WINNT%205.2%20mozilla-inbound%20pgo-build&fromchange=4563a79a7968&tochange=0300f04572b6

The run on rev 51ff7710ccbc tells me that the regression was before glandium's build system changes landed. Below that, bug 1038537 stands out as a definite maybe. Also, bug 1043534 looks like a possibility.

Comment 2

[Ryan VanderMeulen [:RyanVM]]

Backout bug 1038537: https://tbpl.mozilla.org/?tree=Try&rev=fbdfa49e3b0c

Backout bug 1043534: https://tbpl.mozilla.org/?tree=Try&rev=d50998766017

Backout glandium's push: https://tbpl.mozilla.org/?tree=Try&rev=46e80adcc5e2

Comment 3

[Mike Hommey [:glandium]]

Reading the build log, it appears for some reason, libxul is rebuilt during the libs phase. Which means bug 1043865 is going to fix this.

Comment 4

[Ryan VanderMeulen [:RyanVM]]

Retriggers appear to be pointing at Nate's push as the regressor (the push prior ran in 197min):
https://tbpl.mozilla.org/?tree=Mozilla-Inbound&jobname=WINNT%205.2%20mozilla-inbound%20pgo-build&fromchange=4563a79a7968&tochange=0300f04572b6&rev=2246b59a134a

(In reply to Mike Hommey [:glandium] from comment #3)
> Reading the build log, it appears for some reason, libxul is rebuilt during
> the libs phase. Which means bug 1043865 is going to fix this.

Is that going to land soon? This regression is absolutely brutal from a sheriff merge standpoint. Otherwise I'll just back bug 1043534 out and let you two sort it out.

Comment 5

[Mike Hommey [:glandium]]

This was fixed by bug 1043865.