Can no longer do Aurora Win64 PGO builds on Try

RESOLVED FIXED

Status

defect
--
blocker
RESOLVED FIXED
4 years ago
Last year

People

(Reporter: jandem, Assigned: glandium)

Tracking

unspecified
Dependency tree / graph

Firefox Tracking Flags

(firefox42 fixed)

Details

Attachments

(3 attachments)

I need to debug a PGO-only Win64 issue, see bug 1180948 and bug 1167883. The former is keeping Aurora closed, so this is pretty urgent.

Unfortunately all PGO builds I do are timing out on Try:

Try push from yesterday:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=ebd81d3a3722

Try push from last month:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=38a8b8a65b32
Severity: normal → blocker
I've been looking at this today - no leads just yet. The timeouts for try look to be the same as for mozilla-central, so something is causing this to take a lot longer to build than nightlies on trunk.
Disabling sccache seems to have fixed this for me:

https://hg.mozilla.org/try/rev/fa7efed1756d#l2.17

glandium, any ideas why?
Flags: needinfo?(mh+mozilla)
As I said on irc, this might be due to the difference in how debug symbols are treated with sccache enabled. That's not a problem on non-try because MOZ_PGO is set before mozconfig.cache is included, and mozconfig.cache disables sccache when MOZ_PGO is set.
Flags: needinfo?(mh+mozilla)
Why would having sccache enabled make PGO builds so much slower? In particular, the 2nd link of xul.dll is taking more than 2x as long.
Flags: needinfo?(mh+mozilla)
See comment 3.
Flags: needinfo?(mh+mozilla)
Anyways, the reasonable thing to do would be to ensure mozconfig.cache is included last (or at least after mozconfig.common.override) instead of wherever it's included now.
(In reply to Mike Hommey [:glandium] from comment #6)
> Anyways, the reasonable thing to do would be to ensure mozconfig.cache is
> included last (or at least after mozconfig.common.override) instead of
> wherever it's included now.

great - can you get a patch up for that?
Assignee: nobody → mh+mozilla
Attachment #8632009 - Flags: review?(mshal)
Attachment #8632009 - Flags: review?(mshal) → review+
https://hg.mozilla.org/mozilla-central/rev/e6d5c78cb896
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
Depends on: 1183328
It wasn't enough. sccache doesn't get disabled because mk_add_options doesn't have an effect on local variables in mozconfig, so the tests in mozconfig.cache don't work.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
It is useful to be able, during mozconfig execution, to do tests depending
on what was previously added with mk_add_options. Specifically, there is a
need to do this for MOZ_PGO because developers pushing to try may add it to
mozconfig.common.override.

While, ideally, it would be nice if we just defined the variable itself in
the mozconfig execution environment, that is a tedious task, having to jump
through hoops with eval, and handle all cases of variable assigment properly.

The hacky alternative is to just treat MOZ_PGO specially, but meh.

So instead, we set a ${var}_IS_SET variable to 1, indicating that a
mk_add_options defined ${var} to some value.
Attachment #8644180 - Flags: review?(gps)
Comment on attachment 8644180 [details] [diff] [review]
Set ${var}_IS_SET variables for mk_add_options-defined variables

Review of attachment 8644180 [details] [diff] [review]:
-----------------------------------------------------------------

I'm not a huge fan of adding more complexity to mozconfig evaluation. But I don't have a better idea that is just as simple as this.
Attachment #8644180 - Flags: review?(gps) → review+
Attachment #8644181 - Flags: review?(gps) → review+
https://hg.mozilla.org/mozilla-central/rev/06de69c7ea12
https://hg.mozilla.org/mozilla-central/rev/c24075996e1d
Status: REOPENED → RESOLVED
Closed: 4 years ago4 years ago
Resolution: --- → FIXED
Depends on: 1229613
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.