Open Bug 1626390 Opened 5 years ago Updated 2 years ago

stop running gtests (and building libxul-gtest) on Linux and Mac shippable builds

Categories

(Testing :: General, enhancement, P3)

Version 3
enhancement

Tracking

(Not tracked)

People

(Reporter: froydnj, Unassigned)

References

(Blocks 1 open bug)

Details

We already don't run GTests on Windows shippable builds (we run them on opt builds so we still have test coverage). This strategy has saved time/money and doesn't seem to have unduly hurt us, so there's an argument to doing the same for Linux and Mac.

Our Windows shippable builds are ~10 minutes faster on m-c vs. Linux and Mac shippable builds (and they are roughly comparable, since everything is running on a Linux host), so this would save a small amount of time on the build side, as well as running fewer tests.

Priority: -- → P3

wouldn't we want to run on shippable and not on opt?

Flags: needinfo?(nfroyd)

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #1)

wouldn't we want to run on shippable and not on opt?

dmajor knows more about the original decision than I do. shippable is going to take longer to build them than opt would (due to LTO + PGO), and I would expect opt to be a good enough proxy for shippable in this case.

Flags: needinfo?(nfroyd) → needinfo?(dmajor)

Indeed. Linking xul-gtest on shippable builds takes 15 minutes thanks to LTO and PGO -- and shippable builds are already the long pole in our CI even without xul-gtest.

There is a small but nonzero risk that gtests on shippable could catch a miscompilation that we wouldn't see on opt, but all things considered I think moving these to opt is reasonable.

Flags: needinfo?(dmajor)

I don't understand the decision- on autoland (with few exceptions that should be changed soon) we only test on shippable builds- therefore we don't need to build plain opt builds; Forcing gtests to run on opt and not on shippable is another (soon to be the only) requirement for building both opt and shippable.

is this scope for autoland, mozilla-central, beta, try, everything?

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #4)

I don't understand the decision- on autoland (with few exceptions that should be changed soon) we only test on shippable builds- therefore we don't need to build plain opt builds; Forcing gtests to run on opt and not on shippable is another (soon to be the only) requirement for building both opt and shippable.

Obviously I haven't looked at autoland treeherder in some time.

This decision to only test on shippable seems kind of odd...shippable builds take several times longer to finish (over the course of instrument, run, build), and then you're gating tests running on those?

is this scope for autoland, mozilla-central, beta, try, everything?

Everywhere we're currently running gtests, yes.

if the goal is end to end time for developers, then we should run tests on opt for try server; but for autoland having a special build to run tests doesn't provide any value (it only adds cost and potential queues for build machines)- there is little to no benefit of speeding up end to end times on autoland or mozilla-central that I am aware of.

Just to clarify, we'll continue to have tests on debug, right? (And presumably asan-opt?)

I assume this bug is only about opt vs shippable (including the -qr variety)- not asan/debug

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #6)

there is little to no benefit of speeding up end to end times on autoland or mozilla-central that I am aware of.

Cost of doing shippable builds? Test results coming sooner rather than later? Maybe not on m-c, but autoland, surely?

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #8)

I assume this bug is only about opt vs shippable (including the -qr variety)- not asan/debug

The reason I ask is, if we still have gtests on debug and asan-opt, we might be able to get away with disabling gtests on both shippable and regular opt builds.

I wouldn't argue with simplifying that- are there other stakeholders we should query?

I'm not sure.

Severity: normal → S3
You need to log in before you can comment on or make changes to this bug.