1583673 - Reduce our linux32 build spend

Status: NEW

(Testing :: General, task, P3)

Description

[Bobby Holley (:bholley)]

Our current aim is to keep the linux32 testing bill as low as possible, and primarily lean on linux64. To that end, we currently only run the following:

• linux32 builds on autoland
• linux32 builds + WPT on central

For the month of August, the linux32 test spend was ~$800/mo, but the build spend was ~$5500/mo. I think there's substantial room to improve that number.

The simplest approach is to reduce the cost of linux32 builds on autoland. From what I can tell, builds are more expensive than tests in two ways:
(1) They run on more expensive machines (which are generally 4x-5x the cost)
(2) They are immune from SETA (so they run every push)

This is usually the right thing for builds, because so many other tasks depend on them. But in this case, we're not running any of those other tasks on autoland. So we should be able to change either/both of those things, and bring the costs down by at least 5x.

Comment 1

[Geoff Brown [:gbrown]]

Interesting - thanks for filing!

It looks like we build both linux32/debug and linux32/shippable (pgo), with debug builds taking about 20 minutes and shippable taking about 80 minutes (all 3 stages combined).

Not running builds on every push adds the risk of missing breakage, but I would expect the risk of linux32-only or linux32/shippable-only breakage is pretty low.

Running on less expensive machines will increase time per build, which might be a concern for sheriffing / backlog. :aryx -- What's your perspective on that? Could we tolerate longer wait times for linux32 builds?

Comment 2

[Bobby Holley (:bholley)]

Catlee points out that we should consider doing this for all our tier-2-builds-without-tests that run on autoland.

Comment 3

[Marco Castelluccio [:marco]]

There was also another additional idea in bug 1578431 comment 3.

Comment 4

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

(In reply to Geoff Brown [:gbrown] from comment #1)

Running on less expensive machines will increase time per build, which might be a concern for sheriffing / backlog. :aryx -- What's your perspective on that? Could we tolerate longer wait times for linux32 builds?

Yes, up to an hour more because Windows builds and tests take longer and then those tests and the Linux32 builds would finish at the same time.