981433 - Replace Linux talos minis with VMs

Status: RESOLVED WORKSFORME

(Release Engineering :: General, defect)

Description

[(dormant account)]

1) We need to move a bunch of mac linux talos boxes from scl1. That would be wrong for many reasons. 

2) We should consider a new platform instead. We should try to make that platform a VM because Linux is the most VM-friendly 

3) We should test suitability of VMs to running perf tests.
a) Run fio and unixbench on 100 identical nodes on aws to see how consistent the perf is.
b) <vlad> pick a set of 30 or so builds over the past few years, test each build 10x, and repeat each build's test 3 different times during the cycle randomly at different times
c) if above steps fail try testing on our hardware with 2cores per vm with a dedicated harddrive.

I think we should try this ASAP to not block SCL1 migration. I think the risk of us running behind and causing a gap in Linux perf-testing for a week or two is well worth the benefits of a less painful/expensive perf-test setup.

Comment 1

[(dormant account)]

Realized aws part of this plan wont work due to variability of cpu.steal. steal varies depending on how idle/busy the host box is.

We should still pursue the local vm approach.

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

I have some machines that were formerly Ganeti nodes in the relabs network which I can offer for this purpose.  Lots of RAM, lots of cores, and >1 disk.  I'll need to kickstart them and set up VPN access, but can offer them as a basis for 3c.

Comment 3

[Laura Thomson :laura]

I don't know if it's a true dupe, but much of the background for this can be found in https://bugzilla.mozilla.org/show_bug.cgi?id=818968

Comment 4

[Laura Thomson :laura]

See also https://bugzilla.mozilla.org/show_bug.cgi?id=850105

Comment 5

[Chris Cooper [:coop] (he/him)]

r3 minis are dead