Bugzilla

(In reply to Mark Cornmesser [:markco] from comment #9)
> There are a few challenges with using Azure VMs as a performance baseline.
> 
> First, for standard shared Azure VMs, we do not control the hypervisor, host placement, or the underlying physical hardware. The host environment can change due to Azure maintenance, live migration, hardware decommissioning, or other platform-level events, and there may also be differences across regions or zones.
> 
> Second, performance can be affected by host-level scheduling and contention from other workloads running on the same physical hardware. That makes it harder to determine whether a performance shift came from Firefox, the test environment, or the cloud platform itself.
> 
> Also worth noting: we are already exploring options to potentially replace the non-reference hardware.
> 
> For testing, we may also want to create a separate pool isolated to a specific Azure region, and investigate which VM size or type best matches the workload. That would help reduce variability and give us a cleaner signal while we evaluate whether this is a viable path.

Thanks for the detailed breakdown. This came up at the work week as a possibility and we wanted to give it a look... but between the replicate spread we saw in the try runs and the platform-level unpredictability you've described (live migration, host contention, no control over placement), I don't think VMs are a viable baseline right now.

Good to know the non-ref hardware replacement is already being explored. Unless someone has anything to chime in within the next 1-2w I think it makes sense to  close this out as WONTFIX 

if dedicated/region-isolated pool option ever gets traction, that would be interesting.

(In reply to Mark Cornmesser [:markco] from comment #9)
> There are a few challenges with using Azure VMs as a performance baseline.
> 
> First, for standard shared Azure VMs, we do not control the hypervisor, host placement, or the underlying physical hardware. The host environment can change due to Azure maintenance, live migration, hardware decommissioning, or other platform-level events, and there may also be differences across regions or zones.
> 
> Second, performance can be affected by host-level scheduling and contention from other workloads running on the same physical hardware. That makes it harder to determine whether a performance shift came from Firefox, the test environment, or the cloud platform itself.
> 
> Also worth noting: we are already exploring options to potentially replace the non-reference hardware.
> 
> For testing, we may also want to create a separate pool isolated to a specific Azure region, and investigate which VM size or type best matches the workload. That would help reduce variability and give us a cleaner signal while we evaluate whether this is a viable path.

Thanks for the detailed breakdown. This came up at the work week as a possible alternative (partly due to the recent PSU failures on hardware) ... but between the replicate spread we saw in the try runs and the platform-level unpredictability you've described (live migration, host contention, no control over placement), I don't think VMs are a viable baseline right now.

Good to know the non-ref hardware replacement is already being explored. Unless someone has anything to chime in within the next 1-2w I think it makes sense to  close this out as a WONTFIX 

if dedicated/region-isolated pool option ever gets traction, that would be interesting though