Linux Tp3 numbers show machine configurations appearing to change

RESOLVED FIXED

Status

Release Engineering
General
P2
normal
RESOLVED FIXED
10 years ago
5 years ago

People

(Reporter: dbaron, Assigned: alice)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

10 years ago
http://graphs.mozilla.org/graph.html#show=395125,395135,395166,911694,1431032 shows the results of the Tp3 test on Linux machines testing mozilla-central.

This graph shows a number of trends that make it appear that machine configurations are either not stable or prone to random oscillation.  While it looks like problems causing upward drift were fixed, it still appears the machines are varying relative to each other in unexpected ways as recently as September 21.
(Assignee)

Comment 1

10 years ago
Looking at the graph, the most recent divergence was qm-plinux-trunk03.  I believe that this problem was resolved by a machine reboot.

Would it be reasonable to consider this a dupe of bug 379234 (perf test machines should be rebooted regularly)?
(Assignee)

Comment 2

10 years ago
In an attempt to get qm-plinux-trunk01/02/03 in agreement filed bug 457957 to re-image qm-plinux-trunk02.

That should hopefully resolve any lingering configuration difference between machines that should be reporting the same numbers.
(Assignee)

Updated

10 years ago
Assignee: nobody → anodelman
Priority: -- → P2
(Assignee)

Comment 3

10 years ago
Machine (qm-plinux-trunk02) came back up and has been reporting for a while.  Still not as close to reported numbers on qm-plinux-trunk01/03 - but now within 3-5%.  

Considering that these are all running on the same hardware and the same images we may be hitting something more ephemeral (ie, placement in the colo - is it possible that this machine is a couple of degrees hotter than the other two?).  I think that we are as close as we are going to get and that the configuration issues have been resolved as best as we are able.
Status: NEW → RESOLVED
Last Resolved: 10 years ago
Resolution: --- → FIXED

Updated

9 years ago
Component: Release Engineering: Talos → Release Engineering
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.