457464 - Linux Tp3 numbers show machine configurations appearing to change

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

http://graphs.mozilla.org/graph.html#show=395125,395135,395166,911694,1431032 shows the results of the Tp3 test on Linux machines testing mozilla-central.

This graph shows a number of trends that make it appear that machine configurations are either not stable or prone to random oscillation.  While it looks like problems causing upward drift were fixed, it still appears the machines are varying relative to each other in unexpected ways as recently as September 21.

Comment 1

[alice nodelman [:alice] [:anode]]

Looking at the graph, the most recent divergence was qm-plinux-trunk03.  I believe that this problem was resolved by a machine reboot.

Would it be reasonable to consider this a dupe of bug 379234 (perf test machines should be rebooted regularly)?

Comment 2

[alice nodelman [:alice] [:anode]]

In an attempt to get qm-plinux-trunk01/02/03 in agreement filed bug 457957 to re-image qm-plinux-trunk02.

That should hopefully resolve any lingering configuration difference between machines that should be reporting the same numbers.

Comment 3

[alice nodelman [:alice] [:anode]]

Machine (qm-plinux-trunk02) came back up and has been reporting for a while.  Still not as close to reported numbers on qm-plinux-trunk01/03 - but now within 3-5%.  

Considering that these are all running on the same hardware and the same images we may be hitting something more ephemeral (ie, placement in the colo - is it possible that this machine is a couple of degrees hotter than the other two?).  I think that we are as close as we are going to get and that the configuration issues have been resolved as best as we are able.