1014284 - Datazilla Metrics Off since May 19

Status: RESOLVED INCOMPLETE

(Datazilla Graveyard :: General, defect, P1)

Description

[Mason Chang [Inactive] [:mchang]]

Sometime around May 19 - today, datazilla has been reporting wrong results. We've had bug 1013387, bug 1013381, and bug 1014135, but I have been unable to reproduce these locally. 

Jgriffin or davehunt, Did something change in datazilla two days ago? Thanks!

Comment 1

[Jonathan Griffin (:jgriffin)]

Datazilla is just a dashboard and is accurately displaying the data fed to it.  Since this data is generated by 'make test-perf', the only variable on our side that could have changed is something related to flashing or pre-test setup.  I'll leave davehunt needinfo'd and add stephend in case either knows of anything that's changed recently in the environment that could cause this.

Comment 2

[Dave Hunt [:davehunt] [he/him] ⌚BST]

I'm not aware of any changes that would affect the results in Datazilla. It's sometimes worth checking the firmware details of the device as shown to see if there's been an update. It can also be that different devices report different results, so check the machine name to see if there's a correlation there.

Unfortunately we do not yet link back to the Jenkins jobs (bug 984330), but if you need to see any of the last 100 builds they can be found here: http://selenium.qa.mtv2.mozilla.com:8080/view/B2G%20Perf/job/b2g.hamachi.mozilla-central.master.mozperftest/ though you will need VPN, and perhaps even to be added to an ACL to view them.

Comment 3

[Mason Chang [Inactive] [:mchang]]

Thanks for the heads up dave! I Checked the logs and the issue is that we ran the tests in different order between the good / bad gaia revisions. Check bug 1013387 for details. Investigating 1014135.

Comment 4

[Mason Chang [Inactive] [:mchang]]

Resolving and spun off side bugs. Thanks for the help! See two blocker bugs for more details.