1477014 - morph perf-reftests to be "non sheriffed" perf tests that act as a unittest, but run on hardware

Status: RESOLVED WONTFIX

(Testing :: Talos, enhancement, P3)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

developers should be encouraged to write more unittests, not less.  Lets reduce the barrier to entry for writing tests but ensure that they are stable and produce a true/false at the end.  Some simple tools (try syntax recommended) and docs on what to do would be beneficial here.

Since these would be true/false tests, they would be like unittests, although they will be measuring performance and be using hardware.  We wouldn't sheriff them like perf tests, but like unittests.

We should make this a process as easy as adding a new mochitest, but instead it is a perftest in talos/raptor.  I assume a manifest file will be edited to add the test and any expectations.

Comment 1

[Dave Hunt [:davehunt] [he/him] ⌚BST]

:jmaher are you referring to the tests documented here: https://wiki.mozilla.org/Performance_sheriffing/Talos/Tests#perf-reftest? How were you thinking that we could convert these into unittests? Would we set a threshold and fail if it's exceeded?

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

yes, those tests. Ideally yes- we have a limit to not cross and turn the job orange. sort of like we do for the xperf job. Ideally these would run on virtualized machines as well- that introduces other variables.

Comment 3

[Florin Strugariu [:Bebe]]

Gregory Mierzwinski this is similar to something you've mentioned recently. Could you come up with acceptance criteria for this so we can estimate and plan it for a future sprint?

Comment 4

[Greg Mierzwinski [:sparky]]

I don't think we should bother with making this change in Talos. MozPerftest already provides this feature as well.

Comment 5

[Dave Hunt [:davehunt] [he/him] ⌚BST]

:sparky are you okay with resolving this as wontfix? I didn't realise we had this feature in mozperftest.

Comment 6

[Greg Mierzwinski [:sparky]]

Sounds good to me.