559663 - Create new Talos test suite: Measure IOPs instead of time for tracking startup improvements

Status: RESOLVED DUPLICATE of bug 770317

(Testing :: Talos, defect, P5)

Description

[(dormant account)]

Mac(dtrace)/Windows(xperf)/Linux(systemtap+perf?) all provide ways to capture read syscalls + page faults backed by files.
In some cases they even annotate them with backtraces that lead to that io operation. 
These IOPS correlate exactly to startup time, but don't vary as much. IOPS can be measured reliably in a VM. In general they are a much better metric than time.
We need hook into each of the tracing frameworks, produce uniform results for all platforms and produce a graph for every build. It would be nice to have pretty graphs to go with the data, something like http://people.mozilla.com/~tglek/startup/systemtap_graphs/visualize.html?#stock.io.html

Comment 1

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Moving this bug to Testing:General for suite development, as requested by bmoss, ctalbert. 

Once this new suite is developed, please file a separate bug in
mozilla.org/ReleaseEngineering to enable running the new suite on builds
automatically.

Comment 2

[alice nodelman [:alice] [:anode]]

Moving to Talos component.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

:taras, is this resolved by measuring the xperf counters?  We don't run tests in vm's, so measuring IOPs with talos is not an option.

Comment 4

[(dormant account)]

(In reply to Joel Maher (:jmaher) from comment #3)
> :taras, is this resolved by measuring the xperf counters?  We don't run
> tests in vm's, so measuring IOPs with talos is not an option.