721902 - talos, graphserver, pageloader statistics architecture

Status: RESOLVED INVALID

(Testing :: Talos, defect)

Description

[Jeff Hammel]

So this is a longer term goal.  We do statistics all over the place in
Talos-land:

- pageloader calculates the mean, median, min, and max for a run:
http://hg.mozilla.org/build/pageloader/file/beca399c3a16/chrome/report.js#l89

- in talos proper, we do this for both the csv and for our own
  handrolled format for reporting:
  * http://hg.mozilla.org/build/talos/file/9c1b3addb9ee/talos/run_tests.py#l102
  * http://hg.mozilla.org/build/talos/file/9c1b3addb9ee/talos/run_tests.py#l235
  * http://hg.mozilla.org/build/talos/file/9c1b3addb9ee/talos/results.py
  * http://hg.mozilla.org/build/talos/file/9c1b3addb9ee/talos/filter.py
  It is worth noting that in talos we're not even internally
  consistent between csv and the graphserver format

- in graphserver, we also do this in several places:
  * https://github.com/mozilla/graphs/blob/master/server/api.py#L310
  * https://github.com/mozilla/graphs/blob/master/server/analysis/analyze.py
  * https://github.com/mozilla/graphs/blob/master/server/analysis/analyze_talos.py#L128
  * https://github.com/mozilla/graphs/blob/master/server/pyfomatic/collect.py#L166
  * http://hg.mozilla.org/graphs/file/1237d38a299b/server/pyfomatic/collect.py#l208
  And quite possibly more places I don't want to know about.

What we generally want is
- a way to apply statistics (flexibly) from graphserver
- a way to apply statistics (flexibly) locally on a desktop

Current, the latter thing doesn't really exist.  I have a brief sketch
as part of JetPerf,   *
https://github.com/mozilla/graphs/blob/master/server/analysis/analyze_talos\
.py#L128
  *
  https://github.com/mozilla/graphs/blob/master/server/pyfomatic/collect.py#L\
166
  *
  http://hg.mozilla.org/graphs/file/1237d38a299b/server/pyfomatic/collect.py#\
l208
  And quite possibly more places I don't want to know about.

What we generally want is
- a way to apply statistics (flexibly) from graphserver
- a way to apply statistics (flexibly) locally on a desktop

Current, the latter thing doesn't really exist.  I have a brief sketch
as part of JetPerf,   *
https://github.com/mozilla/graphs/blob/master/server/analysis/analyze_talos\
.py#L128
  *
  https://github.com/mozilla/graphs/blob/master/server/pyfomatic/collect.py#L\
166
  *
  http://hg.mozilla.org/graphs/file/1237d38a299b/server/pyfomatic/collect.py#\
l208
  And quite possibly more places I don't want to know about.

What we generally want is
- a way to apply statistics (flexibly) from graphserver
- a way to apply statistics (flexibly) locally on a desktop

Current, the latter thing doesn't really exist.  I have a brief sketch
as part of JetPerf,
http://k0s.org/mozilla/hg/jetperf/file/78da6f98c95a/jetperf/compare.py,
but obviously this is a toy.

What I propose:

1. get rid of the taking of stats in pageloader entirely.  The purpose
of pageloader is to load pages and output raw data; simultaneously,
you will need to correct talos and/or graphserver to not lose these stats

2. build a stats package that can be utilized by graphserver, the
standalone desktop thing that doesn't exist yet, and talos for the
time being

3. make graphserver depend on this package

4. remove all statistics from talos.  talos is in the business of
gathering data, not processing it.  Instead, write the desktop thing
for gathering data locally

Comment 1

[Jeff Hammel]

Not a hard blocker, but see also https://bugzilla.mozilla.org/show_bug.cgi?id=728461

Comment 2

[Jeff Hammel]

So this bug is a bit out of date.  We should figure out what the items still of interest are here and deal with them, probably separately, and then close the bug.

Comment 3

[Jeff Hammel]

So we've mostly ignore this bug in the development of datazilla and talos.  Closing for inapplicability