721097 - post jetperf results to datazilla

Status: RESOLVED WONTFIX

(Testing :: Talos, defect)

Description

[Jeff Hammel]

re https://wiki.mozilla.org/Auto-tools/Projects/JetPerf

We will want the production (buildbot-run) Jetperf tests to upload their data to a graphserver to monitor/enforce regressions.

As far as what to upload, I'm not really sure.  Ideally what one is interested in is (time with cfx generated addons) - (time without) . However, we currently upload all of the data to graphserver, not just the averages.  This is probably a good thing to do, since our data is noisy and we may (read: will) change our processing methodologies which if we discard data is not possible. Relatedly, currently we do not run ts/onload on the build machines, whereas (AIUI) what we want, at least initially, is the startup time hit from a cfx module (or modules), not the MozAfterPaint event.

So maybe the solution is to have the buildbot code that runs the cfx tests also run a baseline Ts?  :shrug:  I'm open to ideas

Comment 1

[Joel Maher ( :jmaher ) (UTC -8) (PTO back normal Nov 17)]

can't we use the existing graph server.

My thoughts would be to:
1) run baseline ts
2) run ts + addon1
3) run ts + addon2

at the end we would have 3 datapoints, and we could either upload the raw values or a diff.

So baseline ts would generate a number, say 314.  with addon1, we might have 350.  with addon2 we might have 345.  So we either upload {'ts': 314, 'a2': 345, 'a1': 350}, or more interesting would be to upload the diff from ts: {'a1': 36, 'a2': 31}.

My thoughts are to do a statistically significant number of runs, say 20.  Ignore the first run, then take the median value.  We then have 3 values and we can diff those and upload the results. 

Whether or not we upload all the data or just the diffs is a different question and something we are questioning with talos in general.  I vote for integrating in with talos proper as it is now and using the existing graph server.

Comment 2

[Jeff Hammel]

(In reply to Joel Maher (:jmaher) from comment #1)
> can't we use the existing graph server.

Sorry, I didn't mean to imply that I wanted a new graphserver instance.  As far as I know, using the existing graphserver is what we want to do.

> My thoughts would be to:
> 1) run baseline ts
> 2) run ts + addon1
> 3) run ts + addon2
> 
> at the end we would have 3 datapoints, and we could either upload the raw
> values or a diff.

I *think* we will want to upload the raw values and have graphserver keep them and average them?  This way we're not losing any data. Other than that, I agree.
 
> So baseline ts would generate a number, say 314.  with addon1, we might have
> 350.  with addon2 we might have 345.  So we either upload {'ts': 314, 'a2':
> 345, 'a1': 350}, or more interesting would be to upload the diff from ts:
> {'a1': 36, 'a2': 31}.
> 
> My thoughts are to do a statistically significant number of runs, say 20. 
> Ignore the first run, then take the median value.  We then have 3 values and
> we can diff those and upload the results. 
> 
> Whether or not we upload all the data or just the diffs is a different
> question and something we are questioning with talos in general.  I vote for
> integrating in with talos proper as it is now and using the existing graph
> server.

Not sure what you mean by integrating in with talos proper.  Could you elaborate?  It is just running ATM talos ts with a cfx-generated addon

Comment 3

[Joel Maher ( :jmaher ) (UTC -8) (PTO back normal Nov 17)]

my integrating with talos proper means lets not write a bunch of custom code right now to get this going.  We are planning this quarter to fix the values we upload to the graph server, so lets worry about those values then.  We can get this done in the real short term if we treat this as a new talos test.

Comment 4

[Jeff Hammel]

So the problem is this: We currently want these run on changes to http://hg.mozilla.org/projects/addon-sdk , not on changes to m-c . I don't know enough about how the infrastructure works to know if graphserver could support something like this or where the Talos tests are shown, though maybe :lsblakk could comment to that?

Comment 5

[Jeff Hammel]

(In reply to Jeff Hammel [:jhammel] from comment #4)
> So the problem is this: We currently want these run on changes to
> http://hg.mozilla.org/projects/addon-sdk , not on changes to m-c . I don't
> know enough about how the infrastructure works to know if graphserver could
> support something like this or where the Talos tests are shown, though maybe
> :lsblakk could comment to that?

or :rhelmer?

Comment 6

[cmtalbert]

(In reply to Jeff Hammel [:jhammel] from comment #4)
> So the problem is this: We currently want these run on changes to
> http://hg.mozilla.org/projects/addon-sdk , not on changes to m-c . I don't
> know enough about how the infrastructure works to know if graphserver could
> support something like this or where the Talos tests are shown, though maybe
> :lsblakk could comment to that?

Graph server is just the reporting arm here.  What schedules talos runs is buildbot. You would have the buildbot run the test when something in hg.mozilla.org/projects/addon-sdk is changed (This buildbot scheduler already exists). And then the tests it runs would report to a specific collection area in graph server for these tests.  It should work just like any other tree with the only caveat being that there will be less "dots" on graph server because there are less changes on the addons-sdk tree than say on mozilla-central.

So in other words, it's a non-issue, but you will need to file a buildbot bug detailing that in addition to the normal jetpack tests, they'll also need to run the jetperf tests on change from hg.mozilla.org/projects/addon-sdk.

Comment 7

[Jeff Hammel]

(In reply to Clint Talbert ( :ctalbert ) from comment #6)
<snip/>
> So in other words, it's a non-issue, but you will need to file a buildbot
> bug detailing that in addition to the normal jetpack tests, they'll also
> need to run the jetperf tests on change from
> hg.mozilla.org/projects/addon-sdk.

That would be bug 720901

Comment 8

[Jeff Hammel]

It has been decided to use datazilla for this bug vs the current graphserver.  This will let us move forward with new technologies.  My inclination is to post the results for all addons (we currently run them all, all three of them) and a baseline.  However, strictly speaking, we do not have to do a baseline if that is already achieved elsewhere.  The important parameter is (with addons)-(baseline), but that can be done by hand until we work out the kinks.

Talos results handling needs to be more extensible for this so I put bug 755527 as a blocker, though it is likely that other bugs will need to be fixed towards this end too.  I'll work out the details with :jeads following the resolution of bug 755527

Comment 9

[Jeff Hammel]

The datazilla server is setup:

http://10.8.73.32/jetperf/load_test

This should be accessible to the MPT vpn:

An example json data structure can be found here https://github.com/mozilla/datazilla/blob/f52e0b00253eef4e3e07fe94b893e8a0af33ef71/datazilla/model/sql/template_schema/schema_perftest.json

Comment 10

[Jeff Hammel]

This will need to use oauth in some form.  Probably the thing to do is just to depend on the datazilla client and fix this properly

Comment 11

[Jeff Hammel]

This will depend on having oauth in mozharness+talos

Comment 12

[Joel Maher ( :jmaher ) (UTC -8) (PTO back normal Nov 17)]

are we still working on jetperf?

Comment 13

[Jeff Hammel]

I am unsure of priority here; AFAIK the project was never cancelled, but neither was it reprioritized.  so WONTFIX seems to be applicable for now