Last Comment Bug 1175295 - Allow third parties to submit perfherder artifacts
: Allow third parties to submit perfherder artifacts
Status: RESOLVED FIXED
:
Product: Tree Management
Classification: Other
Component: Perfherder (show other bugs)
: ---
: Unspecified Unspecified
-- normal
: ---
Assigned To: William Lachance (:wlach) (use needinfo!)
:
:
Mentors:
Depends on:
Blocks: 1156885
  Show dependency treegraph
 
Reported: 2015-06-16 14:35 PDT by William Lachance (:wlach) (use needinfo!)
Modified: 2015-10-08 13:42 PDT (History)
4 users (show)
See Also:
QA Whiteboard:
Iteration: ---
Points: ---


Attachments
PR (46 bytes, text/x-github-pull-request)
2015-09-17 14:37 PDT, William Lachance (:wlach) (use needinfo!)
cdawson: review+
Details | Review | Splinter Review

Description User image William Lachance (:wlach) (use needinfo!) 2015-06-16 14:35:29 PDT
Currently the only way to get performance data into perfherder is via talos. We should support submitting performance artifacts to treeherder and storing / processing them, so arbitrary performance frameworks can submit their own stuff and get it analyzed. This would also be useful for bug 1156885: if we implemented this feature, builds could submit their data to treeherder/perfherder just as they currently submit to graphserver.
Comment 1 User image William Lachance (:wlach) (use needinfo!) 2015-09-17 08:46:54 PDT
I'm going to use this bug to cover creating a more generic performance artifact that people can submit to be processed by treeherder/perfherder, which also allows the specification of which performance framework it is a part of.
Comment 2 User image William Lachance (:wlach) (use needinfo!) 2015-09-17 14:37:57 PDT
Created attachment 8662633 [details] [review]
PR

Not quite ready for review, but I thought I'd attach my wip to this patch as a placeholder. More instructions and docs to come.
Comment 3 User image William Lachance (:wlach) (use needinfo!) 2015-09-21 12:27:05 PDT
Comment on attachment 8662633 [details] [review]
PR

Ok, this is pretty close to being ready for review I'd reckon, but I'll throw this by the autophone people first to see what they think.

So after this patch is applied, to submit performance data records to perfherder, you just need to add an artifact to your pushes that looks like this:

{ "framework": "autophone",
  "suites": [
    {
      "name": <suite name>,
      "value": <summary value for suite, optional>
      "subtests": [
        {
          "name": <subtest name>,
          "value": <summary value for subtest>
        },
        ...
      ]
    },
   ...
  ]
} 

You should be able to plug this into the existing autophone data submission form. Perfherder/Treeherder will take care of creating a unique performance signature corresponding to the reference data you've provided for a job (currently we only use option_collection_hash and machine_platform, but we could add others if needed).

There's a bit of sample code here:

https://github.com/mozilla/treeherder/pull/971/files#diff-af04fadfcca75cc75a13b61b44b5a864R44
Comment 4 User image Joel Maher ( :jmaher) 2015-09-21 14:22:12 PDT
this makes sense, I have questions about the platform, etc.

We have additional meta data about the platform from treeherder, will we use that?  I recall that in autophone we track by specific device ID, not the general class of device (like nexus 7).  Right now this posts under the Android.

here is an example of an android job on inbound:
https://treeherder.allizom.org/#/jobs?repo=mozilla-inbound&revision=045b1309490f&filter-searchStr=autophone
Comment 5 User image Bob Clary [:bc:] 2015-09-22 07:55:13 PDT
As jmaher mentioned, Autophone doesn't currently store aggregated data by device type but stores raw time series for each device.

It appears to me that talos will be able to submit time series for test results but others are expected to submit a summary only without recording information about the variance of the data?
Comment 6 User image William Lachance (:wlach) (use needinfo!) 2015-09-24 10:45:14 PDT
(In reply to Bob Clary [:bc:] from comment #5)
> As jmaher mentioned, Autophone doesn't currently store aggregated data by
> device type but stores raw time series for each device.
> 
> It appears to me that talos will be able to submit time series for test
> results but others are expected to submit a summary only without recording
> information about the variance of the data?

We could create a special platform for every single device currently used by autophone, and then use that in perfherder, then each series would be associated with that device.

As discussed, there are really two things we want to do with autophone:

1. Performance testing across a range of different devices and operating system versions. This is what we currently use autophone for.
2. Replacing Android Talos (i.e.: get fine-grained regression information on a small set of tests). For this, we probably want a large pool of identical devices running the same version of Android, to keep up with the inbound branches, running things on try, etc. 

Since replacing the pandas is a high priority, I think I'd rather focus on (2) first. We can always come up with a strategy for bringing the rest of autophone into the perfherder-fold later. It should be totally possible, I would just rather focus on one thing at a time.

I think I'm going to try to get the attached patch landed, since I think it's something we want anyway and I'd prefer it not bitrot.
Comment 7 User image William Lachance (:wlach) (use needinfo!) 2015-09-29 14:14:36 PDT
Comment on attachment 8662633 [details] [review]
PR

Hey Mauro,

This is mostly just refactoring and testing. The one big new feature is the ability to submit performance artifacts to treeherder with a simpler, more intuitive structure. See the JSON schema here:

https://github.com/mozilla/treeherder/pull/971/files#diff-3a1d40cd29ab5741c74cd2e9ef3e9926R31
Comment 8 User image William Lachance (:wlach) (use needinfo!) 2015-10-01 11:45:04 PDT
Comment on attachment 8662633 [details] [review]
PR

Reassigning review to :camd.
Comment 9 User image Treeherder Bugbot 2015-10-08 13:41:05 PDT
Commits pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/26c6cf0972aa998df08aca8f67eeac05c972e99d
Bug 1175295 - Clean up performance data adapters

* Put talos-specific stuff in the talos data adapter
* Put generic stuff in the generic adapter, in preparation for creating
  a generic perfherder data ingestion path
* Add some explanatory comments
* Use better casing for static defines
* Remove some now-unused code related to json float encoding

https://github.com/mozilla/treeherder/commit/77558b2e86a315ae3b4b99d5e9eb8d0ca36f64a8
Bug 1175295 - Reorganize performance data adapters

The "performance adapter class" never stored any persistent state, so
let's just refactor it into a bunch of standalone methods. Easier to
understand and reason about.

https://github.com/mozilla/treeherder/commit/d3bd244b654d893e742eb294d188bc9507c66537
Bug 1175295 - Add support for submitting generic performance artifacts
Comment 10 User image William Lachance (:wlach) (use needinfo!) 2015-10-08 13:42:13 PDT
Fixed up the travis failures and remaining nits from review and merged. Thanks camd for the helpful feedback. :)

Note You need to log in before you can comment on or make changes to this bug.