1175295 - Allow third parties to submit perfherder artifacts

Status: RESOLVED FIXED

(Tree Management :: Perfherder, defect)

Description

[William Lachance (:wlach)]

Currently the only way to get performance data into perfherder is via talos. We should support submitting performance artifacts to treeherder and storing / processing them, so arbitrary performance frameworks can submit their own stuff and get it analyzed. This would also be useful for bug 1156885: if we implemented this feature, builds could submit their data to treeherder/perfherder just as they currently submit to graphserver.

Comment 1

[William Lachance (:wlach)]

I'm going to use this bug to cover creating a more generic performance artifact that people can submit to be processed by treeherder/perfherder, which also allows the specification of which performance framework it is a part of.

Comment 2

[William Lachance (:wlach)]

Attachment: PR

Not quite ready for review, but I thought I'd attach my wip to this patch as a placeholder. More instructions and docs to come.

Comment 3

[William Lachance (:wlach)]

Comment on attachment 8662633 [details] [review]
PR

Ok, this is pretty close to being ready for review I'd reckon, but I'll throw this by the autophone people first to see what they think.

So after this patch is applied, to submit performance data records to perfherder, you just need to add an artifact to your pushes that looks like this:

{ "framework": "autophone",
  "suites": [
    {
      "name": <suite name>,
      "value": <summary value for suite, optional>
      "subtests": [
        {
          "name": <subtest name>,
          "value": <summary value for subtest>
        },
        ...
      ]
    },
   ...
  ]
} 

You should be able to plug this into the existing autophone data submission form. Perfherder/Treeherder will take care of creating a unique performance signature corresponding to the reference data you've provided for a job (currently we only use option_collection_hash and machine_platform, but we could add others if needed).

There's a bit of sample code here:

https://github.com/mozilla/treeherder/pull/971/files#diff-af04fadfcca75cc75a13b61b44b5a864R44

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

this makes sense, I have questions about the platform, etc.

We have additional meta data about the platform from treeherder, will we use that?  I recall that in autophone we track by specific device ID, not the general class of device (like nexus 7).  Right now this posts under the Android.

here is an example of an android job on inbound:
https://treeherder.allizom.org/#/jobs?repo=mozilla-inbound&revision=045b1309490f&filter-searchStr=autophone

Comment 5

[Bob Clary [:bc] (inactive)]

As jmaher mentioned, Autophone doesn't currently store aggregated data by device type but stores raw time series for each device.

It appears to me that talos will be able to submit time series for test results but others are expected to submit a summary only without recording information about the variance of the data?

Comment 6

[William Lachance (:wlach)]

(In reply to Bob Clary [:bc:] from comment #5)
> As jmaher mentioned, Autophone doesn't currently store aggregated data by
> device type but stores raw time series for each device.
> 
> It appears to me that talos will be able to submit time series for test
> results but others are expected to submit a summary only without recording
> information about the variance of the data?

We could create a special platform for every single device currently used by autophone, and then use that in perfherder, then each series would be associated with that device.

As discussed, there are really two things we want to do with autophone:

1. Performance testing across a range of different devices and operating system versions. This is what we currently use autophone for.
2. Replacing Android Talos (i.e.: get fine-grained regression information on a small set of tests). For this, we probably want a large pool of identical devices running the same version of Android, to keep up with the inbound branches, running things on try, etc. 

Since replacing the pandas is a high priority, I think I'd rather focus on (2) first. We can always come up with a strategy for bringing the rest of autophone into the perfherder-fold later. It should be totally possible, I would just rather focus on one thing at a time.

I think I'm going to try to get the attached patch landed, since I think it's something we want anyway and I'd prefer it not bitrot.

Comment 7

[William Lachance (:wlach)]

Comment on attachment 8662633 [details] [review]
PR

Hey Mauro,

This is mostly just refactoring and testing. The one big new feature is the ability to submit performance artifacts to treeherder with a simpler, more intuitive structure. See the JSON schema here:

https://github.com/mozilla/treeherder/pull/971/files#diff-3a1d40cd29ab5741c74cd2e9ef3e9926R31

Comment 8

[William Lachance (:wlach)]

Comment on attachment 8662633 [details] [review]
PR

Reassigning review to :camd.

Comment 9

[Treeherder GitHub Bugbot]

Commits pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/26c6cf0972aa998df08aca8f67eeac05c972e99d
Bug 1175295 - Clean up performance data adapters

* Put talos-specific stuff in the talos data adapter
* Put generic stuff in the generic adapter, in preparation for creating
  a generic perfherder data ingestion path
* Add some explanatory comments
* Use better casing for static defines
* Remove some now-unused code related to json float encoding

https://github.com/mozilla/treeherder/commit/77558b2e86a315ae3b4b99d5e9eb8d0ca36f64a8
Bug 1175295 - Reorganize performance data adapters

The "performance adapter class" never stored any persistent state, so
let's just refactor it into a bunch of standalone methods. Easier to
understand and reason about.

https://github.com/mozilla/treeherder/commit/d3bd244b654d893e742eb294d188bc9507c66537
Bug 1175295 - Add support for submitting generic performance artifacts

Comment 10

[William Lachance (:wlach)]

Fixed up the travis failures and remaining nits from review and merged. Thanks camd for the helpful feedback. :)