1220432 - Autophone - links to perfherder need to have the correct signatures

Status: RESOLVED FIXED

(Testing Graveyard :: Autophone, defect)

Description

[Bob Clary [:bc] (inactive)]

Looks like that the values which determines the graph changes depending on the platform:

https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=4744545d275c&exclusion_profile=false&filter-searchStr=autophone

The linked graphs are empty for nexus-7-jss15q-2 Android 4.3 API11+ opt.

https://treeherder.mozilla.org/perf.html#/graphs?series=[mozilla-inbound,009a6a521e1498b73899a48ac2aa407adbc5478d,1]

https://treeherder.mozilla.org/perf.html#/graphs?series=[mozilla-inbound,291460acb5a86370d8a88b4d7656e1b4e4a16d1e,1]

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

you can see the two graphs here:
https://treeherder.mozilla.org/perf.html#/graphs?series=[mozilla-inbound,f955baa4c127f70ca253fa72e29a3994966f6637,1]&series=[mozilla-inbound,302e02aaa43603f1d4eb3b38b49938c50d8a9d75,1]

it is all in the signatures.  To get this we will need to do some treeherder api calls.  I believe we already have the treeherder-client in use, so this might be fairly straightforward.  The ideal spot to do this would be in the autophonetreeherder.py.

:bc, let me know if this use of treeherder-client sounds good to you.

Comment 2

[Bob Clary [:bc] (inactive)]

I am all for it if we can look this up and not hard-code it. Which version of the client do we need and where are the features and documentation?

Comment 3

[Bob Clary [:bc] (inactive)]

We shouldn't have to look it up each time we want to submit a result to Treeherder though. I would expect it to better live in PerfTest.

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

we need to look it up for every branch/platform/test combination- that is where the signature comes from. I honestly don't know where this is documented, I just hack around in other code to find the signature usually.

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

would a reasonable solution be to hard code the signatures for fx-team/inbound/central/aurora/beta/try hardcoded, and if we are not on one of those branches we look up the signature?

Comment 6

[Bob Clary [:bc] (inactive)]

I would rank the approaches for the tests which use Perfherder (TalosTest, RoboTest and perhaps any descendant of PerfTest) as:

1. Look up the required signatures when the test object is first initialized in Autophone in autophone.py.

2. Hardcode all of the signatures in a py script that is imported.

3. Mix the two approaches.

If the signatures are constant and not liable to change and if the set of possible repositories is limited, we could just create the script and maintain a list of all of the supported repositories there. The cost will be the maintenance required. It would be a one time cost with the possible failure to keep up with changes.

If we aren't certain the signatures won't change or that the set of repositories will change relatively frequently, then I think looking them up each time we start is a better approach. This has the cost of having to look them up each time we start an Autophone instance but we no longer have the cost of maintaining the list manually nor have the potential downside of having a signature for a repository change over time.

Mixing the two approaches seems like the worst choice. We have the potential costs of both approaches with slightly more complicated code.

/2¢

Comment 7

[Joel Maher ( :jmaher ) (UTC -8)]

Great advice :bc!  the signatures are for the most part static.  If we change the number of pages or other attributes of what we are measuring, then the signatures can change.  I don't see that happening often.  I am leaning towards the static list of signatures as dynamic query of treeherder is one more possible thing to break (if it is down, if the api changes, if we get a timeout, etc.)

If we don't like that, I do like the idea of gaining the signature prior to starting the test.

Let me work on implementing the static signatures.

Comment 8

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: https://github.com/mozilla/autophone/pull/53

as discussed on irc, we have this data automatically in the peformance panel on treeherder.  This means we do not need to have anything hardcoded, nor do we need to have any api call or lookup take place.

Comment 9

[Bob Clary [:bc] (inactive)]

Comment on attachment 8729620 [details] [review]
https://github.com/mozilla/autophone/pull/53

lgtm

Comment 10

[Joel Maher ( :jmaher ) (UTC -8)]

https://github.com/mozilla/autophone/commit/c5b5272b7a9af0c62743394b8be4314004ffcc7d

Comment 11

[Bob Clary [:bc] (inactive)]

deployed 2016-03-15 06:29