1644269 - Unable to add results for the same test but different apps

Status: RESOLVED FIXED

(Tree Management :: Perfherder, defect, P1)

Description

[Dave Hunt [:davehunt] [he/him] ⌚BST]

When adding test data to https://treeherder.mozilla.org/perf.html#/graphs it's not possible to select the same test multiple times for different apps. It appears that we add fields from extra options to distinguish them, but that we're not doing the same for the app name. This makes it difficult to compare results between Firefox and Chrome in Perfherder.

For example, the following results are for amazon-s-cold on Chrome and Fenix:

• chrome-m: https://treeherder.mozilla.org/perf.html#/graphs?series=mozilla-central,2438630,1,13&selected=2438630,1160233696
• fenix: https://treeherder.mozilla.org/perf.html#/graphs?series=mozilla-central,2440234,1,13&selected=2440234,1160203201

It's not possible to add both series results to an empty graph. We should display the app name in the tests, and allow users to distinguish and filter on this field.

Comment 1

[Ionuț Goldan [:igoldan]]

Indeed, this looks like a bug.

Comment 2

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Attachment: Screenshot 2020-07-07 at 17.35.15.png

Could you provide an update on this :igoldan? Could this impact consumers of the Perfherder API? I'm seeing some strange behaviour in arewefastyet.com where results from Fenix are displayed but none from GeckoView. The attached screenshot shows GeckoView results stopped when the platform changed to include '-shippable'. I suspect somehow the Fenix '-shippable' results are preventing the GeckoView results from being displayed.

Comment 3

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Attachment: Screenshot 2020-07-07 at 17.38.39.png

Attached is a screenshot from Perfherder after clicking the link from arewefastyet.com (see previous screenshot).

Comment 4

[Ionuț Goldan [:igoldan]]

(In reply to Dave Hunt [:davehunt] [he/him] ⌚BST from comment #2)

[...]
Could you provide an update on this :igoldan?

I started work on this bug. I haven't been able to quickly pick it up, because of some antivirus issues I had (described in bug 1651299).
I think I've found a (temporary?) workaround for the AV & trying to use it to unblock this bug.

Comment 5

[Ionuț Goldan [:igoldan]]

(In reply to Dave Hunt [:davehunt] [he/him] ⌚BST from comment #2)

[...] Could this impact consumers of the Perfherder API?

Yes, it can. Especially if they're using Perfherder endpoints that rely on the old signature hash mechanic & are interested in the new application field. We haven't got rid of that.

I'm seeing some strange behavior in arewefastyet.com where results from Fenix are displayed but none from GeckoView. The attached screenshot shows GeckoView results stopped when the platform changed to include '-shippable'. I suspect somehow the Fenix '-shippable' results are preventing the GeckoView results from being displayed.

In this particular case, I don't think Fenix '-shippable' results are preventing the equivalent GeckoView results from being displayed.
In this Graphs view I managed to find the missing GeckoView results from the new '-shippable' platform.
As I've mentioned a week ago: we're storing the correct data in the database. We're just not having the identification tools in good shape (some backend endpoints + frontend adjustments on Perfherder).

I think the consumers are not fetching the correct '-shippable' REST resources.
In the meantime, I noticed bug 1645874 which seems to have disabled that GeckoView test from mozilla-central.

Comment 6

[Andrew Creskey [:acreskey]]

I believe this is the bug preventing us from adding applink results for different browsers (e.g. geckoview_example and fenix):

https://treeherder.mozilla.org/perf.html#/graphs?highlightAlerts=1&series=mozilla-central,2450751,1,13&timerange=1209600

Comment 7

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/treeherder/pull/6629

Comment 8

[Ionuț Goldan [:igoldan]]

(In reply to Andrew Creskey [:acreskey] [he/him] from comment #6)

I believe this is the bug preventing us from adding applink results for different browsers (e.g. geckoview_example and fenix):

https://treeherder.mozilla.org/perf.html#/graphs?highlightAlerts=1&series=mozilla-central,2450751,1,13&timerange=1209600

The fix is underway.