1007774 - Add a target line for performance in datazilla

Status: RESOLVED WONTFIX

(Datazilla Graveyard :: Metrics, defect, P1)

Description

[Hubert Figuiere [:hub]]

Add a target line for performance in datazilla

This is a perf Q2 goal. We need to add this target line per test in order to be able to visualize how far we are from target.

Comment 1

[Dave Huseby [:huseby]]

Jeads suggests adding a config file in the datazilla repo that contains the target values for each app+test+device combination.  The UI would then load the config file and just plot the lines.  The UI would also have a checkbox for toggling the lines on/off.

Comment 2

[Hubert Figuiere [:hub]]

That would work. I was thinking something like that.

Comment 3

[Hubert Figuiere [:hub]]

Here what I think we should: For each performance test, attach the performance goal so that it is persisted and datazilla can plot it. Then we need some change on the datazilla side. I have a proof of concept.

Also I do need the performance goal stated.

The testsuite will have them by device.

Comment 4

[Hubert Figuiere [:hub]]

Attachment: Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/22531

The current data set is pure fantasy.

Comment 5

[:Eli Perelman]

The only launch test value that has a performance goal associated with it is hitting "moz-app-visually-complete" by 1000ms. I am unsure if there are goals for other metrics like memory or overfill.

Comment 6

[Mike Lee [:mlee]]

fxOS Performance Criteria & Goals are always accessible here: https://wiki.mozilla.org/FirefoxOS/Performance/Release_Acceptance

We currently have latency for application first launch and are working to also define memory usage for that scenario.

Comment 7

[Hubert Figuiere [:hub]]

Work being done in bug 1049031 will require this to be rebased.

Comment 8

[Hubert Figuiere [:hub]]

Updated the PR.

Comment 9

[Hubert Figuiere [:hub]]

Now we need to get it plotted.

Comment 10

[Hubert Figuiere [:hub]]

Jonathan, do you think we can get this plotted on datazilla?

We have a "mozPerfGoal" value, that is optional, with each perf results. We'd like to have it displayed at the same time as the actual perf value.

Thanks,

(landing will happen soon as gaia is closed right now)

Comment 11

[Hubert Figuiere [:hub]]

Merged
https://github.com/mozilla-b2g/gaia/commit/4c6506064270bc054f70b2923fe2b129227d479c

Comment 12

[Hubert Figuiere [:hub]]

Or maybe :jeads you are the better contact?

Comment 13

[Jonathan Griffin (:jgriffin)]

Jeads, any guesses as to how much work it is?  If it's not trivial, Eli said he'd be willing to step in and implement the front-end side, assuming the mozPerfGoal values are making it into the datazilla db.

Comment 14

[Jonathan Griffin (:jgriffin)]

We're not currently submitting mozPerfGoal to datazilla.  We'll need to modify the mozperf poster to handle that, but to do that, we'll have to decide how we want to handle this.

The easiest way would require no datazilla changes, but would require a little extra work on the part of people who are interested in the goal, and that would be to submit the goal as an extra "app" that could be selected or de-selected.  So, for example, when selecting "startup_>_moz-app-visually-complete" as the test, you might see the following under apps:  "calendar", "calendar-goal", "camera", "camera-goal", etc.

The other option is to add the goal to the metadata we submit with each test, and modify datazilla to display it.  This doesn't fit in with the current data model (see e.g., https://datazilla.mozilla.org/b2g/refdata/objectstore/json_blob/360433), so the work to support this might be non-trivial.

Comment 15

[Jonathan Eads ( :jeads )]

This might not be much work if I understand it correctly. The UI could independently ingest the config file, https://github.com/mozilla-b2g/gaia/blob/master/tests/performance/config.json, and use the device/test names to map it to the correct set of options for display in the performance graph in the datazilla b2g UI. This would require that the exact same device/test nomenclature is used in comparison to what is sent to datazilla. If those match, this approach would work and would probably make more sense then repeatedly sending the same target value in the test data json.

Comment 16

[Hubert Figuiere [:hub]]

Attachment: Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/22893

I caught this testing on hamachi - because we have an empty goal set on hamachi.

Comment 17

[Hubert Figuiere [:hub]]

In master.
https://github.com/mozilla-b2g/gaia/commit/d8452840dab1e8de6e30f9dc615fa14181b00157

Comment 18

[Jonathan Eads ( :jeads )]

The modifications to datazilla to display the goals in https://github.com/mozilla-b2g/gaia/blob/master/tests/performance/config.json#L14 have been made.

Note, the goals will only display for a device name of "flame" because that's the only entry in config.json. If the intention was to apply the goals to the flame-512MB and flame-319MB just add entries for them in the goals structure in config.json and they will show up on the datazilla graph.

Also, there is a discrepancy between the test name listed in config.json and the one sent to datazilla in the json objects:

config.json: "startup event test > * > startup > moz-app-visually-complete"

json objects: "startup_>_moz-app-visually-complete"

To make these match I'm replacing the underscores with spaces:
https://github.com/mozilla/datazilla/blob/master/datazilla/webapp/static/js/b2g_apps/PerformanceGraphComponent.js#L198

This is a bit brittle, it would be great if we could match the test name in config.json exactly without having to modify it. It will work for now, let me know if this gets changed in the future.

Comment 19

[Hubert Figuiere [:hub]]

You shouldn't even be aware of the config.json.

The test should have a value for the perf goal if there is one. Also the json object is transformed AFTER we run the test, so we are not the one changing these name. It is the datazilla submitter that is. See comment 14, :jgriffin knows more about that.

Also, you should be aware that since the perf team has been reorganized (ie disbanded) there is nobody to own this bug / task anymore. So the work may be all pointless.

Comment 20

[Jonathan Griffin (:jgriffin)]

Datazilla is being deprecated this quarter.