1129445 - [Raptor] Support visualizing multiple devices and branches in the Raptor UI

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::PerformanceTest, defect)

Description

[:Eli Perelman]

Right now the Raptor UI only segregates data by suite type, e.g. cold-launch, reboot. We need to find a way to also tag data with a device type and the branch from which it was generated. We should not mingle data of different devices and branches in the default Raptor UI views.

Comment 1

[Robert Wood [:rwood]]

The device type and branch should probably be written to the raptor.log also.

Comment 2

[:Eli Perelman]

We should also separate the memory configurations, i.e. flame-1G should be segregated from flame-319.

Comment 3

[:Eli Perelman]

Bug 1132175 seems to want to do something very similar to this bug. Essentially it comes down to being able to determine a unique identifier for a given device type.

Comment 4

[:Eli Perelman]

This change also needs to be adapted to the Raptor docker container to accept a different schema.

Comment 6

[:Eli Perelman]

For the moment, the implementation of this is going to be via a TAGS environment variable. TAGS can be passed a string which contains space-separated key-value pairs delimited with equals (=) signs. Keys must be valid JavaScript identifiers, with the added constraint that they be lower case and hyphenated at word boundaries.

Examples:

TAGS="memory=319"
TAGS="device=flame memory=512 branch=master"
TAGS="au-name=AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.089 branch=v2.2"

Use of this format will allow us to inject arbitrary data into the current InfluxDB 0.8 schema as columns, and in the future 0.9 schema as tags.

Comment 7

[:Eli Perelman]

Work towards changing the schema to support arbitrary device tags should be in place. Core work towards this bug is being done in the following branches:

https://github.com/mozilla-b2g/raptor/compare/mozilla-b2g:master...eliperelman:bug-1129445
https://github.com/eliperelman/node-mozdevice/compare/bug-1129445

With the current changes, the InfluxDB series name will move to being a solitary metric, e.g. visuallyLoaded, uss, etc. This should allow us greater flexibility when we get proper tags, in exchange for adopting templated dashboards at the present.

I'm also obsoleting comment 6. After talking with James today, it became apparent that constantly passing tags via the environment was not only tedious, but was also not scalable nor intuitive. We would rather see an approach that persists the device configuration/tags with the device for as long as possible, or until the value is changed.

So currently I am moving forward with persisting this data via `setprop`. It may not be the ideal, or even the final solution, but it solves immediately the need to set arbitrary tags for a device which will persist until changed, or the device is reflashed. My changes in the previously mentioned branches already have the necessary changes to support these properties as metadata.

Examples:

adb shell setprop persist.raptor.branch master
adb shell setprop persist.raptor.memory 319
adb shell setprop persist.raptor.device flame-kk
adb shell setprop persist.raptor.branch v2.2
adb shell setprop persist.raptor.qc_tag AU_LINUX_GECKO_LF.BR.1.2.3.00.00.00.000.089

Any property starting with "persist.raptor." is be extrapolated into metadata sent along with the datapoint payload. Maybe we can also support tagging the data without "persist" as well. Like I said, maybe not the final solution, but at least something in place so I can begin to work the new schema into the Raptor UI along with scripted dashboards.

Comment 8

[:Eli Perelman]

Attachment: Link to Github pull-request: https://github.com/eliperelman/node-mozdevice/pull/10

This patch modifies mozdevice to support the new Raptor schema, only needing changes to memory logging.

Comment 9

[:Eli Perelman]

Attachment: Link to Github pull-request: https://github.com/mozilla-b2g/raptor/pull/19

Here's the big one. This one has several changes to schema and data. In order to test this, you'll need the latest docker raptor container, eliperelman/raptor. If you are curious, my changes to the docker container are here [1].

Please ni? me or ping me on IRC with any questions.

[1] https://github.com/eliperelman/docker-raptor/commit/fd12cdc7579f74611f2e0dfb9040adf7539402e5

Comment 10

[:Eli Perelman]

An important note when testing out the stock eliperelman/raptor container: you need to set the Raptor tags prior to doing any testing:

adb shell setprop persist.raptor.branch master
adb shell setprop persist.raptor.memory 319
adb shell setprop persist.raptor.device flame-kk

Rob, to test a device other than the flame-kk, just change the raptor.device property to emulator, emulator-kk, or whatever. Also if you want to visualize emulator data, you'll need an additional panel defined in raptor.json in the docker container:

https://github.com/eliperelman/docker-raptor/commit/fd12cdc7579f74611f2e0dfb9040adf7539402e5#diff-4a036f14c508f243ce654dd2766794ccR17

You should be able to copy/paste the panel object and modify the properties and query string parameters to pass the correct raptor tags. If you need any help or none of this makes sense, ping me.

Comment 11

[Robert Wood [:rwood]]

Comment on attachment 8573606 [details] [review]
Link to Github pull-request: https://github.com/eliperelman/node-mozdevice/pull/10

Tested in conjunction with the raptor patch in this same bug. Ran the launch test on emulator and on flame-kk device, verified the memory logging entries now appear in three parts in console and in the raptor.log.

Comment 12

[Robert Wood [:rwood]]

Comment on attachment 8573609 [details] [review]
Link to Github pull-request: https://github.com/mozilla-b2g/raptor/pull/19

LGTM (just a tiny nit, noted in the PR). Tested with the node-mozdevice patch in this same bug. Built and started up the new visualization container, set persist values on flame-kk, ran the launch test multiple iterations. Verified the branch, memory, and device values appear in the raptor.log and on the visualization; verified the mark and memory results appeared on the visualization details. Modified the visualization docker image locally to add emulator support; set the persist values on the emulator and did the same test. Emulator results show up in raptor visualization on the dashboard alongside device, and details on their own URLs. Awesome!!

Comment 13

[GitHub Autolander Bot]

Attachment: [gaia-node-modules] eliperelman:bug-1129445 > mozilla-b2g:master

Comment 14

[:Eli Perelman]

Comment on attachment 8575567 [details] [review]
[gaia-node-modules] eliperelman:bug-1129445 > mozilla-b2g:master

Carrying over r+ for package bump.

Comment 15

[GitHub Autolander Bot]

Attachment: [gaia] eliperelman:bug-1129445 > mozilla-b2g:master

Comment 16

[:Eli Perelman]

Comment on attachment 8575570 [details] [review]
[gaia] eliperelman:bug-1129445 > mozilla-b2g:master

Carrying over r+ for package bump.

Comment 17

[:Eli Perelman]

In master: https://github.com/mozilla-b2g/gaia/commit/13efa14fe08126933dabe62d8b8b36d226f19328

Comment 18

[:Eli Perelman]

With this landed, it is also possible to visualize using different tags other than branch, device, and memory. Feel free to ping me if you have questions on how to make this possible.