1552464 - Add telemetry to count number of RDM viewport changes

Status: RESOLVED WONTFIX

(DevTools :: Responsive Design Mode, task, P1)

Comment 1

[Razvan Caliman [:rcaliman]]

Attachment: Bug 1552464 - Add telemetry probe to count the number of viewport changes in RDM. r=mtigley,nchevobbe.

This patch adds telemetry instrumentation to count the number of times the RDM viewport properties are changed (dimensions and rotation). This count will be correlated with the panel open count and time spent open to refine the baseline for RDM usage and filter out accidental usage.

A new Redux middleware, telemetryMiddleware, is introduced to the RDM Redux store. This observes actions dispatched to the store. For RESIZE_VIEWPORT and ROTATE_VIEWPORT actions, it increases a numeric value for the new scalar telemetry probe, "devtools.responsive.viewport_change_count".

Other actions may be observed in this middleware for future telemetry instrumentation of RDM.

The RESIZE_VIEWPORT action is a dispatched with a high frequency when dragging to resize. Therefore, we debounce logging for this action. To ensure the test can reliably test counting this action without adding needless complexity to account for the asynchronicity, the debounce() utility is extended with an immediate parameter to cause the very first call to be executed immediately before going into the debounce behaviour.

Comment 2

[Razvan Caliman [:rcaliman]]

Attachment: request-rdm-adau.md

Hi Ben!

May I please have your review on this request to add a new telemetry probe?

It's meant to measure the number of times users take meaningful action in the Responsive Design Mode (RDM) in DevTools. We will correlate this with existing data about RDM open count and time spent in the tool in order to improve our understanding of baseline usage and correct for any accidental use.

Comment 3

[Ben Miroglio [:bmiroglio]]

Comment on attachment 9065745 [details]
request-rdm-adau.md

DATA COLLECTION REVIEW RESPONSE:

    Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate?

Yes. This collection is Telemetry so is documented in its definitions file [Scalars.yaml](https://hg.mozilla.org/mozilla-central/file/tip/toolkit/components/telemetry/Scalars.yaml) and the [Probe Dictionary](https://telemetry.mozilla.org/probe-dictionary/).

    Is there a control mechanism that allows the user to turn the data collection on and off?

Yes. This collection is Telemetry so can be controlled through Firefox's Preferences.

    If the request is for permanent data collection, is there someone who will monitor the data over time?

Yes, [e-mail] is responsible.


    Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 2, Interaction.

    Is the data collection request for default-on or default-off?

Default on for all channels.

    Does the instrumentation include the addition of any new identifiers?

No.

    Is the data collection covered by the existing Firefox privacy notice?

Yes.

    Does there need to be a check-in in the future to determine whether to renew the data?

No. This collection is permanent.


---
Result: datareview r+

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3c2a368a4c8b
Add telemetry probe to count the number of viewport changes in RDM. r=mtigley,nchevobbe.

Comment 5

[Oana Pop-Rus]

Backed out changeset 3c2a368a4c8b (bug 1552464) for xpcshell failures on test_rotate_viewport.js

Push that started the failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=superseded%2Ctestfailed%2Cbusted%2Cexception%2Crunnable&revision=3c2a368a4c8bb527fdd37645b015787a3d3e2613&selectedJob=247746196

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=247746196&repo=autoland&lineNumber=2480

Backout: https://hg.mozilla.org/integration/autoland/rev/b6e9a64b68d92fcb540c0c90550986d0ada13eab

[task 2019-05-22T11:55:38.093Z] 11:55:38 INFO - TEST-START | devtools/client/responsive.html/test/unit/test_rotate_viewport.js
[task 2019-05-22T11:55:38.548Z] 11:55:38 WARNING - TEST-UNEXPECTED-FAIL | devtools/client/responsive.html/test/unit/test_rotate_viewport.js | xpcshell return code: 0
[task 2019-05-22T11:55:38.548Z] 11:55:38 INFO - TEST-INFO took 450ms
[task 2019-05-22T11:55:38.548Z] 11:55:38 INFO - >>>>>>>
[task 2019-05-22T11:55:38.548Z] 11:55:38 INFO - PID 14786 | [14786, Main Thread] WARNING: Couldn't get the user appdata directory. Crash events may not be produced.: file /builds/worker/workspace/build/src/toolkit/crashreporter/nsExceptionHandler.cpp, line 2561
[task 2019-05-22T11:55:38.549Z] 11:55:38 INFO - PID 14786 | JavaScript strict warning: resource://devtools/shared/Loader.jsm, line 231: ReferenceError: reference to undefined property "name"
[task 2019-05-22T11:55:38.550Z] 11:55:38 INFO - (xpcshell/head.js) | test MAIN run_test pending (1)
[task 2019-05-22T11:55:38.550Z] 11:55:38 INFO - (xpcshell/head.js) | test run_next_test 0 pending (2)
[task 2019-05-22T11:55:38.550Z] 11:55:38 INFO - (xpcshell/head.js) | test MAIN run_test finished (2)
[task 2019-05-22T11:55:38.550Z] 11:55:38 INFO - running event loop
[task 2019-05-22T11:55:38.551Z] 11:55:38 INFO - "CONSOLE_MESSAGE: (warn) [JavaScript Warning: "ReferenceError: reference to undefined property "name"" {file: "resource://devtools/shared/Loader.jsm" line: 231}]"
[task 2019-05-22T11:55:38.551Z] 11:55:38 INFO - devtools/client/responsive.html/test/unit/test_rotate_viewport.js | Starting
[task 2019-05-22T11:55:38.551Z] 11:55:38 INFO - (xpcshell/head.js) | test pending (2)
[task 2019-05-22T11:55:38.552Z] 11:55:38 INFO - PID 14786 | console.log: "[DISPATCH] action type:" "ADD_VIEWPORT"
[task 2019-05-22T11:55:38.552Z] 11:55:38 INFO - TEST-PASS | devtools/client/responsive.html/test/unit/test_rotate_viewport.js | - Default width of 320 - 320 == 320
[task 2019-05-22T11:55:38.552Z] 11:55:38 INFO - TEST-PASS | devtools/client/responsive.html/test/unit/test_rotate_viewport.js | - Default height of 480 - 480 == 480
[task 2019-05-22T11:55:38.552Z] 11:55:38 INFO - PID 14786 | console.log: "Unable to post message to window"
[task 2019-05-22T11:55:38.553Z] 11:55:38 INFO - (xpcshell/head.js) | test run_next_test 0 finished (2)
[task 2019-05-22T11:55:38.555Z] 11:55:38 INFO - Unexpected exception TypeError: telemetry is undefined at resource://devtools/client/responsive.html/middleware/telemetry.js:37
[task 2019-05-22T11:55:38.556Z] 11:55:38 INFO - logViewportChange@resource://devtools/client/responsive.html/middleware/telemetry.js:37:5
[task 2019-05-22T11:55:38.557Z] 11:55:38 INFO - telemetryMiddleware/</</<@resource://devtools/client/responsive.html/middleware/telemetry.js:51:9
[task 2019-05-22T11:55:38.559Z] 11:55:38 INFO - exports.history/</</<@resource://devtools/client/shared/redux/middleware/history.js:21:5
[task 2019-05-22T11:55:38.560Z] 11:55:38 INFO - waitUntilService/</<@resource://devtools/client/shared/redux/middleware/wait-service.js:59:20
[task 2019-05-22T11:55:38.562Z] 11:55:38 INFO - promiseMiddleware/</<@resource://devtools/client/shared/redux/middleware/promise.js:16:14
[task 2019-05-22T11:55:38.563Z] 11:55:38 INFO - thunk/</<@resource://devtools/client/shared/redux/middleware/thunk.js:16:9
[task 2019-05-22T11:55:38.564Z] 11:55:38 INFO - task/</<@resource://devtools/client/shared/redux/middleware/task.js:39:12
[task 2019-05-22T11:55:38.565Z] 11:55:38 INFO - @/builds/worker/workspace/build/tests/xpcshell/tests/devtools/client/responsive.html/test/unit/test_rotate_viewport.js:21:3
[task 2019-05-22T11:55:38.567Z] 11:55:38 INFO - run_next_test/_run_next_test/<@/builds/worker/workspace/build/tests/xpcshell/head.js:1437:22
[task 2019-05-22T11:55:38.568Z] 11:55:38 INFO - _run_next_test@/builds/worker/workspace/build/tests/xpcshell/head.js:1437:38
[task 2019-05-22T11:55:38.569Z] 11:55:38 INFO - run@/builds/worker/workspace/build/tests/xpcshell/head.js:688:9
[task 2019-05-22T11:55:38.571Z] 11:55:38 INFO - _do_main@/builds/worker/workspace/build/tests/xpcshell/head.js:227:6
[task 2019-05-22T11:55:38.572Z] 11:55:38 INFO - _execute_test@/builds/worker/workspace/build/tests/xpcshell/head.js:529:5
[task 2019-05-22T11:55:38.573Z] 11:55:38 INFO - @-e:1:1
[task 2019-05-22T11:55:38.575Z] 11:55:38 INFO - exiting test
[task 2019-05-22T11:55:38.576Z] 11:55:38 INFO - PID 14786 | [14786, Main Thread] WARNING: OOPDeinit() without successful OOPInit(): file /builds/worker/workspace/build/src/toolkit/crashreporter/nsExceptionHandler.cpp, line 3055
[task 2019-05-22T11:55:38.576Z] 11:55:38 INFO - PID 14786 | [14786, Main Thread] WARNING: XPCOM objects created/destroyed from static ctor/dtor: file /builds/worker/workspace/build/src/xpcom/base/nsTraceRefcnt.cpp, line 194
[task 2019-05-22T11:55:38.577Z] 11:55:38 INFO - PID 14786 | [14786, Main Thread] WARNING: XPCOM objects created/destroyed from static ctor/dtor: file /builds/worker/workspace/build/src/xpcom/base/nsTraceRefcnt.cpp, line 194
[task 2019-05-22T11:55:38.578Z] 11:55:38 INFO - PID 14786 | nsStringStats
[task 2019-05-22T11:55:38.578Z] 11:55:38 INFO - PID 14786 | => mAllocCount: 12428
[task 2019-05-22T11:55:38.579Z] 11:55:38 INFO - PID 14786 | => mReallocCount: 0
[task 2019-05-22T11:55:38.579Z] 11:55:38 INFO - PID 14786 | => mFreeCount: 12428
[task 2019-05-22T11:55:38.579Z] 11:55:38 INFO - PID 14786 | => mShareCount: 11586
[task 2019-05-22T11:55:38.580Z] 11:55:38 INFO - PID 14786 | => mAdoptCount: 267
[task 2019-05-22T11:55:38.581Z] 11:55:38 INFO - PID 14786 | => mAdoptFreeCount: 267
[task 2019-05-22T11:55:38.581Z] 11:55:38 INFO - PID 14786 | => Process ID: 14786, Thread ID: 140437387760640
[task 2019-05-22T11:55:38.581Z] 11:55:38 INFO - <<<<<<<

Comment 6

[Razvan Caliman [:rcaliman]]

Argh! An xpchsell test fail because I forgot to run test that suite.

Fixed the issue and pushed to try to check before re-attempting to land:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=83e50f0feee8a4f6319e9c1ee658b8211e9e83e1

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/dc38da197244
Add telemetry probe to count the number of viewport changes in RDM. r=mtigley,nchevobbe.

Comment 8

[Noemi Erli[:noemi_erli]]

Backed out changeset dc38da197244 (Bug 1552464) for devtools failures in browser_telemetry_viewport_change.js

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=testfailed%2Cbusted%2Cexception&selectedJob=248193212&revision=dc38da1972445f4a46b1fb7f6d87d7b887bb0b97&searchStr=devtools%2C

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=248165433&repo=autoland&lineNumber=9194

Backout: https://hg.mozilla.org/integration/autoland/rev/01d6a08a12aeca2207b125aeacfa7f57ae2703ed

Comment 9

[Razvan Caliman [:rcaliman]]

Landing a telemetry test which behaves consistently across platforms is proving elusive.

Comment 10

[Razvan Caliman [:rcaliman]]

After closer inspection of the telemetry probe with Product Management, we realized that the number of viewport resize events can be easily misinterpreted when correlated to the number of times RDM is opened which can lead to false conclusions that RDM is not successfully used (a user may be using RDM throughout the course of a day without ever resizing the viewport as they refresh to see changes applied in the same viewport). We will continue to rely on the duration of the RDM being open as a proxy for its active use.

The data about the number of viewport resize events by itself doesn't provide enough value for any other question we currently have. Therefore, we decided to not land this.

We'll keep the code around for later. The Redux middleware for telemetry may be reused when we have new probes to put in.