Closed Bug 1306156 Opened 8 years ago Closed 7 years ago

[webvr] Use telemetry for tracking usage of WebVR API

Categories

(Core :: WebVR, defect, P3)

x86_64
Windows
defect

Tracking

()

RESOLVED FIXED
mozilla56
Tracking Status
firefox56 --- fixed

People

(Reporter: caseyyee.ca, Assigned: daoshengmu)

References

Details

(Whiteboard: [gfx-noted][webvr])

Attachments

(1 file)

We'll want to do this to measure how many people are using WebVR as a KPI.
Please file bugs in relevant components. If webvr needs its own component, please file a bmo bug to have that done.
Component: General → Graphics
Product: Firefox → Core
OS: Unspecified → Windows
Priority: -- → P3
Hardware: Unspecified → x86_64
Whiteboard: [gfx-noted]
à la https://www.chromestatus.com/metrics/feature/popularity
Summary: [webvr] track users entering VR. → [webvr] Use telemetry for tracking usage of WebVR API
Whiteboard: [gfx-noted] → [gfx-noted][webvr]
Component: Graphics → WebVR
I think that we'd like to get info along the lines of:
- MAU on headsets (hopefully by headset)
- Saw WebVR content but did not have a headset 
- Length of sessions in WebVR in a headset
Rebecca, can you help us sanity check this telemetry we are trying to wire up for WebVR? 

We are shipping WebVR 1.1 support in Firefox 55, and it's been requested that we get a sense of usage to help the Firefox team understand priorities and future investment. So we're moving quickly to add in telemetry in 55.

Current thinking is that we'd like to track when:

- A user viewed WebVR content with a headset (MAU)
- A user viewed WebVR content in 2D in the browser (ie, did not use a headset)
- Time spent in WebVR content (with and without headset)

Kip can comment more, but I believe we can do this in the Chrome and is relatively easy to implement. The question is how we can collect this appropriately (ie, anonymized/aggregated), and to make sure we're not stepping over any privacy lines?

AV
Flags: needinfo?(rweiss)
Per Rebecca's ask, answering questions 1-9 from form in bug below.

Form URL: https://docs.google.com/document/d/1F2mnKZiQu76puez1qXvVDNExYMicZIu4VzGdSRmjgKM/edit#

1) What are the motivating questions you want to answer with this data?

Understand use of WebVR hardware and WebVR content in the wild, and track number of users and growth over time. This will help conversations with Firefox org to describe usage of WebVR by early adopters and help guide investment decisions.


2) Why does Mozilla need answers to this question?  Are there benefits for users? Do we need this information to address business requirements?

Establish baselines and track implementation quality/regressions
Provide information necessary to decide future investment levels and supporting OKRs
Determining whether implementation changes has an effect on user usage levels or hardware investment (time in WebVR, or encouraging purchase of hardware for "full" VR experience)



3) List all proposed measurements.  Refer to data collection types for Firefox on the Mozilla wiki and indicate types of measurements for each one.   General characterization of your data collection request will fall the highest type of data.

We wish to measure when:
- A user viewed WebVR content with a headset (MAU)
- A user viewed WebVR content in 2D in the browser (ie, did not use a headset)
- Time spent viewing WebVR content (with and without headset)



4) How long will this data be collected? 
I want to permanently monitor this data. (Lars Bergstrom/Andre Vrignaud)


5) What populations do you need to these measurements for?
Which release channels? - NIGHTLY, BETA, RELEASE
Which countries? - All
Which locales? - All


6) Please provide a general description of how you will analyze this data.
Collect in database, query to graph


7) What alternative methods did you consider to answer this question? Why were they not sufficient?
Data does not exist elsewhere. New technology.


8) Can current instrumentation answer this question?
No, but we can collect data from Firefox Chrome. Larsberg can clarify.


9) Where do you intend to share the results of your analysis?
Internally with WebVR and Firefox teams.
Flags: needinfo?(rweiss) → needinfo?(larsberg)
Rebecca, please see questions above. Lars is also NIed in case we need additional information.
Flags: needinfo?(rweiss)
For question 3, it appears that all are "category 2" types, if I understand https://wiki.mozilla.org/Firefox/Data_Collection correctly.

For question 8, if this means "can we collect with telemetry?" :kip and :daoshengmu have indicate that is the case in our initial conversations.
Flags: needinfo?(larsberg)
Depends on: 1353080
Assignee: nobody → dmu
(In reply to Daosheng Mu[:daoshengmu] from comment #8)
> Created attachment 8889818 [details]
> Bug 1306156 - (wip)Add telemetry to watch MAU and time spent of WebVR users;
> 
> Review commit: https://reviewboard.mozilla.org/r/160910/diff/#index_header
> See other reviews: https://reviewboard.mozilla.org/r/160910/

In this patch, it already can catch the users who view WebVR content in their headsets and collect their time spent.

I still need to find an approach to identify if this user is viewing in 2D not in headsets. When viewing in 2D, users can only view the VR content in a WebGL screen buffer and can't ask for starting presentation.
Comment on attachment 8889818 [details]
Bug 1306156 - Add telemetry for watching MAU and time spent of WebVR users;

https://reviewboard.mozilla.org/r/160910/#review168004

This looks good, thanks!
Attachment #8889818 - Flags: review?(kgilbert) → review+
Pushed by dmu@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/e4f2a707af80
Add telemetry for watching MAU and time spent of WebVR users; r=kip
https://hg.mozilla.org/mozilla-central/rev/e4f2a707af80
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla56
Comment on attachment 8889818 [details]
Bug 1306156 - Add telemetry for watching MAU and time spent of WebVR users;

Approval Request Comment
[Feature/Bug causing the regression]: This is telemetry for tracking users who use WebVR.
[User impact if declined]: We can't get the user data in FF 55.
[Is this code covered by automated tests?]: nope. I test it manually.
[Has the fix been verified in Nightly?]: Yes.
[Needs manual test from QE? If yes, steps to reproduce]: nope.
[List of other uplifts needed for the feature/fix]: nope.
[Is the change risky?]: nope. This is just telemetry.
[Why is the change risky/not risky?]: This is just telemetry.
[String changes made/needed]: nope.
Attachment #8889818 - Flags: approval-mozilla-beta?
I would like to send an uplift request for FF 55, hope we can have this telemetry in time.
Comment on attachment 8889818 [details]
Bug 1306156 - Add telemetry for watching MAU and time spent of WebVR users;

sorry, not taking this in a release candidate.
Attachment #8889818 - Flags: approval-mozilla-beta? → approval-mozilla-beta-
Depends on: 1388274
For the record, r+

1) Is there documentation that describes the schema for the ultimate data set available publicly, complete and accurate?
Yes, there will be, since they will be writing Telemetry probes eventually.

2) Is there a control mechanism that allows the user to turn the data collection on and off? (Note, for data collection not needed for security purposes, Mozilla provides such a control mechanism)  
Yes, standard Telemetry opt out mechanism will apply here.

3) If the request is for permanent data collection, is there someone who will monitor the data over time?
Yes, Lars Bergstrom and Andre Vrignaud.

4) Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?  
This seems like a hybrid between 1 and 2, so default to Category 2 measurements.

5) Is the data collection default-on or default-off? 
Default on, all channels, all locales.

6) Does the instrumentation include the addition of any new identifiers (whether anonymous or otherwise; e.g., username, random IDs, etc.  See the appendix for more details)? 
No.

7) Is the data collection covered by the existing Firefox privacy notice? If unsure: escalate to legal.
Yes.

8) Does there need to be a check-in in the future to determine whether to renew the data? (Yes/No) (If yes, set a todo reminder or file a bug if appropriate)
No.
Flags: needinfo?(rweiss)
You need to log in before you can comment on or make changes to this bug.