1566882 - Enable PDF viewer telemetry on release

Status: RESOLVED FIXED

(Firefox :: PDF Viewer, enhancement, P1)

Description

[Romain Testard [:RT]]

Attachment: request.txt

Enable PDF telemetry on release to help inform priority of addressing PDF issues.

Comment 1

[Romain Testard [:RT]]

Mike, Brendan, Sharon - please let me know if you have anything to add to request.txt before I request data steward review.

Comment 2

[Romain Testard [:RT]]

Comment on attachment 9078753 [details]
request.txt


# Request for data collection review form

**All questions are mandatory. You must receive review from a data steward peer on your responses to these questions before shipping new data collection.**

1) What questions will you answer with this data?
Should we invest in pdf viewer development?
- What share of DAU/MAU uses the pdf viewer?
- How does pdf usage impact retention?

Assuming high pdf usage and lower retention for pdf viewer users, identify pdf characteristics that most correlate with retention drops?

2) Why does Mozilla need to answer these questions?  Are there benefits for users? Do we need this information to address product or business requirements? 

Provide information essential to make resourcing decision for pdf viewer engineering.
Identify high priority areas to address retention issues on pdf usage, therefore contribute to MAU. FYI PDF viewer is a critical Enterprise use case identified through user research. 

3) What alternative methods did you consider to answer these questions? Why were they not sufficient?
Using pre-release data is not sufficient because pre-release population is not representative of release.

4) Can current instrumentation answer these questions?
No

5) List all proposed measurements and indicate the category of data collection for each measurement, using the Firefox [data collection categories](https://wiki.mozilla.org/Firefox/Data_Collection) on the Mozilla wiki.   

Make existing pre-release probes available on release:
PDF_VIEWER_DOCUMENT_GENERATOR
PDF_VIEWER_DOCUMENT_SIZE_KB
PDF_VIEWER_DOCUMENT_VERSION
PDF_VIEWER_EMBED
PDF_VIEWER_FALLBACK_SHOWN
PDF_VIEWER_FONT_TYPES
PDF_VIEWER_FORM
PDF_VIEWER_PRINT
PDF_VIEWER_STREAM_TYPES
PDF_VIEWER_TIME_TO_VIEW_MS
PDF_VIEWER_USED


6) How long will this data be collected?  Choose one of the following:

* I want this data to be collected for 6 months initially (potentially renewable).

7) What populations will you measure?

Release channel
All locales
All countries

8) If this data collection is default on, what is the opt-out mechanism for users?
Standard telemetry opt-out

9) Please provide a general description of how you will analyze this data.
Redash dashboard

10) Where do you intend to share the results of your analysis?
Redash dashboard

11) Is there a third-party tool (i.e. not Telemetry) that you are proposing to use for this data collection? If so:
No

Comment 3

[Romain Testard [:RT]]

Attachment: request.txt

FYI this is the right request file, you can ignore the previous one.

Comment 4

[Mike Kaply [:mkaply]]

lgtm

Comment 5

[Sharon Bautista (she/her) [:sbautista]]

I'm good with this.

Comment 6

[Brendan Dahl [:bdahl]]

Looks good.

Comment 7

[Romain Testard [:RT]]

Hi Chris, can you please help review the request form to bring pre-release pdf telemetry to release?

Comment 8

[Chris H-C :chutten]

Comment on attachment 9078756 [details]
request.txt

DATA COLLECTION REVIEW RESPONSE:

    Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate?

Yes. This collection is Telemetry so is documented in its definitions file [Histograms.json](https://hg.mozilla.org/mozilla-central/file/tip/toolkit/components/telemetry/Histograms.json) and the [Probe Dictionary](https://telemetry.mozilla.org/probe-dictionary/).

    Is there a control mechanism that allows the user to turn the data collection on and off?

Yes. This collection is Telemetry so can be controlled through Firefox's Preferences.

    If the request is for permanent data collection, is there someone who will monitor the data over time?

No. This collection will expire in six months.

    Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 2, Interaction.

    Is the data collection request for default-on or default-off?

Default on for all channels.

    Does the instrumentation include the addition of any new identifiers?

No.

    Is the data collection covered by the existing Firefox privacy notice?

Yes.

    Does there need to be a check-in in the future to determine whether to renew the data?

Yes. :rtestard is responsible for renewing or removing the collection before it expires in six months.

---
Result: datareview+

Comment 9

[Chris H-C :chutten]

Taking my Data Steward hat off and putting my Firefox Telemetry hat on, may I ask if you considered modernizing the measurements as part of this initiative?

• The histograms as written are all very old, which suggests that they might no longer be fit for modern purposes.
• Some of the measurements are actually recording scalar information (pdf viewer used, fallback shown, print), so would be easier to analyze and more efficient to store and send if they were recorded as Scalars
• Others are enumerated histograms when they would benefit quite a lot from being categorical histograms.

Comment 10

[Romain Testard [:RT]]

I did not and feel it would be hard given that pdf viewer is not resourced at the moment.
Are there specific limitations we should be aware of where the current telemetry will limit our ability to answer our questions?

• What share of DAU/MAU uses the pdf viewer?
• How does pdf usage impact retention?
• Identify pdf characteristics that most correlate with retention drops?

Comment 11

[Chris H-C :chutten]

I haven't looked in to what the metrics are actually measuring, so I couldn't say. I recommend asking someone familiar with the pdf viewer code to look into what these are actually measuring, to see if they're fit for your purpose.

Comment 12

[Romain Testard [:RT]]

Thanks Chris.
Brendan, would you be the right person to help there? Otherwise someone else we should ask?

Comment 13

[Brendan Dahl [:bdahl]]

The changes would probably help the readability of the telemetry. I'm not sure it affects what you're trying to accomplish though.

Romain, I can take a stab at updating them if you're interested.

Comment 14

[Romain Testard [:RT]]

(In reply to Brendan Dahl [:bdahl] from comment #13)

The changes would probably help the readability of the telemetry. I'm not sure it affects what you're trying to accomplish though.

Romain, I can take a stab at updating them if you're interested.

That would be great, it sounds like it will make the analysis simpler and storage more efficient so it's a win in any case.

Comment 15

[Brendan Dahl [:bdahl]]

I've opened a pull request upstream for pdf.js, once that lands I'll add the patch to migrate the values here.

Comment 16

[Brendan Dahl [:bdahl]]

Attachment: Bug 1566882 - Update pdf.js telemetry to use more appropriate types. r=chutten

Use the new 'scalar' and 'categorical' telemetry types to make it easier
to read the graphs.

Moves the majority of the data to expire in 75. The following are set to
never expire since they are good to monitor for regressions and general
usage.

• PDF_VIEWER_TIME_TO_VIEW_MS
• pdf.viewer.used

Comment 17

[Brendan Dahl [:bdahl]]

Attachment: Bug 1566882 - Update pdf.js telemetry to use more appropriate types. r=chutten

Use the new 'scalar' and 'categorical' telemetry types to make it easier
to read the graphs.

Moves the majority of the data to expire in 75. The following are set to
never expire since they are good to monitor for regressions and general
usage.

• PDF_VIEWER_TIME_TO_VIEW_MS
• pdf.viewer.used

Comment 18

[Brendan Dahl [:bdahl]]

Attachment: Bug 1566882 - Update pdf.js to version 2.3.71. r=yury

Depends on D40660

Comment 19

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b8cab1a52f0e
Update pdf.js telemetry to use more appropriate types. r=chutten,Snuffleupagus
https://hg.mozilla.org/integration/autoland/rev/79294943c407
Update pdf.js to version 2.3.71. r=yury

Comment 20

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/b8cab1a52f0e
https://hg.mozilla.org/mozilla-central/rev/79294943c407