1242057 - Tune telemetry basics

Status: VERIFIED FIXED

(DevTools :: Responsive Design Mode, defect, P1)

Description

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

The basic shell in bug 1239437 has open / close telemetry, but uses the same probe as the legacy RDM tool.

We should decide whether to track the new one separately, etc.

Comment 1

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

:clarkbw, any thoughts on additional telemetry to collect from the start beyond open / close?

Can always add more as features land, but just wondering if there are some core things we should be sure to have from the start.

Comment 2

[Bryan Clark (DevTools PM) [:clarkbw]]

Currently we have these probes:
DEVTOOLS_RESPONSIVE_OPENED_BOOLEAN
DEVTOOLS_RESPONSIVE_OPENED_PER_USER_FLAG
DEVTOOLS_RESPONSIVE_TIME_ACTIVE_SECONDS

We should be able to see a change based on the release from one setting to the next however we won't see the release channel.

Our most basic questions we need answered are, Are people using this tool?  Are they using it more than before our changes (do they find it more useful)?

DEVTOOLS_RESPONSIVE_OPENED_BOOLEAN

This never reports false so its not really optimal.  Either we convert it over or create a new one that uses a count instead of a bool. Perhaps something like DEVTOOLS_RESPONSIVE_OPENED_COUNT ?

I think this count value should be a release / opt-out probe so we can see release channel values here.

DEVTOOLS_RESPONSIVE_OPENED_PER_USER_FLAG

This one seems ok to me.  Essentially answering how many users ever opened the tool.

DEVTOOLS_RESPONSIVE_TIME_ACTIVE_SECONDS

I'm not sure I trust this metric because I don't understand exactly what the time active is capturing or what is considered active.  If this is working well this would be another value that would be useful to capture from the release channel as well.


I'll be putting together the necessary plans for the telemetry dashboard next week.  Those plans will be required to begin requesting that these probes be available in release.

Comment 3

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

(In reply to Bryan Clark (Firefox PM) [:clarkbw] from comment #2)
> DEVTOOLS_RESPONSIVE_OPENED_BOOLEAN
> 
> This never reports false so its not really optimal.  Either we convert it
> over or create a new one that uses a count instead of a bool. Perhaps
> something like DEVTOOLS_RESPONSIVE_OPENED_COUNT ?

Okay, we've been wanting to switch to count for a while, and we should do so across all tools at once.

I've filed bug 1247985 for this.

Comment 4

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

(In reply to Bryan Clark (Firefox PM) [:clarkbw] from comment #2)
> DEVTOOLS_RESPONSIVE_TIME_ACTIVE_SECONDS
> 
> I'm not sure I trust this metric because I don't understand exactly what the
> time active is capturing or what is considered active.  If this is working
> well this would be another value that would be useful to capture from the
> release channel as well.

This is measuring how long RDM was opened.  It starts a timer when you enter RDM, stops when you exit, and submits the elapsed time.

Comment 5

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

Attachment: repsonsive-telemetry

:clarkbw, does this seem right?  We're just enabled the time active probe for release.  Other than that, we're saying it looks fine as-is.

Comment 6

[Bryan Clark (DevTools PM) [:clarkbw]]

(In reply to J. Ryan Stinnett [:jryans] (use ni?) from comment #5)
> Created attachment 8722477 [details] [diff] [review]
> repsonsive-telemetry
> 
> :clarkbw, does this seem right?  We're just enabled the time active probe
> for release.  Other than that, we're saying it looks fine as-is.

Yes, I think we'll want to do the same for the other time active probes as well but we can do that in another bug.

Comment 7

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

Comment on attachment 8722477 [details] [diff] [review]
repsonsive-telemetry

Review of attachment 8722477 [details] [diff] [review]:
-----------------------------------------------------------------

This enables the DEVTOOLS_RESPONSIVE_TIME_ACTIVE_SECONDS probe on release.

:bsmedberg, the same reasoning from bug 1247985 comment 7 should apply here.

Comment 8

[Benjamin Smedberg]

Comment on attachment 8722477 [details] [diff] [review]
repsonsive-telemetry

This needs to have an expiration. 55 is what the rest of the tools are using, so that would be fine here too.

data-review=me with that change, or come talk to me about user value and monitoring for more permanent collection

Comment 9

[Gabriel [:gl] (ΦωΦ)]

Comment on attachment 8722477 [details] [diff] [review]
repsonsive-telemetry

Review of attachment 8722477 [details] [diff] [review]:
-----------------------------------------------------------------

r+ assuming you also add the expiration

Comment 10

[Pulsebot]

https://hg.mozilla.org/integration/fx-team/rev/0df0a7abe8a5

Comment 11

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/0df0a7abe8a5

Comment 12

[Mihai Boldan, Desktop QA [:mboldan]]

Attachment: sum.png

I managed to verify this issue on Windows 10 x86 and on Firefox 47.0a1 (2016-02-29).
I can confirm that DEVTOOLS_RESPONSIVE_OPENED_COUNT is correctly displayed.
Also, I found some potential issues:
- DEVTOOLS_RESPONSIVE_OPENED_PER_USER_FLAG - is enabled by default(I don't know if this is an expected behavior). And if you can provide me more information about this histogram, because I was not able to modify it's numbers.
-  on DEVTOOLS_RESPONSIVE_TIME_ACTIVE_SECONDS histogram, the sum is not equal with the seconds displayed under each counter. (see the attached screenshot). I've also observed that if the RDM tool is opened under 15 seconds, the correct no. of seconds is added to the sum.   
Is there anything that I should verify around this issue?
Should I log a bug for the issues described above?

Comment 13

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

(In reply to Mihai Boldan, QA [:mboldan] from comment #12)
> - DEVTOOLS_RESPONSIVE_OPENED_PER_USER_FLAG - is enabled by default(I don't
> know if this is an expected behavior). And if you can provide me more
> information about this histogram, because I was not able to modify it's
> numbers.

These "PER_USER_FLAG" probes only record a value once per browser version.  If you reset the pref "devtools.telemetry.tools.opened.version" from about:config which is used to track if we have logged already or not, you should be able to get the probe to fire once more on opening the tool.

> -  on DEVTOOLS_RESPONSIVE_TIME_ACTIVE_SECONDS histogram, the sum is not
> equal with the seconds displayed under each counter. (see the attached
> screenshot). I've also observed that if the RDM tool is opened under 15
> seconds, the correct no. of seconds is added to the sum.   
> Is there anything that I should verify around this issue?
> Should I log a bug for the issues described above?

Hmm, this seems like a possible error in the about:telemetry UI?  I am not sure.  We're only able to report each value separately to the telemetry system, so I am not sure how it computes that sum.

Perhaps file a bug the Toolkit :: Telemetry component.

Seems like that covers everything for this bug, thanks!

Comment 14

[Mihai Boldan, Desktop QA [:mboldan]]

I managed to fire the DEVTOOLS_RESPONSIVE_OPENED_PER_USER_FLAG probe once more, with the steps provided by Ryan in Comment 13.
Also, the bug related to the DEVTOOLS_RESPONSIVE_TIME_ACTIVE_SECONDS probe is invalid - see Bug 1253259 Comment 5.
Since the potential found issues were not confirmed, I am marking this issue Verified_Fixed.