Closed Bug 1242057 Opened 5 years ago Closed 5 years ago

Tune telemetry basics

Categories

(DevTools :: Responsive Design Mode, defect, P1)

defect

Tracking

(firefox46 affected, firefox47 verified)

VERIFIED FIXED
Firefox 47
Iteration:
47.3 - Mar 7
Tracking Status
firefox46 --- affected
firefox47 --- verified

People

(Reporter: jryans, Assigned: jryans)

References

Details

(Whiteboard: [multiviewport] [mvp-rdm])

Attachments

(2 files)

The basic shell in bug 1239437 has open / close telemetry, but uses the same probe as the legacy RDM tool.

We should decide whether to track the new one separately, etc.
Flags: qe-verify?
Flags: qe-verify? → qe-verify+
Priority: -- → P1
:clarkbw, any thoughts on additional telemetry to collect from the start beyond open / close?

Can always add more as features land, but just wondering if there are some core things we should be sure to have from the start.
Flags: needinfo?(clarkbw)
Assignee: nobody → jryans
Status: NEW → ASSIGNED
Iteration: --- → 47.2 - Feb 22
Currently we have these probes:
DEVTOOLS_RESPONSIVE_OPENED_BOOLEAN
DEVTOOLS_RESPONSIVE_OPENED_PER_USER_FLAG
DEVTOOLS_RESPONSIVE_TIME_ACTIVE_SECONDS

We should be able to see a change based on the release from one setting to the next however we won't see the release channel.

Our most basic questions we need answered are, Are people using this tool?  Are they using it more than before our changes (do they find it more useful)?

DEVTOOLS_RESPONSIVE_OPENED_BOOLEAN

This never reports false so its not really optimal.  Either we convert it over or create a new one that uses a count instead of a bool. Perhaps something like DEVTOOLS_RESPONSIVE_OPENED_COUNT ?

I think this count value should be a release / opt-out probe so we can see release channel values here.

DEVTOOLS_RESPONSIVE_OPENED_PER_USER_FLAG

This one seems ok to me.  Essentially answering how many users ever opened the tool.

DEVTOOLS_RESPONSIVE_TIME_ACTIVE_SECONDS

I'm not sure I trust this metric because I don't understand exactly what the time active is capturing or what is considered active.  If this is working well this would be another value that would be useful to capture from the release channel as well.


I'll be putting together the necessary plans for the telemetry dashboard next week.  Those plans will be required to begin requesting that these probes be available in release.
(In reply to Bryan Clark (Firefox PM) [:clarkbw] from comment #2)
> DEVTOOLS_RESPONSIVE_OPENED_BOOLEAN
> 
> This never reports false so its not really optimal.  Either we convert it
> over or create a new one that uses a count instead of a bool. Perhaps
> something like DEVTOOLS_RESPONSIVE_OPENED_COUNT ?

Okay, we've been wanting to switch to count for a while, and we should do so across all tools at once.

I've filed bug 1247985 for this.
QA Contact: mihai.boldan
Assignee: jryans → nobody
Status: ASSIGNED → NEW
Iteration: 47.2 - Feb 22 → ---
Priority: P1 → P2
Flags: needinfo?(clarkbw)
Assignee: nobody → jryans
Status: NEW → ASSIGNED
Iteration: --- → 47.2 - Feb 22
Priority: P2 → P1
(In reply to Bryan Clark (Firefox PM) [:clarkbw] from comment #2)
> DEVTOOLS_RESPONSIVE_TIME_ACTIVE_SECONDS
> 
> I'm not sure I trust this metric because I don't understand exactly what the
> time active is capturing or what is considered active.  If this is working
> well this would be another value that would be useful to capture from the
> release channel as well.

This is measuring how long RDM was opened.  It starts a timer when you enter RDM, stops when you exit, and submits the elapsed time.
:clarkbw, does this seem right?  We're just enabled the time active probe for release.  Other than that, we're saying it looks fine as-is.
Attachment #8722477 - Flags: feedback?(clarkbw)
Iteration: 47.2 - Feb 22 → 47.3 - Mar 7
Attachment #8722477 - Flags: feedback?(clarkbw) → feedback+
(In reply to J. Ryan Stinnett [:jryans] (use ni?) from comment #5)
> Created attachment 8722477 [details] [diff] [review]
> repsonsive-telemetry
> 
> :clarkbw, does this seem right?  We're just enabled the time active probe
> for release.  Other than that, we're saying it looks fine as-is.

Yes, I think we'll want to do the same for the other time active probes as well but we can do that in another bug.
Comment on attachment 8722477 [details] [diff] [review]
repsonsive-telemetry

Review of attachment 8722477 [details] [diff] [review]:
-----------------------------------------------------------------

This enables the DEVTOOLS_RESPONSIVE_TIME_ACTIVE_SECONDS probe on release.

:bsmedberg, the same reasoning from bug 1247985 comment 7 should apply here.
Attachment #8722477 - Flags: review?(gl)
Attachment #8722477 - Flags: feedback?(benjamin)
Comment on attachment 8722477 [details] [diff] [review]
repsonsive-telemetry

This needs to have an expiration. 55 is what the rest of the tools are using, so that would be fine here too.

data-review=me with that change, or come talk to me about user value and monitoring for more permanent collection
Attachment #8722477 - Flags: feedback?(benjamin) → feedback+
Comment on attachment 8722477 [details] [diff] [review]
repsonsive-telemetry

Review of attachment 8722477 [details] [diff] [review]:
-----------------------------------------------------------------

r+ assuming you also add the expiration
Attachment #8722477 - Flags: review?(gl) → review+
https://hg.mozilla.org/mozilla-central/rev/0df0a7abe8a5
Status: ASSIGNED → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
Target Milestone: --- → Firefox 47
Attached image sum.png
I managed to verify this issue on Windows 10 x86 and on Firefox 47.0a1 (2016-02-29).
I can confirm that DEVTOOLS_RESPONSIVE_OPENED_COUNT is correctly displayed.
Also, I found some potential issues:
- DEVTOOLS_RESPONSIVE_OPENED_PER_USER_FLAG - is enabled by default(I don't know if this is an expected behavior). And if you can provide me more information about this histogram, because I was not able to modify it's numbers.
-  on DEVTOOLS_RESPONSIVE_TIME_ACTIVE_SECONDS histogram, the sum is not equal with the seconds displayed under each counter. (see the attached screenshot). I've also observed that if the RDM tool is opened under 15 seconds, the correct no. of seconds is added to the sum.   
Is there anything that I should verify around this issue?
Should I log a bug for the issues described above?
Flags: needinfo?(jryans)
(In reply to Mihai Boldan, QA [:mboldan] from comment #12)
> - DEVTOOLS_RESPONSIVE_OPENED_PER_USER_FLAG - is enabled by default(I don't
> know if this is an expected behavior). And if you can provide me more
> information about this histogram, because I was not able to modify it's
> numbers.

These "PER_USER_FLAG" probes only record a value once per browser version.  If you reset the pref "devtools.telemetry.tools.opened.version" from about:config which is used to track if we have logged already or not, you should be able to get the probe to fire once more on opening the tool.

> -  on DEVTOOLS_RESPONSIVE_TIME_ACTIVE_SECONDS histogram, the sum is not
> equal with the seconds displayed under each counter. (see the attached
> screenshot). I've also observed that if the RDM tool is opened under 15
> seconds, the correct no. of seconds is added to the sum.   
> Is there anything that I should verify around this issue?
> Should I log a bug for the issues described above?

Hmm, this seems like a possible error in the about:telemetry UI?  I am not sure.  We're only able to report each value separately to the telemetry system, so I am not sure how it computes that sum.

Perhaps file a bug the Toolkit :: Telemetry component.

Seems like that covers everything for this bug, thanks!
Flags: needinfo?(jryans)
Whiteboard: [multiviewport] → [multiviewport] [mvp-rdm]
I managed to fire the DEVTOOLS_RESPONSIVE_OPENED_PER_USER_FLAG probe once more, with the steps provided by Ryan in Comment 13.
Also, the bug related to the DEVTOOLS_RESPONSIVE_TIME_ACTIVE_SECONDS probe is invalid - see Bug 1253259 Comment 5.
Since the potential found issues were not confirmed, I am marking this issue Verified_Fixed.
Status: RESOLVED → VERIFIED
Flags: qe-verify+
QA Whiteboard: [qe-rdm].
QA Whiteboard: [qe-rdm]. → [qe-rdm]
Product: Firefox → DevTools
You need to log in before you can comment on or make changes to this bug.