[webide] telemetry-all-the-things

RESOLVED FIXED in Firefox 35

Status

()

Firefox
Developer Tools: WebIDE
P1
normal
RESOLVED FIXED
4 years ago
3 years ago

People

(Reporter: paul, Assigned: jryans)

Tracking

Trunk
Firefox 35
x86
All
Points:
---
Bug Flags:
in-testsuite +

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment, 5 obsolete attachments)

(Reporter)

Description

4 years ago
It would be interesting to know how people use the app manager:
- how many failed connections?
- how many click on Help
- how many projects are managed
- how often does the user update his app
- ...
(Reporter)

Updated

4 years ago
Priority: -- → P2
(Reporter)

Updated

4 years ago
Priority: P2 → P1
Filter on 86b7095e-2bd0-499e-a704-d00f2524aeef / PAUL STOP SETTING QA CONTACT TO THE DEVTOOLS COMPONENT'S WATCHERS EMAIL FOR BUGS YOU FILE :)
QA Contact: developer.tools
(Reporter)

Updated

3 years ago
Blocks: 1007059
Summary: [app manager] telemetry-all-the-things → [webide] telemetry-all-the-things
(Reporter)

Comment 2

3 years ago
I'm lost. I have no idea how to record these data.

First, I use toolOpened and toolClose. How do I check it actually works?

Then, I want to track:
- what runtimes have been discovered (type + number) (Simulators, USB Runtimes, Wifi Runtimes, localRuntime, remoteRuntime)
- number of successful runtime connections (with type)
- number of failed runtime connections (with type)
- how long is the runtime connected
- how often the toolbox is open per connection
- how often the user click "play" per connection

How do I do that? Who can help me?
Flags: needinfo?(mratcliffe)
Flags: needinfo?(jgriffiths)
Mike has done the majority of the telemetry work, and Jordan ( freshly needinfo'd ) has done some for Web Audio and Canvas tools.
Flags: needinfo?(jgriffiths) → needinfo?(jsantell)
For the timing stuff follow the comments in browser/devtools/shared/telemetry.js

For counting things:
1. Create DEVTOOLS_XXX_OPENED_BOOLEAN in toolkit/components/telemetry/Histograms.json (search for boolean to see how).
2. let Telemetry = require("devtools/shared/telemetry")
3. let telemetry = new Telemetry();
4. telemetry.log("DEVTOOLS_XXX_OPENED_BOOLEAN", true); // Do this every time you want to increment the number.

For logging numbers like how often the toolbox is open per connection... do the same as above but use "kind": "exponential" in the histogram.

See https://developer.mozilla.org/docs/Mozilla/Performance/Adding_a_new_Telemetry_probe to see which types to use.

To check that it is working look at how I tested it:
/Users/mratcliffe/Desktop/fx-team/browser/devtools/shared/test/browser_telemetry_sidebar.js // I monkeypatched the log method.
Flags: needinfo?(mratcliffe)
I think Mike addressed all the issues, but ping me if anything else is unclear!
Flags: needinfo?(jsantell)
(Assignee)

Updated

3 years ago
Assignee: nobody → jryans
Status: NEW → ASSIGNED
(Assignee)

Comment 6

3 years ago
Paul, I am making progress here, but I have some questions about what you want to track.

(In reply to Paul Rouget [:paul] (slow to respond. Ping me on IRC) from comment #2)
> - what runtimes have been discovered (type + number) (Simulators, USB
> Runtimes, Wifi Runtimes, localRuntime, remoteRuntime)

The available runtimes can change at any time when devices are unplugged / replugged.  WiFi devices only show up when you scan explicitly, etc.  So, when should data be recorded?  Each time the runtime list changes (this sounds like it would be very noisy data)?  Only once per opening of WebIDE?

More generally, what do we hope to learn from this particular data point?  I am not sure I see the value here...  It *does* seem useful to know what runtime types are used (USB, WiFi, etc.), but we'll have that from the successful / failed connection results you also suggested.

> - how often the toolbox is open per connection

So if I:

* Connect
* Open toolbox
* Close toolbox
* Open toolbox
* Disconnect

We'd record "2", because the toolbox was opened twice?  How is that useful?

> - how often the user click "play" per connection

Same question here as above: how is this data point useful?
Flags: needinfo?(paul)
(Reporter)

Comment 7

3 years ago
(In reply to J. Ryan Stinnett [:jryans] from comment #6)
> > - what runtimes have been discovered (type + number) (Simulators, USB
> > Runtimes, Wifi Runtimes, localRuntime, remoteRuntime)
> 
> The available runtimes can change at any time when devices are unplugged /
> replugged.  WiFi devices only show up when you scan explicitly, etc.  So,
> when should data be recorded?  Each time the runtime list changes (this
> sounds like it would be very noisy data)?  Only once per opening of WebIDE?
> 
> More generally, what do we hope to learn from this particular data point?  I
> am not sure I see the value here...

You're right. Let's ignore that.

> It *does* seem useful to know what
> runtime types are used (USB, WiFi, etc.), but we'll have that from the
> successful / failed connection results you also suggested.
> 
> > - how often the toolbox is open per connection
> 
> So if I:
> 
> * Connect
> * Open toolbox
> * Close toolbox
> * Open toolbox
> * Disconnect
> 
> We'd record "2", because the toolbox was opened twice?  How is that useful?
> 
> > - how often the user click "play" per connection
> 
> Same question here as above: how is this data point useful?

Both for toolbox and play, I'd like to know if people use WebIDE only to push app, or push app and iterate on the code, or push app, debug and iterate on the code.

Maybe people will use WebIDE only to push an app sometimes, maybe they will iterate on the code and not use the toolbox at all.

But all of that can happen in a follow up bug.
Flags: needinfo?(paul)
(Assignee)

Comment 8

3 years ago
Created attachment 8468884 [details] [diff] [review]
Telemetry for WebIDE

Paul, I believe this captures the values you're looking for.  You can test it interactively at about:telemetry, and filter on "webide".  (You have to refresh this tab to get updated histograms after you perform actions.)

Mike, I refactored the telemetry module slightly to expose the timing functionality more generally.  Can you check this over, and also see if I've set up the probes correctly?
Attachment #8468884 - Flags: review?(paul)
Attachment #8468884 - Flags: review?(mratcliffe)
(Assignee)

Comment 9

3 years ago
Try: https://tbpl.mozilla.org/?tree=Try&rev=9d6b90f627bf
Created attachment 8469006 [details] [diff] [review]
Telemetry for WebIDE (v2)

Fixed test failures.

Try: https://tbpl.mozilla.org/?tree=Try&rev=0ce2b99d173d
Attachment #8468884 - Attachment is obsolete: true
Attachment #8468884 - Flags: review?(paul)
Attachment #8468884 - Flags: review?(mratcliffe)
Attachment #8469006 - Flags: review?(paul)
Attachment #8469006 - Flags: review?(mratcliffe)
Comment on attachment 8469006 [details] [diff] [review]
Telemetry for WebIDE (v2)

Review of attachment 8469006 [details] [diff] [review]:
-----------------------------------------------------------------

Nice patch, r+.
Attachment #8469006 - Flags: review?(mratcliffe) → review+
(Reporter)

Comment 12

3 years ago
Comment on attachment 8469006 [details] [diff] [review]
Telemetry for WebIDE (v2)

This covers enough actions. Can't wait to see the results.
Thanks a lot for taking over this patch Ryan.
Attachment #8469006 - Flags: review?(paul) → review+
(Assignee)

Updated

3 years ago
Keywords: checkin-needed
https://hg.mozilla.org/integration/fx-team/rev/5c1ba06b972b
Flags: in-testsuite+
Keywords: checkin-needed
Whiteboard: [fixed-in-fx-team]
https://hg.mozilla.org/mozilla-central/rev/5c1ba06b972b
Status: ASSIGNED → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
Whiteboard: [fixed-in-fx-team]
Target Milestone: --- → Firefox 34
This was causing timeouts that frequently spilled into other tests as shown in the log below. Backed out.
https://hg.mozilla.org/mozilla-central/rev/2a621e79e6b9

https://tbpl.mozilla.org/php/getParsedLog.php?id=45563460&tree=Mozilla-Central
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Target Milestone: Firefox 34 → ---
Created attachment 8471000 [details] [diff] [review]
Telemetry for WebIDE (v3, paul: r+, mratcliffe: r+)

Tried to make the new test more failure-resistant.  Also updated some other tests to check the |DebuggerServer| state.

Try: https://tbpl.mozilla.org/?tree=Try&rev=b59b76ddff84
Attachment #8469006 - Attachment is obsolete: true
Attachment #8471000 - Flags: review+
(Reporter)

Comment 17

3 years ago
green.
Keywords: checkin-needed
https://hg.mozilla.org/integration/fx-team/rev/436e5d00f904
Keywords: checkin-needed
Whiteboard: [fixed-in-fx-team]
Backed out for new mochitest-other failures.
https://hg.mozilla.org/integration/fx-team/rev/ed15a6020e93

https://tbpl.mozilla.org/php/getParsedLog.php?id=45771560&tree=Fx-Team
Whiteboard: [fixed-in-fx-team]
It fails on Linux 32 PGO only... :( I'll try to investigate this more tomorrow.
I am still looking into this via more try debugging.  Adding more logging made the error go away... :(
Created attachment 8476186 [details] [diff] [review]
Telemetry for WebIDE (v4)

I am still blocked on intermittent test issues here.  When I add more logs to debug the issue, the tests start passing. :(

My most recent push[1] at least caused consistent failures on Windows, but I cannot repro this locally.

Paul, can you spot any signs of racy behavior here?  I am not seeing it myself.  From past debugging, when it does fail, it seems to get as far as adding the test app with Cmds.importPackagedApp before hanging and hitting a timeout.

[1]: https://tbpl.mozilla.org/?tree=Try&rev=510e08287517
Attachment #8471000 - Attachment is obsolete: true
Attachment #8476186 - Flags: feedback?(paul)
https://tbpl.mozilla.org/php/getParsedLog.php?id=46276352&tree=Try
09:46:38     INFO -  187 INFO TEST-OK | chrome://mochitests/content/chrome/browser/devtools/webide/test/test_runtime.html | took 863ms
09:46:39     INFO -  188 INFO TEST-START | chrome://mochitests/content/chrome/browser/devtools/webide/test/test_telemetry.html
09:46:39     INFO -  189 INFO Closing WebIDE
09:46:39     INFO -  190 INFO WebIDE closed
09:46:39     INFO -  191 INFO opening WebIDE
09:46:40     INFO -  192 INFO WebIDE open

I can't get why we are seeing "Closing WebIDE" first.
Given my reading of test_telemetry.html, we are calling cycleWebIDE first which immediatly calls openWebIDE.
Any idea why/how we end up trying to close WebIDE first? Could it be just some randomization of logs?
Also, if you have a rebased patch, that may help ;)
Created attachment 8491555 [details] [diff] [review]
Telemetry for WebIDE (v5)

Here's a rebased version.

Note that in past, failures mainly appeared on Linux PGO, which is not run on Try by default.  To make Try do this, you have to add "mk_add_options MOZ_PGO=1" to the "build/mozconfig.common.override" file.

It seems to be some kind of interaction with a previous WebIDE test.  Sprinkling many requestCompleteLog and dump / print calls in the past seemed to suggest it would hang inside the "importPackagedApp" method, but once I added further logging calls, the problem vanished.
Attachment #8476186 - Attachment is obsolete: true
Attachment #8476186 - Flags: feedback?(paul)
Attachment #8491555 - Flags: feedback?(poirot.alex)
I tried to push just test_runtime.html to try, but I'm not sure I rebased correctly (I had conflict with my recent patches... [autoselect and app front refactoring]):
  https://tbpl.mozilla.org/?tree=Try&rev=563d83bd3430
That ended up failing with the same error with the whole patch:
  https://tbpl.mozilla.org/?tree=Try&rev=53c46ebf6e7d
Here's a branch with my patch, as requested: https://github.com/jryans/gecko-dev/tree/wi-telemetry-reg
I got a green try with all the patch rebased:
  https://tbpl.mozilla.org/?tree=Try&rev=80c3ca8a9af5
Is it only intermittent?
Could it be that we fixed something?
Comment on attachment 8491555 [details] [diff] [review]
Telemetry for WebIDE (v5)

Review of attachment 8491555 [details] [diff] [review]:
-----------------------------------------------------------------

I launched a bunch of try runs at various times, especially the last one during pacific time wake up.
https://tbpl.mozilla.org/?tree=Try&rev=80c3ca8a9af5

Looks like it has been fixed by some recent change.
That doesn't surprise me that much as we fixed various tiny issues recently.

The last patch doesn't apply with very last changes I made to webide, here is a rebase of it if that can help:
  https://github.com/ochameau/mozilla-central/commit/c08302783b8f36103dc548fe876249137e704251
  https://github.com/ochameau/mozilla-central/commit/7ef9524c42732e0f30768116c2c439fffd0a7d97

Could you try to rebase it and push it yourself to confirm it is really green?
Attachment #8491555 - Flags: feedback?(poirot.alex) → feedback+
Created attachment 8498501 [details] [diff] [review]
Telemetry for WebIDE (v6)

Thanks Alex for your help!

Both regular and PGO try are green (Mac PGO failure is expected):

Non-PGO: https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=d1765a288884
PGO:     https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=9e5f09c50656
Attachment #8491555 - Attachment is obsolete: true
Attachment #8498501 - Flags: review+
(Assignee)

Updated

3 years ago
Keywords: checkin-needed
https://hg.mozilla.org/integration/fx-team/rev/f960230483c6
Keywords: checkin-needed
Whiteboard: [fixed-in-fx-team]
https://hg.mozilla.org/mozilla-central/rev/f960230483c6
Status: REOPENED → RESOLVED
Last Resolved: 3 years ago3 years ago
Resolution: --- → FIXED
Whiteboard: [fixed-in-fx-team]
Target Milestone: --- → Firefox 35
You need to log in before you can comment on or make changes to this bug.