916804 - [webide] telemetry-all-the-things

Status: RESOLVED FIXED

(DevTools Graveyard :: WebIDE, defect, P1)

Description

[Paul Rouget [:paul]]

It would be interesting to know how people use the app manager:
- how many failed connections?
- how many click on Help
- how many projects are managed
- how often does the user update his app
- ...

Comment 1

[Nick Fitzgerald [:fitzgen] [⏰PST; UTC-8]]

Filter on 86b7095e-2bd0-499e-a704-d00f2524aeef / PAUL STOP SETTING QA CONTACT TO THE DEVTOOLS COMPONENT'S WATCHERS EMAIL FOR BUGS YOU FILE :)

Comment 2

[Paul Rouget [:paul]]

I'm lost. I have no idea how to record these data.

First, I use toolOpened and toolClose. How do I check it actually works?

Then, I want to track:
- what runtimes have been discovered (type + number) (Simulators, USB Runtimes, Wifi Runtimes, localRuntime, remoteRuntime)
- number of successful runtime connections (with type)
- number of failed runtime connections (with type)
- how long is the runtime connected
- how often the toolbox is open per connection
- how often the user click "play" per connection

How do I do that? Who can help me?

Comment 3

[Jeff Griffiths (:canuckistani) (:⚡︎)]

Mike has done the majority of the telemetry work, and Jordan ( freshly needinfo'd ) has done some for Web Audio and Canvas tools.

Comment 4

[Mike Ratcliffe [:miker] [:mratcliffe] [:mikeratcliffe]]

For the timing stuff follow the comments in browser/devtools/shared/telemetry.js

For counting things:
1. Create DEVTOOLS_XXX_OPENED_BOOLEAN in toolkit/components/telemetry/Histograms.json (search for boolean to see how).
2. let Telemetry = require("devtools/shared/telemetry")
3. let telemetry = new Telemetry();
4. telemetry.log("DEVTOOLS_XXX_OPENED_BOOLEAN", true); // Do this every time you want to increment the number.

For logging numbers like how often the toolbox is open per connection... do the same as above but use "kind": "exponential" in the histogram.

See https://developer.mozilla.org/docs/Mozilla/Performance/Adding_a_new_Telemetry_probe to see which types to use.

To check that it is working look at how I tested it:
/Users/mratcliffe/Desktop/fx-team/browser/devtools/shared/test/browser_telemetry_sidebar.js // I monkeypatched the log method.

Comment 5

[Jordan Santell [:jsantell] [@jsantell]]

I think Mike addressed all the issues, but ping me if anything else is unclear!

Comment 6

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

Paul, I am making progress here, but I have some questions about what you want to track.

(In reply to Paul Rouget [:paul] (slow to respond. Ping me on IRC) from comment #2)
> - what runtimes have been discovered (type + number) (Simulators, USB
> Runtimes, Wifi Runtimes, localRuntime, remoteRuntime)

The available runtimes can change at any time when devices are unplugged / replugged.  WiFi devices only show up when you scan explicitly, etc.  So, when should data be recorded?  Each time the runtime list changes (this sounds like it would be very noisy data)?  Only once per opening of WebIDE?

More generally, what do we hope to learn from this particular data point?  I am not sure I see the value here...  It *does* seem useful to know what runtime types are used (USB, WiFi, etc.), but we'll have that from the successful / failed connection results you also suggested.

> - how often the toolbox is open per connection

So if I:

* Connect
* Open toolbox
* Close toolbox
* Open toolbox
* Disconnect

We'd record "2", because the toolbox was opened twice?  How is that useful?

> - how often the user click "play" per connection

Same question here as above: how is this data point useful?

Comment 7

[Paul Rouget [:paul]]

(In reply to J. Ryan Stinnett [:jryans] from comment #6)
> > - what runtimes have been discovered (type + number) (Simulators, USB
> > Runtimes, Wifi Runtimes, localRuntime, remoteRuntime)
> 
> The available runtimes can change at any time when devices are unplugged /
> replugged.  WiFi devices only show up when you scan explicitly, etc.  So,
> when should data be recorded?  Each time the runtime list changes (this
> sounds like it would be very noisy data)?  Only once per opening of WebIDE?
> 
> More generally, what do we hope to learn from this particular data point?  I
> am not sure I see the value here...

You're right. Let's ignore that.

> It *does* seem useful to know what
> runtime types are used (USB, WiFi, etc.), but we'll have that from the
> successful / failed connection results you also suggested.
> 
> > - how often the toolbox is open per connection
> 
> So if I:
> 
> * Connect
> * Open toolbox
> * Close toolbox
> * Open toolbox
> * Disconnect
> 
> We'd record "2", because the toolbox was opened twice?  How is that useful?
> 
> > - how often the user click "play" per connection
> 
> Same question here as above: how is this data point useful?

Both for toolbox and play, I'd like to know if people use WebIDE only to push app, or push app and iterate on the code, or push app, debug and iterate on the code.

Maybe people will use WebIDE only to push an app sometimes, maybe they will iterate on the code and not use the toolbox at all.

But all of that can happen in a follow up bug.

Comment 8

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

Attachment: Telemetry for WebIDE

Paul, I believe this captures the values you're looking for.  You can test it interactively at about:telemetry, and filter on "webide".  (You have to refresh this tab to get updated histograms after you perform actions.)

Mike, I refactored the telemetry module slightly to expose the timing functionality more generally.  Can you check this over, and also see if I've set up the probes correctly?

Comment 9

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

Try: https://tbpl.mozilla.org/?tree=Try&rev=9d6b90f627bf

Comment 10

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

Attachment: Telemetry for WebIDE (v2)

Fixed test failures.

Try: https://tbpl.mozilla.org/?tree=Try&rev=0ce2b99d173d

Comment 11

[Mike Ratcliffe [:miker] [:mratcliffe] [:mikeratcliffe]]

Comment on attachment 8469006 [details] [diff] [review]
Telemetry for WebIDE (v2)

Review of attachment 8469006 [details] [diff] [review]:
-----------------------------------------------------------------

Nice patch, r+.

Comment 12

[Paul Rouget [:paul]]

Comment on attachment 8469006 [details] [diff] [review]
Telemetry for WebIDE (v2)

This covers enough actions. Can't wait to see the results.
Thanks a lot for taking over this patch Ryan.

Comment 13

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/integration/fx-team/rev/5c1ba06b972b

Comment 14

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/5c1ba06b972b

Comment 15

[Ryan VanderMeulen [:RyanVM]]

This was causing timeouts that frequently spilled into other tests as shown in the log below. Backed out.
https://hg.mozilla.org/mozilla-central/rev/2a621e79e6b9

https://tbpl.mozilla.org/php/getParsedLog.php?id=45563460&tree=Mozilla-Central

Comment 16

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

Attachment: Telemetry for WebIDE (v3, paul: r+, mratcliffe: r+)

Tried to make the new test more failure-resistant.  Also updated some other tests to check the |DebuggerServer| state.

Try: https://tbpl.mozilla.org/?tree=Try&rev=b59b76ddff84

Comment 17

[Paul Rouget [:paul]]

green.

Comment 18

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/integration/fx-team/rev/436e5d00f904

Comment 19

[Ryan VanderMeulen [:RyanVM]]

Backed out for new mochitest-other failures.
https://hg.mozilla.org/integration/fx-team/rev/ed15a6020e93

https://tbpl.mozilla.org/php/getParsedLog.php?id=45771560&tree=Fx-Team

Comment 20

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

It fails on Linux 32 PGO only... :( I'll try to investigate this more tomorrow.

Comment 21

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

I am still looking into this via more try debugging.  Adding more logging made the error go away... :(

Comment 22

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

Attachment: Telemetry for WebIDE (v4)

I am still blocked on intermittent test issues here.  When I add more logs to debug the issue, the tests start passing. :(

My most recent push[1] at least caused consistent failures on Windows, but I cannot repro this locally.

Paul, can you spot any signs of racy behavior here?  I am not seeing it myself.  From past debugging, when it does fail, it seems to get as far as adding the test app with Cmds.importPackagedApp before hanging and hitting a timeout.

[1]: https://tbpl.mozilla.org/?tree=Try&rev=510e08287517

Comment 23

[Alexandre Poirot [:ochameau]]

https://tbpl.mozilla.org/php/getParsedLog.php?id=46276352&tree=Try
09:46:38     INFO -  187 INFO TEST-OK | chrome://mochitests/content/chrome/browser/devtools/webide/test/test_runtime.html | took 863ms
09:46:39     INFO -  188 INFO TEST-START | chrome://mochitests/content/chrome/browser/devtools/webide/test/test_telemetry.html
09:46:39     INFO -  189 INFO Closing WebIDE
09:46:39     INFO -  190 INFO WebIDE closed
09:46:39     INFO -  191 INFO opening WebIDE
09:46:40     INFO -  192 INFO WebIDE open

I can't get why we are seeing "Closing WebIDE" first.
Given my reading of test_telemetry.html, we are calling cycleWebIDE first which immediatly calls openWebIDE.
Any idea why/how we end up trying to close WebIDE first? Could it be just some randomization of logs?

Comment 24

[Alexandre Poirot [:ochameau]]

Also, if you have a rebased patch, that may help ;)

Comment 25

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

Attachment: Telemetry for WebIDE (v5)

Here's a rebased version.

Note that in past, failures mainly appeared on Linux PGO, which is not run on Try by default.  To make Try do this, you have to add "mk_add_options MOZ_PGO=1" to the "build/mozconfig.common.override" file.

It seems to be some kind of interaction with a previous WebIDE test.  Sprinkling many requestCompleteLog and dump / print calls in the past seemed to suggest it would hang inside the "importPackagedApp" method, but once I added further logging calls, the problem vanished.

Comment 26

[Alexandre Poirot [:ochameau]]

I tried to push just test_runtime.html to try, but I'm not sure I rebased correctly (I had conflict with my recent patches... [autoselect and app front refactoring]):
  https://tbpl.mozilla.org/?tree=Try&rev=563d83bd3430
That ended up failing with the same error with the whole patch:
  https://tbpl.mozilla.org/?tree=Try&rev=53c46ebf6e7d

Comment 27

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

Here's a branch with my patch, as requested: https://github.com/jryans/gecko-dev/tree/wi-telemetry-reg

Comment 28

[Alexandre Poirot [:ochameau]]

I got a green try with all the patch rebased:
  https://tbpl.mozilla.org/?tree=Try&rev=80c3ca8a9af5
Is it only intermittent?
Could it be that we fixed something?

Comment 29

[Alexandre Poirot [:ochameau]]

Comment on attachment 8491555 [details] [diff] [review]
Telemetry for WebIDE (v5)

Review of attachment 8491555 [details] [diff] [review]:
-----------------------------------------------------------------

I launched a bunch of try runs at various times, especially the last one during pacific time wake up.
https://tbpl.mozilla.org/?tree=Try&rev=80c3ca8a9af5

Looks like it has been fixed by some recent change.
That doesn't surprise me that much as we fixed various tiny issues recently.

The last patch doesn't apply with very last changes I made to webide, here is a rebase of it if that can help:
  https://github.com/ochameau/mozilla-central/commit/c08302783b8f36103dc548fe876249137e704251
  https://github.com/ochameau/mozilla-central/commit/7ef9524c42732e0f30768116c2c439fffd0a7d97

Could you try to rebase it and push it yourself to confirm it is really green?

Comment 30

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

Attachment: Telemetry for WebIDE (v6)

Thanks Alex for your help!

Both regular and PGO try are green (Mac PGO failure is expected):

Non-PGO: https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=d1765a288884
PGO:     https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=9e5f09c50656

Comment 31

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/integration/fx-team/rev/f960230483c6

Comment 32

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/f960230483c6