1124371 - Loop Reporting JSON that includes rooms

Status: RESOLVED WONTFIX

(Cloud Services :: Operations: Metrics/Monitoring, task)

Description

[Katie Parlante]

We need to revist the "callers" metrics, given that there's a new rooms api, so a new way to be a successful caller. As before, the loop server doesn't observe the success of the media, so we're only measuring success in connecting the two users.

Some definitions for our approach:
- Unique users ==> unique hawk ids (uid, calleeId)
- Successful call user ==> same as our "caller" metrics before. This counts both direct and call-url callers.
- Successful room user ==> users who have been in a conversation/room with another user.

- Successful session ==> successful calls (same as before) + successful room sessions. For rooms, this means the number of times a person has joined a room with another person already in it. (This will be an overcount; we have unique rooms as part of the rooms flow).

Comment 1

[Katie Parlante]

To start, lets do parallel "rooms" counts for all of the caller metrics (we can skip fortnightly):

https://metrics.services.mozilla.com/loop-server-dashboard/data/loop_new_callers.json
https://metrics.services.mozilla.com/loop-server-dashboard/data/loop_cumulative_callers.json
https://metrics.services.mozilla.com/loop-server-dashboard/data/loop_daily_unique_callers.json
https://metrics.services.mozilla.com/loop-server-dashboard/data/loop_weekly_recurring.json
https://metrics.services.mozilla.com/loop-server-dashboard/data/loop_monthly_recurring.json

In each case, we care about unique uid's for this query:
method:post AND path:rooms AND action:join AND errno:0 AND participants:2

Also, a "room sessions", which is the moral equivalent of "call setups" (only we don't have parallel failure data). The proposed approach is to just count:
method:post AND path:rooms AND action:join AND errno:0 AND participants:2

Comment 2

[Wesley Dawson [:whd]]

I'm going to append "rooms" to the names for the rooms counts e.g.

https://metrics.services.mozilla.com/loop-server-dashboard/data/loop_monthly_recurring_rooms.json

Do we want cumulative callers etc. to start from 0 or from the current value? I'm guessing 0.

I'm going to put the rooms sessions data at:

https://metrics.services.mozilla.com/loop-server-dashboard/data/loop_rooms_sessions.json

Comment 3

[Wesley Dawson [:whd]]

The following filters have been set up, and should start to see activity once 0.15.1 goes to prod (possibly tomorrow):

https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/LoopRoomsSessionsMetrics (json)
https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/LoopRoomsSessionsCount (cbuf of same)
https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/LoopTotalUsersWithRooms
https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/LoopActiveDailyCallersWithRooms

Recurring counts is an offline computation but the code is the same (only the message matcher differs). Once we have some data those numbers should be available at:

https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/LoopRoomsMetrics

Comment 4

[Katie Parlante]

Looks like 0.15.1 landed in production (I'm seeing "participants" in kibana), so should be able to add this to the dashboard tomorrow.

Comment 5

[Katie Parlante]

(In reply to Katie Parlante from comment #1)
> In each case, we care about unique uid's for this query:
> method:post AND path:rooms AND action:join AND errno:0 AND participants:2
> 
> Also, a "room sessions", which is the moral equivalent of "call setups"
> (only we don't have parallel failure data). The proposed approach is to just
> count:
> method:post AND path:rooms AND action:join AND errno:0 AND participants:2

Looking at the actual data in kibana, looks like "participants" is the number of people in the room when you try to join. So these queries should change to:
method:post AND path:rooms AND action:join AND errno:0 AND participants:1

Comment 6

[Rémy Hubscher (:natim)]

Yes true, this might be something we want to change, should we?

https://github.com/mozilla-services/loop-server/blob/master/loop/middlewares.js#L90-L92

Comment 7

[Katie Parlante]

(In reply to Rémy Hubscher (:natim) from comment #6)
> Yes true, this might be something we want to change, should we?
> 
> https://github.com/mozilla-services/loop-server/blob/master/loop/middlewares.
> js#L90-L92

No need to change.

Comment 8

[Katie Parlante]

Ok, I have more use cases working locally with the production data, but I'm not pushing live tonight -- the combination of so few data points & the load testing snafu renders it more confusing than useful.

Changes still needed:
- Add the "Fields[user_agent_browser] != NIL" filter to the rooms variants & reprocess
- Make 7 day & 30 day active (not recurring) available to the dashboard (for both calls and rooms)
- Make room session json available to the dashboard

Comment 9

[Wesley Dawson [:whd]]

"Fields[user_agent_browser] != NIL" has been added to the live filters, but still needs reprocessing.
Today I wrote a generic "actives" filter at https://github.com/mozilla-services/puppet-config/blob/loop_rooms_metrics/shared/modules/shared/files/hekad/lua_filters/active_counts.lua which I'll configure for 7 & 30 day actives. I'll also do the reprocessing and dashboard proxy rules tomorrow.

Comment 10

[Wesley Dawson [:whd]]

Rooms variants should be recomputed and running live. The 7 day and 30 day counts will not show up until 7 days and 30 days have passed, but the other data should look better. When the 7 and 30 day counts are available, they will be at:

https://metrics.services.mozilla.com/loop-server-dashboard/data/loop_room_weekly_unique_callers.json
https://metrics.services.mozilla.com/loop-server-dashboard/data/loop_room_monthly_unique_callers.json

I also tweaked the dashboard name for the rooms variants for consistency (s/rooms/room/); where they were:

/loop-server-dashboard/data/loop_rooms_$METRIC.json

They are now:

/loop-server-dashboard/data/loop_room_$METRIC.json

7 and 30 day counts for "calls" are also live and backfilled with data from 2014-12; that data is available at:

https://metrics.services.mozilla.com/loop-server-dashboard/data/loop_weekly_unique_callers.json
https://metrics.services.mozilla.com/loop-server-dashboard/data/loop_monthly_unique_callers.json

Session JSON available at:

https://metrics.services.mozilla.com/loop-server-dashboard/data/loop_room_sessions.json

(feel free to suggest better names for any of the above)

Per discussion in bug #1122506 I'm going to modify "Fields[user_agent_browser] != NIL" to be "Fields[agent] != $LOADS_USER_AGENT" for all cases where we use the former, which I will do tomorrow.