1003174 - Server needs to report an active user

Status: VERIFIED FIXED

(Hello (Loop) :: Server, defect, P1)

Comment 1

[Romain Testard [:RT]]

Stable ID will allow identifying unique users as part of FHR, we may be able to leverage this.

Comment 2

[Romain Testard [:RT]]

A user is defined as someone who has a successful call.

Comment 3

[:shell escalante]

Believe we can gather from server rather than client.

Comment 4

[Alexis Metaireau (:alexis)]

Once the server will now that a call was accepted / rejected, we will have a way to build this on the server, yes.

I'm marking this bug as a duplicate of the "server reporting" one.

Comment 5

[Romain Testard [:RT]]

How will you identify unique and recurring users on the server?

Comment 6

[Alexis Metaireau (:alexis)]

I believe this is the work of the aggregator to tell which users are recurring and which ones are unique. The server will just say that user-<uuid> was active, and then the aggregator will aggregate the information the right way.

Comment 7

[Romain Testard [:RT]]

After discussion with Alexis, the server will store the uuid and the number of links generated.

Comment 8

[Alexis Metaireau (:alexis)]

It will actually send these to statsd and not store them.

Comment 9

[Romain Testard [:RT]]

This has to be reported per user agent - i.e windows, Linux, Mac, FFOS, ...

Comment 10

[James Bonacci [:jbonacci]]

Adding QA and OPs

Comment 11

[Benson Wong [:mostlygeek]]

We can do with this statsd but it would be better to do this with heka. The latest version supports bloom filters and hyperloglog for counting unique items. We use HyperLogLog in production already. It consumes very little memory and can uniquely count billions of items to a very low probability of error.

Heka does this by looking for specific events in the log stream and increments a counter if deemed unique.

Comment 12

[Alexis Metaireau (:alexis)]

Benson, I'm not sure to understand. 

We're currently using heka as a transport for statsd, so wouldn't it work out the box for us?

Comment 13

[Tarek Ziadé (:tarek)]

Can you point us to an example of such a log consumed by heka/HyperLogLog ?

Comment 14

[Benson Wong [:mostlygeek]]

:tarek https://heka.fxa.us-west-2.prod.mozaws.net/#plugins/filters/FxaActiveDailyUsers

Comment 15

[Katie Parlante]

More context/info: 

The fxa-auth-server generates two log streams (to stdout/stderr): the nginx logs and the "application" logs. In particular, it logs a summary line for every request. Example:

{"name":"fxa-auth-server","hostname":"ip-10-220-156-235","pid":29233,"level":30,"op":"request.summary","code":200,"errno":0,"rid":"1403895904787-29233-33476","path":"/auth/v1/account/destroy","lang":"en-US,en;q=0.5","agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:30.0) Gecko/20100101 Firefox/30.0","remoteAddressChain":["75.37.29.214","10.245.46.178","127.0.0.1"],"t":579,"email":"[e-mail]","msg":"","time":"2014-06-27T19:05:05.366Z","v":0}

Heka has a logstreamer input, it takes the log data from all of the nodes running the servers and sends them to heka nodes, which feed them to a heka aggregator. The heka nodes & aggregator do a few things:
- filters out pii
- parses User Agent string
- uses hyperloglog algorithm for counting unique daily uids on one particular endpoint (/certificate/sign)
- displays realtime graphs for some endpoints/measures (like the one benson linked to)
- triggers email alerts (mostly anomaly detection, also a couple of fraud/abuse related alerts)
- outputs pii-stripped data to elastic search, which can then be viewed by/explored with Kibana dashboards
- outputs log data to file for long term storage (eventually to canonical data store)
- outputs a json file to be used by "all mozilla" dashboard

Heka and Kibana dashboards are only available to qa/ops/dev working on servers, for privacy/data security reasons. (Stackdriver pulls data from aws/cloudwatch, so similar to nginx logs + cpu/memory utilization etc.)

Here's the dashboard that's visible to everyone at mozilla: https://metrics.fxa.us-west-2.prod.mozaws.net/accounts-dashboard/

The plan is to do a similar dashboard for loop, to meet this use case. I'm assuming the plan is also to have heka and kibana dashboards, alerts, stackdriver, etc. similar to what we're doing for fxa & sync.

More info on the fxa-auth summary log line: https://github.com/mozilla/fxa-auth-server/pull/565

Comment 16

[Tarek Ziadé (:tarek)]

Thanks for all the info. 

> The plan is to do a similar dashboard for loop, to meet this use case. I'm assuming the plan is also to 
> have heka and kibana dashboards, alerts, stackdriver, etc. similar to what we're doing for fxa & sync.

Do you have an idea about the time it would take to set everything up for this reporting system ?

Comment 17

[Tarek Ziadé (:tarek)]

(once we provide all the logging info)

Comment 18

[Romain Testard [:RT]]

Please note that after discussions with TEF, we need this statistic both for Desktop Loop users and FFOS Loop users in order to be able to differentiate service take-up on both platforms.
Please confirm if you will have the client identifier necessary to provide this.

Comment 19

[Rémy Hubscher (:natim)]

Hello Katie,

I am looking at the log function we are doing to match the Heka log stream protocol.

You said we need to return something like:

{
    "name": "fxa-auth-server",
    "hostname": "ip-10-220-156-235",
    "pid": 29233,
    "level": 30,
    "op": "request.summary",
    "code": 200,
    "errno": 0,
    "rid": "1403895904787-29233-33476",
    "path": "/auth/v1/account/destroy",
    "lang": "en-US,en;q=0.5",
    "agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:30.0) Gecko/20100101 Firefox/30.0",
    "remoteAddressChain": [
        "75.37.29.214",
        "10.245.46.178",
        "127.0.0.1"
    ],
    "t": 579,
    "email": "[e-mail]",
    "msg": "",
    "time": "2014-06-27T19:05:05.366Z",
    "v": 0
}

Do you know what is the v parameter?

What is the email for? Can we provide the user unique Id instead?
Also do you know what is the level parameter?

Can we add other fields to it or are we tight to this fields?

Thank you very much.

Comment 20

[Romain Testard [:RT]]

I added a requirement for "monthly recurring" statistics  in the user story given that most communication services share data related to their monthly recurring users - so we can compare.

Comment 21

[Katie Parlante]

Hi Rémy,

> Do you know what is the v parameter?

This is the version of the log format, in case it changes later. You can just set to 1.

> What is the email for? Can we provide the user unique Id instead?

"email" was specific to the auth server. You should include whatever fields you want to log for each loop event that you are logging.

Yes, provide the unique user id. Using the identifier "uid" is probably a good idea, I think "uuid" is used to uniquely identify the heka message.

> Also do you know what is the level parameter?

"level" is used to rank importance of different log entries. You can ignore it if you don't need/want to rank types of log entries.

> Can we add other fields to it or are we tight to this fields?

Yes, you should add other fields. 

> Thank you very much.
yw

Comment 22

[Tarek Ziadé (:tarek)]

(In reply to Romain Testard [:RT] from comment #18)
> Please note that after discussions with TEF, we need this statistic both for
> Desktop Loop users and FFOS Loop users in order to be able to differentiate
> service take-up on both platforms.
> Please confirm if you will have the client identifier necessary to provide
> this.

Fernando, how do we make a difference between desktop users and mobile users ?

Comment 23

[Tarek Ziadé (:tarek)]

^ is the user-agent good enough ?

Comment 24

[Tarek Ziadé (:tarek)]

(In reply to Rémy Hubscher (:natim) from comment #19)
> I am looking at the log function we are doing to match the Heka log stream
> protocol.

Thanks! 

Please synchronize with me if you are going to work on this so we don't duplicate efforts.

Comment 25

[Katie Parlante]

Rémy & Tarek,

I should list more clearly which fields are required/important for heka processing:

- "time" : this should be a UTC timestamp 
- "op" : type of log entry, you can have multiple types if you want/need. We might filter or process types differently
- "hostname" : os.hostname() -- allows qa/ops to notice problems on specific aws instances
- "v" : should be 1, allows us to process different log versions differently in the future
- "lang" : allows segmentation of the data by locale
- "agent" : user agent, allows segmentation of the data by desktop/fxos, segmentation by browser

Other conventions that might be useful for following:

- "uid" : as you noted, we'll need uid for the active daily, recurring, etc. counts
- "errno" : other servers have found it useful to have a number for each error type. 0 for success.
- "path" : request url, assuming the event being logged is a request that is being handled

And of course any other fields that might be useful specifically for loop.

Comment 26

[Katie Parlante]

(In reply to Tarek Ziadé (:tarek) from comment #22)
> (In reply to Romain Testard [:RT] from comment #18)
> > Please note that after discussions with TEF, we need this statistic both for
> > Desktop Loop users and FFOS Loop users in order to be able to differentiate
> > service take-up on both platforms.
> > Please confirm if you will have the client identifier necessary to provide
> > this.
> 
> Fernando, how do we make a difference between desktop users and mobile users
> ?

We're using user agent for device segmentations with other servers (to distinguish between Desktop/FxOS/Android). The user agent from FxOS should look like this: https://developer.mozilla.org/en-US/docs/Web/HTTP/Gecko_user_agent_string_reference#Firefox_OS

Comment 27

[Tarek Ziadé (:tarek)]

(In reply to Katie Parlante from comment #25)
> I should list more clearly which fields are required/important for heka processing

Thanks. The final list should not be that different from what you have in fxa. I guess agent/uid/path are the 3 main ones to be able to build aggregated views.

I'll push a first version this week so ops/then you can hook it up.

Comment 28

[Fernando Jiménez Moreno [:ferjm]]

(In reply to Tarek Ziadé (:tarek) from comment #23)
> ^ is the user-agent good enough ?

Yes, I believe the UA should be enough.

Comment 29

[Alexis Metaireau (:alexis)]

https://github.com/mozilla-services/loop-server/commit/61da2e98999f74eefe29b36d6d45c61eeb678631

Comment 30

[Romain Testard [:RT]]

I don't see the required stats on https://metrics.fxa.us-west-2.prod.mozaws.net/accounts-dashboard/
Is there another dashboard to access it?

Comment 31

[Rémy Hubscher (:natim)]

I don't think it should be on Firefox Account stats page.

Comment 32

[James Bonacci [:jbonacci]]

yea, that's not the right dashboard.

:mostlygeek can you help here?

Comment 33

[Wesley Dawson [:whd]]

We don't have a high level dashboard for this yet. I'm working on total users and JSON output (should be done early this week), and :trink is working on the recurring users (hopefully by end of week). We're targeting having a dashboard available before Beta which I believe is the beginning of September. At least one app code change needs to go out before that (see bug #1046236).

Comment 34

[James Bonacci [:jbonacci]]

OK, will hold off on verifying this bug until then.

Comment 35

[Katie Parlante]

FWIW, the bug for high level dashboard itself is here: https://bugzilla.mozilla.org/show_bug.cgi?id=1036059

Comment 36

[Romain Testard [:RT]]

(In reply to Wesley Dawson [:whd] from comment #33)
> We don't have a high level dashboard for this yet. I'm working on total
> users and JSON output (should be done early this week), and :trink is
> working on the recurring users (hopefully by end of week). We're targeting
> having a dashboard available before Beta which I believe is the beginning of
> September. At least one app code change needs to go out before that (see bug
> #1046236).

Thanks Wesley. We have a product review early next week, if you can it would be great if you could send me by e-mail any data you have on total users and recurring users so I can have an idea (rough current numbers or ideally some form of historical data).

Comment 37

[James Bonacci [:jbonacci]]

Where are we with this bug?
This does not feel "Resolved" to me...

Comment 38

[Rémy Hubscher (:natim)]

At least the code has been made on server side: https://github.com/mozilla-services/loop-server/commit/61da2e98999f74eefe29b36d6d45c61eeb678631

Comment 39

[Wesley Dawson [:whd]]

Indeed, the latest metrics code is running in production as of yesterday. :jbonacci I would say this bug can be closed as it is mostly tracking the server-side logging, which is in place.

The static prototype dashboard fueled by this logging is currently living at https://metrics.fxa.us-west-2.prod.mozaws.net/loop-server-dashboard/ but that has its own tracking bugs.

Comment 40

[James Bonacci [:jbonacci]]

Thank you.