669275 - Need more info about synch pings

Status: RESOLVED INCOMPLETE

(Firefox :: Sync, defect)

Description

[Nelson]

The way we currently monitor Synch usage is by capturing pings with the structure

GET /%x.x.x%/%username%/info/collections?v=%version%
or
GET /%x.x.x%/%username%/info/collections?client=FxHome&v=%version%

and rely on this ping structure to get both the username and weave version.

However, due to an unrelated matter, I had to browse manually through 2h of weave logs and noticed that there are way too many requests with a different structure, where the request field has just 

GET /%x.x.x%/%username%/info/collections 

and all the necessary info about the weave client comes in the UA string.


So, my question is: what's the structure of the various pings from weave clients, what do they mean and what info can we extract from them. Should we refactor the way we count weave users to account for the changes seen in Firefox, Fennec and Firefox Home?

Comment 1

[:Atoll]

The new user agent structure arrived when bug 568156 was landed on mozilla-central on 2011-04-10.  I don't know if/when ?v= pings are schedule to be removed.

Client team is best equipped to provide the breakdown and should be able to assist you further.

Comment 2

[Nelson]

Thanks.

So for now we're not missing out on any data, but if/when ?v= is removed we need to refactor our current data extraction.

Comment 3

[:Atoll]

At some point in the future, the change to remove ?v= will land on m-c, likely without much (if any) advance warning.

I advise refactoring the process ahead of that change, something like as follows:

1. Given the existing ?v=X pings, find out from Client what the earliest value of X is that supports the new User-Agent mechanism.
2. Start processing all user-agent pings.
3. Reprocess all logs since 2011-04-10, discarding all ?v=X pings where the above condition is true.

Pending confirmation from the client team, this can presumably be done any time without waiting for the ?v= removal.

Comment 4

[Philipp von Weitershausen [:philikon]]

(In reply to comment #3)
> At some point in the future, the change to remove ?v= will land on m-c,
> likely without much (if any) advance warning.

What makes you think that a) we have plans to remove it and b) if we did it would happen without much or any warning? We don't have a bug on file for removing it, and the related bug 568156 was landed with plenty of notice, with heads ups to both Ops and Metrics.

> I advise refactoring the process ahead of that change, something like as
> follows:
> 
> 1. Given the existing ?v=X pings, find out from Client what the earliest
> value of X is that supports the new User-Agent mechanism.

As you can tell from the Target Milestone, bug 568156 was landed in Firefox 5.0 which shipped with Sync 1.8.0 (so X=1.8.0).

> 2. Start processing all user-agent pings.
> 3. Reprocess all logs since 2011-04-10, discarding all ?v=X pings where the
> above condition is true.

You could do this, or you could just keep processing ?v=X pings for now. As said, we have no plans to change it right now. If you want us to, please file a bug against us.


P.S.: I really don't understand why this is a bug and not an email to services-all.