659019 - Define Goals and Measurements for Mozillians.org 1.0

Status: VERIFIED FIXED

(Participation Infrastructure :: Phonebook, defect)

Description

[Austin King [:ozten]]

Before this app launches, let's document how we will measure it's adoption.

Create a wiki page to document these numbers at:
http://wiki.mozilla.org/Websites/Mozillians/Goals

Please record:
* What are the goals for this launch?
* How do we plan to measure this launch?
** Which goals will be measured in Webtrends?
** Document requirements in Bug#659017.
** Which goals need to have a stats area in the app's admin panel?
*** Create tickets for stats area in admin panel as needed
* How will we determine whether this launch was a success?
* When do we need to schedule a post-mortem for this project?

Each of these goals should be measurable.  Provide a start date and an end date by which to measure each goal.  Each goal will need to have a documented way of how of that data will be tracked and measured.  

Starting numbers that will be maintained externally (Twitter followers, Facebook fans, etc.) must be recorded before the site launches.

Comment 1

[David Boswell]

* What are the goals for this launch?

To allow all active contributors to create a profile for themselves on the site.

* How do we plan to measure this launch?

By checking the number of profiles (both vouched and unvouched) on the site.

** Which goals will be measured in Webtrends?

None.  WebTrends will be useful in determining how people are using the site, but we won't directly track success by these web stats.

** Which goals need to have a stats area in the app's admin panel?

We will need a place to list total profiles (and how they break out by vouched or unvouched) but I'm not sure if the best location for this is in an admin panel.  

Unless there's a reason for not making it public, I'd prefer having these stats be public.

We can discuss the best location, but one thought is perhaps on the search page with a note that says 'Search for one of the ____ Mozillians on this site.'

Longer term we want to have a place to show interesting visualizations and this data would fit there, but in the short term limited scope approach of the 1.0 we might not need a dedicated page for this.

*** Create tickets for stats area in admin panel as needed

Based on the comment above, let me know if we want a separate bug for this or should track it as part of an existing page.

* How will we determine whether this launch was a success?

By determining how many people have signed up on the site and have gone through the vouching process.

* When do we need to schedule a post-mortem for this project?

Is there a recommendation for timing?  Happy to follow best practices here.

Comment 2

[Austin King [:ozten]]

(In reply to comment #1)
> ** Which goals will be measured in Webtrends?
>> None.  WebTrends will be useful in determining how people are using the site, 
>> but we won't directly track success by these web stats.

If we want to identify a couple key events, we can add special code so that the Webtrends reports highlight these events. If not, no worries. It's (minor) extra work for both webtrends and webdev. After 1.0 we can zone in on measuring suspected problem areas.

> We will need a place to list total profiles (and how they break out by vouched 
> or unvouched) but I'm not sure if the best location for this is in an admin 
> panel. 

Cool, we can make this a public url, which is not linked. We do this on apps for various reasons, like a /status page for app health.

/stats could be a short-term metrics page. Polling it day over day we could record these key performance indicators and track them in a spreadsheet or whatever. Not automagic, but very useful.

> * How will we determine whether this launch was a success?
> By determining how many people have signed up on the site and have gone 
> through the vouching process.

Let's use this bug to set a goal like 500 people within 2 weeks, etc. Without a goal and/or stretch goals, it's hard to measure and improve.

> Is there a recommendation for timing?  Happy to follow best practices here.

Good call, we can use this bug as a reminder, basically once we've shipped we should schedule one.

Comment 3

[David Boswell]

(In reply to comment #2)

> If we want to identify a couple key events, we can add special code so that
> the Webtrends reports highlight these events. If not, no worries. It's
> (minor) extra work for both webtrends and webdev. After 1.0 we can zone in
> on measuring suspected problem areas.

Agreed.  That sort of fine tuning seems like it can come after 1.0.

> Cool, we can make this a public url, which is not linked. We do this on apps
> for various reasons, like a /status page for app health.

Sounds good.
 
> Let's use this bug to set a goal like 500 people within 2 weeks, etc.
> Without a goal and/or stretch goals, it's hard to measure and improve.

CC'ing Mary for input on specifics of goals.  500 people within 2 weeks certainly seems in the right ballpark.
 
> Good call, we can use this bug as a reminder, basically once we've shipped
> we should schedule one.

Sounds good.

Comment 4

[Austin King [:ozten]]

(In reply to comment #1)
> By checking the number of profiles (both vouched and unvouched) on the site.

This turns out to be technically difficult with the privacy model of the backend.

We can do a couple of things:
1) Use webtrends to track usage for 1.0. Add event tracking for 
1.1) Cold Registration
1.2) Pre-Vouched Registration
1.3) User Vouched
2) Add a metrics agent Bug#676336
3) Setup a very simple cronjob that runs as root and avoids backend privacy controls and writes to a text file the # of people and vouches

I'd recommend we do #2 in 1.0. We nail down the requirements and figure out how much scope this adds. #1 is a good option, if Webtrends data is viable as a long term solution.

> We can discuss the best location, but one thought is perhaps on the search 
> page with a note that says 'Search for one of the ____ Mozillians on this site.'

This requirement is also technically difficult with our current privacy constraints.

> Longer term we want to have a place to show interesting visualizations and 
> this data would fit there, but in the short term limited scope approach of the 
> 1.0 we might not need a dedicated page for this.

I think this visualization belongs in the metrics dashboard.

Comment 5

[Aakash Desai [:aakashd]]

Ah, that's unfortunate to hear! Option #2 works best as that should help the Metrics folks have a jumpstart on the work they'd like to accomplish going forward with the Phonebook too. 

The other pieces can be done with a later release in mind and with Metrics help once they're in the loop.

Thanks for the heads up, Austin. I've got a meeting with Daniel and I'll close the loop with him there on some of the other options that aren't currently possible. Is there anything you'd like me to do here to help out the process?

Comment 6

[Austin King [:ozten]]

Dave Dash also suggested:

4) Use statsd from our code for the key events listed in 1.1, 1.2, 1.3, etc.

This would create a datasource outside of the phonebook that metrics could consume.

Comment 7

[Austin King [:ozten]]

@Daniel - Is statsd a suitable way for you to collect incremental metrics and meet aakashd's requirements?

Comment 8

[Daniel Einspanjer [:dre] [:deinspanjer]]

@Austin -- We haven't done any work with statsd so far.  It seems to me that UDP might not be a good fit for the low volume and high importants of measurements that you are going to collect.  Does that sound wrong to you?

We should be able to have the backend service quickly and easily submit payloads of metrics to our existing data collection service via REST.  Would that be a solution that you feel wouldn't be a good fit?

As far as the events listed in 1.1, 1.2, and 1.3, we do similar types of queries on the Sync service's LDAP.  Is that not feasible in this case?

Comment 9

[Austin King [:ozten]]

(In reply to Daniel Einspanjer :dre [:deinspanjer] from comment #8)

Right, the issue is without doing more work (metricsAgent), LDAP queries aren't an option, because the ACL on this project are very tight. No use can retrieve more than 50 results.

It's possible that we could use statsd to provide a REST interface.

I'd be interested to learn more about Sync's queries are the checekd into a code repo somewhere?

Comment 10

[Daniel Einspanjer [:dre] [:deinspanjer]]

The development of the metrics agent would actually be on us to developer and hand over to you guys for validation and installation in production.  The application would perform the necessary queries and package up the data to be delivered to our data warehouse.  This is what we did for Sync.  Take a look at this page:

https://intranet.mozilla.org/Metrics/Weave/ETL_Configuration

The actual code is a series of XML files that are metadata about the data processing to be performed. These files are used by Pentaho Data Integration (Kettle).  Basically, a JRE and the kettle app is installed on a machine and then you just cronjob a call to invoke it.

I'd be happy to walk through the code and discuss it.  Hopefully we can meet on Tuesday?

Comment 11

[Austin King [:ozten]]

(In reply to Daniel Einspanjer :dre [:deinspanjer] from comment #10)

Great I scheduled a meeting. My guess would be that LDAP is used to authenticate, but that no metrics about data in LDAP is gathered.

Comment 12

[Aakash Desai [:aakashd]]

There's some useful chatter on this bug, but I don't think any of it is pertinent anymore. Closing this out; feel free to re-open if there is a substantive reason to do so.

Comment 13

[Tobias (:Tobbi) Markus]

Verifying tracker bug as QA-