Closed Bug 685992 Opened 13 years ago Closed 12 years ago

[Tracking Bug] Grow Mozilla dashboard

Categories

(Mozilla Metrics :: Frontend Reports, defect)

x86
macOS
defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED
Moved to JIRA

People

(Reporter: davidwboswell, Unassigned)

References

()

Details

(Whiteboard: [JIRA METRICS-79])

Attachments

(3 files)

I'd love to get a dashboard set up for the Get Involved program, an set of pages at www.mozilla.org/contribute that is bringing in new contributors to all different areas of the project.

Specifically, I'd like to track:

* how many people are expressing interest in getting involved through the 'Want to help?' form?

* what project areas they are interested in (localization, support, etc)?

* how many of those people end up actually contributing?

We currently have some very basic stats that are being pulled monthly from mailman at:

https://wiki.mozilla.org/Mozilla.org/Contribute/Form_stats

There are several issues with the existing mailman-based data, including the fact that it is incomplete and can conflate different project areas when one person is responding to more than one type of inquiry.

As soon as the phonebook is up, we should be able to answer the conversion question by checking to see if the people coming through this form also end up in the phonebook.  And there could be other interesting things to ask -- for instance, how long does it take someone to ramp up and go from being interested to actually contributing?

I'd be happy to talk through this more at whatever point works well for the metrics team.  I imagine you're pretty busy with lots of other projects :)
sounds perfect, find me next week!
Cool.  I'll keep an eye out for you :)
Notes from the meeting here: http://etherpad.mozilla.com:9000/contribute-metrics


Nuno / Joana - we'll need to meet about this to build a mockup proposal for it and get back to David.
Assignee: nobody → pedro.alves
Attached image Get Involved Dashboard
Hi,

attached you will find the mockup dashboard.
Looking forward to your feedback.

Joana
Very exciting to see the mockup.  I'll take a look and post feedback and questions soon.

Would you mind if I blogged about this?
Happy you liked it.

Regarding your question, I think you should ask that to someone from PR team.
Paula, I would just share this within the community so I don't think PR would have any concerns with that.  I'll share this around and get some feedback and will post back here soon.
Thus may be off topic, from a process perspective I think we have a problem if PR needs to approve blog posts about ideas that are being discussed/worked on in Bugzilla.
Couldn't agree more. From our perspective (metrics team) it's great to have our things blogged and discussed, so we'd never oppose... Paula was just saying that if anyone would about you blogging on this, wouldn't be us :)
I had just asked about blogging this since this specific bug seems to have defaulted to being confidential.  

I wasn't sure if there was a general issue around new metrics projects needing to be vetted for potential privacy issues before sharing publicly.  

If not, could we also make this bug public?
And done - go ahead! ( and tell me when you blog about it, so I can make some noise too ;)  )
Group: metrics-private
Pedro, thanks for opening this up and for making some noise :)

I just blogged about this:

http://davidwboswell.wordpress.com/2011/10/04/making-more-resources-available-to-bring-more-people-into-mozilla/

and posted the mockup on the Contribute Group on Yammer:

https://www.yammer.com/contributegroup/

Please feel free to follow along with any comments there and post this anywhere else that makes sense.  If you do, please add the link in this bug so we can track all the comments.
Here's my feedback on the current dashboard mockup:

* I think the most important information is how many total inquiries have come in each month and what project areas people are asking about.  For reference, there are currently 15 different project areas people can choose from in the form, but that has changed over time and may grow to around 20.

* Getting information about the number of inquiries answered and the number of replies received back is important, but I think it's secondary for a few reasons: 

** The rate of replies is not a great proxy for a success rate -- a potential volunteer could write back and then not end up contributing, or someone could get the information they needed to make a contribution and not write back.

** We recently set up auto-responses that reply to all messages coming in to some (but not all) areas of interest, so the number of inquiries answered isn't very interesting for those areas.

** It's not clear what a better success metric would be.  I had thought it might be the number of people who sent inquiries and then created a profile on mozillians.org but that has potential issues too.  Pedro and I talked through options at the All Hands and it seems like it would be worth thinking through this more.

* It's cool to see a leaderboard here -- love that idea.

* I like the idea of tagging inquiries as helpful or unhelpful (or promising or not promising) and tracking that, but we don't have that data now.  Would it make sense to have two phases to this dashboard process -- first phase would create a dashboard for the data we have now and the second phase would create a dashboard with all the information we'd like to have?  I ask since I'd be happy getting a very simple dashboard going soon and working toward a full dashboard later.

* Someone mentioned that this looked like a SUMO dashboard because of the references to 'Questions'.  I've been referring to the messages coming in from the Get Involved page as 'Inquiries' so changing the wording may avoid some confusion.

Thanks again for working on this with me.  It's going to be really great having this dashboard :)
Attached image Get Involved workflow
Hi,

attached you will find the update to the dashboard, navigation aspects for private and public area.
(In reply to joana granja from comment #15)
> Created attachment 566974 [details]
> Get Involved Dashboard_v02

This looks great.  Thanks for addressing my feedback.  The only feedback I have on this is that it would be great to still have the Breakdown by Username available from the original mockup.

Setting up dashboards is new to me, so what is the next step after getting the mockups in place?
(In reply to Pedro Alves from comment #14)
> Created attachment 566163 [details]
> Get Involved workflow

Just to make sure I understand this attachment, this workflow would be for a new system to handle the inquiries that would give us the data we need for a full version of the dashboard (for instance, being able to rate specific inquiries)?

If I'm reading it right, this looks great to me.  At first pass I just see small things to tweak, such as also adding a 'Support Question' to the Set Rating section.  I'll need to take a closer look though to make sure I'm understanding this fully.

What would the next steps be for this workflow?  And to confirm, we could move forward with a simple version of the dashboard in parallel to sorting out the more complete dashboard?
Ping me on irc and I'll chat with you a bit. The biggest issue with the dashboard is that we have very few data currently, and we need to fix that asap

As for "what's next", you don't  have to do anything. As soon as it's approved it will enter our pipeline. We do have a lot of stuff going on at the moment so I can't promise to start working immediately, but hopefully soon enough!
(In reply to Pedro Alves from comment #18)
> Ping me on irc and I'll chat with you a bit. The biggest issue with the
> dashboard is that we have very few data currently, and we need to fix that
> asap

OK, I'll look for you on IRC tomorrow or Thursday.

I know there's not lots of data now, but there is enough to get started if we approached this in phases.  For instance, the data in mailman would give us everything we need to do the Overview portion of the latest mockup.

I think it's great to rethink the whole process of fielding and responding to inquiries, but that is a big project.  If the Metrics team has a lot going on right now, I'm happy to start small to get something going and then we can build on it over time.
Don't worry - we won't hold one for the other.

But I honestly don't think it'll be that big... we'll check!
FYI, I haven't gotten feedback on my blog posts about the dashboard mockups so I just posted about this on the Mozillians mailing list at

http://groups.google.com/group/mozilla.mozillians/browse_thread/thread/d12a4b19299a4a36#

Feel free to follow along with any discussion there or post with any comments or ideas.
Ping.  Any update on the status of this dashboard?
Hi David,

We plan to start working on this dashboard next week. 
We are in the SFO offices today and Friday until 11am. Do you want to meet up if you want to discuss something face to face?
Paula, thanks for the update.  If you have any questions for me, I'm happy to meet up while you're here.  I'll be in today and maybe tomorrow.  If you don't have any questions, let's plan to meet when there's something to review.
Assignee: pedro.alves → Pedro.Martins
Whiteboard: [METRICS-79]
Alright, I'm on it. First step will be to get a draft of the "Get Involved" global dashboard. If we can't have the data warehouse ready, some dummy data might be used for layout/behaviour checking.

Tasks have been created in JIRA (METRICS-79, METRICS-138, METRICS-139)
There's now a preliminary version of the public dashboard here: https://metrics.mozilla.com/pentaho/content/pentaho-cdf-dd/Render?solution=metrics2&path=%2FgetInvolved&file=getInvolved.wcdf

Regarding the access to the data, can you please give me access to the server where the mails are stored? The data extraction will have to run daily, in order to have updated data on the dashboards, unless the gzipped files available on https://mail.mozilla.org/private/contribute are updated every day.

Thanks!
I forgot to mention that the data in the preliminary version is DUMMY! 
My bad
(In reply to Pedro Martins from comment #26)
> There's now a preliminary version of the public dashboard here:

Cool :)

> Regarding the access to the data, can you please give me access to the
> server where the mails are stored? The data extraction will have to run
> daily, in order to have updated data on the dashboards, unless the gzipped
> files available on https://mail.mozilla.org/private/contribute are updated
> every day.

I just filed bug 715175 for this.
I have updated the dashboard. It has now data only for 2011 (and part of Jan 2012, I believe)

- A couple of values (ranking, thread info, etc) won't appear until the data warehouse work is complete. That is on-going but it will quickly stall until Bug 715175 is solved.

- The display of the month names is also missing, that's to be fixed quickly.

Any feedback is appreciated:

Link: https://metrics.mozilla.com/pentaho/content/pentaho-cdf-dd/Render?solution=metrics2&path=%2FgetInvolved&file=getInvolved.wcdf


(In reply to David Boswell from comment #28)
> (In reply to Pedro Martins from comment #26)
> > There's now a preliminary version of the public dashboard here:
> 
> Cool :)
> 
> > Regarding the access to the data, can you please give me access to the
> > server where the mails are stored? The data extraction will have to run
> > daily, in order to have updated data on the dashboards, unless the gzipped
> > files available on https://mail.mozilla.org/private/contribute are updated
> > every day.
> 
> I just filed bug 715175 for this.
> I have updated the dashboard. It has now data only for 2011 (and part of Jan
> 2012, I believe)

Great :)
 
> - A couple of values (ranking, thread info, etc) won't appear until the data
> warehouse work is complete. That is on-going but it will quickly stall until
> Bug 715175 is solved.

OK.  Looking forward to trying out the more complete dashboard experience :)
 
> - The display of the month names is also missing, that's to be fixed quickly.
> 
> Any feedback is appreciated:

For feedback, we can present this at the Contribute Group meeting this Thursday at 10 pacific.  Pedro, if you can make it, it would be great for you to present it and lead a discussion.  If that time doesn't work, I can gather feedback.  Meeting details and agenda at:

https://wiki.mozilla.org/Mozilla.org/Contribute/Group/1_19_12

Here is some of my feedback from looking at things so far:

* How far back will we be able to pull data from mailman?  The contribute list has been active since September 2009 so it could be interesting to see some long term trends.

* For the list of projects areas, I'm assuming the 'N/A' category is for inquiries that are coming in through the 'Other' option in the form?  There is an 'other' area listed but it isn't showing any data.  Would it make sense to remove that 'other' and rename 'N/A' to 'Other'?

* Maybe I'm reading things wrong, but I'm not seeing the right numbers for total inquiries for a given area.  For instance, if I click into Webdev's numbers it says there are 131 inquiries, but the graph shows four months of data with more inquiries than that (65 in Oct, 192 in Nov, 186 in Dec, 59 in Jan).  

Very exciting to see this coming along.  I'll keep looking over this and will post any other comments or questions.
(In reply to David Boswell from comment #30)
> > I have updated the dashboard. It has now data only for 2011 (and part of Jan
> > 2012, I believe)
> 
> Great :)
>  
> > - A couple of values (ranking, thread info, etc) won't appear until the data
> > warehouse work is complete. That is on-going but it will quickly stall until
> > Bug 715175 is solved.
> 
> OK.  Looking forward to trying out the more complete dashboard experience :)
>  
> > - The display of the month names is also missing, that's to be fixed quickly.
> > 
> > Any feedback is appreciated:
> 
> For feedback, we can present this at the Contribute Group meeting this
> Thursday at 10 pacific.  Pedro, if you can make it, it would be great for
> you to present it and lead a discussion.  If that time doesn't work, I can
> gather feedback.  Meeting details and agenda at:
> 
> https://wiki.mozilla.org/Mozilla.org/Contribute/Group/1_19_12

- It will be a pleasure to give you guys an update on this dashboard (keeping in mind that this is WIP) :)

 
> Here is some of my feedback from looking at things so far:
> 
> * How far back will we be able to pull data from mailman?  The contribute
> list has been active since September 2009 so it could be interesting to see
> some long term trends.

As far as you want. I had to manually download the 12 files to have 2011 in there, but once we get access, we can go as far as the files allows to go.
 
> * For the list of projects areas, I'm assuming the 'N/A' category is for
> inquiries that are coming in through the 'Other' option in the form?  There
> is an 'other' area listed but it isn't showing any data.  Would it make
> sense to remove that 'other' and rename 'N/A' to 'Other'?

N/A actually correspond to empty entries, while "Other" actually present the user with the chance of specifying some other project area. I'll check again, but "N/A" seems to be the plain "Firefox" project area.

 
> * Maybe I'm reading things wrong, but I'm not seeing the right numbers for
> total inquiries for a given area.  For instance, if I click into Webdev's
> numbers it says there are 131 inquiries, but the graph shows four months of
> data with more inquiries than that (65 in Oct, 192 in Nov, 186 in Dec, 59 in
> Jan).  

The graphs are always a 12 month look (where the last date is the reference date you chose). The top number on the "details row" are actually referring to the date range that you have selected above. The names on the date aren't 100% correct. If you select the 11th Jan, the range obtained with "previous month" will actually be from the 12th Dec to 11th Jan (30 days)
 
> Very exciting to see this coming along.  I'll keep looking over this and
> will post any other comments or questions.

Please do!
(In reply to Pedro Martins (PMartins) from comment #31)

> - It will be a pleasure to give you guys an update on this dashboard
> (keeping in mind that this is WIP) :)

Thanks for presenting at the meeting.

> As far as you want. I had to manually download the 12 files to have 2011 in
> there, but once we get access, we can go as far as the files allows to go.

Great.  If it's not a big trouble, it would be interesting to have all of the historical data.

> N/A actually correspond to empty entries, while "Other" actually present the
> user with the chance of specifying some other project area. I'll check
> again, but "N/A" seems to be the plain "Firefox" project area.

We may be using the term 'Other' in different ways.

In the project area drop down box there is an option for 'Other' and since this is a required field the inquiry is either about a real project area or for Other.  I think that's what's being tracked here with the N/A field.  

We do get a lot of inquiries that aren't about a specific area, so having 0 inquiries showing up for the current 'Other' field doesn't seem right, but having 304 inquiries showing up associated with N/A does seem to fit.

For reference, emails that are sent with 'Other' selected in that field have a subject line of just 'Inquiry about Mozilla' without Other in the subject line.

> The graphs are always a 12 month look (where the last date is the reference
> date you chose). The top number on the "details row" are actually referring
> to the date range that you have selected above. The names on the date aren't
> 100% correct. If you select the 11th Jan, the range obtained with "previous
> month" will actually be from the 12th Dec to 11th Jan (30 days)

Thanks for the explanation.  I guess I'm not seeing how to change the date range then.  How do I change the date range to see the total for the last 12 months?
I just heard one suggestion about how we're tracking replies in the dashboard.  Right now there is a mix of auto-responses and manual responses to replies.  For instance, when someone selects the Marketing area of interest the form sends a generic auto-response and then Chelsea manually follows up with the promising leads.

If we tracked auto-responses and manual responses together it seems like we'd lose some useful information.  For instance, saying that there was a 100% response rate isn't relevant information if Chelsea only thought 25% of the inquiries were qualified enough to get a response.

Would it make sense to have another column that just showed if an area has an auto-response turned on or not?  For reference, the auto-response is optional for each project area and some people have one and some don't.  The auto-response also doesn't CC the list so it's not showing up in the list data, but we can pull what the auto-response settings are from the form itself.

That would mean the '% answered' and '% unanswered' responses would be specifically tracking manual responses which we could take as a proxy for now of the quality of the leads.

Thoughts?

(In reply to David Boswell from comment #33)
> Great.  If it's not a big trouble, it would be interesting to have all of
> the historical data.

No problem. We will be working on the Data warehouse in the next 2-3 days, so expect some things to be disrupted or even wrong at some point. I'll update this bug as soon as things are fully loaded and stable.

> We may be using the term 'Other' in different ways.
> 
> In the project area drop down box there is an option for 'Other' and since
> this is a required field the inquiry is either about a real project area or
> for Other.  I think that's what's being tracked here with the N/A field.  
> 
> We do get a lot of inquiries that aren't about a specific area, so having 0
> inquiries showing up for the current 'Other' field doesn't seem right, but
> having 304 inquiries showing up associated with N/A does seem to fit.
> 
> For reference, emails that are sent with 'Other' selected in that field have
> a subject line of just 'Inquiry about Mozilla' without Other in the subject
> line.

In this refactoring of the datawarehouse, we'll take into account the info that you provided for the "other" field and try to put things into the right buckets. :)

> Thanks for the explanation.  I guess I'm not seeing how to change the date
> range then.  How do I change the date range to see the total for the last 12
> months?

"Previous year" should give you the total for the last 365 days. I'll update that section in order to describe better what is happening and with some extra ranges that might be useful for you
Status: NEW → ASSIGNED
There's a plan of integrating the "FAQ" or frequent/auto replies into the data that is going to be analyzed.
Nevertheless, assuming that all the emails that we have right now are the result of manual replies, then it seems to me like this is a good indicator of the quality of the leads. I didn't take a close look at the content of the emails. I will check them now and see how easy it is to distinguish between auto and manual emails :)

We'll keep you posted!


(In reply to David Boswell from comment #34)
> I just heard one suggestion about how we're tracking replies in the
> dashboard.  Right now there is a mix of auto-responses and manual responses
> to replies.  For instance, when someone selects the Marketing area of
> interest the form sends a generic auto-response and then Chelsea manually
> follows up with the promising leads.
> 
> If we tracked auto-responses and manual responses together it seems like
> we'd lose some useful information.  For instance, saying that there was a
> 100% response rate isn't relevant information if Chelsea only thought 25% of
> the inquiries were qualified enough to get a response.
> 
> Would it make sense to have another column that just showed if an area has
> an auto-response turned on or not?  For reference, the auto-response is
> optional for each project area and some people have one and some don't.  The
> auto-response also doesn't CC the list so it's not showing up in the list
> data, but we can pull what the auto-response settings are from the form
> itself.
> 
> That would mean the '% answered' and '% unanswered' responses would be
> specifically tracking manual responses which we could take as a proxy for
> now of the quality of the leads.
> 
> Thoughts?
(In reply to Pedro Martins (:PMartins) from comment #36)
> Nevertheless, assuming that all the emails that we have right now are the
> result of manual replies, then it seems to me like this is a good indicator
> of the quality of the leads. I didn't take a close look at the content of
> the emails. I will check them now and see how easy it is to distinguish
> between auto and manual emails :)

Yes, the only mail in the mailman archive is from manual responses.  I also agree that a reply is a reasonable proxy for the quality of the lead (not perfect but reasonable).
Hi,
I'll update tomorrow the data tables for the dashboard. This means that the data could be wrong for a few hours. I'll post here when that process is finished and when the dashboard is ready :)
(In reply to Pedro Martins (:PMartins) from comment #38)
> Hi,
> I'll update tomorrow the data tables for the dashboard. This means that the
> data could be wrong for a few hours. I'll post here when that process is
> finished and when the dashboard is ready :)

What is going to happen is that the emails from 2009 until 2012 will be there. Also, we'll be putting in place an auto-update system that will check for more emails every 15 minutes.
Pedro, very cool.  Looking forward to seeing it.
Alright, the incremental ETL (auto update) is now running every 15 minutes on our server.
I reorganized the "null" entries into the "Other" field as well. There's still a couple of things (Rating, thread status) that will only make sense once we have the next dashboard (Get Involved - mailbox) in place.

The available options for "project area" are fixed, I took the ones from the website and we can easily add more, if needed. All other (invalid) Project Areas are now going into a specific "other_specified" field, which is not taken into account here.

David, feel free to talk with me about this and the next dashboard. There are a couple of things that for sure deserve some thinking before implementing them (access to email content, rating, etc)

Any feedback is welcome :)
Pedro, thanks for the update.  Things look great to me.  

The only feedback I have right now is what we talked about on IRC.  It would be nice to have a way to see all previous data instead of just being able to see just the previous year's data.  For total inquiries, it would also be helpful to sort out the original inquiries from any responses.  These are small tweaks though and not urgent.

I'm definitely interested in talking with you about next steps, but I would like to wrap up this phase by making this public so Stewards and others can make use of it.  I just opened bug 728016 to track making this public.  As soon as that's done, let's meet up to talk about the next phase.
I'd love to contribute to all of this, but I think the dashboards are behind firewalls I cannot access.
We have now added this dashboard to the public server for a security screening. David will give you the access details on a more private channel, I guess.
This is being tracked on bug 728016
(In reply to Pedro Martins (:PMartins) from comment #44)
> We have now added this dashboard to the public server for a security
> screening. David will give you the access details on a more private channel,
> I guess.  This is being tracked on bug 728016

Pedro, thanks for setting this up.  David Eaves is on bug 728016 so he'll be able to see that login info (David, let us know your thoughts when you have a chance to review it).

The process for getting this made public might end up taking more time than I had expected at first, so it may make sense for us to start talking through next step ideas while we're waiting.

For instance, can we do a test for tracking conversions by comparing the information from Support inquiries with activity in the SUMO forums?  Forum activity is how Michelle Luna, one of the Support Steward, is measuring when someone becomes a new contributor and she keeps good records of that.  Seems like we should be able to hook those together to get a conversion number.
In a similar vein, I'd love to find out how many people who email about Coding end up creating bugzilla accounts.
(In reply to Josh Matthews [:jdm] from comment #46)
> In a similar vein, I'd love to find out how many people who email about
> Coding end up creating bugzilla accounts.

Good point.  This is another great possibility for trying out conversion tracking -- the Bugzilla data maps to an actual contribution event (coders need to be active in Bugzilla to contribute) and the data is available.
Pedro suggested that we move off the discussion about conversion rates to a separate bug to make it easier to track, so I just opened bug 736109.  Let me know if you'd like to be copied on that bug.
This bug has evolved into a tracking bug for additional enhancements to the Get Involved dashboard.  If it's better to close this out since the dashboard is now live and open a new tracking bug, let me know.
I'm terribly late to the party here, but… shouldn't the dashboard be public? As far as I can see everything under metrics.mozilla.com is tied to the MoCo database which means even volunteers with LDAP accounts can't see it. Isn't it supposed to be publicly available?
You're in fact early to the party, it's still under develoment but pans are to make it public indeed
(In reply to Reuben Morais [:reuben] from comment #50)
> I'm terribly late to the party here, but… shouldn't the dashboard be public?
> As far as I can see everything under metrics.mozilla.com is tied to the MoCo
> database which means even volunteers with LDAP accounts can't see it. Isn't
> it supposed to be publicly available?

Yes, it should be and that's the goal with bug 728016.  The dashboard needs to go through a review first though to make sure there are no privacy or security concerns.
Depends on: 744826
+1 Reuben.

I'd go further. Shouldn't the *underlying data* be open as well? That way, if someone wants to craft a dashboard that is useful to them, they can go ahead and do that?
Summary: Get Involved program metrics dashboard → Grow Mozilla program metrics dashboard
Get Involved changed to Grow Mozilla, but only in the dashboard's names and selectors, not in the file name. Is this enough or should we change also the name of the files?
(In reply to Paulo Pires from comment #54)
> Get Involved changed to Grow Mozilla, but only in the dashboard's names and
> selectors, not in the file name. Is this enough or should we change also the
> name of the files?

Just to help keep track of things, I'll respond to this in bug 747967 about renaming the dashboard.
Summary: Grow Mozilla program metrics dashboard → [Tracking Bug] Grow Mozilla dashboard
migrated to JIRA https://metrics.mozilla.com/projects/browse/METRICS-79
Assignee: pmartins → nobody
Whiteboard: [METRICS-79] → migrated to JIRA
CLosing this bug out, as original ask is done. If there are new asks, please open a new task.
Status: ASSIGNED → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
New JIRA migration identifier - 55 BZ issues being considered in current planning for Q3
Target Milestone: Unreviewed → Targeted - JIRA
Whiteboard: migrated to JIRA → [JIRA METRICS-79]
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: