627570 - need some ways to calibrate input system feedback

Status: RESOLVED WONTFIX

(Input :: General, defect)

Description

[chris hofmann]

axel made the point today in the input review session that we need ways to calibrate input reports for any particular area or topic. 

for example back in july we had 200k firefox 4 beta testers now we have 1.2 million

if we look at data that says we got 10 pieces of input last summer, and now we get 100 pieces of input on the same topic that could indicate a couple of things.

one is that the problem or praise for a particular has increased dramatically

but if we have added a million more users it might just mean we are getting the same amount of feedback, but a larger pool of users contributing.

we have similar kinds of calibration problems on crash-stats.  

to help there we express the feedback (crash reports) as a pct. of total reports for a given time period/product release sample, or we express the number of reports relative to active daily users.

e.g.

http://input.mozilla.com/en-US/themes/

should/could look like

http://crash-stats.mozilla.com/topcrasher/byversion/Firefox/4.0b9

and we could create a summary report that tells the number of feedback reports that we are getting per active daily users to let us know if users are more, or less, active in submitting feedback against any particular release.

e.g.

http://crash-stats.mozilla.com/daily?form_selection=by_version&p=Firefox&v[]=4.0b8&throttle[]=10.00&v[]=4.0b9&throttle[]=100.00&v[]=4.0b10pre&throttle[]=100.00&v[]=&throttle[]=10.00&hang_type=any&os[]=Windows&os[]=Mac&os[]=Linux&date_start=2011-01-01&date_end=2011-01-19&submit=Generate

Comment 2

[Matt Grimes [:Matt_G]]

We have a number of new ways of slicing and dicing data landing in Input in Q4. This should help us to better understand the historical trends and as you say "calibrate" our output to ensure the data is accurate.

Comment 4

[Will Kahn-Greene [:willkg] ET needinfo? me]

I suspect this bug is moot now since we've moved much of the data analysis off of Input and onto the UA systems.

Matt: Do you think we can close this out as WONTFIX?

Comment 5

[chris hofmann]

where are the "UA systems", is it possible for me to get access?

Comment 6

[Will Kahn-Greene [:willkg] ET needinfo? me]

You'll need to talk to the User Advocacy folks. https://wiki.mozilla.org/Advocacy

Comment 7

[Matt Grimes [:Matt_G]]

Will is correct. We've built a series of tools that live on the UA server that allow us to do calibration as described above AND some other really interesting things. 

Chris: I'll send you some details on the UA server. We can discuss any additional requests via email.