644264 - Feedback channels are different for 4.0 beta users and 4.0 release users - leading to misclassified data

Status: RESOLVED FIXED

(Input :: General, defect)

Description

[chris hofmann]

It looks like there is misclassification of the data now that a
new interface is has been launched on i.m.o

the majority of feedback now looks like its coming in as ideas now... at least
on some topics. maybe something to do with the changing defaults?

 stuff like https://input.mozilla.com/en-US/release/opinion/1809763 is
not an idea

Also we can't do *any* queries for to search for common terms in the feedback from 4.0 users.

The later problem is more critical, but if we can fix both of these problems with a back out of recent changes to input, then we should be consider it.

I'm open to spot fixes that can address the later problem as a temporary solution.

We need to get this fixed as quickly as possible.

Comment 1

[Dave Dash [:davedash, :dd] (assign all bugs to mbrandt)]

These are not quick fixes - nor were they recent changes to Input.  Release users go through a different mechanism to submit feedback than beta users.

What you've highlighted in your first point is potentially a hole in our reporting system.  Release users will still be having issues that might be similar to what beta users would report.  E.g. their frustrations may not just be broken web sites, it might be problems with the browser overall.

Comment 2

[Ryan Snyder [:ryansnyder] [:rsnyder] [:rysny]]

IMO, we should not be collecting different data for separate product types.  Users should be able to submit the exact same feedback regardless of whether they are using a release product or a beta product.  

There are a few issues:

* Users of a beta product will expect to be able to submit the same feedback once they adopt the release product.  

* There is a completely different flow in the UI on the website, and most people, including developers like myself, get confused as to why there are not seeing specific types of feedback.  They may not realize that you can switch to a different product type (Beta or Release).

* Not collecting statements and sentiments from our users for Release product is a mistake.  These are different users than our Beta users, and we should collect similar feedback from them to gain an understanding of how that set of users experiences our products.  We really need to be able to compare the statements and sentiments of beta users to release users.

Comment 3

[Dave Dash [:davedash, :dd] (assign all bugs to mbrandt)]

> * Not collecting statements and sentiments from our users for Release product
> is a mistake.  These are different users than our Beta users, and we should
> collect similar feedback from them to gain an understanding of how that set of
> users experiences our products.  We really need to be able to compare the
> statements and sentiments of beta users to release users.

Yes.  I feel like the "beta" feedback has now stopped so the more interesting/tweetable/personable/relatable feedback is no longer coming in.  We need to open that door again.

Also, with our rapid release schedule, the issues people find are hopefully not blocker-like issues, but they are issues that might warrant a .x release or at the very least help us understand what we should prioritize for Fx5/Fx7.

Comment 4

[Dave Dash [:davedash, :dd] (assign all bugs to mbrandt)]

Clarifying the bug title.
Removing regression as this was as intended, we just had an unanticipated consequence of "bad" incoming data.

Comment 5

[Fred Wenzel [:wenzel]]

(In reply to comment #3)
> Yes.  I feel like the "beta" feedback has now stopped so the more
> interesting/tweetable/personable/relatable feedback is no longer coming in.  We
> need to open that door again.

Agreed. I feel like we don't get a lot of useful feedback from release users so far, which might partly be because the feature is somewhat hidden but also because we don't solicit their "sentiments" as directly anymore as with the betas.

Comment 6

[chris hofmann]

(In reply to comment #4)
> Clarifying the bug title.
> Removing regression as this was as intended, we just had an unanticipated
> consequence of "bad" incoming data.

not sure that we have "bad" data coming in.   we do have mis-classified data (e.g.  serious feedback on product issues that are classified as "ideas".)

> I feel like we don't get a lot of useful feedback from release users so
> far, which might partly be because the feature is somewhat hidden but also
> because we don't solicit their "sentiments" as directly anymore as with the
> betas.

agree here also, but users are also going around the way the we positioned the feedback tool for final and are continuing to provide us with lots of good comparision data to IE and chrome, as well as many other areas.

Check some of the analysis reports in Bug 640676.

We need to get widespread access to the comments so we can get more eyeballs on the feedback figure out what users are trying to tell us.

here is an example:

we put off lots of analysis on the beta population about if users really needed a confirmation button to save tabs at the end of a session because we really didn't believe the beta testers were reflective of the widerspread user population.   now we have wider spread use from a wider variety of users and we need to see things like if a query for "not saving tabs on close" and various permuations of those words on the increase or decline as the composition of the user base changes.

please just get me a way to query for comments from 4.0 RC and final users.

Comment 7

[Aakash Desai [:aakashd]]

> please just get me a way to query for comments from 4.0 RC and final users.

I wholeheartedly agree and we're going to devote 5.x to make sure we focus on this once nightly support and an API are available.



...As for changing the feedback types back to happy/sad/ideas, we have to go to the context of the argument. RC users were skewed back from what they were used to with happy/sad/ideas, which they were supposed to see, and offered feedback that was a jumble between those within "ideas". Now, if you look at the feedback coming in over the past day, you'll start noticing the quality of feedback we're receiving. That's the quality we expected because it matches the type of user who is submitting it (i.e. mainstream, non-webby and folks who care more about getting a good browser to use rather than helping to build a great browser). 

The feedback types are the right approach, but dashboard implementation is the wrong one. We're going to focus on this for 5.x and we'll hash it out together. If you have questions about this, lets bring it up during the weekly meetings.

Comment 8

[chris hofmann]

> I wholeheartedly agree and we're going to devote 5.x to make sure we focus on
> this once nightly support and an API are available.

can you explain why these two features are taking priority over just getting a way to look at 4.0 comments?  I'm still missing the point there.

nightly users generally know how to file bugs and provide input that way.

a full blown set of APIs to extract the data and do lots of interesting things are not needed for this simple request of providing the same kind of queries that existed just a few weeks ago.

The request is to just make this query and others like it

http://input.mozilla.com/en-US/beta/search?product=firefox&q=tab+not

return results for the feedback that 4.0 RC and final users are sending us, and show me the trend of reports that match that query.

Comment 9

[chris hofmann]

or in other words make this work.

http://input.mozilla.com/en-US/release/?product=firefox&version=4.0&q=tab+not

Comment 10

[chris hofmann]

talked to aakashd today and figured something that might be a short term solution to getting better and wider access to to the 100,000 pieces of feedback that we have received on firefox 4 final so far.

Lets just add to the list of terms that we are currently clustering.

We clearly need to add these terms

== not     12899    
     stuff that we aren't doing correctly/.
     firefox does not, will not, doesn't work, does not load, does not open, cannot.
      
== IE       9630         
     basically the same thing...   people are comparing  us to IE in extremely large numbers.  what are they saying?

== 3.6      6908         
      stuff that does 4.0 to 3.6 comparisons or things that help us to understand why people are referencing 3.6 so much:
      3.6 new, Firefox 4 tabs like 3.6,  firefox 3.6 works ok -- firefox 4 does not.2  

== version          3011 
    again, anything that mentions versions of other browsers or firefox helps us to find regressions.        

== load     7517   
      stuff like:
        not  load  loading,   download downloading,  
        Firefox  site  will  does  not load. 
        I was loading or I loaded -- firefox/site/video

== bar      5792         bar  not  like  Firefox  search  la  tab  bar.  new  tabs  should  back  firefox  can  bookmarks  page  no  up  was  right  click  make  use  barra  bar,

== work     5552
      stuff like
            not  working  -- doesn't  work  
            don't  like  
            worked in IE/Firefox 3.6  
            used to work

= able     3185      

  stuff like:
    not able to, unable to...


== video            3155   

      all things video:
        video  does not play,  no  flash Video, full screen video
         loading video, watching video, playing video

== Window           3118         
    stuff like:
      new window,  new tab,  close window,
  


these are the key places where we need to do some analysis and identify bugs that need to get fixed before we do major update.

Comment 11

[chris hofmann]

Attachment: sorted-opinions-20110323.txt for firefox 4

here is the list I derived those custering terms from.

left column is instances of the single word, and horizontal list is a break down of frequently use terms where that word is seen.

Comment 12

[Aakash Desai [:aakashd]]

Hey Dave, there's a problem with Norton that Cheng and Chris are trying to figure out. Would it be possible to cluster messages within the subset of broken/idea messages that include "norton" over a time period of 3/22 until now?

Chris, the messages listed in comment #10 are a bit too general, are there any specific issues you're looking for past the Norton issue that we can do a quick cluster around?

Comment 13

[Dave Dash [:davedash, :dd] (assign all bugs to mbrandt)]

Cheng said that there isn't really much data that Input would have since there's no happy/sad data - and that even if there was, people aren't able to get to their browser in the first place.

Also these clustering one-offs are not easy to do - we're clearly losing an avenue of data here.

Comment 14

[chris hofmann]

(In reply to comment #13)
> Cheng said that there isn't really much data that Input would have since
> there's no happy/sad data - and that even if there was, people aren't able to
> get to their browser in the first place.
> 
> Also these clustering one-offs are not easy to do - we're clearly losing an
> avenue of data here.


I'm not sure thats true about having happy/sad for a lot of these cases.

Very few people if any are going to give us happy news or comments about the combination of Norton and Firefox.  Its going to be people that have run into problems.   We can ignore the happy/sad/or rating and just need a way to query or view the comments.

Comment 15

[chris hofmann]

Attachment: norton feedback on 4.0 RC and final up though 3/23

Comment 16

[chris hofmann]

Attachment: just the 4.0 norton comments

Comment 17

[[:Cww]]

To be clear: I didn't say that people wouldn't post to input if they want Norton to work, I said that I don't think there's anything that Input would have that the nearly 1000 people on SUMO wouldn't have said.  Especially given that we a) know that Norton toolbar doesn't work and b) know that Norton was actively quarantining plugin-container.exe.  Having an extra channel of feedback that's limited to 200 characters isn't useful when we already have hundreds of reports from other channels -- especially where we already have tons of details about the problem.

This particular issue isn't something that having happy-sad or having input would have caught.  I still love input, it's just not the right tool here, clustering or not.

Back to the point of the bug: chofmann is right, we need a better way to filter the data.  Clustering is too clever for its own good, it'd be better to just let us search if we know the gist of what we're looking for.

Comment 18

[Dave Dash [:davedash, :dd] (assign all bugs to mbrandt)]

cww/choffman,

we're just not collecting the right data for FF4 release at the moment - we'll get search for ideas, but really we need to collect happy/sad for release as well to get searchable data.

Comment 19

[chris hofmann]

yeah, it sounded like we were on track to get happy/sad/idea classification for all releases going again soon, but after today's feedback discussion session and seeing Aakash's slide  I'm not so sure.

those slides indicated different classifications on the aurora channels (happy /sad/idea) and different classification for beta and final.  there were some discussions and several of us advocated for happy/sad/ideas on all releases but I'm not sure where we left this.

Comment 20

[Dave Dash [:davedash, :dd] (assign all bugs to mbrandt)]

(In reply to comment #19)
> yeah, it sounded like we were on track to get happy/sad/idea classification for
> all releases going again soon, but after today's feedback discussion session
> and seeing Aakash's slide  I'm not so sure.
> 
> those slides indicated different classifications on the aurora channels (happy
> /sad/idea) and different classification for beta and final.  there were some
> discussions and several of us advocated for happy/sad/ideas on all releases but
> I'm not sure where we left this.

For the time being: Happy/Sad/Idea for everything.  In the medium term a revisit to nightly feedback to see that we're getting what we want out of it, or if we want a "File a Report" type which will have no implied feelings.

Resolving Fixed, as the above mentioned strategy will happen early next week and is covered in bug 648401.  Other bugs refer to streamlining the dashboard and being able to query everything.  We're blocked on product-details, at the moment.