Closed Bug 694075 Opened 13 years ago Closed 13 years ago

collect information about why an article wasn't helpful

Categories

(support.mozilla.org :: Knowledge Base Software, task)

Product:
support.mozilla.org
Component:
Knowledge Base Software
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
VERIFIED FIXED
Milestone:
2012-01-10

People

(Reporter: cbeasley, Assigned: rrosario)

References

Details

(Keywords: ux-control) 

When a user tells us a KB article isn't helpful by clicking the No button, a small non-modal popup will ask what sort of not helpful. They will be checkboxes so they can select more than one.

[] out-of-date content
[] inaccurate
[] not complete
[] too confusing
[] wrong topic
[] flag as inappropriate
[] other

And then a textarea for comments.

I will provide mockups. James - I think we need another tracker bug for medium-term stuff like this that have design dependencies.
(In reply to Crystal Beasley [:skinny, :crystal] from comment #0)

General comment: I love the idea of learning more about our articles but I'm not sure the choices proposed will tell us much that we don't already know. 

> [] out-of-date content
> [] inaccurate

We're aware of nearly everything that's out of date or inaccurate. Keeping things up-to-date is an issue, knowing what isn't up-to-date is not. :(


> [] wrong topic

If someone thought the article was of the wrong topic it would be an indication that our search, keywords, taxonomy, etc. weren't working as expected. Won't we have other, better ways to know that? All this would tell us is that something is broken but not what is broken. Or am I wrong and/or missing the point? Is this like an early warning system?


> [] flag as inappropriate

I don't think we've ever had inappropriate content in a live sumo article. I don't think this one is necessary.


> [] other
> And then a textarea for comments.

In tikiwiki we had a text area for comments and it resulted in thousands of comments/misplaced support requests a week with a handful (usually 1 -3) that were somewhat useful. This implementation is a bit different and probably wouldn't result in thousands of comments a week but I'm worried that it will create a lot of noise and misplaced support requests. We currently get a few helpful posts from readers in the article discussion tab (similar to when it was a free for all in tikiwiki) but without anywhere near the amount of noise. So, mission accomplished?


[] not complete
[] too confusing

I think these kinds of choices are the key. For me this is where we might be able to get people to distinguish problems with the article but I think we should be much more specific (why was it confusing?). For example, maybe something like this (brainstorming here):
[] the information was too technical
[] the steps were too hard to follow
[] My Firefox doesn't look the same
[] the article was too long
[] I tried the solution but it didn't work
[] the information wasn't detailed enough
This internal survey that's constantly running. We should include all dimensions of why a user doesn't find the content to be helpful. Even if we don't want to use the data immediately, we should still collect it.

[] out of date
[] inaccurate

We may feel that we do know these, but it's still important to check. If we find that zero people click them, then I'm fine with taking them out. If it's a small number, say a few per week, it's still worth collecting as a measure of health of the system.

[] wrong topic
I agree this is a measure of the effectiveness of the search, related articles feature and other improvements. These are things I'm very invested in assessing. Every data point is useful. This one is basically free, so why would we throw it away?

Textarea - Let's try it. It's a different implementation. If we have problems with misdirected questions, that's a signal that it's too hard to post in the forum or that forums are badly placed in the nav. If we fix forums and we still have a problem, we can kill it. For now, until we know how useful those comments are, we can dump it all in a database and analyze it as raw logs.

I like your added options. I'll incorporate them in the design.
(In reply to Crystal Beasley [:skinny, :crystal] from comment #2)
> 
> [] out of date
> [] inaccurate
> 
> We may feel that we do know these, but it's still important to check. If we
> find that zero people click them, then I'm fine with taking them out. If
> it's a small number, say a few per week, it's still worth collecting as a
> measure of health of the system.
> 

Ok gotcha. Also, looking back at what I wrote I realize that I kind of included those options -  "out of date" is basically "My Firefox doesn't look the same" and "inaccurate" is pretty much "I tried the solution but it didn't work"

The textarea - I'm assuming the goal is get good, detailed feedback. What if the text area was a followup to the survey? So you check one or more things and we invite you to help us by providing details about what didn't work. This way we give some context instead of just a free response area.
(In reply to Verdi from comment #3)
> The textarea - I'm assuming the goal is get good, detailed feedback. What if
> the text area was a followup to the survey? So you check one or more things
> and we invite you to help us by providing details about what didn't work.
> This way we give some context instead of just a free response area.

Either way, we need some plan for dealing with this feedback. A rough estimate of the volume would help. It can happen in follow-up bugs, but we probably shouldn't just sack it away in a DB table/email forever without some idea of looking at it again.
I think we need to set users' expectation that there will not be an answer to this feedback. 

Yes, there does need to be a plan for what to do with this data. I don't feel like I know enough about this data to know what to do with it.
(In reply to James Socol [:jsocol, :james] from comment #4)
> (In reply to Verdi from comment #3)
> A rough
> estimate of the volume would help. It can happen in follow-up bugs, but we
> probably shouldn't just sack it away in a DB table/email forever without
> some idea of looking at it again.

In Tikiwiki we used to get upwards of 2000 comments a week. Now we get about 20 a week. I would guess this would land us somewhere in the hundreds of comments a week.

My first thought about what to do with the data is to somehow (!!) add it to the helpfulness graphs on an article. The wrong topic thing feels like it needs to be treated kind of like the least helpful docs chart.

Another selection for not helpful could be (especially since we'll move the survey to the top):
[] Didn't read/didn't seem helpful

Also, would it be bad to have a follow up survey for people who say the article was helpful?
Love to hear what idea you have about the "helpful" survey. What sort of questions would you want to ask?
Building on what Michael suggested in comment 6, how about a '[] tl;dr' option? ;)

We can get started here, but we can't get too far without having a decent sense of the options and how it'll be displayed.
OS: Mac OS X → All
Hardware: x86 → All
Prototype doesn't show the success screen yet, but it's worth having a look at for the list of checkboxes. Clicking Yes works the same as on current. Clicking no brings up the survey.

http://people.mozilla.com/~cbeasley/SUMO/prototypes/filtering-forum/article-sidebar.html
if you click on "No" on the bottom, the page doesn't scroll up to expose the survey in the TOP right sidebar, please fix that! Other than that I like the prototype so far.
Roland, the intent was for the buttons on the bottom to generate that little popup right next to those buttons. It'll have to be above them though since it's at the bottom of the page and will float over the article content. Thanks for pointing it out so I could clarify!

Another implementation detail is that clicking on the label of the checkbox should check the box. So, for example, if you click on the words "too confusing" it checks the corresponding box.
Thinking about how to capture the sentiment for users who are angry because they don't like the feature or change. How should we word it? 

[] dislike this functionality
(In reply to Crystal Beasley [:skinny, :crystal] from comment #12)
> Thinking about how to capture the sentiment for users who are angry because
> they don't like the feature or change. How should we word it? 

We're talking about Firefox features, right?
No, that's what makes this hard to word. An article could be about a fx feature, a marketplace app issue, a thunderbird issue, etc.
@jsocol - This needs to be rolled out to 2% of users so we can measure how much useless commentary the text box will gather.
Hokay. Whoever takes this, put it behind a flag.
Assignee: cbeasley → nobody
Ready to implement 

See prototype for example 
http://people.mozilla.com/~cbeasley/SUMO/prototypes/filtering-forum/article-sidebar.html

* make second instance of the helpful yes/no question in the right sidebar
* when no is clicked, show survey as shown in prototype
* when submit is clicked, replace survey with text "Thanks for making us better!" as shown http://cl.ly/473d072A2o2d3L213w1H
Target Milestone: --- → 2011Q4
Depends on: 687871
Crystal, I think we talked about why we won't have this survey at the end of the page anymore, but I can't remember, could you explain again? For me it seems like having it in the sidebar at the top *and* at the bottom of the article would lead to more results.

Regarding the items:

* What is the difference between "incorrect info" and "solution didn't work"?
* What does "Wrong topic" mean? Is that to figure out wrong search results?
* The text input for "other" is extremely small. That will probably be an issue for a lot of people with less than perfect eye sight. I think we should increase the text size, expand the text box a bit, and at the same time limit the input to about 250 characters. 

One thing that we have to keep in mind: The votes will be divided between all the available options, the more options we add the more votes we will need to get a significant number of votes for any of the options. With regards to the text box: Is that going to be active for all users or only those that choose "other"?


Also, we still need to talk about how the data should be presented. I'd say we should put it into the article history page together with the helpfulness votes chart. Ideally you'd click on "no" in that chart and it would explode to give you the lines for all the options. The thing we have to keep in mind is that those results need to be shown in context. If suddenly the rate of people rating the article as too long goes up 200% that could be either because the article got a new revision or because it's suddenly one of the top articles. The helpfulness chart does a great job in providing that context.

That still leaves the issues of the "other" box. We'd still need a way to display those submissions in a readable form, especially if those are not only for the "other" option, but for all of them. We'd then need that context too when we display the free form text. Chrystal, have you already given thought to this?
(In reply to Kadir Topal [:atopal] from comment #18)

Good call, I see where you're going with this but per comments 10&11 we addressed your question about the bottom survey.

I think this bug is still ready to implement since we're past review. Let's get the a/b survey up and we can polish with what we learn there.
Assignee: nobody → rrosario
https://github.com/mozilla/kitsune/commit/e7f0c9c41d248471eb6b1057a257084b873ba1c3

This is the last of the bugs behind the 2% waffle flag (called 'editing-tools-show-hide'). Once this lands we can enable the flag?
Status: NEW → RESOLVED
Closed: 13 years ago
Resolution: --- → FIXED
Target Milestone: 2011Q4 → 2012-01-10
Verified implementation with waffle flag
Status: RESOLVED → VERIFIED
Depends on: 1204251
You need to log in before you can comment on or make changes to this bug.