Closed Bug 1173430 Opened 9 years ago Closed 9 years ago

analyze success of Thank You page project phase 1

Categories

(Input Graveyard :: Submission, defect, P2)

Product:
Input Graveyard
Component:
Submission
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: willkg, Assigned: willkg)

References

Details

(Whiteboard: u=user c=feedback p= s=input.2015q2) 

On June 10th, 2015, we pushed out and turned on the new Thank You page suggestions which currently have a single suggestions provider which uses SUMO Suggest API to find relevant kb articles for user's feedback.

In bug #1133774, we covered what questions we're using to think about success, what metrics we're gathering and how to manipulate those metrics to determine how things went.

This bug covers doing that analysis work and writing up a report with conclusions and possible next steps.

This bug shouldn't be worked on for a couple of weeks so we can accumulate enough data. Maybe it's best to do at the end of June 2015.
This blocks the Thank You page project phase 1 tracker.
Blocks: 1007840
Priority: -- → P2
Grabbing this one to work on this week.
Assignee: nobody → willkg
Status: NEW → ASSIGNED
Checked the Event Flow for sumo_suggest1 events in Google Analytics for the last week.

Total engagement:

    605 / 1690    = 35.7% people clicked on something

That's well north of 20%, so I think this is a success and that we are solving a real problem here.

Success of the suggestions heuristics:

    530 / 1690    = 31.3% people clicked on a suggested kb link

vs.:

     75 / 1690    =  4.4% people clicked on "Ask a question in our support forum"

31.3% vs. 4.4% suggests our suggestion heuristic is decent--definitely not completely terrible. I suspect we can do better than that. Not many people choose a second link to click on, so it seems like of the three suggested kb links only one is attractive.

Given all that, I claim this project is a success. I have some thoughts on next steps and there's the trigger-keyword suggestion provider, too. All that in other bugs.

I wrote up a blog post:

http://bluesock.org/~willkg/blog/mozilla/input_thankyou_phase1.html

Closing this out.
Status: ASSIGNED → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Product: Input → Input Graveyard
You need to log in before you can comment on or make changes to this bug.