817540 - [tracker] AJAXify the refine+focus panel

Status: RESOLVED INVALID

(support.mozilla.org :: Knowledge Base Software, task, P3)

Description

[Kadir Topal [:atopal]]

Currently, when you chose sub-topics from the focus+refine panel, the whole page gets reloaded and scrolled to the top. So each time you select an item you have to wait for the reload and scroll down to see the resulting list. This effectively eliminates the benefit of having such a feature, namely: shortening the list of articles shown for quick browsing of the available options.

The list should be dynamically updated based on the selected sub-topic.

Comment 1

[Ricky Rosario [:rrosario, :r1cky]]

Do we have any metrics on the usage of the panel so far?

Comment 2

[Kadir Topal [:atopal]]

I don't think we have usage data. What would be your question?

Comment 3

[Ricky Rosario [:rrosario, :r1cky]]

I think you can figure that out in Google Analytics. My point was going to be, if nobody every uses this then it doesn't matter if it's AJAX or not. Ajaxifying isn't trivial if we want to have the metrics be right in analytics. We'd have to figure out how to trigger page views.

Comment 4

[Ibai Garcia [:ibai]]

I can take the measuring part. 

Should we open another bug to track that and make this one dependent of that one?

Comment 5

[Ricky Rosario [:rrosario, :r1cky]]

Moving to the next sprint while we discuss further.

Comment 6

[Kadir Topal [:atopal]]

Some clarification upfront to make sure everyone following this is on the same page: Focus and Refine was not part of our original design as such it was never paper prototyped and never user tested. During the implementation of the new iA it became clear that some of our topics had such a high number of articles assigned to them that manually scanning the list of articles was infeasible. Focus and Refine follows the philosophy that people are okay with deep hierarchies as long as they feel they are getting closer to their goal. 

Engagement is higher when the reaction times are low, that's why I suggested Ajaxifying the focus and refine panel. 

But Ricky is right, we should first figure out if the panel is used at all. That said, in its current form engagement will be low, so we should first figure out what percentage of people even attempt to use the functionality of focus and refine. 

Let's first do some A/B testing on how this is being used when it's shown/hidden by default. If usage of the panel is low, it wouldn't matter whether we Ajaxifying it or not, since they can't tell if it is dynamic or not. 

If usage is high, but people abandon the usage of the panel after the first click, it might indicate that people expect a better interaction and we can justify Ajaxifying the panel to test that assumption.

Ibai, it would be great if you could help with this. We fist need to measure usage per unique visitor, preferably on some of the topics that have a lot of articles, where this panel makes the most sense.

I'm making this bug a tracker to test all of the above. And I'll file a follow up bug to a/b test showing/hiding the panel by default.

Comment 7

[Ibai Garcia [:ibai]]

I was certain that I did report on the numbers. I can give the specifics in a minute again, but what I remember is that the actual usage is tiny. Almost no one uses the refinement feature.

The hypothesis that I read from the previous answer is that this is because it's not expanded. 

So here is my proposal, make an A/B test (we can do this one with Analytics) to learn if there's a larger consumption of articles (bigger CTR on Articles) from the Category pages if we expand the Refine and Focus box.

The benefit of having the box open is that it make refining easier and because of that it will make finding articles easier. The problem with expanding the box is that the list of articles is "hidden" and the user will need several scrolls to see article titles.

So, we can set up a Test where we show a box expanded or not expanded (we can even make it without the box) and measure how many articles are visited.

The main thing here is, that if we proof that the expanded box has better performance, we don't proof that making the filtering client-side will have any benefit.

Comment 8

[Ibai Garcia [:ibai]]

I cooked the data again. Since the 28th of December (I excluded the 27th for a reason that I will explain later in the bug):

1,322,5778 category pages where loaded

From those, only 

19,095 where triggered by the search and refine box.

In other words, only 1.44% of the category pages is loaded from the search and refine box. Pretty small fraction. If the assumption is that users who use search and refine use it multiple times, then the visitors using it is less than 1.44% of the total visitors. 

In my opinion, we should try to demonstrate that there's any value on this implementation of the article filter before moving to something more complex like the client side filtering that we are talking here.

Speaking of, the 27th of December we broke the site and we were not displaying articles in the Category menus. This doesn't show up an impact in many of the metrics, but one of the ones that gets significantly impacted is the amount of users who tried Search and Refine, in particular it went from the average 1000 usages a day to almost 12,000. Meaning..users understood they need to use that to find more articles...but just because we didn't show them any. If we show them some...they prefer to read through the list than trying to guess the sub-category. 

My 2 cents, as long as you show them content...users will jump into it instead of trying to keep refining it until there's only one. In other words, they trust their criteria to categorize better than ours. Just a hypothesis though.

Comment 9

[Kadir Topal [:atopal]]

Ibai, did you see bug 830753? I think we are on the same page.

Comment 10

[Ricky Rosario [:rrosario, :r1cky]]

Q1 is in the past. Moving to the future.

Comment 11

[Ibai Garcia [:ibai]]

Ricky+Kadir...I think that this bug is obsolete now. 

Marking it as such, reopen if you feel otherwise.