Closed Bug 1115303 Opened 10 years ago Closed 5 years ago

Most of the support forum is not being indexed by search engines.

Categories

(support.mozilla.org :: Questions, task)

Product:
support.mozilla.org
Component:
Questions
Platform:
x86
All
Type:
task
Priority:
Not set
Severity:
major

Tracking

(Not tracked)

Status:
RESOLVED WONTFIX

People

(Reporter: unicorn.consulting, Unassigned)

References

Details

(Keywords: regression) 

In bug 1081933 a user story of "I as a visitor would like to only see questions with solutions when searching for things, so I don't waste time reading things that don't actually help me." was used to make only answered questions on support forums indexable on search engines.

As a visitor to the site I would be using the site search,  and I would have expected the bug to address the many complaints about the internal search engine.

Result:  It is becoming ever more difficult to research anything support without doing many searches internally and externally.  This decision increased volunteers work loads without any sort of consultation or compensation.  If your going to make our jobs harder, we do expect something to offset that, and I am simply not seeing it.

Expected result: all information on Support.mozilla.org to be indexed for search engines, restricting data indexed offers no material gain for asnyone and considerable pain for volunteers.
I have mixed feelings on this. No doubt bug 1081933 is well intentioned. On the one hand, yes, it would be great for users seeking solutions to get only references to topics which are solved.

But, realisticly only 28% of Firefox topics are *marked* solved (and 25% of Thunderbird). 
* Of the other ~75%, what % do we estimate have good answers/responses but have not been marked solved, and could be of high value to users?  (If it's similar to what we saw in the Thunderbird getsatisfaction forums, it's a very significant percentage)
* We have hurt users who are seeking confirmation that they are not the only poor slob with the problem they are seeing. 

What metrics are available to show the impact of bug 1081933?
Is there a middle ground that can be explored, where some topics that are not marked solved can still be indexed?
Blocks: 1081933
Keywords: regression
Yes, some metrics would be good to check the impact, I'll look into that.

Matt, over 90% of our visitors come Google, they do not use our internal search. I can understand that this change is making it harder for contributors to do their research, this is why some improvements have been made to Advanced Search. 

I'm open for middle ground if it brings more value but we need data to confirm that.
ps: Wayne thanks for the feedback, indeed the intention is good, the purpose of it being trying to increase the satisfaction rate of visitors (there are around 150.000 visitors/day in the forums and their satisfaction is very low). This is why looking at some data and figuring out the impact is important.
There are several groups of users using the forums (visitors, users, contributors, moderators) so maintaining a balance and trying to keep everybody happy can be difficult.
(In reply to madalina from comment #2)
> Yes, some metrics would be good to check the impact, I'll look into that. 

Madalina ?
Flags: needinfo?(mana)
Thanks for the reminder Alice, hope to get some metrics beginning of March
Flags: needinfo?(mana)
Sorry I meant end of March beginning of April
 (In reply to madalina from comment #5)
> Thanks for the reminder Alice, hope to get some metrics beginning of March

(In reply to madalina from comment #6)
> Sorry I meant end of March beginning of April

Anything yet?
Flags: needinfo?(mana)
I'm currently doing a postmortem on previous projects, should have the info soon.
Flags: needinfo?(mana)
That will no doubt answer whether there was an increased use of sumo standard and sumo advanced search after the Google change and initial search improvement. 

Further increases may take time and need us to make progress with:
1.) Promote the advanced search more. (Including Bug 1121625) 
2.) Fix the abysmal advanced search interface (Bug 1118375 Bug 886554)
3.) Change the standard search (I've asked for this repeatedly now we have bug 1167449 )
*) See  also listing https://trello.com/b/3TyzEOvn/search-quality-draft
(In reply to John Hesling [:John99] from comment #9)
> That will no doubt answer whether there was an increased use of sumo
> standard and sumo advanced search after the Google change and initial search
> improvement. 
> 
> Further increases may take time and need us to make progress with:
> 1.) Promote the advanced search more. (Including Bug 1121625) 
> 2.) Fix the abysmal advanced search interface (Bug 1118375 Bug 886554)
> 3.) Change the standard search (I've asked for this repeatedly now we have
> bug 1167449 )
> *) See  also listing https://trello.com/b/3TyzEOvn/search-quality-draft

Thanks for citing these in the interest of moving search forward.
Looking forward to madalina's feedback.
Severity: normal → major
OS: Windows 7 → All
(In reply to Wayne Mery (:wsmwk, use Needinfo for questions) from comment #11)
> Another factor ... are "Helpful Reply"s indexed by google?
>  
No, only "solved" threads are indexed by search engines.  Related discussion:
https://support.mozilla.org/en-US/forums/contributors/710806 
Change in support forum threads indexed by Google or other search engines: Only solved threads will be shown.

Helpful replies *are* included in the internal SuMo basic search, however: See KadirTopal's comment here: 
https://support.mozilla.org/en-US/forums/contributors/709852#post-56327

I don't understand why questions with helpful replies can't be included in Google and other search engine results.
Flags: needinfo?(mana)
(In reply to Alice Wyman from comment #12)
> (In reply to Wayne Mery (:wsmwk, use Needinfo for questions) from comment
> #11)
> I don't understand why questions with helpful replies can't be included in
> Google and other search engine results.

Helpful replies aren't always necessary helpful. They could me used by spammers, or others users saying that x is helpful because they are experiencing the same issue (treated like an upvote)
(In reply to Andrew Truong [:feer56] from comment #13)
> (In reply to Alice Wyman from comment #12)
> > (In reply to Wayne Mery (:wsmwk, use Needinfo for questions) from comment
> > #11)
> > I don't understand why questions with helpful replies can't be included in
> > Google and other search engine results.
> 
> Helpful replies aren't always necessary helpful. They could me used by
> spammers, or others users saying that x is helpful because they are
> experiencing the same issue (treated like an upvote)

I haven't paid particular attention to posts that have been marked "helpful". But I can't imagine what you suggest is a serious issue.

Every possible choice for exposing data is going to have a downside.  Even topics marked solved. For example, there are plenty of posts marked as solutions whic are merely something that helped the user in some non-optimal way, such as a workaround or something that is not a real "fix".
 (In reply to Andrew Truong [:feer56] from comment #13)
> (In reply to Alice Wyman from comment #12)
> > (In reply to Wayne Mery (:wsmwk, use Needinfo for questions) from comment
> > #11)
> > I don't understand why questions with helpful replies can't be included in
> > Google and other search engine results.
> 
> Helpful replies aren't always necessary helpful. They could me used by
> spammers, or others users saying that x is helpful because they are
> experiencing the same issue (treated like an upvote)

As I wrote in comment 12 
> Helpful replies *are* included in the internal SuMo basic search, however:
> See KadirTopal's comment here: 
> https://support.mozilla.org/en-US/forums/contributors/709852#post-56327
> I don't understand why questions with helpful replies can't be included in
> Google and other search engine results.

To rephrase my question, which is directed at Madalina and/or Kadir Topal:
Since threads with helpful replies ARE included in SuMo basic search results, why can't they be included in google or other search engine results?

Hopefully Madalina or Kadir will respond soon.
Flags: needinfo?(a.topal)
If we are to include helpful replies in external searches maybe we should reconsider:

Bug 928905 
Because we are specifically adding unsolved threads but focussing on the helpful answers
And although
> The forum report revealed that more than 90% of all threads are no longer than a single page.
I wonder what happens if that is adjusted for the number of question views.
Highly viewed; & likely highly SEO rated questions; may well be multi page. 
So useful metrics may be:
What percentage of VIEWS are of multi page questions ?
And maybe; but of less importance; what % of unsolved questions with helpful votes are multi page ?  

Bug Bug 700060  
Because that was wontfixed due to karma being  deprecated.
The issue here now is whether the answer is genuinely helpful or not.
Not whether a person has earned karma.
(In reply to Andrew Truong [:feer56] from comment #13)

> Helpful replies aren't always necessary helpful. They could me used by
> spammers, or others users saying that x is helpful because they are
> experiencing the same issue (treated like an upvote)

Helpful. Solved. Unsolved.  Really the whole concept of solved is the root cause of this silliness.  We are asking someone with the technical skills of a lamppost to decide when and if the question is solved and therefore suitable for indexing by search engines.

Please what is Solved?
Can Firebox fold my washing?  No!   (Solved? Is there anything to actually solve?  The assumption that a question requires a solution is fundamentally flawed.)
Or a real life.
I want Thunderbird to filter mail based on clicking the unread heading it did until this last upgraded.  This person refuses to accept that the bug files for that in 2000 to sort is still outstanding.  They are frustrated that we will not accept the palpably incorrect statement that it used to work.  That thread is still open.  Will it ever be solved to the posters satisfaction?  I doubt it.  Should it appear in a Google search?  Most definitely.

I have serious issues with SUMO.  This is one of those I find most difficult to cope with. Either exclude the whole forum or include it.  The arbitrary solved is at best a poor definition of the status of a topic,  but completely useless is my gut feeling.  I don't know how often I have gone back to a question after getting an email saying "yep, that solves it"  only to find my original response asking for troubleshooting information was marked as the solution a fortnight ago.
(In reply to Matt from comment #17)

> Helpful. Solved. Unsolved.  Really the whole concept of solved is the root
> cause of this silliness.  We are asking someone with the technical skills of
> a lamppost to decide when and if the question is solved ...

We are going to be told that is drifting off topic. 

I agree it is a big obstacle that we repeatedly run up against in sumo: Having only two states with a default of unsolved, and reliant on the OP to post back and it solved ! 

I think even some Admins recognise the flaw Michael Verdi for instance saying:
>I like John99's idea that sort of filters down what is actionable. 
>It sounds like things marked as incomplete or known issues wouldn't count against the solved rate. 
>In this way we're measuring the solved rate of the "solvable" questions.
> [https://support.mozilla.org/en-US/forums/contributors/709311#post-52937]
So I tried looking into some metrics. Unfortunately due to the fact that we only started to measure visitor satisfaction on the forums a few months before the indexing was change there's not a lot of data that we can play with. But the current satisfaction rate looks to be at 4 out of 5 so it looks like we're on the right track.

But we need to also recognize that there are indeed a lot of threads out there that might have high value and are not being highlighted to the user simply because they have not been marked as"solved". And this is an issue that we need to address. 

*Taking helpful votes into consideration might be a good idea. The only downside that I can see right now (maybe there are more) is that helpful votes are sometimes used by users as a way to say "I have the same issue" . I wonder if there is a way to run a few tests to check how many of the helpful votes were actually "me too" upvotes. In this way we should be able to tell if the data is accurate enough to be used.

*We need to figure out the what "solved" really means  and think of ways to calculate that more accurately. Right now this is highly depending on the user taking action which doesn't always happen nor is always very accurate.

The new advanced search should also be available by the end of this quarter. This should hopefully eliminate all the issues around doing research.
Flags: needinfo?(mana)
Since there is a needinfo for me, I don't know why we have two different set of filters for the internal search and other search engines. I agree that we should unify it.
Flags: needinfo?(a.topal)
Here is a perfect example of why our filtering of search is a joke

Advanced Search SUMO for MYOB.  6 results 5 Thunderbird 1 Firefox  None solved and none likely to be.

Normal search from the seach SUMO dialog I get
"We couldn't find any results for MYOB in English. Maybe one of these articles will be helpful? "

Google search
https://www.google.com/search?q=myob+site%3Asupport.mozilla.org&ie=utf-8&oe=utf-8

Zero results.

Now I do not care what sort of user satisfaction metrics are applied. Some things are obvious to me.
1.  Those that have no idea we withheld results will say they were satisfied.  Ask them again after they become aware the search did not return all results, only the ones we had sanitised for them.  Shades of the satisfaction people has with the NSA before they were aware of what they actually did.
2. We have not supported the person typing MYOB into a search box.  We have mislead them into thinking it might just be them.  We have also not shown them their own previous question which they did not click "Solved" on  See this user https://support.mozilla.org/en-US/user/Biggeew as an example.

Those that do not know they do not know will always be happy.  They are not aware they are missing out on a thing.  What have they to be unhappy about.
See Also: → 1047825

Closed as per https://blog.mozilla.org/sumo/2019/06/07/sumo-platform-roadmap/

Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.