501880 - [ForumUX] Advanced forum and KB search with more options

Status: VERIFIED FIXED

(support.mozilla.org :: Search, defect, P1)

Description

[Paul Craciunoiu [:paulc]]

As per https://wiki.mozilla.org/Support/ForumUX/PRD contributors should be able to search the forums using a more advanced search options.

Contributors: this would be the place where you start mentioning options you'd like to see.

Comment 1

[Paul Craciunoiu [:paulc]]

Implementation-wise, my proposal is to add these options on the tiki-newsearch.php page, for contributors (=logged-in users), and, if needed, modify the newsearch indexer to account for new fields.

Comment 2

[Paul Craciunoiu [:paulc]]

Cheng, what would be some needed search options? If you could prioritize them that would be great.
* Post status/type (solved, sticky, etc)
* Last post date (show posts later than / before / between dates)
* Parent post date
* # of replies
* author (already done; is it in the UI?)

Others?

Comment 3

[[:Cww]]

The most important ones are:

Post status: Combo box: solved, proposed, requesting more info, open (status = n or last reply is a user), no replies. (most important)

Posted by user:

Sort results by: (dropdown): relevance, last post date, original post date, # of replies

Posted after: (original post date)

The rest aren't that important IMO.

We should also have

Comment 4

[Paul Craciunoiu [:paulc]]

I've been working on this today. I will implement the functionality with a first-draft UI, and then we can adjust the UI (either for this milestone or a future one).

I will focus on Cheng's requirements for forum searches. It would be good to also think about KB searches (although those already have implemented the author, and category, for example).

I will file a patch late this week or early next that rewrites a good deal of the current newsearch page to make it easier to maintain, etc.

I'd rather make this bug about p10n applied to the newsearch page and shoot it for 1.3 instead.

Comment 5

[Paul Craciunoiu [:paulc]]

(In reply to comment #4)
Also, the rewrite aspect deals with the implementation -- the page will look the same (aside from the search options UI).

Comment 6

[Paul Craciunoiu [:paulc]]

So, I have newsearch mostly rewritten (and I must say, it looks much better!) but I would rather wait on the UI input from Neil until I post anything.

Comment 7

[[:Cww]]

Fields needed:     
*  Forum search
                + Search terms
                + thread status (pick one or many)
                + post author
                + tags
                + created/last posted after date (or in the last XXX)
                + sort by (relevance, date of last post, date of original post) 
* Advanced KB search
                + Search terms
                + language (dropdown)
                + category (Firefox 3, 3.5, Help, troubleshooting) should be able to select multiple
                + tags 
    * We can merge forum and kb search on one page but they have different options so we need to be very clear on what they're searching for.

Comment 8

[Neil Lee [:neilio]]

Just so there's something to look at and comment on in advance of Thursday's meeting, here's what I've got so far. Let me know if you think this is moving in the right direction:

Rejigged "normal" search box with updated text, new advanced search link
http://snaps.beatnikpad.com/sumo-normalsearch-20090810-222618.png

Advanced search page (set to "forum" options):
http://snaps.beatnikpad.com/sumo-advsearch-20090810-222920.png

I still need to do the KB search options and could use the final list of options for each of these (such as what thread states there are, language options, etc.) so I know what to design for.

Comment 9

[[:Cww]]

Thread statuses are:
* Thread has no replies
* Thread is waiting on more info from user (requesting more info)
* Thread contains a proposed solution
* Thread is solved
* The user has replied in this thread

We'd possibly add more but those are the main usecases

language options: we have 30+ languages on SUMO so you should be able to search in just one language or multiple.


As far as comments, will the text in the right box change as you focus on different fields?

Comment 10

[Neil Lee [:neilio]]

> As far as comments, will the text in the right box change as you focus on
different fields?

It could, if what you wanted was contextual help -- my original thinking was it'd be a "how do I actually *use* this to find something?". I find most advanced search pages just dump the fields in front of the user without much help, and seeing how some people might not know how to really take advantage of these options, a bit of in-page help might be useful.

Comment 12

[Neil Lee [:neilio]]

Here's the advanced search mockup with all of the forum advanced fields in place:

http://snaps.beatnikpad.com/advancedsearch-20090821-151525.png

The Knowledge base is an identical design except with those fields swapped in.

Comment 13

[Paul Craciunoiu [:paulc]]

I'd like to see this finalized, but I've got a head start on the implementation already.

Comment 14

[Neil Lee [:neilio]]

Attachment: Search design notes

Here are the design notes for the search design - as discussed, I'd like us to use box-shadow and border-radius for the drop shadow and rounded corners. The button uses the same bg graphic and design as the "ask a question" buttons.

Comment 15

[Laura Thomson :laura]

I think that's everything from Neil.  Reassigning to Paul for implementation.

Comment 16

[Paul Craciunoiu [:paulc]]

I'm adding all bugs that will be fixed by this as blocked by it.

Comment 17

[Paul Craciunoiu [:paulc]]

The patch is ready, I'm waiting for the merge to see if there are any conflicts. We should do the merge first so I can update this patch accordingly.

Comment 18

[Paul Craciunoiu [:paulc]]

Attachment: giant patch, v1

Sorry. This is really a huge patch. Listing details here:

-- List of files it adds:
webroot/tiki-newsearch_advanced.php
webroot/tiki-newsearch.php
webroot/images/search
webroot/lib/search/sphinxlib.php
webroot/lib/cache/memcachelib.php
webroot/styles/jui/smoothness/jquery-ui-1.7.1.custom.css (moved to webroot/styles/jquery-ui-1.7.1.custom.css)
webroot/styles/mozcommon.css
webroot/styles/mozsadvanced.css
webroot/templates/tiki-admin_newsearch.tpl
webroot/templates/tiki-newsearch_advanced.tpl
webroot/templates/tiki-newsearch.tpl
scripts/sphinx/search.conf.php.dist
scripts/sphinx/indexer.php
scripts/sphinx/indexer-forums.php
scripts/minify/minify.conf.php.dist
webroot/js/mozsadvanced.js
-- Also add images in:
webroot/images/search (attached in next comment)

After applying the patch:
* Reindex Sphinx, both KB and forums
* Run these:
cp scripts/minify/minify.conf.php.dist scripts/minify/minify.conf.php
cp scripts/sphinx/search.conf.php.dist scripts/sphinx/search.conf.php
* Run the attached SQL (to add the permission)
* Copy the images over from the next attachment

Explanation of changes:
I'm sure I'll miss stuff. Here we go...
* Fixes all the search bugs dependent on this bug
* Rewrites sphinxlib, tiki-newsearch almost entirely
* Adds required XML tags to indexer and indexer-forums
* Removes "session" cache from admin panel, as it is not yet implemented
* Updates search.conf.php.dist with lots of constants used in search. Also crucial to make search productizable
* Adds a useful memcache function to build a cache key from the $_GET array
* Updates the paths in jquery-ui-1.7.1.custom.css, the images wouldn't be showing up through minify (relative paths become incorrect)
* Updates minify.conf.php.dist with the new css files

Comment 19

[Paul Craciunoiu [:paulc]]

Attachment: images, go in webroot/images/search

Comment 20

[Paul Craciunoiu [:paulc]]

Attachment: permission SQL

Comment 21

[Paul Craciunoiu [:paulc]]

Attachment: updated sphinx.conf

I forgot: the final step in the indexing consist of running /usr/local/sphinx/bin/indexer
This is the sphinx.conf that goes with the updated indexer.php and indexer-forums.php

Comment 22

[Paul Craciunoiu [:paulc]]

If permissions are not set for advanced search, go to tiki-admingroups.php to set them.

Comment 23

[Laura Thomson :laura]

Comment on attachment 400143 [details] [diff] [review]
updated sphinx.conf

Would have preferred to see it as a patch but this is ok for me

Comment 24

[Laura Thomson :laura]

Comment on attachment 400137 [details] [diff] [review]
giant patch, v1

Took a long time to go through all this! Looks really good but I'm sure I haven't tested it out enough.  QA will need to do a lot on this one.

Comment 25

[James Socol [:jsocol, :james]]

r51742 / r51743

Comment 26

[Stephen Donner [:stephend] Not actively reading bugmail]

Reopening; https://support-stage.mozilla.org/tiki-admin_newsearch.php is blank right now -- James has a fix.

Comment 27

[James Socol [:jsocol, :james]]

Attachment: extra patch

patch to the giant patch, fixes the blank admin page

Comment 28

[James Socol [:jsocol, :james]]

r51801 / r51802

Comment 29

[James Socol [:jsocol, :james]]

Backed out in r51894

Comment 30

[Paul Craciunoiu [:paulc]]

Attachment: indexer patch

I was testing locally and some stuff lurked in. The "LIMIT 2500" in particular would be a problem on production.

Comment 31

[Paul Craciunoiu [:paulc]]

I was setting up a fresh SUMO, and noticed another issue with a dump from Aug 13. If no search results show up, it can be because of the weights. Advanced search removes these fields:
* polls (pollyes, pollcoeff)
* answered
These fields need to be removed from the weights at tiki-admin_newsearch.php

Comment 32

[Laura Thomson :laura]

This is backed out, moving to 1.4.1

Comment 33

[James Socol [:jsocol, :james]]

Further backed out in r52135

Comment 34

[Paul Craciunoiu [:paulc]]

Comment on attachment 401642 [details] [diff] [review]
indexer patch

Just going through my r? list

Comment 35

[Paul Craciunoiu [:paulc]]

Attachment: search branch patch

Search branch now has basic search frontend code committed. Patch attached.
r55392 (branches/search)

Comment 36

[Paul Craciunoiu [:paulc]]

I should add that the (basic) search will now be at /search.php, and the patch above updates all templates accordingly. We are left to update the sidebar modules.

James will take care of updating the sphinx.conf file and setting up search branch to work with sphinx so we can test.

Comment 37

[Paul Craciunoiu [:paulc]]

Attachment: remove translation calls

r55847 (search branch)
This removes calls to Google translate and shows result counts separately for current locale versus default (English). Patch attached for reference.

Comment 38

[Paul Craciunoiu [:paulc]]

Attachment: advanced search, v1

This patch adds advanced search to the same file.

While doing this, I also noticed that searching for "firefox" suggests "Firefox" on my machine. Maybe we should ignore case?

Comment 39

[Paul Craciunoiu [:paulc]]

A few notes on the patch:
* It does not yet add search for *any* poster in a thread -- just the original author
* I think we also mentioned adding some other filters?

I only made the original stuff (that was in the mockups) work. So James, you can either make me add the other filters in this same patch, or we can land this and then add more :)

Comment 40

[James Socol [:jsocol, :james]]

Comment on attachment 412340 [details] [diff] [review]
advanced search, v1

The form looks great, and seems to be working fine, but...

- Including the form on the results page pushes the results below the fold. A better behavior is to add an "Edit your search" link below the search/refine form (bug 527859) on advanced search. (IE: refining your search should persist the options you selected, clicking "Edit your search" should take you back to the advanced form with your choices pre-made.)

- The language on the KB search defaults to Arabic if no locale is set (eg: /search.php?advanced=1) English is preferable as a fallback.

- Searching the forums still results in "0 results in and ### results in English". I don't know if that's in scope of this bug or not, but thought I'd mention it.

- 1/3 of this seems unnecessary:
+    $smarty->assign('title', tra("Advanced Search"));
+    $smarty->assign('page', tra("Advanced Search"));
+    $smarty->assign('current_lang', $search_parameters['lang']);
+    $smarty->assign('current_locale', $search_parameters['locale']);
+    $smarty->assign('title', tra("Advanced Search"));
+    $smarty->assign('page', tra("Advanced Search"));

Comment 41

[Paul Craciunoiu [:paulc]]

All good points. I will do the simple search form Friday and finish this up after.

Comment 42

[James Socol [:jsocol, :james]]

This is an absolute must before we can push search. Please prioritize this.

Comment 43

[Paul Craciunoiu [:paulc]]

I'm almost done with this, fixed everything except I'm waiting to integrate it with the form in bug 527859 (e.g. I also have a ready link for "refine your search" that toggles the advanced form to make it visible). I'd rather wait for bug 527859 to file the patch, but let me know if you want me to file it anyway.

Comment 44

[James Socol [:jsocol, :james]]

I will get a patch to bug 527859 first thing in the morning, so you can hold off til then.

Comment 45

[Paul Craciunoiu [:paulc]]

Attachment: v2

This should do it.

Comment 46

[James Socol [:jsocol, :james]]

Comment on attachment 413493 [details] [diff] [review]
v2

Woah, not sure what happened here. After applying the patch, going to search.php with no parameters tells me there are 1300 results in English (and lists them) and adding ?advanced=1 doesn't seem to do anything.

Comment 47

[Paul Craciunoiu [:paulc]]

Attachment: v3, no search on empty query

Comment 48

[Paul Craciunoiu [:paulc]]

Attachment: v4

This should have all the fixes as we discussed today. Outline of the behavior:
* No search if query is empty, and replaced with slightly larger search form.
** Exception: in advanced search, if filters are set (all filters count except article language and forum results sorting.
* Added "advanced search" link for logged-in users, which takes you to the advanced form.
* After searching by using the advanced form, you can refine your search by clicking "refine search" and you are again taken to the advanced form, with the values pre-filled.

From a dev/QA standpoint, these URL parameters determine behavior:
* refine=1, advanced form shows up, for refining results; no search results shown
* advanced=1, use advanced search for the given parameters
* if both are present, refine takes predecence (i.e. advanced=1 makes no difference)

Comment 49

[James Socol [:jsocol, :james]]

Comment on attachment 413719 [details] [diff] [review]
v4

Woo, ok!

Comment 50

[Paul Craciunoiu [:paulc]]

r56619
Finally!

Comment 51

[Stephen Donner [:stephend] Not actively reading bugmail]

Verified FIXED; we've verified all associated bugs, as well as done ad-hoc testing.  Will continue to file new issues as we discover them.