470378 - New search stopwords can't be changed on production

Status: RESOLVED FIXED

(support.mozilla.org :: Knowledge Base Software, task)

Description

[Nelson Ko [:nkoth]]

On http://support.mozilla.com/tiki-admin_newsearch.php?admin=stopwords, it is not possible to add/remove stopwords (or to see existing stopwords for that matter).

The current stopwords are: a firefox he she the we

The reason for this problem is because /usr/local/sphinx/index/stopwords.txt is only on the subset of sphinxd webheads.

The prefered solution to this in my opinion is to refactor to put stopwords in the database instead, or alternatively in some shared space. (usr/local/sphinx/index is not on shared space right now in case of performance issues with Netapp - hypothetical, not known for sure if there are any)

Comment 1

[Laura Thomson :laura]

I would probably recommend putting them in the database.

Comment 2

[E. Cooper [:ecooper]]

Attachment: New functions to allow stopwords to be saved into database

Comment 3

[E. Cooper [:ecooper]]

Attachment: This is a .sql file to add the se_stopwords table

I followed the basic schema for the other se tables for this table.

Comment 4

[E. Cooper [:ecooper]]

stopwords has to be a file apparently.

http://www.sphinxsearch.com/docs/current.html#conf-stopwords

Comment 5

[E. Cooper [:ecooper]]

Attachment: Save stopwords to scripts/sphinx

IT will need to update the sphinx.conf so it points to the correct location for the stopwords (scripts/sphinx/stopwords.txt).

Comment 6

[E. Cooper [:ecooper]]

Comment on attachment 355704 [details] [diff] [review]
Save stopwords to scripts/sphinx

>Index: tiki-admin_newsearch.php
>===================================================================
>--- tiki-admin_newsearch.php	(revision 21446)
>+++ tiki-admin_newsearch.php	(working copy)
>@@ -302,7 +302,7 @@
> }
> elseif(isset($_GET['admin']) && $_GET['admin']=='stopwords')
> {
>-	$stopwords = file_get_contents('/usr/local/sphinx/index/stopwords.txt');
>+	$stopwords = file_get_contents('../scripts/sphinx/stopwords.txt');
> 	$stopwords = explode(' ',$stopwords);
> 
> 	if(isset($_GET['remove']))
>@@ -311,7 +311,7 @@
> 		$removed = $stopwords[$remove];
> 		unset($stopwords[$remove]);
> 		sort($stopwords,SORT_STRING);
>-		file_put_contents('/usr/local/sphinx/index/stopwords.txt',trim(implode(" ",$stopwords)));
>+		file_put_contents('../scripts/sphinx/stopwords.txt',trim(implode(" ",$stopwords)));
> 		header('Location: tiki-admin_newsearch.php?admin=stopwords&removed='.urlencode($removed));
> 	}
> 	if(!empty($_POST['stopword']))
>@@ -321,7 +321,7 @@
> 		{
> 			$stopwords[] = $stopword;
> 			sort($stopwords,SORT_STRING);
>-			file_put_contents('/usr/local/sphinx/index/stopwords.txt',trim(implode(" ",$stopwords)));
>+			file_put_contents('../scripts/sphinx/stopwords.txt',trim(implode(" ",$stopwords)));
> 			header('Location: tiki-admin_newsearch.php?admin=stopwords&added='.urlencode($stopword));
> 		}
> 	}

Comment 7

[E. Cooper [:ecooper]]

Attachment: Using the proper path now

Patch uses the correct relative path of ../scripts/sphinx/stopwords.txt now.

I also learned what the Edit Attachment as Comment button does.

Comment 8

[Nelson Ko [:nkoth]]

Comment on attachment 355709 [details] [diff] [review]
Using the proper path now

in r21447/r21448. Note also that I added scripts/sphinx/stopwords.txt.dist which should be deployed to scripts/sphinx/stopwords.txt and set read/writable by apache

Comment 9

[E. Cooper [:ecooper]]

Marking this as resolved, but this won't be able to be verified to after push since it's a production-only bug.

Comment 10

[David Tenser [:djst]]

Part of newsearch.