234464 - Design an intuitive syntax for useful "QuickSearch" features

Status: NEW

(Bugzilla :: Query/Bug List, enhancement, P4)

Description

[Andreas Franke (gone)]

On [e-mail], it has been proposed to find a more intuitive syntax
for some useful features from QuickSearch (see
<http://bugzilla.mozilla.org/quicksearch.html> and
<http://bugzilla.mozilla.org/quicksearchhack.html>) and add them to the fulltext
search textbox (see <http://bugzilla.mozilla.org/query.cgi?format=specific>) 
which was introduced in bug 145588.  Provided that most useful features are
preserved, this step could render QuickSearch obsolete.

The goal of this bug is to answer the following two questions:
1. Which QuickSearch features are useful, and which aren't?
2. What is an intuitive syntax for them?

I have not split them into two separate bugs because I consider all features
offered on the full query form at http://bugzilla.mozilla.org/query.cgi to be
more or less useful, and the actual usefulness of a specific feature in a
textbox entry field can differ dramatically, depending on the availability of a
convenient syntax.

In an attempt to foster a systematic discussion in spite of these interrelated
questions, I will file sub-bugs for individual features.  (Of course, this way
there is a huge potential for conflicting syntax.  But if we try to find the
most intuitive syntax for each of them first and resolve conflicts later, we can
still drop some features if its intuitive syntax is already taken and no good
alternative can be found.)

Comment 1

[Dave Miller [:justdave]]

Google has a pretty good syntax that a lot of people are familiar with already...

that being if you want to search on a specific field, prefix the keyword with
that field name and a colon.

On Google for example, you can search on "link:http://www.bugzilla.org/" to see
what sites link to that url, or "site:www.bugzilla.org" to restrict the search
results to items found in that domain.

That would extend to our system very easily using "keyword:" "product:"
"component:" etc as prefixes.

Comment 2

[Andreas Franke (gone)]

Yes, but even the google syntax has many aspects. Prefixes are one of them; this
issue was going to be one of the sub-bugs. Others are e.g. the logical
connectives AND / OR / NOT, or escaping strings with special chars. If you give
me some time, I will file sub-bugs for each issue I have in mind. You will see
that my proposals are intended to be compatible with Google.

Comment 3

[Gervase Markham [:gerv]]

I suggest we try and avoid falling into the trap of making a language which can
describe any query you might want to do. IMO, this will unavoidably make it too
complicated to easily explain. Remember the "quick" in QuickSearch should apply
to learning it and remembering how to use it as well.

Gerv

Comment 4

[Jesse Ruderman]

The most important QuickSearch features for me are:

* By default, it searches product/component/whiteboard in addition to summary,
and searches keyword and URL fields when appropriate.
* By default, it does *not* search comments, so there are few bogus results.
* Disjunction for synonyms: "url,address,location".
* All-words-as-substrings search: "bookmark", "auto compl".
* Fields: "votes:2".

Things I find "arcane" about QuickSearch:

* Symbol prefix shortcuts (:product, @owner, !keyword, #summaryword).
* The "ALL" prefix.  I think there should be a radio group outside of the
textbox: "All", "Open bugs and their duplicates", "Open".  When I want some
combination of resolutions not in that list, I can type "resolution:fixed" or
"res:inva,wont".
* It complains when I leave off a closing quote instead of inferring it at the
end of the search string.

I like some other shortcuts in QuickSearch, like "enh" standing for
"severity:enh", but I could live without that feature.

I don't like http://bugzilla.mozilla.org/query.cgi?format=specific 's default of
full-text searches because:

* With format=specific, I get 200 results no matter what I search for, so if my
bug isn't in the top 5, I don't know whether I need to expand my query, restrict
my query, add synoyms, read through the entire list, or give up searching.
* It feels like Altavista before Google: you have to + each term to get useful
results.  (4% of *Google* referers to www.squarefree.com yesterday had + in
front of every term!)

I think Bugzilla's most prominent search mechanism should be a QuickSearch-like
textbox plus:

* a radio group: "Open", "Open bugs and their duplicates", and "All".
* a dropdown for choosing sorting method, with "rank based on fulltext" as one
option.
* a checkbox for grouping by resolution ("before" sorting).
* a simple syntax for doing a fulltext search on a single word or phrase, since
that's occasionally useful.  Maybe "any:" or "fulltext:" in addition to "desc:"
(which should be renamed to "comment:").
* the hints at http://www.squarefree.com/bugzilla/quicksearch-help.html.
* a hint for how and when to use fulltext search.

Comment 5

[Andreas Franke (gone)]

Gerv: Let's assume that the quicksearch documentation at quicksearch.html has
been replaced with http://www.squarefree.com/bugzilla/quicksearch-help.html ,
ok?  Now the problem is that we can't go back in time and undo the design of
quicksearch since that happened a few years ago.  We can only ignore it, or look
at it and pick the good things and throw away the bad ones.

I don't share your skepticism about powerful query languages. For one, I think
Jesse's page *is* an easy explanation of quicksearch.  If you think it isn't,
maybe you can point out the problems with it?  Secondly, it seems to me that the
generalized, more powerful features that were added late in the quicksearch
development (like component:foo,bar,baz) are generally considered much more
intuitive than (not so powerful) arcane symbol shortcuts from the early days.
And last but not least: in the dependent bugs I have given a fairly complete
description of the Quicksearch features.  Comparing the complexity of the Google
search features with Quicksearch, it does not occur to me that Google is
significantly less complex.  You may disagree with this statement, but if you
don't, then you are saying that Google search isn't easy to explain, aren't you?

My hypothesis is that the real challenge is to integrate quicksearch with
fulltext search, which can be extremely useful as a computationally inexpensive
way to look up bugs where a certain word occurs in the description/comments,
and for sorting the results. In other words, the question is if fulltext search
can be made to play nice with the rest of the bugzilla query.cgi interface.

Jesse: Inferring the closing quote is a WONTFIX because IMO the danger of a
wrong guess is too big to allow it to go unnoticed. For symbol prefix shortcuts,
see my comments in bug 235622. The status/resolution issue might indeed have a
lot of potential for improvements; note however that the space constraints on
the bugzilla front page were such that having a radio box was not an option at
that time at all, so everything had to be done in a textbox. This might be
different if quicksearch became another tab on the query page and every user
could set a pref for her favourite one.

Comment 6

[Myk Melez [:myk] [@mykmelez]]

>In other words, the question is if fulltext search
>can be made to play nice with the rest of the bugzilla query.cgi interface.

It already does for the most part; note its presence in the boolean charts on
the advanced search page.

Comment 7

[Gervase Markham [:gerv]]

afranke: I only suggested removing quicksearch because I didn't want the effort
of converting it to a page.cgi page, and thought we had lots of other search
mechanisms. I have no agenda: if people like it, it's fine with me if it stays.
Have fun redesigning it :-)

I do think it should be server-side, in Perl, though.

Gerv

Comment 8

[Andreas Franke (gone)]

If "content" search behaves just like another search option, then it is trivial
to integrate this into quicksearch.  One could e.g. introduce a new content:...
prefix, and/or recycle the # symbol shortcut for it so that  #mouse #wheel 
behaves as if you typed  mouse wheel  into the fulltext search form.  Replacing
the default behaviour (of terms without a prefix) with fulltext search would be
another option, of course, but then you'd have to deal with comment #4.

I can try and see if I can find some time in the next three months to revive the
code to make Quicksearch server-side and improve the docs.  But this is
orthogonal to the issue of making it more intuitive and integrating it with
fulltext search.  If fulltext search is going to be extended with some
Quicksearch-like features with the goal of replacing Quicksearch, then it would
be nice to know the price in terms of stripped functionality early in the
process.  In this case, I would like to know which features will be removed and
which features will have their syntax changed, as opposed to just reorganizing
and improving the docs.  If Quicksearch continues to be available somewhere as
an alternative to fulltext search, then this is a non-issue, of course.  But
having one well-documented and well-supported Google-like textbox search tool is
obviously better than having two slightly different ones with incompatible
feature sets and syntax.  Thus constructive criticism and suggestions are very
welcome.

Comment 9

[Myk Melez [:myk] [@mykmelez]]

Attachment: patch v1: basic implementation of intuitive syntax

Here's a basic implementation of enhancements for the fulltext search that give
it quicksearch-like capabilities.  With this patch applied, users can type
"field:value" into the "find a specific bug" text field (or any other "content
search" text field), and Bugzilla will add the criterion "<field> equals
<value> to the query.

Multiple space-separated instances of "field:value" are ANDed together, and
multiple comma-separated instances of "value" or ORed together, so a user can
say f.e. "severity:blocker product:Bugzilla,Webtools" to find Bugzilla and
Webtools blockers.

Field terms can be shortened, f.e. "pro:Bugzilla" is the same as
"product:Bugzilla".  Terms are localizable and obey the Bugzilla "terms" hash,
so users can search for "problemid:###" on an installation that calls bugs
"problems".

Comment 10

[Marc Schumann [:Wurblzap]]

Wouldn't it make sense to let this one be blocked by bug 70907, and then let the
two start using each other's syntax?

Comment 11

[Myk Melez [:myk] [@mykmelez]]

Attachment: patch v2: fixes issues with first patch

This version of the patch moves some of the code into buglist.cgi so that
script can accurately determine the value of $fulltext (content searches with
only triplets shouldn't set $fulltext to true).  It also uses the standard
module Text::SplitWords to accurately split words, valued-quoted triplets, and
triplets whose value contains the type (":").

>Wouldn't it make sense to let this one be blocked by bug 70907, and then let 
>the two start using each other's syntax?

On a technical level, the two features are independent, since neither depends
on the other to provide its functionality.  They're also targeted at different
audiences, since this feature focuses on a simple Google-like syntax that many
users will be able to learn and use.  And they also have different consequences
for Bugzilla's featureset, since the quicksearch patch is about internally
migrating the location of an existing feature's code, while this patch is about
adding a new feature.  So I don't think either patch should depend on the
other.

Comment 12

[Max Kanat-Alexander]

Comment on attachment 192886 [details] [diff] [review]
patch v2: fixes issues with first patch

OK, I have a few thoughts on this.

First off: I've always personally found quicksearch more accurate and useful than the fulltext search, mostly because I rarely care about comments (summaries are about 100x more relevant to me than comments) and because the relevance sorting makes it hard for me to find what I'm looking for. (Any search with more than 20 results is difficult to parse, mentally, anyhow.)

With that said, I have a few general architectural comments about the patch.

First, instead of changing the format of field_descs to add a terms item to each item, why not (a) make the "default term" just be a case-insensitive version of the field name without spaces and (b) have a field_terms hash to add additional ways that a user could refer to a field.

Also, I really am not too fond of moving more code into buglist.cgi -- would it be somehow possible (even if perhaps in another bug) to instead move the relevant buglist code into Search.pm?

The field-descs.pl is a good idea. In fact, perhaps it should even contain a constant and go into Bugzilla::Constants somehow. At the least, it would be nice if we could have a way to make it easily and generally available to all of Bugzilla, since we've needed something like that for a long time. Of course, ideally, the CGI code would never need to know anything from the templates.