15809 - buglist.cgi URL character length optimization

Status: RESOLVED FIXED

(Bugzilla :: Query/Bug List, enhancement, P2)

Description

[Matthew Tuck [:CodeMachine]]

At the moment buglist.cgi takes huge URLs.  Quite by accident I've noticed that
you can remove part of this and they'll take defaults.

It would be really nice is the default buglist.cgi URLs that query.cgi generates
were minimised.  This would make it easier to determine the nature of the query
from the URL, and also to quote the URL in things like bug reports and emails.

Comment 1

[Terry Weissman]

I would love to do this.  I don't think it's possible, given the way HTML forms
work.  Let me know if I'm wrong!  But meanwhile, I am marking this bug INVALID.

Comment 2

[Matthew Tuck [:CodeMachine]]

You could use a minimisation intermediate URL with redirect I imagine, although
would slow things down.

Maybe you could include at the bottom -

The shortform of this URL is
<A...>http://bugzilla.mozilla.org/buglist.cgi?...</A>.  This would at least
allow obtaining the short form for quoting purposes if not easy bookmarking.

Or I could say I suspect this could be done with JS (?), but then you might have
to kill me.

Comment 3

[Matthew Tuck [:CodeMachine]]

Maybe allow a user to have a preference saying whether they want this feature
done with a redirect, slightly slowing things down?  I would turn it on.

Comment 4

[Matthew T (active 1999-2002)]

I'd use this feature, too. Especially handy when you're trying to paste Bugzilla 
query URLs in e-mail/Usenet messages.

Comment 5

[Matthew Tuck [:CodeMachine]]

Reopening for reconsideration by Tara.  There are a number of possibilities that
might work:

(a) Javascript. (?)
(b) A redirect, or at least a preference to allow this for those who want it.
(c) A link on the bug list that does the minimisation .  At the very least it
could display the URL on the page, if not in the URL bar.

Comment 6

[Matthew Tuck [:CodeMachine]]

Reassigning ...

Comment 7

[Matthew Tuck [:CodeMachine]]

*** Bug 38489 has been marked as a duplicate of this bug. ***

Comment 8

[Jesse Ruderman]

Bug 38489 duplicates possibility #3.

Comment 9

[Jesse Ruderman]

See also bug 65391, "Give brief summary of bug list query".

Comment 10

[Matthew Tuck [:CodeMachine]]

One problem with this is that AIUI if you "reduce" a URL and share it, it could
give different results if the other person has a different default query.

Comment 11

[Jacob Steenhagen]

IIRC, it only loads the default query if no other parameters are passed to 
query.cgi, but I could be wrong so that's certainly worth looking into.

Comment 12

[Matthew Tuck [:CodeMachine]]

Moving to new Bugzilla product ....

Comment 13

[GavinS]

I'll attach a patch in a moment which provides a link containing a shortened
version of the URL for bookmarking/copying.


(And also, as an aside, query.cgi only seems to use the default query if there
are no parameters specified in the URL, as Jake suggested in an earlier comment.)

Comment 14

[GavinS]

Attachment: patch against 2.14 to provide a shortened URL to link/bookmark

Comment 15

[Gervase Markham [:gerv]]

Review comments:

This function can be about 1/3 the length by:

a) using a prefix relationship on the params to be deleted.

@list = ('short_desc', 'long_desc', ...)

foreach @list
  if !defined($useful_params{$_})
    remove all members of $useful_params beginning with $_

Then, you can delete multiple entries from $useful_params using a list.

+        $url .= "?";

You can add that unconditionally because it'll get chopped if it's all there is.

+        my $aref = $M{$key};
+        if (defined($aref)) {
+            if (scalar(@$aref) > 0) {
+                foreach (@$aref) {
+                    $url .= "$key=" . url_quote($_) . "&";
+                }
+            }
+        }
+    }

I'm sure there's a better way to do this using map and join.

+    chop($url);
+
+    return $url;

Combine these.

Gerv

Comment 16

[Dave Miller [:justdave]]

We are currently trying to wrap up Bugzilla 2.16.  We are now close enough to
release time that anything that wasn't already ranked at P1 isn't going to make
the cut.  Thus this is being retargetted at 2.18.  If you strongly disagree with
this retargetting, please comment, however, be aware that we only have about 2
weeks left to review and test anything at this point, and we intend to devote
this time to the remaining bugs that were designated as release blockers.

Comment 17

[Brian 'netdragon' Bober]

*** Bug 148570 has been marked as a duplicate of this bug. ***

Comment 18

[Gagan]

Wow. I didn't realize there was such an old bug for this already (till myk cc'd
me on an email refering to this bug) Anyhow... my idea for fixing this was that
we could at least add some minimal javascript at the top of the query page that
looked at all the fields marked "dirty" (from catching onchange of all fields)
and just added those as the query fields in a constructed URL. It could "on
Submit", do a document URL replace for that constructed URL (which hopefully
would be a heck of a lot smaller than the default) Makes sense? 

FWIW an year back (or maybe more) I had written some test to see if there was a
difference in bugzilla's response time for long or unedited version versus
condensed versions. Not surprisingly the smaller queries did shave off some
seconds. So this bug may very well help improve bugzilla performance as well. 

Comment 19

[Brian 'netdragon' Bober]

Gagan: Personally, I'd like the full form data to be shown by default so you can
manually edit it. It would be nice to have a link or button at the top of the
page that says "Create linkable URL" or "Strip unused header fields".

I would be ok, I guess, with :-/ the reverse too, if need be, where we strip by
default and add the extra headers if they ask for it.

> Not surprisingly the smaller queries did shave off some
> seconds. So this bug may very well help improve bugzilla performance as well.

I'm not sure, but that seems to be a performance issue that can (Can't test that
because I don't have a bugzilla up atm) be solved without fixing this bug, and
IMHO should be if that is the case. A field such as:
&order=& should intuitively not slow down the CGI because it should do the same
thing as if it didn't exist.

Comment 20

[Gagan]

> I'd like the full form data to be shown by default so you can
> manually edit it. 

Well... that's what the Bugzilla query page is for. You are saying you prefer
editing the URL by hand more than going to the page and editing the fields? How
many other people like to do that? 

> A field such as: &order=& should intuitively not slow down the CGI because it 
> should do the same thing as if it didn't exist.

But it does. I'm no expert at server stuff but my thinking is that it still has
to be parsed to be considered empty/no-op and that work over several fields and
over several queries probably starts to add up.

Comment 21

[GavinS]

Comment on attachment 51856 [details] [diff] [review]
patch against 2.14 to provide a shortened URL to link/bookmark

Not ready for review -- needs updating to cvs tip

Comment 22

[GavinS]

Comment on attachment 51856 [details] [diff] [review]
patch against 2.14 to provide a shortened URL to link/bookmark

Not ready for review -- needs updating to cvs tip

Comment 23

[Dave Miller [:justdave]]

*** Bug 196356 has been marked as a duplicate of this bug. ***

Comment 24

[Jesse Ruderman]

sirLurxalot wrote a bookmarklet that does this:
http://www.squarefree.com/bookmarklets/mozilla.html#shorten_bug_query

Comment 25

[Joel Peshkin]

These bugs appear to be abandoned.  Retargeting to 2.20

Comment 26

[Dave Miller [:justdave]]

Bugzilla 2.20 feature set is now frozen as of 15 Sept 2004.  Anything flagged
enhancement that hasn't already landed is being pushed out.  If this bug is
otherwise ready to land, we'll handle it on a case-by-case basis, please set the
blocking2.20 flag to '?' if you think it qualifies.

Comment 27

[:glob ✱]

Attachment: javascript v1

here's a first cut of a javascript solution to this problem.

it's not ready for review as there's a lot of simliar tests that i'd like to
change to a lookup from an array/hash.

there's also a problem where if your query is optimised to nothing, editing the
query resets it back to the default.

Comment 28

[Myk Melez [:myk] [@mykmelez]]

Note that bug_status=NEW&bug_status=ASSIGNED can be shortened to
bug_status=NEW,ASSIGNED.

Comment 29

[Joel Peshkin]

That works suprisingly well

FWIW, I tried it in Mozilla with and without JS enabled and also in IE5. 
Everything I tried worked well until I added another boolean chart.

http://basebasebase/query.cgi?query_format=advanced&bug_status=NEW&bug_status=REOPENED&negate0=1&field0-0-0=component&type0-0-0=notregexp&value0-0-0=deq2fd&cmd-add0-0-1=Or&cmd-add0-1-0=And&cmd-add1-0-0=Add%20another%20boolean%20chart&negate1=1&field1-0-0=votes&type1-0-0=lessthan&value1-0-0=5&cmd-add1-0-1=Or&cmd-add1-1-0=And&cmd-add2-0-0=Add%20another%20boolean%20chart#chart

Comment 30

[Andrew Pinski]

It would be nice to get this bug fixed, as it is a pain in the but to share query URLS over IRC or any 
other chat.

Comment 31

[Frédéric Buclin]

The trunk is now frozen to prepare Bugzilla 2.22. Enhancement bugs are retargetted to 2.24.

Comment 32

[Matthias Kienle]

I don't understand why you would use javascript or redirects. The problem (so as i understand) is that the url string is huge because you use the get methode and all parameters are in the url. I would suggest to use the post methode instead of the get methode or why isn't this an allowed solution for this bug

Comment 33

[Jesse Ruderman]

Simply using POST would break bookmarking, reloading, and sharing URLs of buglists.

Comment 34

[:glob ✱]

another approach, which is probably better, is to use POST, but provide a "bookmarkable link" on the search results page that contains the optimised form.

Comment 35

[Frédéric Buclin]

We are freezing the code for 3.0 in two weeks and we don't expect this bug to be fixed on time.

Comment 36

[Jens Petersen]

As a first step how about at least adding a cleaned-up link somewhere for
the resulting returned page.

Comment 37

[Markus Keller]

(In reply to comment #28)
> Note that bug_status=NEW&bug_status=ASSIGNED can be shortened to
> bug_status=NEW,ASSIGNED.

I guess this shortening would already solve the problem in most cases. This trick was the only way I found that allowed me to save a query with a lot of &target_milestone= segments (to find all open bugs for past milestones).

Comment 38

[Paul]

It would be nice if we kept records of queries run.  Possibly storing the query in a table via POST data and getting a short URL that references it is the best possible solution.

Comment 39

[Emmanuel Seyman]

Attachment: trimming the query part of a buglist

Taking.

Comment 40

[Max Kanat-Alexander]

Comment on attachment 322174 [details] [diff] [review]
trimming the query part of a buglist

Nooooo, don't make this spaghetti code inside of buglist.cgi.

Did you notice the already-existing Bugzilla::CGI::clean_search_url?

Also, this doesn't accomplish the intention of this bug, which is to trim the URL that is submitted to the *browser* by query.cgi.

Comment 41

[Emmanuel Seyman]

(In reply to comment #40) :
>
> Also, this doesn't accomplish the intention of this bug, which is to trim the
> URL that is submitted to the *browser* by query.cgi.

I'm seeing two different requests in this bug (and comments from other bugs describing this one).

1) Trim the URL upon submit
- Uncheck options that apply to other options that haven't been set (e.g. email*1 for email1)
- Uncheck selections that don't change the result (e.g. deselect the products if all of them have been selected).

2) Trim the URL further (with stuff like bug_id=1,2 instead of bug_id=1&bug_id=2) server-side (because it can't be done client-side, to the best of my knowledge but I'ld love to be proven wrong).

Is this one, the other or both ?

Comment 42

[Matthew T (active 1999-2002)]

Either of those approaches would help in the overall goal of making Bugzilla search result URLs easier to share. But approach #1 -- stripping entire unused parameters -- probably would produce the greatest average URL shortening, since it's more common not to use a parameter than to enter multiple values for a parameter.

Comment 43

[Emmanuel Seyman]

(In reply to comment #42)
> striping entire unused parameters -- probably would produce the greatest
> average URL shortening

I don't know that we can strip parameters.
We can definitely unset their value(s), though.

> it's more common not to use a parameter than to enter multiple values for a
> parameter.

I ran into this bug when a user wanted to search across the whole database,
selected all 200 products and hit an apache error (the URL was too long).
I'ld say selecting all products or all components must be a somewhat common thing to do.

Comment 44

[Emmanuel Seyman]

Attachment: combine multiple params, clean up the url and paste a short link in the results

I'm a bit stuck on this one.
I don't see any parameters whose value I would feel comfortable removing (the only one that comes close is "order").

This patch deselects menus if all the values inside have been selected and,
on the server side, generates a minimised url for which a link is given on
the results page.

I'ld appreciate comments.

Comment 46

[miketosh]

it probably doesn't help this issue that some of the links, such as 'change columns', include both the &<fieldname>=value, as well as &field-1-0-0=<fieldname>&value-1-0-0=value.  Those links should ignore either the -1 boolean chart, or the declared values.

Comment 48

[Mikhail Gusarov]

Attachment: Cleanup the search URL by redirect. v1

Last patch in this bug had no review flags, so here's implementation from (duplicate) bug 453707.

There is already cleanup_search_url method, so let's use it.

Comment 49

[Max Kanat-Alexander]

Comment on attachment 336923 [details] [diff] [review]
Cleanup the search URL by redirect. v1

Yeah, actually, this looks find to me. I have to think a tiny bit about two things, though:

1) Is POST really a good idea here?

2) Do we really want to do a temp redirect there? I don't want the POST page to appear in the browser's History--will it?

Comment 50

[Mikhail Gusarov]

(In reply to comment #49)

> 1) Is POST really a good idea here?

The other idea is the hidden field "needscleanup".

> 2) Do we really want to do a temp redirect there? I don't want the POST page to
> appear in the browser's History--will it?

Tested in FF3 and Opera - it doesn't. However with GET and hidden field it may.

Comment 51

[Max Kanat-Alexander]

Comment on attachment 336923 [details] [diff] [review]
Cleanup the search URL by redirect. v1

Okay, this is great. :-) I tested it, and it really works nicely.

I also have another patch that makes clean_search_url even better, and I'll attach it to a separate bug.

Comment 52

[Max Kanat-Alexander]

Thanks, Mikhail! :-) It's funny, about a month before you posted this patch, I had that exact idea on how to fix this, though I hadn't thought of the POST. I think I said something about it in IRC, maybe, but I'm not sure I ever wrote it down anywhere.

Checking in buglist.cgi;
/cvsroot/mozilla/webtools/bugzilla/buglist.cgi,v  <--  buglist.cgi
new revision: 1.384; previous revision: 1.383
done
Checking in template/en/default/search/search-advanced.html.tmpl;
/cvsroot/mozilla/webtools/bugzilla/template/en/default/search/search-advanced.html.tmpl,v  <--  search-advanced.html.tmpl
new revision: 1.33; previous revision: 1.32
done

Comment 53

[miketosh]

Max, please reference that related bug number here for an improved clean_search_url.  I, like others, may want to backport these fixes to solve our existing problems.  :-)

Comment 54

[miketosh]

This isn't exactly complete though.  The links provided on the pages (Change Columns, etc) still contain the negative-numbered boolean searches, which can double or more than double the URL length.  Is there any way to also improve these links?

http://server/colchange.cgi?bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&chfieldto=Now&emailassigned_to1=1&emailassigned_to2=1&emailcc2=1&emailreporter2=1&field-1-0-0=product&field-1-1-0=bug_status&product=bugzilla&query_format=advanced&remaction=&type-1-0-0=anyexact&type-1-1-0=anyexact&value-1-0-0=bugzilla&value-1-1-0=NEW%2CASSIGNED%2CREOPENED&query_based_on=

Comment 55

[miketosh]

Since I'm getting myself so involved...

Adding the following line to the canonicalise_query in Bugzilla/CGI.pm will remove the negative-numbered boolean charts from the links to solve the other URL length problem:

--- CGI.pm.old  2008-09-12 09:56:02.388007334 -0400
+++ CGI.pm.new  2008-09-12 09:55:47.710174516 -0400
@@ -120,6 +120,7 @@
     foreach my $key (sort($self->param())) {
         # Leave this key out if it's in the exclude list
         next if lsearch(\@exclude, $key) != -1;
+        next if ($key =~ /-\d-\d-\d/);
 
         my $esc_key = url_quote($key);

Comment 56

[Max Kanat-Alexander]

(In reply to comment #54)
> The links provided on the pages (Change Columns, etc) still contain 
> the negative-numbered boolean searches

  Those aren't negative-numbered, those are valid boolean charts. You're talking about some other bug entirely--please report it as a separate bug.

Comment 57

[Max Kanat-Alexander]

OH, and the other bug is bug 454892.

Comment 60

[Max Kanat-Alexander]

Added to the release notes for Bugzilla 3.4 in bug 494037.