81920 - Bugzilla is search-engine unfriendly

Status: RESOLVED WONTFIX

(Bugzilla :: Documentation, enhancement, P3)

Description

[Emlyn Shannon]

Most search engine "spider" programs will not follow a link which to a CGI 
script, particularly if GET data is given  (eg show_bug.cgi?id=xxxxxx), 
because of the increased possibility that the link will not work in future. 
Search engines tend to avoid dynamic pages in general. 

It may be desirable to have all Bugzilla bug reports - the pages that are 
served by show_bug.cgi indexed by the search engines. Some search 
engines are faster than the bugzilla search, and allow for a more 
generalised search of everything, rather than requiring particular 
parameters.

There are certain things that can be done to allow search engines to index 
and file a particular page. One of them is the alias or map a fake set of 
.html URLs to a cgi script.

For instance, you could instutute a convention to refer to the page 
shaowing any particular bug by 

http://bugzilla.mozilla.org/show_bug/xxxxx.html

Since this page doesn't exist, bugzilla's 404 errordocument will be 
invoked. If that errordocument is itself a CGI script, then it can parse the 
URL given to find the bug #, then output what you would normally get from 
show_bug.cgi?id=xxxxx.

If all links and references to bugs are then changed to using the 
imaginary html files, search engines will eventually crawl all across 
bugzilla and cache and index it all.

I'm not sure how big a re-write this would be, for questionable gains - I 
know how difficult it would be in php (not very) but bugzilla doesn't use 
php...

Comment 1

[Dave Miller [:justdave]]

-> Bugzilla product

This would be a piece of cake to due with an Apache rewrite rule.  Since we
can't set up rewrite rules as part of Bugzilla it'd have to be a docs thing.

RewriteRule ^/show_bug/(\d+)\.html$ /show_bug.cgi?id=$1

Comment 2

[Zach Lipton [:zach]]

Why can't we generate the rewrite rule when we create .htaccess files?

Comment 3

[Jacob Steenhagen]

The real question is, would you want search engines indexing your pages?  I'm
reasonably sure that mozilla.org would not simply because of the sudden server
load a spidering program would put on the server.  In fact, that's the main
reason bugzilla ships w/a robots.txt configured to tell all robots to "keep away".

Of course, if you want to subject your server to this flooding, it's certainly
something you could set up :)

Comment 4

[Dave Miller [:justdave]]

We are currently trying to wrap up Bugzilla 2.16.  We are now close enough to
release time that anything that wasn't already ranked at P1 isn't going to make
the cut.  Thus this is being retargetted at 2.18.  If you strongly disagree with
this retargetting, please comment, however, be aware that we only have about 2
weeks left to review and test anything at this point, and we intend to devote
this time to the remaining bugs that were designated as release blockers.

Comment 5

[Dave Miller [:justdave]]

The docs should always be up-to-date when we release.  Forgot to exclude
Documentation when I mass-retargetted.

Comment 6

[Dave Miller [:justdave]]

Barnboy changed his email address and opened a new account instead of having the
address changed on his existing one.  Reassigning all docs bugs to his new account.

Comment 7

[Gervase Markham [:gerv]]

Bugzilla should stay search-engine-unfriendly IMO. If someone wants to work out
and write up how to use a rewrite rule to make it not so, add the writeup here
and we'll document it. But there's no way we are doing it. :-)

Regardless, this is not a 2.16 blocker.

Gerv

Comment 8

[Gervase Markham [:gerv]]

Clearing out documentation bugs. Let's face it - we aren't going to fix this.

Gerv

Comment 9

[Dave Miller [:justdave]]

clearing target in DUPLICATE/WORKSFORME/INVALID/WONTFIX bugs so they'll show up
as untriaged if they get reopened.

Comment 10

[Benedikt Kantus]

*** Bug 238750 has been marked as a duplicate of this bug. ***