Doctor's robots.txt isn't giving the intended result



11 years ago
2 years ago


(Reporter: nelson, Assigned: reed)





(1 attachment)

Google searches for mozilla documenation should NEVER return pages on  But they do, all too often.  

The search feature on does a google search on 
servers.  When searching for mozilla documentation using that search, 
very often I get a page back where many of the google links point to  

For example, go to and in the search box enter

The output SHOULD include this page:

But often it does not.  Instead, it contains

Because google tries to suppress duplicate pages, it suppressed the page
on (which is the page users should see), and instead it 
gives the url for  This is really frustrating.

Please, let's keep out of google.
I suspect that a simple robots.txt file would solve the problem

Comment 1

11 years ago
erm, something's missing there.  It was supposed to say
"in the search box, enter SEC_ERROR_BAD_KEY and click Go."
It has one, added by bug 359129, it's just almost completely broken. Since /doctor.cgi is the root of *every* doctor URL, 

User-agent: *
Allow: /doctor.cgi
Disallow: /

works out to allowing everything, as far as Google is concerned - if you allow them a file, you allow them any query string for that file.

It's actually possible to block them from query strings, *if* we're only interested in Google: they treat

Disallow: /*?

as blocking any URL with a query string. However, they can still choose to include it, with no summary, if their algo decides for a particular query that the doctor URL is a better result (based on factors from pages linking to the doctor URL).

If the goal is to not have doctor URLs appear in Google results, rather than to not have Google crawl doctor, then robots.txt is the wrong tool: rather than keeping it out, we need to let it in, and include a noindex meta tag in every result that isn't the front page.
Assignee: server-ops → myk
Blocks: 359129
Component: Server Operations → Doctor
Product: → Webtools
QA Contact: justin → doctor
Summary: put robots.txt on → Doctor's robots.txt isn't giving the intended result

Comment 3

11 years ago
Why not having:

Disallow: /doctor.cgi
Since doctor.cgi is the index page, amusingly enough that works - Disallow: /doctor.cgi will actually allow doctor.cgi, as a request for /. However, it's very fragile - one link to and you're sunk, and it's crawled.

It's also not guaranteed to do what you want: if enough pages it trusts say that /doctor.cgi?file=foo/bar is the best page evar about "foo bar" then even though it hasn't crawled it, Google will still include it in results for that search. If doctor's robots.txt was there to avoid the database hit of crawling, I'd say the thing to do is add an index.cgi that only has the content you want indexed and a form that submits to doctor.cgi, and only Allow: /index.cgi, but as the tail end of says, if your goal is to not have the URLs appear in results, you instead have to let Google crawl them, so that it can see that it's not supposed to index them.
I signed up for the Google Webmaster Tools, and used their site preferences to request removal of the entire site.  I also changed the robots.txt file to Disallow: /
But that's only for our local copy, that does nothing for the one in CVS (in case of other people using it)

Comment 7

11 years ago
Created attachment 276312 [details] [diff] [review]
patch - v1

Remove doctor.cgi as an allowed file (already in production).
Assignee: myk → reed
Attachment #276312 - Flags: review?(timeless)


11 years ago
Attachment #276312 - Flags: review?(timeless) → review+

Comment 8

11 years ago
Checking in robots.txt;
/cvsroot/mozilla/webtools/doctor/robots.txt,v  <--  robots.txt
new revision: 1.2; previous revision: 1.1
Last Resolved: 11 years ago
Resolution: --- → FIXED
Product: Webtools → Webtools Graveyard
You need to log in before you can comment on or make changes to this bug.