Closed Bug 1393145 Opened 7 years ago Closed 7 years ago

Add crawl-delay directive to robots.txt

Categories

(bugzilla.mozilla.org :: General, enhancement)

Product:
bugzilla.mozilla.org
Component:
General
Version:
Production
Type:
enhancement
Priority:
Not set
Severity:
normal

Tracking

()

Status:
RESOLVED FIXED

People

(Reporter: dylan, Assigned: umohm12)

Details

Attachments

(1 file)

PR
45 bytes, text/x-github-pull-request
dylan
: review+
Details | Review 
I think there's an instruction that can be added to robots.txt that causes reasonable search engines to crawl more slowly -- can you look this up and add it?

thanks!
Attached file PR — 

        Attachment #8900535 -
        Flags: review+

    
  
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED

    
  
Summary: See if we can add a directive/rule to robots.txt to get search engines to crawl more slowly → Add crawl-delay directive to robots.txt

          You need to log in
          before you can comment on or make changes to this bug.
        






  

    
  







  

    

      
Attachment

      

      
      
      
      
    

    

      

        

          

            
General

            

              Creator: 



            

            
Created: 

            
Updated: 

            
Size: