Closed Bug 1196997 Opened 10 years ago Closed 9 years ago

Baiduspider is spidering production even though robots.txt disallows all

Categories

(Tree Management :: Treeherder: Infrastructure, defect)

Product:
Tree Management
Component:
Treeherder: Infrastructure
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED WONTFIX

People

(Reporter: emorley, Unassigned)

References

Details

Seen in: https://rpm.newrelic.com/accounts/677903/applications/4180461/traced_errors/4eb47b-9d49fb99-4779-11e5-97ea-b82a72d22a14 Maybe we need a specific Baiduspider entry in robots.txt, if it's ignoring the wildcard. Or else an apache block. (Although that doesn't help with Heroku)
Depends on: 1199046
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.