Closed Bug 1410308 Opened 8 years ago Closed 7 years ago

pastebin.mozilla.org's robots.txt is set to Disallow, but doesn't whitelist the Internet Archive

Categories

(Websites :: Other, defect)

Product:
Websites
Component:
Other
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED INCOMPLETE

People

(Reporter: ranking.purpose.7z, Unassigned)

Details

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0 Build ID: 20170614204913 Steps to reproduce: Hi! Pastebin.mozilla.org's robots.txt is set to Disallow, but doesn't whitelist the Internet Archive (a nonprofit digital library: https://archive.org/). This prevents saving or browsing the pages through their library — please fix this! Additionally, it would be good to upload data dumps of all pastes as they come in to the Internet Archive, so they have the complete set (otherwise, it relies on individuals hitting the "save page" button and non-comprehensive scraper robots to find things, so it's likely to miss things). Thanks!
Status: UNCONFIRMED → NEW
Ever confirmed: true
Pastebin was shut down. https://wiki.mozilla.org/Pastebin
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → INCOMPLETE
You need to log in before you can comment on or make changes to this bug.