742599 - Tinderboxpushlog (tbpl) should use a robots.txt

Status: RESOLVED FIXED

(Tree Management Graveyard :: TBPL, defect)

Description

[Justin Wood (:Callek)]

TBPL is serving *parsed* full logs for every build to google.

I don't see much value in this and it likely increases load on TBPL and our infra, I suggest we just make TBPL have a robots.txt and noindex/nofollow/etc. everything

It is likely finding these logs from pastes in orange bugs.

Comment 1

[(no longer active)]

Does it make sense to be indexed by search engines at all?  If not, we can add a robots.txt which disallows all bots.

Comment 2

[Justin Wood (:Callek)]

IMO, nothing on tbpl needs to/should be indexed. We can might want the site itself to be findable, but not sure what type(s) of google/etc searches would be good to find it with without making everything overcomplicated anyway.

Comment 3

[(no longer active)]

Attachment: Patch (v1)

Comment 4

[Justin Wood (:Callek)]

Comment on attachment 612442 [details] [diff] [review]
Patch (v1)

I don't feel I'm a capable/appropriate reviewer here, looks good though.

Unfortunately I'm not sure who owns/is-in-charge of TBPL, if you are a peer/owner feel free to treat that as r+ though

Comment 5

[Arpad Borsos [:Swatinem]]

Comment on attachment 612442 [details] [diff] [review]
Patch (v1)

Review of attachment 612442 [details] [diff] [review]:
-----------------------------------------------------------------

Typo in the filename: roboST.txt?

I actually have no idea how the robots syntax is, so I just trust you and Justin on that.

Comment 6

[(no longer active)]

http://hg.mozilla.org/users/mstange_themasta.com/tinderboxpushlog/rev/14656443a76d