1501848 - (bmo-aggressive-stance) Adopt a more aggressive stance to spiders and bots

Status: NEW

(bugzilla.mozilla.org :: General, enhancement)

Description

[Dylan Hardison [:dylan] (he/him)]

I couldn't find a good tracker for this, and I've spent a fair bit of research into this so here we are.


1) The IP blocked page needs to recapture its previous speed. Since we made it not be a blank page, it regressed 0.030 seconds on average, it needs to be more like 0.00 as it was before.
2) The IP blocked page should use a recaptcha to allow people / employees to get around it. Alternatively, we could prompt for 2fa?
3) To discourage spammers, we should be more actively limiting those accounts (bug 1501834 does this). A lot of the non-search-engine traffic has been people downloading all the bugs to build a corpus to make spam comments with links to various sites.
4) In addition to IP-based blocking, we should do user-agent based blocking.

Comment 1

[Kohei Yoshino [:kohei]]

Is it probably time to reconsider hiding email addresses in API responses from logged out users (or spammers, I mean) ?

Comment 2

[Dylan Hardison [:dylan] (he/him)]

Doing this in a non-breaking way is really hard (in a way that won't break legit API consumers). The best I think we can do is allow people to set their login_name to the new nick field, opt-in, and also have all new users do the same. 

Right now the sort of spam we're seeing is all about putting links on bmo, luckily.

Comment 3

[Dylan Hardison [:dylan] (he/him)]

(to do this means I need to get around adding the 'emails' table, that maps profile to one or more email addresses)