1537954 - Turn off Akismet checking of page changes

Status: RESOLVED WONTFIX

(developer.mozilla.org Graveyard :: User management, task, P2)

Description

[Julien G]

What feature should be changed? Please provide the URL of the feature if possible.

The Askimet spam detection which is currently in use for spam classification.
The rate of false positives for spam attempts on localized content (newly created or existing) is too high. If possible, one should tweak Akismet to reduce this rate.

What problems would this solve?

Getting a bad noise vs. signal idea about spam attempts.
Decreased motivation about spam handling.
Decreased quality of spam handling including en-US content

Who would use this?

Spam handlers (Janet and I, maybe others)

What would users see?

Nothing

What would users do? What would happen as a result?

Impacted users would spend less time classifying ham attempts.

Is there anything else we should know?

I don't have time to build stats at the moment but
/documentspamattempt/ and https://developer.mozilla.org/en-US/dashboards/spam could help.

Currently pondering if it's still worh to classify such "attempt": training Akismet does not seem to do anything.

Comment 1

[Florian Scholz (Open Web Docs)]

John, can you update us on akismet settings?

Comment 2

[John Whitlock [:jwhitlock]]

TL;DR - We are not using Akismet effectively, so we shouldn't expect to see it improve with more data. I'd support turning off Akismet tracking, until we notice another automated spam attack. We may need a few round of changes to retain user banning and bulk revert.

Akismet is designed to block comment spam, for example on a blog. For example, let's say you have a blog about interesting developments in the web standards process. You would expect the comments to mention CSS features and browser vendors, include sample CSS, and link to demonstration web sites. You would not expect them to mention discount medicines, legal services, or inheritance from unknown distant royalty. The key is that "good" comments will tend to use certain phrases, and "bad" comments will use other words, perhaps ones that are being posted to many blogs. Akismet uses the words in the content as a signal, as well as other information.

However, if your blog is about medical discounts, your accepted word profile would be different. Training helps Akismet learn about the expected comments for your site.

Reviewing the recent spam submissions on MDN, we are using "Spam" to mean "I didn't like this change". For example, if a change just appears to add or remove whitespace, we mark as spam. If a change uses a <pre> section instead of a <div>, it's spam. Layout change? Spam. We are driving Akismet slowly insane as it tries to interpret what we are telling it is important or not important to us.

In order for Akismet to prevent automated spam, identification of spam needs to be limited to the addition of irrelevant advertising content. A change that is only whitespace is not spam, it's a possible unfortunate side effect of editing. A change to style is not spam, it's a violation of page style consistency. Adding a link to your blog article about a topic isn't spam. Adding a link to your related product may not be spam either, just against our policies. We didn't sufficiently train people as we gave them the permissions to revert content and mark spam, so we now have a useless spam filter with too many false positives.

I've recently learned some useful terminology, precision and recall for the fitness of binary classifiers (such as spam/no spam). They are more useful than other methods when the positive case (spam) is rare (less than 1% of edits).

Precision is the chance that, if the classifier calls a revision spam, then it is spam. When the precision is high, we'd feel comfortable automatically blocking revisions identified as spam. When it is low, we'd risk blocking many good changes. It is the ratio between true positives (the classifer says it is spam, and it is spam) and all positives (the classifier says it is spam, and it may be right or wrong). For the last 90 days, we report 9 blocked spam (true positives) and 202 blocked ham (false positives), for a precision of 9 / (9 + 202) = 4%. We're right to not automatically block edits.

Recall is the chance that, if a revision is spam, then the classifier will identify it as spam. If our goal is to avoid all spam, then a high recall is useful. Airport security is an example of a system optimizing for high recall. The measure is the ratio between true positives (the classifer says it is spam, and it is spam) and all relevant elements (it is spam). For the last 90 days, we report 9 blocked spam (true positives) and 34 published spam (false negatives), for a recall of 9 / (9 + 34) = 21%. We've got lousy recall as well, but I'm not certain that all those "published spams" are actually spam, or just whitespace edits, style changes, etc.

A good classifier balances both. A simple classifier that always returns "it is spam" would have 100% recall and a positive precision. The F score is a way to combine into two measurements, and is 2 * (precision * recall) / (precision + recall). Our F-score is 0.07, where 1.0 would be a perfect classifier. This matches our experience, that Akismet is not really helping distinguish spam from non-spam, even after a few years of training.

For this reason, I'd support turning it off.

There's a lot of feature toggles for spam. We have the switch wiki_spam_training on, which means that we send changes to Akismet, but do not block publishing of the content. The next step is to turn off the flagspam_submissions_enabled, so that the spam/ham buttons will go away, and the flag (which should be a switch) spam_checks_enabled, to disable Akismet checking of content.

Alternatively, we could restrict spam_submissions_enabled to a small number of staff (2-3 people) that are trained to know the consequences of marking an edit as spam, and will use it appropriately. We may ask Akismet to reset our learned profile, to clear out the garbage, since we've probably trained Akismet that HTML markup and non-English edits are spam.

In either case, the "ban user" workflow, along with bulk reverting, should remain. It may have been written to assume that spam submission is available, so it may need adjustments for when it is turned off.

Comment 3

[Julien G]

Thanks a lot for both this detailed explanation and the corresponding metrics.

I too am in support of turning it off.

Comment 4

[BMO Automation]

MDN Web Docs' bug reporting has now moved to GitHub. From now on, please file content bugs at https://github.com/mdn/sprints/issues/ and platform bugs at https://github.com/mdn/kuma/issues/.