Add more flags controlling Akismet submission

RESOLVED FIXED

Status

RESOLVED FIXED
3 years ago
2 years ago

People

(Reporter: jwhitlock, Assigned: jwhitlock)

Tracking

({in-triage})

Details

(Whiteboard: [specification][type:change])

(Assignee)

Description

3 years ago
What feature should be changed? Please provide the URL of the feature if possible.
==================================================================================
There are currently three flags controlling Akismet submission:

* spam_submissions_enabled - Controls if SPAM and HAM buttons are available on revision dashboard
* spam_check_enabled - A site-wide flag to control checking of edits (maybe should be a switch)
* wiki_spam_exempted - A targeted flag to control exemption of checking for edits.

After discussion with Akismet engineers, some additional flags should be implemented:

* spam_admin_mode - Edits are submitted to Akismet, but configured to always return "not spam". Errors do not block saving the edit. Useful for staff and trusted writers.
* spam_spammer_mode - Edits are submitted to Akismet, but configured to always return "spam". Useful for testing system response to spam edits.
* spam_testing_mode - Edits are submitted to Akismet, but a "spam" determination does not block the edit.

What problems would this solve?
===============================
"wiki_spam_exempted" is used for trusted writers and staff, but means that Akismet does not see the content and behavior of known good writers.  This will result in a higher false positive rate.

"spam_check_enabled" is used to turn off Akismet tracking completely.  "spam_check_enabled" plus "spam_testing_mode" would allow tracking production edits with Akismet, for the purpose of training on content and reducing the false positive rate.

Who would use this?
===================
Contributors, staff content reviewers, and QA testers

What would users see?
=====================
This assumes "spam_check_enabled" is True, and "wiki_spam_exempted" is False.

Users with "spam_admin_mode" turned on would always have edits accepted. If Akismet is unavailable, the error would not prevent their edit from being submitted.

Users with "spam_spammer_mode" turned on would always have their edits flagged as spam.

Users with "spam_testing_mode" turned on would have their edits accepted, but admin-visible triggers, like "new spam" emails, would still be generated if the edit was judged to be spam by Akismet.

What would users do? What would happen as a result?
===================================================
With "spam_testing_mode" on, Contributors could continue making edits, but not be blocked while we train Akismet. When the false positive rate is acceptable, we could flip "spam_testing_mode" to False, for all or for a subset of users.

Staff would measure the effectiveness of Akismet, submit training data, and confidently turn off "spam_testing_mode".  Trusted writers could be added to "spam_admin_mode", so that their content is never blocked.

QA staff could temporarily turn on "spam_spammer_mode" for their account, to measure the system's response to spam without guessing what Akismet considers spam.

Is there anything else we should know?
======================================
If this works as planned, "wiki_spam_exempted" will be deprecated.
(Assignee)

Updated

3 years ago
Blocks: 1188029
(Assignee)

Comment 1

3 years ago
We've had mixed success in manually changing user names to "viagra-test-123". My guess is this is due to caching of "wiki_spam_exempted" flags, which can happen in javascript, sessions, and other places. This should be tested with these changes, and a new bug opened if it continues.
(Assignee)

Comment 2

3 years ago
I'm listing the requirements in another way, to determine better names for the flags.

The existing flag names stay the same:

* spam_submissions_enabled - Controls if SPAM and HAM buttons are available on revision dashboard
* spam_check_enabled - A site-wide flag to control checking of edits (maybe should be a switch)
* wiki_spam_exempted - A targeted flag to control exemption of checking for edits.

The new flag names are:

* spam_admin_override - Sets "role=administrator" in Akismet submission, forcing ham determination
* spam_spammer_override - Sets "username=viagra-test-123" in Akismet submission, forcing spam determination
* spam_testing_mode - Sets "is_test=True" in Akismet submission
* spam_training_mode - Submits to Akismet and records the result, but does not block edits when spam is detected or errors are raised.

Here are the use cases and the related flags:

Case 1: Normal user's edits are be sent to Akismet. The content may be good or bad. Edits should be submitted as normal, and a "spam" determination blocks the edit. If Akismet returns an error, the edit is also blocked. This is the "normal" path.

Flags set:
* spam_check_enabled

Case 2: Edits are sent to Akismet, but spam and errors do not block saving edits. This allows training Akismet and measuring the false positive / false negative rates, and builds staff confidence in Akismet.

Flags set:
* spam_check_enabled
* spam_training_mode

Case 3: Trusted writers need to be able to submit content. The content should be good, so it is useful for Akismet training as well. Edits should be submitted with user_role=administrator. If Akismet returns an error, we also allow the edit.

Flags set:
* spam_check_enabled
* spam_admin_override
* spam_training_mode

Case 4: Staff is performing manual tests of the editing system's response to spam blocks. They need Akismet to identify the content as spam (username=viagra-test-123, is_test=True)

* spam_check_enabled
* spam_spammer_override
* spam_testing_mode

Case 5: Integration tests are being run against the server. The test account needs to create content without being blocked as spam. Either Akismet needs to be skipped entirely, or it needs to allow the content but not record it as useful content (role=administrator, is_test=True)

Flags set:
* spam_check_enabled
* spam_admin_override
* spam_testing_mode

Case 5: Integration tests are being run against the server. The test account needs to create content and for it to be blocked as spam by Akismet.  This is done by setting the username to viagra-test-123, and setting is_test to True.

* spam_check_enabled
* spam_spammer_override
* spam_testing_mode

Case 7: Akismet is broken, for example because their server is down. Normal user's edits are always being blocked. In this case, we want to skip Akismet checks entirely, and use other methods (such as closing down new accounts) to mitigate spam.

Flags set: None

Note, the wiki_spam_exempted flag is never set in these scenarios.
Assignee: nobody → jwhitlock
Keywords: in-triage

Comment 3

3 years ago
Commits pushed to master at https://github.com/mozilla/kuma

https://github.com/mozilla/kuma/commit/36f07ce4ff406178e6efadb702a2954779d6611e
bug 1259233 - Switch to py.test assert style

In preparation for adding additional tests, convert form spam tests from
unittest self.assertX style to py.test assert style.

https://github.com/mozilla/kuma/commit/b5cd99020aaae0e1fc38207ff656f2780287d2c4
bug 1259233 - Add helper for Akismet check tests

https://github.com/mozilla/kuma/commit/8502d54aa752364d09cc9de1066fef87c3b61636
bug 1259233 - More Akismet waffle flags

Add some generic flags to allow per-user alternate Akismet payloads:
* spam_admin_override - Set "role=administrator", never spam
* spam_spammer_override - Set "username=viagra-test-123", always spam
* spam_testing_mode - Set "is_test=True", don't train on content

And one per-user flag to control the wiki editing response:
* wiki_spam_training - Don't block save on spam or Akismet errors

https://github.com/mozilla/kuma/commit/283f0fa3ddbedccc14c9157c1fa4bb8a532bcade
bug 1259233 - Update features document

Add the new waffle flags, and clean up existing switches and flags to
reflect the current code and production use.

https://github.com/mozilla/kuma/commit/e928dd499f1f40c4bc4c609c63b6e514084348b5
Merge pull request #3827 from mozilla/spam-flags-1259233

bug 1259233 - Add more waffle flags for Akismet tuning

r=jezdez
(Assignee)

Comment 4

3 years ago
The code is in production, next the flags need to be added to the database
Blocks: 1260253
(Assignee)

Updated

3 years ago
Blocks: 1264390
No longer blocks: 1260253
(Assignee)

Comment 5

2 years ago
Flags are added to staging and production databases. "spam_training_mode" became "wiki_spam_training", but otherwise comment #2 is still accurate.
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.