What feature should be changed? Please provide the URL of the feature if possible. ================================================================================== There are currently three flags controlling Akismet submission: * spam_submissions_enabled - Controls if SPAM and HAM buttons are available on revision dashboard * spam_check_enabled - A site-wide flag to control checking of edits (maybe should be a switch) * wiki_spam_exempted - A targeted flag to control exemption of checking for edits. After discussion with Akismet engineers, some additional flags should be implemented: * spam_admin_mode - Edits are submitted to Akismet, but configured to always return "not spam". Errors do not block saving the edit. Useful for staff and trusted writers. * spam_spammer_mode - Edits are submitted to Akismet, but configured to always return "spam". Useful for testing system response to spam edits. * spam_testing_mode - Edits are submitted to Akismet, but a "spam" determination does not block the edit. What problems would this solve? =============================== "wiki_spam_exempted" is used for trusted writers and staff, but means that Akismet does not see the content and behavior of known good writers. This will result in a higher false positive rate. "spam_check_enabled" is used to turn off Akismet tracking completely. "spam_check_enabled" plus "spam_testing_mode" would allow tracking production edits with Akismet, for the purpose of training on content and reducing the false positive rate. Who would use this? =================== Contributors, staff content reviewers, and QA testers What would users see? ===================== This assumes "spam_check_enabled" is True, and "wiki_spam_exempted" is False. Users with "spam_admin_mode" turned on would always have edits accepted. If Akismet is unavailable, the error would not prevent their edit from being submitted. Users with "spam_spammer_mode" turned on would always have their edits flagged as spam. Users with "spam_testing_mode" turned on would have their edits accepted, but admin-visible triggers, like "new spam" emails, would still be generated if the edit was judged to be spam by Akismet. What would users do? What would happen as a result? =================================================== With "spam_testing_mode" on, Contributors could continue making edits, but not be blocked while we train Akismet. When the false positive rate is acceptable, we could flip "spam_testing_mode" to False, for all or for a subset of users. Staff would measure the effectiveness of Akismet, submit training data, and confidently turn off "spam_testing_mode". Trusted writers could be added to "spam_admin_mode", so that their content is never blocked. QA staff could temporarily turn on "spam_spammer_mode" for their account, to measure the system's response to spam without guessing what Akismet considers spam. Is there anything else we should know? ====================================== If this works as planned, "wiki_spam_exempted" will be deprecated.
I'm listing the requirements in another way, to determine better names for the flags. The existing flag names stay the same: * spam_submissions_enabled - Controls if SPAM and HAM buttons are available on revision dashboard * spam_check_enabled - A site-wide flag to control checking of edits (maybe should be a switch) * wiki_spam_exempted - A targeted flag to control exemption of checking for edits. The new flag names are: * spam_admin_override - Sets "role=administrator" in Akismet submission, forcing ham determination * spam_spammer_override - Sets "username=viagra-test-123" in Akismet submission, forcing spam determination * spam_testing_mode - Sets "is_test=True" in Akismet submission * spam_training_mode - Submits to Akismet and records the result, but does not block edits when spam is detected or errors are raised. Here are the use cases and the related flags: Case 1: Normal user's edits are be sent to Akismet. The content may be good or bad. Edits should be submitted as normal, and a "spam" determination blocks the edit. If Akismet returns an error, the edit is also blocked. This is the "normal" path. Flags set: * spam_check_enabled Case 2: Edits are sent to Akismet, but spam and errors do not block saving edits. This allows training Akismet and measuring the false positive / false negative rates, and builds staff confidence in Akismet. Flags set: * spam_check_enabled * spam_training_mode Case 3: Trusted writers need to be able to submit content. The content should be good, so it is useful for Akismet training as well. Edits should be submitted with user_role=administrator. If Akismet returns an error, we also allow the edit. Flags set: * spam_check_enabled * spam_admin_override * spam_training_mode Case 4: Staff is performing manual tests of the editing system's response to spam blocks. They need Akismet to identify the content as spam (username=viagra-test-123, is_test=True) * spam_check_enabled * spam_spammer_override * spam_testing_mode Case 5: Integration tests are being run against the server. The test account needs to create content without being blocked as spam. Either Akismet needs to be skipped entirely, or it needs to allow the content but not record it as useful content (role=administrator, is_test=True) Flags set: * spam_check_enabled * spam_admin_override * spam_testing_mode Case 5: Integration tests are being run against the server. The test account needs to create content and for it to be blocked as spam by Akismet. This is done by setting the username to viagra-test-123, and setting is_test to True. * spam_check_enabled * spam_spammer_override * spam_testing_mode Case 7: Akismet is broken, for example because their server is down. Normal user's edits are always being blocked. In this case, we want to skip Akismet checks entirely, and use other methods (such as closing down new accounts) to mitigate spam. Flags set: None Note, the wiki_spam_exempted flag is never set in these scenarios.
Assignee: nobody → jwhitlock
Commits pushed to master at https://github.com/mozilla/kuma https://github.com/mozilla/kuma/commit/36f07ce4ff406178e6efadb702a2954779d6611e bug 1259233 - Switch to py.test assert style In preparation for adding additional tests, convert form spam tests from unittest self.assertX style to py.test assert style. https://github.com/mozilla/kuma/commit/b5cd99020aaae0e1fc38207ff656f2780287d2c4 bug 1259233 - Add helper for Akismet check tests https://github.com/mozilla/kuma/commit/8502d54aa752364d09cc9de1066fef87c3b61636 bug 1259233 - More Akismet waffle flags Add some generic flags to allow per-user alternate Akismet payloads: * spam_admin_override - Set "role=administrator", never spam * spam_spammer_override - Set "username=viagra-test-123", always spam * spam_testing_mode - Set "is_test=True", don't train on content And one per-user flag to control the wiki editing response: * wiki_spam_training - Don't block save on spam or Akismet errors https://github.com/mozilla/kuma/commit/283f0fa3ddbedccc14c9157c1fa4bb8a532bcade bug 1259233 - Update features document Add the new waffle flags, and clean up existing switches and flags to reflect the current code and production use. https://github.com/mozilla/kuma/commit/e928dd499f1f40c4bc4c609c63b6e514084348b5 Merge pull request #3827 from mozilla/spam-flags-1259233 bug 1259233 - Add more waffle flags for Akismet tuning r=jezdez
The code is in production, next the flags need to be added to the database
Flags are added to staging and production databases. "spam_training_mode" became "wiki_spam_training", but otherwise comment #2 is still accurate.
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.