1260264 - Provide Baseline Metrics To Evaluate Akismet

Status: RESOLVED FIXED

(developer.mozilla.org :: Security, enhancement)

Description

[Ben (:bensternthal)]

What problem would this feature solve?
======================================
We will need the following metrics to evaluate how well Akismet is doing. This is the initial set of metrics.

False Positive % - How annoying are we being to good editors?
False Negative % - How much spam does staff have to mark?

Who has this problem?
=====================
Staff contributors to MDN

How do you know that the users identified above have this problem?
==================================================================
N/A

How are the users identified above solving this problem now?
============================================================
N/A

Do you have any suggestions for solving the problem? Please explain in detail.
==============================================================================
N/A

Is there anything else we should know?
======================================
N/A

Comment 1

[John Whitlock [:jwhitlock]]

Further details from Toronto meeting with Kadir, with my own elaboration.

The dimensions for metrics are:
A) Change type (edit English page, new English page, new translation, edit translation)
B) Account type (Persona, Github)
C) Time (day, 7 days, 30 days, 90 days). Stats will be generated after 24 hours, to allow staff time to identify spam / rescue ham.

1) Estimated time to "handle spam", per incident. This includes reporting to Akismet, blocking a user account, and removing spam by reverting or deleting. This does not include monitoring the revision dashboard or developer time.
2) Total published change count
3) Published spam count
4) Published spam rate (#3 / #4 as percent)
5) Monthly staff time deleting spam (#3 * #1), our KPI for the spam mitigation initiative

These metrics are valid with and without Akismet integration. With Akismet enabled, further stats should be calculated:

6) Blocked change count
7) Total attempted change count
8) False positive count (blocked as spam by Akismet, identified ham by staff)
9) False negative count (allowed as ham by Akismet, identified spam by staff)
10) False positive rate (#8 / #7, as percentage), 1st KPI for retaining Akismet
11) False negative rate (#9 / #7, as percentage), 2nd KPI for retaining Akismet

Comment 2

[John Whitlock [:jwhitlock]]

Some changes over the last few months, or that seemed like a good idea while I'm coding this:

Since we're dropping Persona soon (bug 1273028, bug 1260257), I'm not segmenting spam by account type.  Initial study shows that both account types were used for automated spam.

I've adjusted the time buckets to reflect the weekly trends on MDN (high traffic and edits during work week, etc.).  "Monthly" is now 28 days (4 weeks), and "Quarterly" is 91 days (13 weeks).

I've generated the by-time statistics, and they feel like numbers without meaning. It may be more useful to generate weekly numbers for 13 or 26 weeks, and color the numbers by increase / decrease over the previous period.  This may be future work.

The KPI has evolved from "Monthly staff time deleting spam" to "Number of visitors viewing spam".  Staff time spent on spam is still important (bug 1259776), but mostly as a means for quickly detecting and removing spam to reduce impact on users.  So, I'm not including estimates of staff time.

Tracking viewers of spam will require code and/or infrastructure, tracked in a to-be-written bug.  For now, omitting estimate from the dashboard, but showing time that recent published spam was visible.  Initial testing shows an average of 1-2 visitors view spam before it is removed.

Comment 3

[John Whitlock [:jwhitlock]]

After discussions with Kadir, this has evolved further.  The important metric is "estimated viewers of spam" over an undefined time period.  Initial research did not find a good model to estimate this metric. Instead the current proposal is:

* Send page revision IDs to Google Analytics as a custom dimension
* Identify revisions as spam on MDN
* Query GA for estimated viewers of a particular revision of a document, store in the MDN database
* Aggregate and display spam viewers in a dashboard on MDN

This will require additional work, and won't be available until them.  We will publish some metrics that we can computer from just MDN data on a spam dashboard:

* False positive count (published spam)
* False negative count (blocked ham)
* True positive count (blocked spam)

These counts will be aggregated over the time periods:

* Previous day
* Previous week
* Previous month (rounded to 4 complete weeks, 28 days)
* Previous quarter (rounded to 13 complete weeks, 91 days)

They will be reported as counts, not rates or ratios.

When the Google Analytics integration is complete, the page visitor data will be included as well.

Comment 4

[John Whitlock [:jwhitlock]]

After another prototype round, the desired stats are:

Desired statistics as of August 2nd:

* Spam Viewers (data to come in US #351)
* Daily Spam Viewers (derived)
* % Change in Spam viewers (derived)
* False positive count (published spam)
* False negative count (blocked ham)
* True positive count (blocked spam)
* True positive rate (blocked spam / spam)
* True negative rate (published ham / ham)
* Stats needed for above calculations (total spam, total ham) but not displayed

Counts are over the time periods:

* Previous day
* Previous week
* Previous month (rounded to 4 complete weeks, 28 days)
* Previous quarter (rounded to 13 complete weeks, 91 days)
* The 4 "n-1" periods, for the % change in spam viewers

Comment 5

[MDN Team (:mdn-dev)]

Commits pushed to master at https://github.com/mozilla/kuma

https://github.com/mozilla/kuma/commit/982d4e9a57587865789d60c8df12f66eff67d535
bug 1260264 - Added the skeleton of the spam dashboard view.

https://github.com/mozilla/kuma/commit/7149a06a7615d82612f4fc1dc8e3fbc83b24c5fe
bug 1260264 - Brought in the spam_dashboard_recent_events utility function.

https://github.com/mozilla/kuma/commit/b03afa56b495885bcda6b5d0b5873ca37046b3fe
bug 1260264 - Brought in the SpamDashboardRecentEvents job, and render it.

https://github.com/mozilla/kuma/commit/bad913748d0d3d4575a5186a2e130d3101cb6c24
bug 1260264 - Added a simplied version of spam_day_stats.

https://github.com/mozilla/kuma/commit/65872c627cbbaa766361f2b51071d884dec29cdc
bug 1260264 - Added utility function and job to calculate spam statistics over periods of time.

Simplified version of the prototype version.

https://github.com/mozilla/kuma/commit/9583684f270b9cbc9d5e64abc30ffcbc164a7288
bug 1260264 - Added the html for the spam statistics table.

https://github.com/mozilla/kuma/commit/5fcdc0c46abd1144b50d006fd03394d1dbf1d3e2
bug 1260264 - Added the viewer counts to each recent spam item.

https://github.com/mozilla/kuma/commit/28377fd816511783f93641271584462b4319f684
bug 1260264 - Added the spam viewer statistics.

https://github.com/mozilla/kuma/commit/6376de9f62a37f58e4a94ff354d8d0fb75ea15df
bug 1260264 - minor tweaks to parameter names, table id, permission name

https://github.com/mozilla/kuma/commit/f2fc3076302c9ee0b3575941158aeeb1bad61218
bug 1260264 - add tests for the spam dashboard

https://github.com/mozilla/kuma/commit/fd5c547afa252cce19024d65d9bc68c1fd8c8f0f
bug 1260264 - Pass through the ImproperlyConfigured message to the admin user.

https://github.com/mozilla/kuma/commit/3ca842d0b1196702c50ccf82216e6f7f6a7fb43c
bug 1260264 - Added basic styling.

https://github.com/mozilla/kuma/commit/d0cca3e527d023a2903490ef8d9ee0ae72accf41
bug 1260264 - Fixed and added to the tests.

https://github.com/mozilla/kuma/commit/13aea5bda9b17bc91f12832f495472f7a5bfd624
bug 1260264 - Set the job version numbers back to 1.

https://github.com/mozilla/kuma/commit/b21d03d64d183ad7c039758417ff0330627817fa
bug 1260264 - generate the timestamps for the jobs more consistently.

https://github.com/mozilla/kuma/commit/dc9b6245eafa5cbc90ebe1d29f88d541ecabaf59
Merge pull request #3944 from caktus/spam-dashboard-679

bug 1260264 - Spam dashboard

Merged with some to do items:
* Improve test coverage
* Improve storage of unique pageviews

Comment 6

[John Whitlock [:jwhitlock]]

The dashboard is live at:

https://developer.mozilla.org/en-US/dashboards/spam

Quarterly numbers (2016-05-26 to present):

66 spam viewers, 0.7 daily viewers (GA estimate)
32 published spam (false negative)
12 blocked spam (true positive)
189 blocked ham (false positive)
27.3% True positive rate (blocked spam / total spam)
99.1% True negative rate (allowed ham / total ham)

The original requested stats (False Positive %, False Negative %) are not there, but can derived from the information given (False positive % is 100% - True positive rate, or 72.7%)

There is very little spam at the movement, and the system relies on administrators using the tools to mark and revert spam rather than manual cleanup, and that the spam is identified within three days.

We're delaying further improvements and actions until a later date, since the spam rate is currently very low.

Comment 7

[MDN Team (:mdn-dev)]

Commits pushed to master at https://github.com/mozilla/kuma

https://github.com/mozilla/kuma/commit/143df9c5cb46068c80ac4e65f9782e6f778d7a16
Bug 1260264: Formating Change For Spam Dashboard

https://github.com/mozilla/kuma/commit/00fc05b101658f863f58d7fc2b1aefecbcf71cf8
Merge pull request #3971 from mozilla/1260264-spam-table-format

Bug 1260264: Formating Change For Spam Dashboard