1356923 - Add content experiment validating examples on top

Status: RESOLVED FIXED

(developer.mozilla.org Graveyard :: Wiki pages, enhancement)

Description

[John Whitlock [:jwhitlock]]

What problem would this feature solve?
======================================
Measure the impact of changing the layout of reference pages to include a short usage example on top.

Who has this problem?
=====================
Staff contributors to MDN

How do you know that the users identified above have this problem?
==================================================================
It is generally agreed that action-oriented developers would appreciate a short example at the top of pages.  It is a labor-intensive project to re-write all the reference pages in this way.  By measuring the impact of the change for a number of high-traffic page, it will easier to decide if to continue the project, and if additional resources should be requested.

How are the users identified above solving this problem now?
============================================================
A number of JS reference pages were adjusted several months ago. It should be possible to compare traffic in the month prior and month after the change.

Do you have any suggestions for solving the problem? Please explain in detail.
==============================================================================
We added a content experiment framework (bug 1341770) explicitly for the purpose of segmenting traffic for this experiment.

Is there anything else we should know?
======================================
The technical specification is at:

https://docs.google.com/document/d/1CUe6-EqNbpSkLMLxfd0RWW6-TbPZFRtLc0rMh2Uu3G0/edit?usp=sharing

Comment 1

[[github robot]]

Commits pushed to master at https://github.com/mozilla/kuma

https://github.com/mozilla/kuma/commit/6c83216216476632589fbca2ac0bb9808a7a12dd
bug 1356923: Content experiment data to json file

Move the content experiment configuration to a .json file, loaded at
startup.

https://github.com/mozilla/kuma/commit/8217a84050599a02d79d6386f5b4bfe202979de2
bug 1356923: Add static-examples-on-top experiment

https://github.com/mozilla/kuma/commit/68b1bad3fe6dc11b5d754dcba1de36ca2e0fc66a
bug 1356923: Configure Traffic Cop for experiment

https://github.com/mozilla/kuma/commit/ec0cde0d092e13941aa9675d7c439e97cf96c461
bug 1356923: Adjust CONTENT_EXPERIMENTS format

Change to locale:slug keys, so that checking if a page is in an
experiment is faster (25x speedup).  Also change variants to a
dictionary, since weights are handled on the JS side.

https://github.com/mozilla/kuma/commit/26b993ef451ef03cf3be0bccec778efdb8a4aafa
bug 1356923: Drop experiment-wiki-content

This was a placeholder for the first "real" experiment, which is not the
static examples on top experiment.

https://github.com/mozilla/kuma/commit/de5d0813a52a3662e82531ba88e16dc475b0780f
bug 1356923: Add Mozilla to the inner call.

If we're passing in window.Mozilla to the IIFE, it should be aliased
here

https://github.com/mozilla/kuma/commit/9a1062bb101521a7f98d1f1f55cb24663b40e27c
bug 1356923: Access variants once

https://github.com/mozilla/kuma/commit/a62cdc656c1c7a10dcdb3bda711456b383f07c09
bug 1356923: Update templates test for new format

https://github.com/mozilla/kuma/commit/16fb7bf9a139f90d8c7a6a40609569637f99973f
Merge pull request #4183 from jwhitlock/content_exp_static_1356923

Bug 1356923: Content experiment "static-examples-on-top"

Comment 2

[[github robot]]

Commits pushed to master at https://github.com/mozilla/kuma

https://github.com/mozilla/kuma/commit/c2fdf2d2c437eef64e5b51ac0ce7d9faa2f00489
bug 1356923: Exclude 'Array' from the experiment

The 'Array' page has had examples on top for years, and doesn't have a
variant without them.

https://github.com/mozilla/kuma/commit/842668822937f58b9a2324cd415bf4ba571c0d0e
Merge pull request #4185 from jwhitlock/ce_static2_1356923

bug 1356923: Exclude 'Array' from the experiment

Comment 3

[Kadir Topal [:atopal]]

The results are in: https://docs.google.com/presentation/d/1LKiP8e2Jo0P9fJDaI0Ce5Q95OmfQ73UIIVqNc2l62ds/edit#slide=id.g1cf070bb1e_0_166

We can turn the A/B test off now. Thanks for all the work on this!

Comment 4

[John Whitlock [:jwhitlock]]

OK, I'll create a PR to turn off this experiment.

Comment 5

[[github robot]]

Commits pushed to master at https://github.com/mozilla/kuma

https://github.com/mozilla/kuma/commit/8035f0fbd92fcf14bf3a0f67a5a982a90a73b172
bug 1356923: Retire content experiments

Remove the framework-test and static-examples-on-top content experiments
and related function tests.

https://github.com/mozilla/kuma/commit/d2ba18c869ce44e5af46e19710b1a3c390b38371
Merge pull request #4266 from jwhitlock/stop-static-examples-on-top-1356923

bug 1356923: Retire content experiments

Comment 6

[John Whitlock [:jwhitlock]]

Pushed to production, experiment is over.