Add content experiment validating examples on top



a year ago
a year ago


(Reporter: jwhitlock, Assigned: jwhitlock)




(Whiteboard: [specification][type:feature])



a year ago
What problem would this feature solve?
Measure the impact of changing the layout of reference pages to include a short usage example on top.

Who has this problem?
Staff contributors to MDN

How do you know that the users identified above have this problem?
It is generally agreed that action-oriented developers would appreciate a short example at the top of pages.  It is a labor-intensive project to re-write all the reference pages in this way.  By measuring the impact of the change for a number of high-traffic page, it will easier to decide if to continue the project, and if additional resources should be requested.

How are the users identified above solving this problem now?
A number of JS reference pages were adjusted several months ago. It should be possible to compare traffic in the month prior and month after the change.

Do you have any suggestions for solving the problem? Please explain in detail.
We added a content experiment framework (bug 1341770) explicitly for the purpose of segmenting traffic for this experiment.

Is there anything else we should know?
The technical specification is at:
Keywords: in-triage

Comment 1

a year ago
Commits pushed to master at
bug 1356923: Content experiment data to json file

Move the content experiment configuration to a .json file, loaded at
bug 1356923: Add static-examples-on-top experiment
bug 1356923: Configure Traffic Cop for experiment
bug 1356923: Adjust CONTENT_EXPERIMENTS format

Change to locale:slug keys, so that checking if a page is in an
experiment is faster (25x speedup).  Also change variants to a
dictionary, since weights are handled on the JS side.
bug 1356923: Drop experiment-wiki-content

This was a placeholder for the first "real" experiment, which is not the
static examples on top experiment.
bug 1356923: Add Mozilla to the inner call.

If we're passing in window.Mozilla to the IIFE, it should be aliased
bug 1356923: Access variants once
bug 1356923: Update templates test for new format
Merge pull request #4183 from jwhitlock/content_exp_static_1356923

Bug 1356923: Content experiment "static-examples-on-top"

Comment 2

a year ago
Commits pushed to master at
bug 1356923: Exclude 'Array' from the experiment

The 'Array' page has had examples on top for years, and doesn't have a
variant without them.
Merge pull request #4185 from jwhitlock/ce_static2_1356923

bug 1356923: Exclude 'Array' from the experiment
The results are in:

We can turn the A/B test off now. Thanks for all the work on this!

Comment 4

a year ago
OK, I'll create a PR to turn off this experiment.
Assignee: nobody → jwhitlock

Comment 5

a year ago
Commits pushed to master at
bug 1356923: Retire content experiments

Remove the framework-test and static-examples-on-top content experiments
and related function tests.
Merge pull request #4266 from jwhitlock/stop-static-examples-on-top-1356923

bug 1356923: Retire content experiments

Comment 6

a year ago
Pushed to production, experiment is over.
Last Resolved: a year ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.