1368729 - A/B Testing TOC usage

Status: RESOLVED FIXED

(developer.mozilla.org Graveyard :: General, enhancement)

Description

[Stephanie Hobson [:shobson]]

It's hard to asses the value of the TOC (aka "In this article") links based on analytics data.

Hypothesis

If we decrease the number of links in the TOC, then more people will click on them because it is faster for them to assess if there is useful information.

Method

Create variants of these two English pages which have the TOC restricted to "h2 and higher":
https://developer.mozilla.org//en-US/docs/Web/HTTP/Access_control_CORS
https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API/Using_Fetch

We already have event tracking in place for the TOC.

Running it for a week should be enough data.

Comment 1

[Stephanie Hobson [:shobson]]

I changed the TOC depth on https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array I'll compare data and revert it on July 12th.

I feel it is important to do this test now before the redesign is applied so we have enough data points to do a proper comparison and so we can decide how important it is to include the TOC in the new design.

Comment 2

[Stephanie Hobson [:shobson]]

I did find something interesting in the analytics. When we launched the beta TOC usage dropped off significantly. The only explanation I can think of for this is that half of our regular TOC users went and signed up for the beta. I couldn't filter them out of the analysis because they weren't beta users before that date. This makes sense because these users would be the ones most engaged with the site. 

I had the smaller TOC on the array page for 7 days but I only compared the middle 5 days to be sure the TOC had been there for the entire day. I compared to the 5 previous matching days of the week (after the beta was launched) and in both cases I filtered out beta testers.

The difference was not statistically significant. Only about 3% of our users click the TOC links so I guess we need to make a bigger change than length have an effect.