883967 - offline content: smoke test the feature

Status: RESOLVED FIXED

(developer.mozilla.org Graveyard :: General, defect)

Description

[Luke Crouch [:groovecoder]]

We can give the users existing solutions to MDN offline docs. I.e., links to running dochub locally [1] or, for Mac visitors, to Dash [2]. *And* we can link them to the bug to add their own comments on how they'd like to get the MDN docs offline.

That way, we don't piss users off - we give them what they click on, but we still gather metrics about user interest and input on feature development.

My estimate - lots of users will click the link, use one of the existing options, and not click thru to the bug.

But I could be wrong ... :)

Comment 1

[Luke Crouch [:groovecoder]]

[1] https://github.com/rgarcia/dochub
[2] https://itunes.apple.com/us/app/dash-docs-snippets/id458034879?mt=12

Comment 2

[Les Orchard [:lorchard]]

A little confused by this bug. 

Sounds more like an experiment than a "smoke test". That is, I've usually seen that to mean alpha test of a completed project - i.e. "blow smoke through the finished plumbing to see if it leaks" or "power up the finished circuit and see if it smokes"

If it's an experiment... What's the hypothesis / question(s) we're trying to answer? When do we know the experiment is finished? What are some possible the next steps? Is the point to evaluate 3rd party services / apps for offline MDN docs?

Comment 3

[Luke Crouch [:groovecoder]]

Sorry, yeah - I'm using "smoke test" in the MVP sense. [1]

The hypothesis is that "lots" of MDN users want offline content features. I propose we should see at least 5-10% of visitors click the offline link (general interest), and we should receive at least 5-10 comments on the bugzilla bug (significant interest). But I'm just making those numbers up - we really need to weigh the amount of demand and interest with the effort it will take to develop it. No real formula for that. :/

Let's time-box the experiment to 2 weeks of traffic.

Next steps would be either:
1. Leave the bug alone and let it fall back down towards the bottom of the backlog like it's already doing
2. Participate with the comments on the bug and file the appropriate blocker bugs to implement the feature

The primary point is to evaluate average MDN reader interest in offline content. Anything after that is just gravy.

[1] https://en.wikipedia.org/wiki/Minimum_viable_product#Techniques

Comment 4

[Luke Crouch [:groovecoder]]

Attachment: link from article to smoke page

Comment 5

[Luke Crouch [:groovecoder]]

Attachment: offline content smoke page

Comment 6

[[github robot]]

Commits pushed to master at https://github.com/mozilla/kuma

https://github.com/mozilla/kuma/commit/b19a20c496e5ef6c8a06642b7b64fc0bfa2cf13b
fix bug 883967 - Add offline content notice

https://github.com/mozilla/kuma/commit/d7f2d9082ff1e923186481615b6f5a27ee54bc17
Merge pull request #1217 from darkwing/smoke-test-883967

fix bug 883967 - Add offline content notice

Comment 7

[Eric Shepherd [:sheppy]]

How exactly does dochub.io have anything to do with reading offline? It's still an online tool for reading, it's just different from MDN.

Comment 8

[Luke Crouch [:groovecoder]]

Yeah, we might want to change that link to https://github.com/rgarcia/dochub#runnning-locally - the instructions for scraping MDN and running dochub.io locally.

Comment 9

[Luke Crouch [:groovecoder]]

Of course that oversight is also a honeypot for people to make a comment on bug 665750 ;)

Comment 10

[Luke Crouch [:groovecoder]]

:davidwalsh can double-check my analytics, but 2 weeks later ...

* 886k unique visitors
* 2.9M pageviews in en-US/docs/
* 114 clicks on the Offline Content Dialog
* 0 new comments on bug 665750

Conversion ratio of 114/2.9M = 0.00003%

I think that's definitive - this feature is not in high demand. Safe to ignore bug 665750 until we see more demand.

Comment 11

[Luke Crouch [:groovecoder]]

Attachment: Analytics developer.mozilla.org (production) Pages 20130710-20130724.pdf

Comment 12

[Luke Crouch [:groovecoder]]

Attachment: Analytics developer.mozilla.org (production) Events Overview 20130710-20130724.pdf

Comment 13

[Luke Crouch [:groovecoder]]

Sorry, that's 2.9M pageviews in */docs/*

Comment 14

[John Karahalis [:openjck]]

Let's check these numbers again in a couple of months. If little changes, I would be happy to WONTFIX bug 665750.

Comment 15

[Ben Bucksch (:BenB)]

Crouch [:groovecoder] from comment #10)

> * 0 new comments on bug 665750
> * 114 clicks on the Offline Content Dialog
> 
> this feature is not in high demand. Safe to ignore bug 665750

For the record, bug 665750 has 7 votes, and only 9 bugs in this product have this many votes. There you go for popularity.

I completely ignored that feature because it needs third party software or services, which I don't want.
Dash doesn't even work for me, and neither of them seem to provide full MDN content.


FWIW, I can't even find the "offline" link, even though I'm actively searching for it right now. https://github.com/mozilla/kuma/blob/master/apps/wiki/templates/wiki/document.html suggests that it should still be here, but I can't see it. I did see it once, but saw that it uses third party services, and ignored it. It's simply not what I need.
Suggesting to close the essential bug 665750 just because I and others didn't use some third party service or software is shocking.

Sorry for the noise here, but you are directly asking for it!

-----------

Bug 665750 is critically important, because MDN is open source and I must be able to get both the source (bug 561470) and the result (bug 665750) of the community work.

Comment 16

[Francois Guerraz]

(In reply to Luke Crouch [:groovecoder] from comment #10)
> :davidwalsh can double-check my analytics, but 2 weeks later ...
> 
> * 886k unique visitors
> * 2.9M pageviews in en-US/docs/
> * 114 clicks on the Offline Content Dialog
> * 0 new comments on bug 665750

So yes, you're basically saying that the people who *can* access the website don't care about the off-line content and that those who can't don't click on the Offline Content Dialog which is not easy to find even with intense browsing?

That's flawless reasoning...

Comment 17

[Luke Crouch [:groovecoder]]

The traffic for the time period included 156k visits from the 9 countries with the worst internet connection speeds according to Akamai [1]:

* 132,468 visits from India
* 12,709 visits from Indonesia
* 6,116 visits from Iran
* 1,328 visits from Nepal
* 1,186 visits from Nigeria
* 1,179 visits from Bolivia
* 1,101 visits from Kazakhstan
* 345 visits from Syria
* 193 visits from Libya

These visitors reached the site and still did not click the offline content dialog.

[1] http://www.huffingtonpost.com/2012/05/08/slowest-internet-countries_n_1475397.html

Comment 18

[John Karahalis [:openjck]]

(In reply to Francois Guerraz from comment #16)
> So yes, you're basically saying that... those who can't don't click on the
> Offline Content Dialog which is not easy to find even with intense browsing?
> 
> That's flawless reasoning...

The offline browsing notice is no longer being shown. While I do not have eye tracking numbers to independently confirm this, I can say that when it was being shown, it was quite prominent. Even annoyingly so.

Comment 19

[Luke Crouch [:groovecoder]]

I've re-enabled the offline content notice - it's directly beneath the article title of every page.