Closed Bug 861981 Opened 11 years ago Closed 6 years ago

Support external sources of documentation (GitHub and more)

Categories

(developer.mozilla.org Graveyard :: General, enhancement, P3)

Product:
developer.mozilla.org Graveyard
Component:
General
Platform:
All
Other
Type:
enhancement

Tracking

(Not tracked)

Status:
RESOLVED WONTFIX

People

(Reporter: openjck, Unassigned)

References

Details

(Whiteboard: [specification][type:feature][dev-ecosystem][triaged]) 

What problems would this solve?
===============================
Many projects host documentation on GitHub only. If we do not support GitHub as a content source, we could lose credibility as a trusted resource for developers.

https://github.com/mozilla/addon-sdk
https://github.com/mozilla/pdf.js
https://github.com/mozilla/shumway
https://github.com/mozilla/collusion
https://github.com/mozilla/browserid
https://github.com/mozilla/popcorn-js
https://github.com/mozilla/openwebapps
https://github.com/mozilla/sweet.js
https://github.com/mozilla/task.js
https://github.com/mozilla/mortar-* (about 5 repos)
https://github.com/mozilla/openbadges
https://github.com/mozilla/rust
https://github.com/mozilla/servo

Who would use this?
===================
Anyone who primarily publishes documentation on GitHub. See the list above.

What would users see?
=====================
We have not decided exactly how this feature will look and behave (we will discuss in this bug), but we frequently name readthedocs.org as inspiration.

What would users do? What would happen as a result?
===================================================
See above.

Is there anything else we should know?
======================================
Whiteboard: [specification][type:feature] → [specification][type:feature][dev-ecosystem]
Summary: Support GitHub documentation sources → Support external sources of documentation (GitHub and more)
Priority: -- → P2
Pointed bug 780227 to this one, as we might consider supporting switchable HTML renderers per project (ie. MediaWiki, reStructuredText, Markdown, etc)
(In reply to Les Orchard [:lorchard] from comment #1)
> Pointed bug 780227 to this one, as we might consider supporting switchable
> HTML renderers per project (ie. MediaWiki, reStructuredText, Markdown, etc)

Apologies for drive-by linking, but thought pandoc might bare worth investigation (first glance looks awesome, save haskell not being much in use here):

http://johnmacfarlane.net/pandoc/
(In reply to Jeff Hammel [:jhammel] from comment #2)
> (In reply to Les Orchard [:lorchard] from comment #1)
> > Pointed bug 780227 to this one, as we might consider supporting switchable
> > HTML renderers per project (ie. MediaWiki, reStructuredText, Markdown, etc)
> 
> Apologies for drive-by linking, but thought pandoc might bare worth
> investigation (first glance looks awesome, save haskell not being much in
> use here):
> 
> http://johnmacfarlane.net/pandoc/

That sounds like a potentially huge help...
Don't forget l10n here. We can translate every page now, we should not regress here.
There are good Python implementations of Markdown, ReST, Textile, and most other common documentation markup languages. I suspect we'd rather avoid adding new languages to our stack if at all possible.
(In reply to Jean-Yves Perrier [:teoli] from comment #4)
> Don't forget l10n here. We can translate every page now, we should not
> regress here.

I don't think we have a strategy for this yet. This is a fairly large project, and we have many things to figure out.
Thinking out loud - this should be somehow connected to/from MDN, and uses ReadTheDocs.org:

    http://firefox-marketplace-api.readthedocs.org/en/latest/

That uses a Mozilla Sandstone theme for Sphinx:

    https://github.com/ametaireau/mozilla-sphinx-theme

A quick-and-dirty idea for this bug is to build an MDN-centric theme for RTD/Sphinx. That is, one that shares the MDN design and has top-level nav pointing back to MDN.

It might also be interesting to somehow include RTD docs in our search indexing.

And, to make this even more official, we can look into using a developer.mozilla.org domain for docs on RTD
Oh, and the source for the Marketplace API docs is here:

    https://github.com/mozilla/zamboni/tree/master/docs/api
I agree that leveraging RTD could be very helpful here. It may not solve all of our problems, but it could get us 80% of the way there with 10% of the effort of building our own platform, no maintenance costs, and free (or almost free) improvements over time.
Porting this over from an email thread:

On 5/15/13 12:56 PM, Jeff Griffiths wrote:
> I don't think gut feels and doubts are the most productive way to
> make decisions. Up-front I'd want to see that someone has at least:

The reason I say "gut feel" is that I've looked casually into the history of requests for Markdown in Sphinx, and most of the answers are "Meh, reST is good enough / better, why bother with Markdown?"

The features of readthedocs.org are very, very compelling versus building our own parallel & competing system. Additionally, readthedocs.org is being worked on by some Mozillians, so we'd be competing against ourselves to boot :)

> 1. looked into conversion of markdown to rest to see what the issues
> are - do we convert once? can we convert as part of a build step?

Haven't done that yet, no. I would think that conversion - even as part of a build step - is a much more achievable goal than reinventing the Sphinx + RtD wheel. And, I'd bet that a one-time conversion and usage of reST into the future would be better overall than reliance on convert-at-build.

Proving out the above is part of what's needed for this bug, though.

> 2. asked whoever is working on Sphinx about markdown support -
> specifically, if we implemented this would they be interested?

I haven't asked personally, no, but others have. I don't think there's a great deal of interest.

> 3. given an assessment of the easiest way forward vs the 'right' way
> forward in terms of up-front effort *and* long-term cost

readthedocs.org is easiest, but probably even the right way: It's in use by a lot of projects, many at Mozilla. It's often the path to bypass MDN entirely, because we've not provided a satisfactory service for those projects.

> I'm not stuck on Markdown over rest because of personal preference or
> a concern over Will's preference, I'm interested in it because it is
> built into github and therefore in already familiar to all of our
> potential contributors. It's pretty important.

FWIW, reST is also built into github, and is already in use by a number of other Mozilla projects for documentation. That's why I'm so keen on it, overall. Basically, I feel like we're being lapped by Read the Docs in many ways, and we should make friends to catch up.
I think choosing whether to support Markdown, or to use RTD+Sphinx+ReST, is a fundamental choice in this project. Restricting ourselves to Sphinx/ReST would make things much simpler, but would probably be less attractive to products.

I've had a bit of a look at RTD/Sphinx/ReST, and there seems to be a strong interdependency there. Supporting Markdown is not just a matter of switching out a source format: ReST has features that just don't exist in Markdown, and Sphinx (and therefore RTD) depends on these features.

On the other hand, if we're looking at converting the Add-on SDK to Sphinx/ReST, that's not just a format conversion either: it looks to me like a ground-up rewrite to use Sphinx/ReST. Suggesting we could keep the Markdown source and convert as a build step looks very optimistic to me.

Also, there are things that the SDK docs system does that would be difficult to replicate in Sphinx/ReST (for example, the SDK docs system automatically discovers API docs and adds them to the sidebar and various docs summary pages, and it reads metadata from the source code for API stability).

Having said that, I'm not at all against rewriting the SDK docs system to use Sphinx/ReST, or at least looking into it much more closely to see if there are major problems. The benefits of being more normal would almost certainly outweigh the compromises we'd have to make, and for me it would be a practical way I could move this project forward.
(In reply to Will Bamberg [:wbamberg] from comment #11)
...
> Also, there are things that the SDK docs system does that would be difficult
> to replicate in Sphinx/ReST (for example, the SDK docs system automatically
> discovers API docs and adds them to the sidebar and various docs summary
> pages, and it reads metadata from the source code for API stability).
...

I think so far, we (I?) have been assuming a rather naive model for docs. (eg. a pile of source files that get rendered into HTML somehow and served up from MDN)

What would be very handy are some more requirements from the addon SDK, because I'm not super-familiar with what all it does.

Short of lobbying for Markdown as an alternate syntax overall, we *can* do selective augmentations for Read the Docs - eg. do granular Markdown-to-reST conversions automatically (eg. for content extracted from code, etc)

FWIW, overall I'm operating from the perspective of seeing Mozilla projects skip MDN entirely, and taking note of what they're using:

    https://readthedocs.org/search/project/?q=mozilla

641 results - granted, not all projects, but that suggests a corpus of doco that I really wish was better represented in MDN
(In reply to Les Orchard [:lorchard] from comment #12)
> What would be very handy are some more requirements from the addon SDK,
> because I'm not super-familiar with what all it does.

Not sure if you saw this yet. Will's requirements from the Add-on perspective.

https://wiki.mozilla.org/GitHub_Integration
We all (well, almost all) agree that we should do this. What is the first step? What is the first thing that would get in our way if we tried building this tomorrow?

Let's open a dependent bug for that thing and get it on the Kanban board.
(In reply to John Karahalis [:openjck] from comment #13)
> (In reply to Les Orchard [:lorchard] from comment #12)
> > What would be very handy are some more requirements from the addon SDK,
> > because I'm not super-familiar with what all it does.
> 
> Not sure if you saw this yet. Will's requirements from the Add-on
> perspective.
> 
> https://wiki.mozilla.org/GitHub_Integration

Yes, that's part of what spurred this renewed discussion. 

That doc is fairly high level though. What I'm looking for are more specifics that have bubbled up like when will wrote...

"Also, there are things that the SDK docs system does that would be difficult to replicate in Sphinx/ReST (for example, the SDK docs system automatically discovers API docs and adds them to the sidebar and various docs summary pages, and it reads metadata from the source code for API stability)."

So what I'm looking for is just what kinds of special things does the SDK docs system do, that it doesn't seem like Sphinx + RtD would do.
(In reply to John Karahalis [:openjck] from comment #14)
> We all (well, almost all) agree that we should do this. What is the first
> step? What is the first thing that would get in our way if we tried building
> this tomorrow?
> 
> Let's open a dependent bug for that thing and get it on the Kanban board.

IMO, two first steps:

1. Build an MDN-flavored Sphinx theme and run something like the Marketplace API docs through it (because that's already written in reST). This will probably be a throwaway prototype at least in part, because we're also planning a redesign for MDN

2. Experiment with converting a set of Markdown docs to reST and see what breaks
Will is currently looking into porting the SDK docs to Sphinx/ReST. He should have more information on special cases once he is done.

Another next step is to determine how many products we would leave out if we supported only Sphinx/ReST.
Depends on: 873256
Opened bug 873256 for the Sphinx theme. We can work on the rest right here, unless we think separate bugs would be helpful.

To answer my own question, here is a list of some important projects at Mozilla that are documented in source control. All of these projects use Markdown. Some of these projects host additional documentation on a project website, but none of them use Read the Docs.

For each item in the list, a small/medium/large indicator is provided to reflect the size of their Markdown documentation library.

* [small] https://github.com/mozilla/pdf.js
* [small] https://github.com/mozilla/shumway
* [small] https://github.com/mozilla/sweet.js
* [small] https://github.com/mozilla/openbadges
* [small] https://github.com/mozilla/servo
* [medium] https://github.com/mozilla/collusion
* [medium] https://github.com/mozilla/mortar-*
* [large] https://github.com/mozilla/browserid
* [large] https://github.com/mozilla/openwebapps
* [large] https://github.com/mozilla/rust
We also have interest from the buildingfirefoxos group:

http://buildingfirefoxos.com/
https://github.com/buildingfirefoxos/

Would likely be another GitHub+Markdown source.
Great! Just as a reminder, the next steps are outlined in comment 16. The first of those steps is tracked with bug 873256, the other can be worked on here.

If anyone is looking for something to do, please keep these in mind.
I wonder if we could get some incremental gains from these notions:

* Don't fix what's not broken - that is, don't try to get groups with existing docs (and working sites) to convert over to an MDN solution (eg. Read the Docs or whatever). In my mind, this is even more "be where the writers are"

Instead, provide an API whereby those sites can push content into MDN's search engine. We don't want to be Google, but I could see this being handy nonetheless.

Alternatively (or additionally), we could offer a way to register collections of docs not hosted by MDN, so we can at least publish a searchable directory.

MDN can be a discovery hub for all Mozilla docs, whether they live on MDN or not.

* Develop a kind of "Tabzilla" drop-in component that lets external-to-MDN doc sites display navigation that links back to MDN.

* Offer MDN themes for Read the Docs and others to help provide some unity in identity across Mozilla docs. That could help with the "experience cliff" involved in jumping from MDN to external sources.
(In reply to John Karahalis [:openjck] from comment #20)
> Great! Just as a reminder, the next steps are outlined in comment 16. The
> first of those steps is tracked with bug 873256, the other can be worked on
> here.
> 
> If anyone is looking for something to do, please keep these in mind.

FWIW, I don't think those are the only or even the right next steps. I think we're still talking this through. We don't have a straightforward solution yet that seems satisfying
(In reply to Les Orchard [:lorchard] from comment #21)
> I wonder if we could get some incremental gains from these notions:
> 
> * Don't fix what's not broken - that is, don't try to get groups with
> existing docs (and working sites) to convert over to an MDN solution (eg.
> Read the Docs or whatever). In my mind, this is even more "be where the
> writers are"
> 
> Instead, provide an API whereby those sites can push content into MDN's
> search engine. We don't want to be Google, but I could see this being handy
> nonetheless.
> 
> Alternatively (or additionally), we could offer a way to register
> collections of docs not hosted by MDN, so we can at least publish a
> searchable directory.
> 
> MDN can be a discovery hub for all Mozilla docs, whether they live on MDN or
> not.

This might work for some projects, but I would like the SDK docs to be actually hosted on MDN.

I'd also be happy for the SDK doc build system to be more normal.In fact if I had to choose between (1) the SDK docs remaining in Markdown, but the build system being entirely bespoke or (2) having the rewrite the SDK docs in ReST, but the build system being mostly normal, I would choose (2).

I've spent the last couple of days looking at Sphinx/ReST, and I'm tentatively hopeful that it could do everything we might want to do, although I might eventually need some help from real experts to make it less hacky/more efficient.

But of course, other projects might have different priorities.
 
> * Develop a kind of "Tabzilla" drop-in component that lets external-to-MDN
> doc sites display navigation that links back to MDN.
> 
> * Offer MDN themes for Read the Docs and others to help provide some unity
> in identity across Mozilla docs. That could help with the "experience cliff"
> involved in jumping from MDN to external sources.

For the SDK I think MDN theming is v. important.
Quick update about bug 873256 - I've very hastily hacked together a Sphinx theme based on MDN and used it to build docs for one of my Django projects.

If you want to see how the transition from MDN to RtD looks, try clicking on the "Check out this doc" link on my User page:

    https://developer.mozilla.org/en-US/docs/User:lmorchard

The theme still needs refinement, and will need a complete overhaul whenever we have a redesign. But, it didn't take much work to get it into its current state. Devil's in the details, of course.
I'm very impressed.

But now I'm worried about the *opposite* effect of the "experience cliff" - if a link from MDN to readthedocs looks *this* similar, people will be confused about other stuff:

* where'd the search box go? why isn't it searching all of MDN now?
* why am I logged out? how do I edit this page?

It will be hard to get this right ...
(In reply to Luke Crouch [:groovecoder] from comment #25)
> I'm very impressed.
> 
> But now I'm worried about the *opposite* effect of the "experience cliff" -
> if a link from MDN to readthedocs looks *this* similar, people will be
> confused about other stuff:
> 
> * where'd the search box go? why isn't it searching all of MDN now?
> * why am I logged out? how do I edit this page?
> 
> It will be hard to get this right ...

Yup, that's what I mean by the devil being in the details - need lots of feedback like this.

But, the search box can be re-added, especially when we're on elasticsearch. It didn't quite work off-site as-is, using the Google API. So, I just slashed it.

And, for login, we can offer a tabzilla-like component that lives on MDN which reflects the login status along with other links back to MDN.
FWIW, I imagine each of our external sources of docs could be a "content zone" on MDN, and I've been looking at jquery.com - the way they have their "content zones" set up across domains is pretty good.
Just wanted to say that this is beyond awesome. These pages look so natural I had a hard time believing they were generated. I didn't even realize they were on readthedocs.org until Luke pointed it out.

I agree that we will need to think hard about branding. I really like the approach RIT takes -- all sites that are part of the RIT "family" have a similar header...

http://cias.rit.edu/
http://www.se.rit.edu/
http://www.cs.rit.edu/

Tabzilla is another variation on the same concept, and so is the black "Google bar" that appears at the top of all of their websites. Even the Demo Studio has some nice branding that makes it clear that the site is related to, but not identical to, the wiki. Maybe one of these approaches would work.
Another option, for Markdown-formatted docs, is Github Pages [0] and Jekyll [1].

[0]: http://pages.github.com/
[1]: http://jekyllrb.com/

Not sure why I hadn't mentioned it before, since I've used it myself for my old blog. Sitting in a GlueCon session discussing "Creating Sustainable Documentation with Jekyll" reminded me of it.

I think this is another option worth considering. Github itself will generate static sites from Markdown-formatted docs in a git repo. The output can also be themed like Sphinx, so we could offer an MDN theme for that route as well.

I also don't think ReadTheDocs / Sphinx vs Github Pages / Jekyll is an either-or thing. We could possibly support MDN themes and search engine / discovery support for both.
Depends on: 877189
This bug is getting big and seems worth splitting at least into two more - bug 877189 and bug 877197 are now blockers, for Markdown & reST docs respectively
Depends on: 877197
No longer depends on: 873256
No longer depends on: 877197
Just filed bug 877212 with an idea - we could build a browseable & searchable directory of external sources of docs.
Depends on: 877197, 877212
Depends on: 877217
Depends on: 881859
Blocking bug 861991, because I think supporting loosely-coupled external sources of docs with static publishing could be a good & fast way to get some collections of materials up & running & embraced by MDN
Blocks: 861991
Triaged in post-launch bug review on 1/10/2013. Moving to P3 as we don't have someone to champion using this functionality on MDN. Will review in a few weeks after Sheppy talks to engineering teams
Priority: P2 → P3
Severity: normal → enhancement
Whiteboard: [specification][type:feature][dev-ecosystem] → [specification][type:feature][dev-ecosystem][triaged]
developer.mozilla.org has moved away from being the central repository of Mozilla documentation (Mozilla Developer Network) to a focus on vendor-neutral web platform documentation (MDN Web Docs):

https://discourse.mozilla.org/t/the-future-of-mdn-a-focus-on-web-docs/16128

We're no longer interested in aggregating All The Docs.
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → WONTFIX
Product: developer.mozilla.org → developer.mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.