793926 - Documentation generator needs to recognize new file layout

Status: RESOLVED FIXED

(Add-on SDK Graveyard :: Documentation, defect, P1)

Description

[Irakli Gozalishvili [:irakli] [:gozala] [@gozala]]

Documentation generator will need to recognize new file layout and how these files map to the modules in the firefox.

For details see Bug 787346

Comment 1

[Dave Townsend [:mossop]]

Will are you going to be working on this?

Comment 2

[Will Bamberg [:wbamberg]]

I was intending to, yes. I'm not quite sure what the implications of it are yet though, I'll need to talk to Irakli about it.

Comment 3

[Will Bamberg [:wbamberg]]

1) is the layout actually decided yet? The wiki page at https://github.com/mozilla/addon-sdk/wiki/JEP-lib hasn't been updated for 4 months, it seems, but the discussion on the etherpad: https://etherpad.mozilla.org/JEP-lib is more recent.

2) is there a branch I can try stuff out on?

3) how will we coordinate landing this change? It seems like there will be a lot of moving parts.

4) where will docs files live? In a mirrored /doc tree, or alongside the js files themselves under /lib? (I'd prefer the latter.)

5) presumably cfx/packaging.py won't be usable any more to get the set of SDK modules. We'll need an equivalent that builds the tree of modules, and reads the metadata (esp. stability). This will need to be callable form the Python code that builds the docs. Is writing this in scope for this bug? (I'm happy to have a go at it.)

6) since there won't be any packages, we won't organize modules in the docs by package any more. But we also still talk about high-level and low-level modules, and the proposals discussed in https://etherpad.mozilla.org/JEP-lib have various ways of indicating that. Is this distinction orthogonal to module stability? (I think it is).

If it is, should we also try to reflect this distinction in the way we present modules in the docs, for example by grouping modules into high-level and low-level as we've done before?

Actually, I think it would be better to organize by stability instead, like this:

--------------------
Stable APIs
--------------------
    page-mod
    tabs
    core/heritage (just for the sake of argument, I know it's not *actually* stable yet...)
    ...
-------------------
Experimental APIs
-------------------
    indexed-db
    window/utils
    ...
--------------------
Deprecated APIs
--------------------
    window-utils
    ...

I think this would be more helpful to developers. What do you think?

Comment 4

[Irakli Gozalishvili [:irakli] [:gozala] [@gozala]]

(In reply to Will Bamberg [:wbamberg] from comment #3)
> 1) is the layout actually decided yet? The wiki page at
> https://github.com/mozilla/addon-sdk/wiki/JEP-lib hasn't been updated for 4
> months, it seems, but the discussion on the etherpad:
> https://etherpad.mozilla.org/JEP-lib is more recent.
> 

Yeah I'm working on implementation of a winner option form the etherpad. Also cfx-js is implemented with that layout in mind.


> 2) is there a branch I can try stuff out on?
> 

Work in this area is happening here:
https://github.com/mozilla/addon-sdk/pull/569

Currently it's possible to run example add-ons, test don't yet work.
I'm working on tests, hopefully soon they'll be all passing.

>
> 3) how will we coordinate landing this change? It seems like there will be a
> lot of moving parts.
>


I'll be working on this with ochameau next week from Paris. Plan is to nail all the moving part during this week. Doc generator code is one of the moving parts that we
don't know much about yet :(


> 
> 4) where will docs files live? In a mirrored /doc tree, or alongside the js
> files themselves under /lib? (I'd prefer the latter.)
>


I had mirrored doc tree in mind (in fact pointed pull request already has docs
moved, still need to make sure that links are up to date too painting to new locations etc.. If you could help out there, it would be great.


> 
> 5) presumably cfx/packaging.py won't be usable any more to get the set of
> SDK modules. We'll need an equivalent that builds the tree of modules, and
> reads the metadata (esp. stability). This will need to be callable form the
> Python code that builds the docs. Is writing this in scope for this bug?
> (I'm happy to have a go at it.)
> 

In fact ochameau did some tricks with cfx to enable layout changes without major
rewrites of cfx. Main idea is that cfx thinks of folders in lib as if they were packages, so it still maybe usable for docs. Metadata recognition is not part of
this, the rest is. Basically we need to be able to generate docs to land layout changes, doing whatever necessary for that is part of this bug.

>
> 6) since there won't be any packages, we won't organize modules in the docs
> by package any more. But we also still talk about high-level and low-level
> modules, and the proposals discussed in https://etherpad.mozilla.org/JEP-lib
> have various ways of indicating that. Is this distinction orthogonal to
> module stability? (I think it is).
> 

Yeap I think so.


>
> If it is, should we also try to reflect this distinction in the way we
> present modules in the docs, for example by grouping modules into high-level
> and low-level as we've done before?
>

Don't really know. I think target audiences for high level modules is different from the audience of stable APIs. I think add-on devs would rather use high level APIs
while lib devs would use low-level but stable ones. Also if ff will be using SDK modules likely it will be low-level stable ones. How to reflect all this in docs I don't know. Have not thought of this yet either to be honest.

> 
> Actually, I think it would be better to organize by stability instead, like
> this:
> 
> --------------------
> Stable APIs
> --------------------
>     page-mod
>     tabs
>     core/heritage (just for the sake of argument, I know it's not *actually*
> stable yet...)
>     ...
> -------------------
> Experimental APIs
> -------------------
>     indexed-db
>     window/utils
>     ...
> --------------------
> Deprecated APIs
> --------------------
>     window-utils
>     ...
> 
> I think this would be more helpful to developers. What do you think?


I think it would be best to have an option for devs to decide weather they want
to be presented with docs organized by stability or (high / low) level. Maybe that's overkill too. Either way at this point I really want to focus on implementing layout change itself to unblock landing to FF work, there for I would just go with whatever is easiest to get working and would improve presentation / organization in followup steps.

Comment 5

[Irakli Gozalishvili [:irakli] [:gozala] [@gozala]]

Oh almost forgot:

> Python code that builds the docs. Is writing this in scope for this bug?
> (I'm happy to have a go at it.)


If you don't mind getting your hands dirty with this, that would really help, yes please!!

Comment 6

[Will Bamberg [:wbamberg]]

I think you're right, I should not parse the metadata and attempt to represent it in this bug. I'll keep with organizing by high level/low level for the time being, and just remove all the packages stuff.

More questions! And comments.

1) In the current mode, when you drop third-party packages into the SDK, their docs show up in the doc browser. Will we stop supporting this feature in the future, and if not, where should I look for them? FOr the time being I just ripped all that out.

2) What lives in "toolkit"? Is this stuff a third category, along with high-level and low-level?

3) I'm not sure I like having deprecated modules placed in a specific location. We already have metadata to represent deprecated-ness, isn't it redundant to encode that status in a specific location in the tree as well? Also, if we have to move modules when we deprecate them, won't that break add-ons that are using deprecated modules?

Comment 7

[Irakli Gozalishvili [:irakli] [:gozala] [@gozala]]

(In reply to Will Bamberg [:wbamberg] from comment #6)
> I think you're right, I should not parse the metadata and attempt to
> represent it in this bug. I'll keep with organizing by high level/low level
> for the time being, and just remove all the packages stuff.
> 
> More questions! And comments.
> 
> 1) In the current mode, when you drop third-party packages into the SDK,
> their docs show up in the doc browser. Will we stop supporting this feature
> in the future, and if not, where should I look for them? FOr the time being
> I just ripped all that out.

Honestly I have not took into account. I guess in a future we could look into
addons `node_modules` folder which is were dependencies will be installed by
a package manager. For the time being I guess looking into packages folder and
using it should be fine.


> 2) What lives in "toolkit"? Is this stuff a third category, along with
> high-level and low-level?

This is typically modules that we share with other projects. Also these modules
are in all xul based apps. Mainly separation is to enable easy collaboration with
other teams. Also it's like very generic / platform agnostic modules.

> 
> 3) I'm not sure I like having deprecated modules placed in a specific
> location. We already have metadata to represent deprecated-ness, isn't it
> redundant to encode that status in a specific location in the tree as well?
> Also, if we have to move modules when we deprecate them, won't that break
> add-ons that are using deprecated modules?
>

Yeah it's redundant and that in a future we won't be moving such modules around.
In this case we just wanted to make it more clear for FF engineers that this is
something we plan on removing.

Comment 8

[Jeff Griffiths (:canuckistani) (:⚡︎)]

(In reply to Irakli Gozilalishvili [:irakli] [:gozala] [@gozala] from comment #7)
...
> > 1) In the current mode, when you drop third-party packages into the SDK,
> > their docs show up in the doc browser. Will we stop supporting this feature
> > in the future, and if not, where should I look for them? FOr the time being
> > I just ripped all that out.
> 
> Honestly I have not took into account. I guess in a future we could look into
> addons `node_modules` folder which is were dependencies will be installed by
> a package manager. For the time being I guess looking into packages folder
> and
> using it should be fine.

Nit: if we can, would really rather this be a 'modules' folder, instead of 'node_modules'.

Comment 9

[Will Bamberg [:wbamberg]]

(In reply to Jeff Griffiths (:canuckistani) from comment #8)
> (In reply to Irakli Gozilalishvili [:irakli] [:gozala] [@gozala] from
> comment #7)
> ...
> > > 1) In the current mode, when you drop third-party packages into the SDK,
> > > their docs show up in the doc browser. Will we stop supporting this feature
> > > in the future, and if not, where should I look for them? FOr the time being
> > > I just ripped all that out.
> > 
> > Honestly I have not took into account. I guess in a future we could look into
> > addons `node_modules` folder which is were dependencies will be installed by
> > a package manager. For the time being I guess looking into packages folder
> > and
> > using it should be fine.

If you need this bug fixed today, I won't be able to support docs for third party packages, because it means I'll have to do two quite different things for internal rather than external packages. It'll take me at least a copule more days to do that and make sure works properly. The current fix for this bug (https://github.com/wbamberg/jetpack-sdk/tree/793926) just doesn't support third-party packages at all.

Also, I'm wondering what the story will be in general for third-party package docs, when the SDK lands in Firefox. Since the docs will no longer be generated locally at all, will we just no longer support this feature? In which case, is it worthwhile to hack around to support it for the little while until we land in Firefox?

Comment 10

[Will Bamberg [:wbamberg]]

Attachment: fixes doc generation

This fixes doc generation to work in a packageless world. Some things to note:

- it doesn't support generation of docs for third party packages any more
- I haven't updated the content, which still talks about packages. This needs to be fixed, but not necessarily in this bug.
- cfx testcfx tests still fail, but I'm pretty sure it's not because of the doc generating code - all the doc tests pass

Maybe we should ask Mark Hammond to review all the Python code changes, which are quite substantial?

Comment 11

[Will Bamberg [:wbamberg]]

I'm closing this bug although it's not been r+'ed, because it got landed as part of the layout-change stuff in https://github.com/mozilla/addon-sdk/pull/569.