636693 - cfx docs throws "TypeError: 'int' object is not iterable"

Status: VERIFIED FIXED

(Add-on SDK Graveyard :: General, defect)

Description

[Irakli Gozalishvili [:irakli] [:gozala] [@gozala]]

It seems that cfx docs is broken. Also its seems that following commit caused the breakage:

https://github.com/mozilla/addon-sdk/commit/5512c7a143bfeac2cc9bbcffda744e8cd4c5cbd4#diff-28

P.S: Cherry picking really messes up history and it was hard to find the commit that caused the problem.

Comment 1

[Irakli Gozalishvili [:irakli] [:gozala] [@gozala]]

This is an error output:

(addon-sdk)➜  api-utils  (5512c7a...) cfx docs                                                                            

One moment.
Traceback (most recent call last):
  File "/Users/gozala/Projects/addon-sdk/python-lib/cuddlefish/server.py", line 447, in <module>
    httpd, port = fault_tolerant_make_httpd(env_root)
  File "/Users/gozala/Projects/addon-sdk/python-lib/cuddlefish/server.py", line 262, in fault_tolerant_make_httpd
    httpd = make_httpd(env_root, host, port, quiet=True)
  File "/Users/gozala/Projects/addon-sdk/python-lib/cuddlefish/server.py", line 250, in make_httpd
    web_docs = webdocs.WebDocs(env_root)
  File "/Users/gozala/Projects/addon-sdk/python-lib/cuddlefish/webdocs.py", line 63, in __init__
    self.base_page = self._create_base_page(root, base_url)
  File "/Users/gozala/Projects/addon-sdk/python-lib/cuddlefish/webdocs.py", line 130, in _create_base_page
    self._create_package_summaries(self.packages_json, is_low_level)
  File "/Users/gozala/Projects/addon-sdk/python-lib/cuddlefish/webdocs.py", line 112, in _create_package_summaries
    text += self._create_module_list(package_json)
  File "/Users/gozala/Projects/addon-sdk/python-lib/cuddlefish/webdocs.py", line 92, in _create_module_list
    package_json['files']['lib'])
  File "/Users/gozala/Projects/addon-sdk/python-lib/cuddlefish/webdocs.py", line 30, in get_documented_modules
    modules = get_modules(modules_json)
  File "/Users/gozala/Projects/addon-sdk/python-lib/cuddlefish/webdocs.py", line 24, in get_modules
    sub_modules = get_modules(modules_json[module])
  File "/Users/gozala/Projects/addon-sdk/python-lib/cuddlefish/webdocs.py", line 24, in get_modules
    sub_modules = get_modules(modules_json[module])
  File "/Users/gozala/Projects/addon-sdk/python-lib/cuddlefish/webdocs.py", line 20, in get_modules
    for module in modules_json:
TypeError: 'int' object is not iterable

Comment 2

[Myk Melez [:myk] [@mykmelez]]

(In reply to comment #0)
> It seems that cfx docs is broken.

Hmm, I can't reproduce the problem on Mac OS X 10.6.6 with Python 2.6.6, Ubuntu Linux 10.04 with Python 2.6.5, nor Windows 7 with Python 2.6.5.  With which OSes and Python versions are you noticing the problem?


> P.S: Cherry picking really messes up history and it was hard to find the commit
> that caused the problem.

How so?

Comment 3

[Will Bamberg [:wbamberg]]

Are there any extra packages/modules in your tree?

The code there uses the JSON structure returned from 'packaging.py' to build the list of modules. It assumes that in the 'lib' list in that JSON, items are either modules or are themselves lists of modules, i.e.:

'lib': 
  {
  'securable-module.js': {'size': 25326}, 
  'memory.js': {'size': 4754}, 
  'tab-browser.js': {'size': 25192}, 
  'content': 
    {
    'symbiont.js': {'size': 6334}, 
    'worker.js': {'size': 12798}, 
    'loader.js': {'size': 6691}
    }, 

This assumption is true for the packages and modules in the SDK, I think. If something in there caused this assumption to fail you might see an error like that. I'm not very familiar with the code in 'packaging.py' though, so I'm not sure what exactly would cause that.

Comment 4

[Irakli Gozalishvili [:irakli] [:gozala] [@gozala]]

So I have noticed that if I make a fresh clone it works fine again for some time but then it get's broken again :(

Also I noticed that cloning this repo: https://github.com/Gozala/jetpack-net into packages folder breaks cfx docs regardless of how fresh my clone is.

Mac OS X 10.6.6
Python 2.6.1

Comment 5

[Irakli Gozalishvili [:irakli] [:gozala] [@gozala]]

Also I do have additional packages but if I go back to the version mentioned above everything works fine again. Also removing all extra packages does not solves an issue. Only fresh clone seems to work.

Comment 6

[Will Bamberg [:wbamberg]]

(In reply to comment #4)
> So I have noticed that if I make a fresh clone it works fine again for some
> time but then it get's broken again :(

Is there any obvious trigger which breaks it again, after you get a fresh clone?
 
> Also I noticed that cloning this repo: https://github.com/Gozala/jetpack-net
> into packages folder breaks cfx docs regardless of how fresh my clone is.

So, to be clear, this sequence of commands:

~ > cd mozilla/
~/mozilla > git clone git://github.com/mozilla/addon-sdk.git
~/mozilla > cd addon-sdk
~/mozilla/addon-sdk(master) > source bin/activate
(addon-sdk)~/mozilla/addon-sdk/packages(master) > git clone git://github.com/Gozala/jetpack-net.git
(addon-sdk)~/mozilla/addon-sdk/packages(master) > cfx docs


? This works for me. (At least: it's not showing the modules inside the 'net' package, but the doc browser launches, and everything else seems like it's working.)

Comment 7

[Will Bamberg [:wbamberg]]

(In reply to comment #6)
> (In reply to comment #4)
> > So I have noticed that if I make a fresh clone it works fine again for some
> > time but then it get's broken again :(
> 
> Is there any obvious trigger which breaks it again, after you get a fresh
> clone?
> 
> > Also I noticed that cloning this repo: https://github.com/Gozala/jetpack-net
> > into packages folder breaks cfx docs regardless of how fresh my clone is.
> 
> So, to be clear, this sequence of commands:
> 
> ~ > cd mozilla/
> ~/mozilla > git clone git://github.com/mozilla/addon-sdk.git
> ~/mozilla > cd addon-sdk
> ~/mozilla/addon-sdk(master) > source bin/activate
+ (addon-sdk)~/mozilla/addon-sdk(master) > cd packages/
> (addon-sdk)~/mozilla/addon-sdk/packages(master) > git clone
> git://github.com/Gozala/jetpack-net.git
> (addon-sdk)~/mozilla/addon-sdk/packages(master) > cfx docs
>

Comment 8

[Will Bamberg [:wbamberg]]

One thing that might help me, if it's not too much trouble, would be to see the package index. You can generate it using cuddlefish.packaging:

(addon-sdk)~/mozilla/addon-sdk(master) > python
>>> import cuddlefish.packaging, os, simplejson
>>> pkg_cfg = cuddlefish.packaging.build_pkg_cfg(os.getcwd())
>>> pkg_index = cuddlefish.packaging.build_pkg_index(pkg_cfg)
>>> print simplejson.dumps(pkg_index, indent=2)

Then I could compare them to see if there are any interesting differences.

Comment 9

[Irakli Gozalishvili [:irakli] [:gozala] [@gozala]]

Output was huge so I created this gist out of it.

https://gist.github.com/52cb8a55cf25be538d22

Also I guess that json does not really needs to contain all the files from the nested git repositories or *.un~ files that are generated by VIM.

Comment 10

[Brian Warner [:warner :bwarner]]

I found it: to reproduce on a fresh addon-sdk checkout, just do:

  touch packages/addon-kit/lib/IAMNOTJAVASCRIPT
  cfx docs

The problem is that the test in get_modules() assumes that the JSON structure contains only javascript files (*.js) and directories, and mistakenly concludes that an entry that doesn't have ".js" in the name must be a directory. The JSON structure is basically just a recursive walk of the filesystem, so any random file (editor temp files, the funny .DS_Store file that OS-X likes to leave around, .gitignore files) will show up there.

We need to tolerate lots of random files in those directories. I'll write up a patch.

Comment 11

[Brian Warner [:warner :bwarner]]

Attachment: better data structure, tolerate non-.js files in packages

This patch fixes the crash, and adds some tests.

Comment 12

[Irakli Gozalishvili [:irakli] [:gozala] [@gozala]]

Brian,

Thanks for the patch. I think there is a different issue now, it looks like `keywords` property in package.json causes some troubles. Here is my output:

Traceback (most recent call last):
  File "/Users/gozala/Projects/addon-sdk/python-lib/cuddlefish/server.py", line 447, in <module>
    httpd, port = fault_tolerant_make_httpd(env_root)
  File "/Users/gozala/Projects/addon-sdk/python-lib/cuddlefish/server.py", line 262, in fault_tolerant_make_httpd
    httpd = make_httpd(env_root, host, port, quiet=True)
  File "/Users/gozala/Projects/addon-sdk/python-lib/cuddlefish/server.py", line 250, in make_httpd
    web_docs = webdocs.WebDocs(env_root)
  File "/Users/gozala/Projects/addon-sdk/python-lib/cuddlefish/webdocs.py", line 66, in __init__
    self.base_page = self._create_base_page(root, base_url)
  File "/Users/gozala/Projects/addon-sdk/python-lib/cuddlefish/webdocs.py", line 129, in _create_base_page
    self._create_package_summaries(self.packages_json, is_high_level)
  File "/Users/gozala/Projects/addon-sdk/python-lib/cuddlefish/webdocs.py", line 109, in _create_package_summaries
    if not include(package_json):
  File "/Users/gozala/Projects/addon-sdk/python-lib/cuddlefish/webdocs.py", line 53, in is_high_level
    return not is_low_level(package_json)
  File "/Users/gozala/Projects/addon-sdk/python-lib/cuddlefish/webdocs.py", line 56, in is_low_level
    return 'jetpack-low-level' in package_json['keywords']
  File "/Users/gozala/Projects/addon-sdk/python-lib/cuddlefish/bunch.py", line 13, in __getitem__
    item = dict.__getitem__(self, key)
KeyError: 'keywords'

Comment 13

[Irakli Gozalishvili [:irakli] [:gozala] [@gozala]]

Also removing packages that contain keywords: ["foo", "bar"] in package.json everything works. I think we should tolerate all the properties in package.json regardless if we understand them or not.

Also I've noticed that API-utils no longer shows up in docs and I'm not sure if that's intentional.

Comment 14

[Myk Melez [:myk] [@mykmelez]]

(In reply to comment #13)
> Also removing packages that contain keywords: ["foo", "bar"] in package.json
> everything works. I think we should tolerate all the properties in package.json
> regardless if we understand them or not.

Can you file that as a separate bug?

Comment 15

[Brian Warner [:warner :bwarner]]

Attachment: better data structure, tolerate non-.js files in packages, tolerate lack of 'keywords' property in package.json

here's an updated patch, which also tolerates the lack of a 'keywords' property in the package.json, which seems to be the problem in that traceback

Comment 16

[Brian Warner [:warner :bwarner]]

Landed in https://github.com/mozilla/addon-sdk/commit/345b68ec7e443f3aef85dc65b285a22ab229de75

Comment 17

[Brian Warner [:warner :bwarner]]

ok, I think that closes this one. I think we *do* tolerate all properties in package.json .. the problem was that webdocs.py was relying upon having a 'keywords' property (whereas for almost everything else it touches, it checks to see if the property is present before using it). The only remaining requirement is the ".name" property: if a package doesn't have that, webdocs.py will probably crash, but I think it's reasonable to require a .name property on all packages.

Comment 18

[Myk Melez [:myk] [@mykmelez]]

Thanks for the fix!

Cherry-pick commit:

https://github.com/mozilla/addon-sdk/commit/e47587d7fa845ea2074f15df4d020899e1ecb7be