1426456 - Add analyzer for python code via scip-python

Status: RESOLVED FIXED

(Webtools :: Searchfox, enhancement)

Description

[Kartikaya Gupta (email:[e-mail])]

It should be possible to add analysis for python the same way we do c++ and js code. Some useful links:

https://docs.python.org/2/library/ast.html
https://greentreesnakes.readthedocs.io/en/latest/

And sample code to get the AST:

import ast
m = ast.parse(open("some-python-file.py", "r").read(), "some-python-file.py")
ast.dump(m)

Comment 1

[Jeff Muizelaar [:jrmuizel]]

See also https://github.com/mozilla/dxr/tree/master/dxr/plugins/python

Comment 2

[Kartikaya Gupta (email:[e-mail])]

https://github.com/davidhalter/jedi might also be useful

Comment 3

[Andrew Sutherland [:asuth] (he/him)]

https://github.com/sourcegraph/scip-python (based on pyright) is I think now the clear choice for us to use.

Comment 4

[Andrew Sutherland [:asuth] (he/him)]

I turned on some stuff in https://github.com/mozsearch/mozsearch-mozilla/pull/227 building on some work in https://github.com/mozsearch/mozsearch/pull/690 plus of course a ton of the SCIP work by Emilio, myself, and Nicolas, but with many, many thanks to the team at sourcegraph for their work on scip and scip-python and of course the underlying pyright team.

A lot of python code is now indexed, but not all of it. scip-python just falls down and eats CPU forever if we let it try and ingest the wpt tree and there was no easy way to get any insight about what was going on inside scip-python because the interaction with pyright ends up being to just turn a crank as type inference logic happens, but there are no instrumentation hooks to easily understand what data it's processing that is overwhelming it.

I dropped a note about this in the slack python channel but do not plan to update the root help.html matrix or send an update to dev-platform anytime soon because the level of python semantic indexing people should expect is none. I'm also going to stick to this being m-c only for now.