842547 - Generated files missing on DXR (e.g. headers/sources generated from webidl/xpidl/ipdl)

Status: RESOLVED WONTFIX

(Webtools Graveyard :: DXR, defect, P3)

Description

[Georg Fritzsche [:gfritzsche]]

Apparently since the recent change in DXR (at least the frontend changed) the generated files don't appear in search results anymore.
This includes e.g. <objdir>/ipc/ipdl/PPluginInstanceChild.cpp

Searching for PPluginInstanceChild.cpp before the change took me to something like __GENERATED__/ipc/ipdl/PPluginInstanceChild.cpp, while now it just says "Sorry, no results!".

Comment 1

[Georg Fritzsche [:gfritzsche]]

While DXR works fine again (yay!), the generated files are apparently still not being indexed.

Comment 2

[Erik Rose [:erik][:erikrose]]

Note to self: see about moving the indexing phase to after the compilation phase. Not sure if we need the files table prior to compilation for any reason.

Comment 3

[Erik Rose [:erik][:erikrose]]

I'd kind of like to do that anyway so we can get build errors up front, without having to first wait for trigram indexing.

Comment 5

[(no longer active)]

So note that all of our generated code is created during the export phase, so technically we don't need to index after the full build is finished.  All we need to do is to make sure that we run make export first, and then run the indexing phase.  With the clang plugin, for example, it should be able to see all of this generated code since it runs after the make export phase.  It might be the case that we just fail to store the information about code outside of the source directory when generating the index.

Note that this might be one of the big reasons why developers cannot use DXR right now.  It would be really great if we can prioritize this.  Thanks!

Comment 6

[Erik Rose [:erik][:erikrose]]

Those are just the kinds of reasons we want to squash, so I'm sure I'll have follow-up questions in a bit. Thanks!

Comment 7

[Ryan VanderMeulen [:RyanVM]]

Saw complaints on IRC about this, figured it was worth getting on Laura's radar :)

Comment 8

[(no longer active)]

Do you guys have any plans to fix this in the near future?  As a long time advocate for DXR in the past few years, this is literally _the_ only bug that prevents me to switch from mxr to dxr for most of my code searches.  I'd appreciate if you could please bump the priority on this.

Comment 9

[Ting-Yu Lin [:TYLin] (PDT, UTC-7)]

Please help fix this bug. It will make DXR a a lot more useful.

Comment 10

[Erik Rose [:erik][:erikrose]]

Thanks for the votes, all; that really helps prioritize the roadmap. I've bumped this up from a 5 (and was surprised it was so low) to 3, which brings it into the realm of possibility for next quarter: https://wiki.mozilla.org/DXR_Roadmap#Indexing.

It would be great if those of you who commented could comment again citing your specific use cases. Previously, I hypothesized that simply making DXR understand IDLs natively would suffice, so I'm particularly interested to have that hypothesis invalidated.

Cheers!

Comment 11

[Benjamin Smedberg]

See https://ask.mozilla.org/question/536/can-dxr-give-me-all-the-c-callers-of-nsifilecontains/

Comment 12

[Joshua Cranmer [:jcranmer]]

(In reply to Erik Rose [:erik][:erikrose] from comment #10)
> Thanks for the votes, all; that really helps prioritize the roadmap. I've
> bumped this up from a 5 (and was surprised it was so low) to 3, which brings
> it into the realm of possibility for next quarter:
> https://wiki.mozilla.org/DXR_Roadmap#Indexing.
> 
> It would be great if those of you who commented could comment again citing
> your specific use cases. Previously, I hypothesized that simply making DXR
> understand IDLs natively would suffice, so I'm particularly interested to
> have that hypothesis invalidated.

It's worth noting that the most pressing part of this bug could be summarized as "make it possible to search on functions defined in generated code" (such as nsIFile::Contains, etc.). While the display of generated code is still a very valuable resource in itself, it is the inability to search on things like nsIFile::Contains that greatly hampers usability of DXR.

Comment 13

[Erik Rose [:erik][:erikrose]]

When you attack this, remember to change the bit in InclusionDirective() in the clang plugin where we ignore generated files. Search for "Support generated files".

Comment 14

[Peter Elmers [:new_one]]

I am not sure what exactly changed in the past few months, but it looks like the generated files do appear. I got around to running a full index, and the search `path:PPluginInstanceChild.cpp` brings up `obj-x86_64-unknown-linux-gnu/ipc/ipdl/PPluginInstanceChild.cpp` as the result.
I notice that the staging server has an almost empty object directory (https://dxr.allizom.org/mozilla-central/source/obj-x86_64-unknown-linux-gnu), and I suspect this happens if the build and the file indexing stages are run separately. See http://git.io/vkHjD, which clears the object directory; this step happens even if `skip_stages = build`.

Comment 15

[Erik Rose [:erik][:erikrose]]

Hmm, that's odd. I can tell you for sure that we don't use skip_stages in staging or production.

Comment 17

[Peter Elmers [:new_one]]

Looks like we have generated files again. https://dxr.allizom.org/mozilla-central/source/obj-x86_64-unknown-linux-gnu

What's changed in the build process since last month?

Comment 18

[Kendall Libby [:fubar] (he/him)]

older builds had object_folder and MOZ_OBJDIR set to build out of the src tree (e.g. obj/mozilla-central), then it was "broken" until yesterday-ish when I noticed and returned it to how it had been.

historically, it always looked like this:
[tree]
source_folder=src/tree
object_folder=obj/tree

so aiui dxr wouldn't have indexed obj. it's not clear to *me* what it ought to be, only how it was. erik?

Comment 19

[Erik Rose [:erik][:erikrose]]

Let's put the object_folder *inside* the source_folder. That way, DXR will index the generated files, which we actually want, since some of them are C files we want to search and cross-reference to. On my box, object_folder=/home/vagrant/src/mozilla-central/obj-x86_64-unknown-linux-gnu. I must have picked that up from a default; obviously, it's arch-specific and so may differ for you. I suppose we ought to name it something less arch-specific in case we change archs someday; otherwise, we'd break all the links. Suggestions? OTOH, everybody would instantly know what it was if we left the name alone. We're unlikely to move off x86_64 or Linux anytime soon. Anybody remember what "unknown" signifies?

Comment 20

[Erik Rose [:erik][:erikrose]]

This is done on staging: https://dxr.allizom.org/mozilla-central/source/obj-x86_64-unknown-linux-gnu.

Comment 21

[Nika Layzell [:nika] (ni? for response)]

(In reply to Erik Rose [:erik][:erikrose] from comment #19)
> Anybody remember what "unknown" signifies?

The "unknown" is the vendor part of the Target Triple. It is <arch>-<vendor>-<sys>-<abi>, so we are using x86_64, unknown vendor, linux system, and the gnu ABI.

(In reply to Erik Rose [:erik][:erikrose] from comment #20)
> This is done on staging:
> https://dxr.allizom.org/mozilla-central/source/obj-x86_64-unknown-linux-gnu.

That's awesome :). Unfortunately, it breaks, for example, typing in the name of a file and jumping directly to it in many cases. For example, if I type in `nsIURI.idl`, I would expect to be loaded into `netwerk/mase/nsIURI.idl`, however now that the object directory is also being indexed, I don't get redirected anywhere, because there is also an `nsIURI.idl` at `obj.../dist/idl/nsIURI.idl`. The file in the dist directory is a symlink to the other one (at least on my system), so it may make sense to make references to the file in the object directory become references to the file in netwerk, and not include the object directory file in DXR.

Comment 22

[Peter Elmers [:new_one]]

Michael, I would say that what you mention is a separate issue. Feel free to file a bug if you think of a good way to handle symlinks. Note that in https://bugzilla.mozilla.org/show_bug.cgi?id=1181772, Nihanth describes a workaround that would exist if we fix that bug.

Comment 23

[Peter Elmers [:new_one]]

(In reply to Michael Layzell [:mystor] from comment #21)
> (In reply to Erik Rose [:erik][:erikrose] from comment #19)
> > Anybody remember what "unknown" signifies?
> 
> The "unknown" is the vendor part of the Target Triple. It is
> <arch>-<vendor>-<sys>-<abi>, so we are using x86_64, unknown vendor, linux
> system, and the gnu ABI.
> 
> (In reply to Erik Rose [:erik][:erikrose] from comment #20)
> > This is done on staging:
> > https://dxr.allizom.org/mozilla-central/source/obj-x86_64-unknown-linux-gnu.
> 
> That's awesome :). Unfortunately, it breaks, for example, typing in the name
> of a file and jumping directly to it in many cases. For example, if I type
> in `nsIURI.idl`, I would expect to be loaded into `netwerk/mase/nsIURI.idl`,
> however now that the object directory is also being indexed, I don't get
> redirected anywhere, because there is also an `nsIURI.idl` at
> `obj.../dist/idl/nsIURI.idl`. The file in the dist directory is a symlink to
> the other one (at least on my system), so it may make sense to make
> references to the file in the object directory become references to the file
> in netwerk, and not include the object directory file in DXR.


I went ahead and filed a bug on hiding symlinks in search results: https://bugzilla.mozilla.org/show_bug.cgi?id=1185687

Comment 24

[Sylvestre Ledru [:Sylvestre]]

DXR is no longer available. Searchfox is now replacing it.
See meta bug 1669906 & https://groups.google.com/g/mozilla.dev.platform/c/jDRjrq3l-CY for more details.