1505869 - Windows indexing has not indexed mozglue/misc/TimeStamp_windows.h

Status: RESOLVED FIXED

(Webtools :: Searchfox, defect)

Description

[Markus Stange [:mstange]]

This file doesn't show any indexed understanding: https://searchfox.org/mozilla-central/source/mozglue/misc/TimeStamp_windows.h

Similarly, in a different file which accesses e.g. mUsedCanonicalNow, the context menu on mUsedCanonicalNow does not have a "go to definition" item:
https://searchfox.org/mozilla-central/source/ipc/glue/IPCMessageUtils.h#830

Comment 1

[Kartikaya Gupta (email:[e-mail])]

I looked into the issue with TimeStamp_windows.h. What happens is that the analysis gets generated into __GENERATED__/dist/include/mozilla/TimeStamp_windows.h rather than into mozglue/misc/TimeStamp_windows.h, because when TimeStamp_windows.cpp and other files are compiled, the #include picks it up from <objdir>/dist/include/mozilla instead of the source dir. So the tarball of analysis files has the analysis data, but for the "generated source" file of dist/include/mozilla/TimeStamp_windows.h. Of course, that's not actually a generated source file, and so isn't part of the generated sources tarball, and so we throw away the analysis data.

For comparison, both linux and macosx produce analysis for mozglue/misc/TimeStamp* and so everything there works fine. I suspect this might be because linux and macosx use symlinks in the objdir, so all the things in <objdir>/dist/include/mozilla/ are just symlinks to the original source file, and the location that clang gives us dereferences the symlink. On Windows presumably the files are actually copied and so we get the objdir path in the location.

No ideas off the top of my head how to resolve this, suggestions welcome.

Comment 2

[Kartikaya Gupta (email:[e-mail])]

Nathan, do you know of any way (e.g. a mach command) for the build system to generate a mapping of all the files linked/copied into <objdir>/dist/include and where they come from in the source tree?


I tried to see if I could just hack around this problem by finding a file in the source tree with the same name as that in dist/include and hoping they were the same, and while most filenames (e.g. "TimeStamp_windows.h") are unique enough that this works, there are a number that are not (e.g. "Path.h" or "HashTable.h"), so it would be good to have a more robust solution if possible.

Comment 3

[Kartikaya Gupta (email:[e-mail])]

Actually even better than a mach command would be if I can emit the mapping into a file that is a build artifact on the searchfox build job.

Comment 4

[Nathan Froyd [:froydnj]]

(In reply to Kartikaya Gupta (email:[e-mail]) from comment #3)
> Actually even better than a mach command would be if I can emit the mapping
> into a file that is a build artifact on the searchfox build job.

This idea seems completely reasonable; it ought to be roughly as straightforward as the generated-files.json file we emit for GENERATED_FILES.  I don't think there's any pre-existing mach command for this, as you asked in comment 2, though.

Comment 5

[Kartikaya Gupta (email:[e-mail])]

Cool. After poking around a bit it looks like there is already a file at <objdir>/_build_manifests/install/dist_include which seems to have what I want. It's emitted at [1] and the file format is pretty easy to parse (the same python file contains a parser too if we want to use that). I'll see if I can copy this file to the artifacts folder for a --enable-mozsearch-plugin build, and then use in the indexer setup to merge the analysis data accordingly.

[1] https://searchfox.org/mozilla-central/rev/06d5d5ae4396be85f26e8548323ee6c12e7bce4e/python/mozbuild/mozpack/manifests.py#223

Comment 6

[Kartikaya Gupta (email:[e-mail])]

This is fixed on dev.searchfox.org but I need to investigate some more misplaced analysis that turned up before I'll be ready to put patches up.

Comment 7

[Kartikaya Gupta (email:[e-mail])]

For posterity:

[1] is the try push where I export the dist-include manifest file. I wrote a patch on the searchfox file to iterate through all the dist/include files and use this manifest to map them to source files, and then merge the analysis data accordingly. That works fine.

I also made it emit warnings if it found other analysis data in dist/include that couldn't be mapped to a source file, to catch any remaining cases of this class of problem. It found a bunch (list below). Most of them are generated files, generated at [2].

In the example of AccessibleComponent_i.c, it is generated into <objdir>/accessible/interfaces/ia2/ but then also copied into <objdir>/dist/include, and it's not clear to me what does that copy. Also it's not clear how it manages to skip being in the manifest, because I would expect it to hit the code at [3] and should still end up in the manifest as an optional file, but it's not there at all.

./dist/include/AccessibleComponent_i.c on win64
./dist/include/AccessibleHypertext2_i.c on win64
./dist/include/AccessibleApplication_i.c on win64
./dist/include/ISimpleDOM.h on win64
./dist/include/AccessibleTable.h on win64
./dist/include/xpcom-config.h on win64
./dist/include/AccessibleImage_i.c on win64
./dist/include/Accessible2_3_i.c on win64
./dist/include/xpcAccEvents.h on win64
./dist/include/AccessibleDocument_i.c on win64
./dist/include/Accessible2_3.h on win64
./dist/include/AccessibleText_i.c on win64
./dist/include/AccessibleAction_i.c on win64
./dist/include/nsCSSPropertyID.h on win64
./dist/include/Accessible2_2.h on win64
./dist/include/Accessible2.h on win64
./dist/include/IA2CommonTypes.h on win64
./dist/include/AccessibleEventId.h on win64
./dist/include/necko-config.h on win64
./dist/include/AccessibleStates.h on win64
./dist/include/AccessibleTableCell_i.c on win64
./dist/include/AccessibleTable_i.c on win64
./dist/include/AccessibleAction.h on win64
./dist/include/AccessibleEditableText_i.c on win64
./dist/include/AccessibleHypertext2.h on win64
./dist/include/AccessibleRole.h on win64
./dist/include/AccessibleValue.h on win64
./dist/include/ErrorList.h on win64
./dist/include/AccessibleValue_i.c on win64
./dist/include/ISimpleDOM_i.c on win64
./dist/include/cairo/cairo-features.h on win64
./dist/include/Accessible2_i.c on win64
./dist/include/AccessibleRelation.h on win64
./dist/include/AccessibleDocument.h on win64
./dist/include/mozilla/TelemetryProcessEnums.h on win64
./dist/include/mozilla/ServoStyleConsts.h on win64
./dist/include/mozilla/TelemetryEventEnums.h on win64
./dist/include/mozilla/TelemetryHistogramEnums.h on win64
./dist/include/mozilla/CompositorAnimatableProperties.h on win64
./dist/include/mozilla/Services.h on win64
./dist/include/mozilla/dom/UseCounterList.h on win64
./dist/include/mozilla/ServoCSSPropList.h on win64
./dist/include/mozilla/TelemetryScalarEnums.h on win64
./dist/include/AccessibleTable2.h on win64
./dist/include/nsGkAtomList.h on win64
./dist/include/AccessibleRelation_i.c on win64
./dist/include/IGeckoCustom.h on win64
./dist/include/AccessibleEditableText.h on win64
./dist/include/AccessibleHypertext_i.c on win64
./dist/include/AccessibleHypertext.h on win64
./dist/include/AccessibleHyperlink_i.c on win64
./dist/include/HandlerData_i.c on win64
./dist/include/AccessibleImage.h on win64
./dist/include/js-config.h on win64
./dist/include/nsGkAtomConsts.h on win64
./dist/include/AccessibleTable2_i.c on win64
./dist/include/AccessibleComponent.h on win64
./dist/include/AccessibleApplication.h on win64
./dist/include/AccessibleTableCell.h on win64
./dist/include/AccessibleHyperlink.h on win64
./dist/include/Accessible2_2_i.c on win64
./dist/include/CrashAnnotations.h on win64
./dist/include/HandlerData.h on win64
./dist/include/AccessibleText.h on win64

[1] https://treeherder.mozilla.org/#/jobs?repo=try&group_state=expanded&revision=60cd2730f92013420ecf4e946a6b94161d434d13 
[2] https://searchfox.org/mozilla-central/rev/7f7c353e969e61a6a85201cc8ad3c3de12ac30d8/accessible/interfaces/ia2/moz.build#21
[3] https://searchfox.org/mozilla-central/rev/7f7c353e969e61a6a85201cc8ad3c3de12ac30d8/python/mozbuild/mozbuild/backend/recursivemake.py#1526

Comment 8

[Kartikaya Gupta (email:[e-mail])]

I haven't had the chance to look into this further. I think I'll spin it out into a follow-up bug and land what I have since that already fixes the problem for hundreds of headers that are currently missing windows analysis.

Comment 9

[Kartikaya Gupta (email:[e-mail])]

Attachment: Bug 1505869 - Expose the dist-include manifest as a taskcluster artifact for mozsearch builds. r?froydnj

Comment 10

[Kartikaya Gupta (email:[e-mail])]

Filed bug 1507419 as the follow-up.

Comment 11

[Kartikaya Gupta (email:[e-mail])]

https://github.com/mozsearch/mozsearch-mozilla/pull/26

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/0d6a2a521efe
Expose the dist-include manifest as a taskcluster artifact for mozsearch builds. r=froydnj

Comment 13

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/mozilla-central/rev/0d6a2a521efe

Comment 14

[Kartikaya Gupta (email:[e-mail])]

Fix is deployed.

Comment 16

[Markus Stange [:mstange]]

Thanks!!