Closed Bug 553490 Opened 10 years ago Closed 9 years ago

Separate the build process from everything else in DXR


(Webtools :: DXR, defect)

Not set


(Not tracked)



(Reporter: jcranmer, Assigned: jcranmer)



(3 obsolete files)

Right now, DXR requires you to modify several files in order to build anything that's not a Mozilla-based project. This bug is to allow configurations to specify any build procedure.

Some assorted thoughts (hashed it in an IRC conversation):
* Basic build procedure:
  1. Update source
  2. Patch source
  3. Clobber build
  4. Configure build
  5. Make build
  6. Collect DXR information

Step 3 is probably just remove the build directory. Any project complex enough to warrant DXR is most likely complex enough to handle separate build directories. Besides, DXR currently depends on there being a separate build directory.

In theory, most projects should be suitable for the first steps by supporting standard VCS systems (svn, cvs, hg, git, bzr); standard patch utilities (likely most Linux distributions maintain stuff for package building on their systems); standard configuration utilities (autoconf and cmake are the two I know of); and make in the build directory for the latter.

* For small scope, we can limit the build procedure here to the following:
  a. Clobber old build, create new build directory, configure
  b. Make

* Implementation-wise, humph suggested using Python scripts to do the building. So there would be a set of "standard" classes for the standard steps, and people can include custom steps for custom processes (e.g., mozilla).

* "the dxr.js I use should be broken into pieces, and probably in js I should do callbacks from a generic analysis script that the mozilla stuff layers on"

* For good non-Mozilla projects, humph suggested KDE (they already have expressed interest in the matter). I've got a hacked-together setup for vbam, which is a non-trivial program I can do DXR on without killing my poor computer.
A little bit more concrete details:

In terms of the config file, I would imagine a tree would look like this:

So far, this doesn't account for postprocessing step variation. In terms of config, only sourcedir would be necessary: objdir would default to sourcedir, steps to dxr.steps.StandardConfigure,dxr.steps.StandardMake; other attributes would be optional.

More thorough explanation:
The steps option is ultimately defined as a space/whitespace-separated list of python classes which represent the steps.

Options like:
StandardMake.environment=REPORT_BUILD='$(if $(filter %.cpp,$<),$(CXX) -dM -E $(COMPILE_CXXFLAGS) $< > $(subst .o,.macros,$@) 2>&1,@echo $(notdir $<))'
would be per-step options known only to a single step. The base class for the steps would know about a few options (so all steps would recognize): arguments and environment are the two that I can think of.

Some other special options, like environment, would be known to all steps, so if I set environment=DISABLE_DEBUG=1, every step will have that as an extra environment variable.

Each step would have some sort of read-access to the tree: SvnUpdate, for example, uses the upstream option to know which repository to pull from; it can therefore pull a source tree from scratch. Source directory, build directory, and environment would probably also be separate globally-known attributes. Environment, in particular, would need to be amenable to adding random stuff to.

The languages would be a special attribute that tells DXR which language tools it would be needing so it can set standard environment options. If not present, either a simple languages=c,c++ could be assumed or options could be enabled for all recognized languages. There is another section, [languages], which sets the global parameters for all language tools. For example:


The C++ language class would read g++-plugin and dehydra-loc to know what to set CXX and CXXFLAGS to for environments, for example.

An example of what Mozilla could look like under a simple version of this bug:

# By default, Clobber does make clean
# Clobber.targets would enable you to merely change the target for make
# The .1 says it's the first CustomShell step -p %(objdir)s/config
                 cp %(dxrscripts)s/ %(objdir)s/config
                 mkdir -p %(objdir)s/js/src/config
                 cp %(dxrscripts)s/ %(objdir)s/js/src/config
# Of course, if it's not there, it's implied to be .1
# Or, perhaps, it applies to all of them?
CustomMake.environment=REPORT_BUILD='$(if $(filter %.cpp,$<),$(CXX) -dM -E $(COMPILE_CXXFLAGS) $< > $(subst .o,.macros,$@) 2>&1,@echo $(notdir $<))'
# Everybody has this environment
# Note that it references another option in here

It's not the simplest thing in the world, but it works.

Things to note:
* This only worries about the process through making a build. I'm not sure yet how to worry about postprocessing issues.
* Thinking ahead, we probably want some sort of extensible control for plugins. I've left room for plugins in build steps and at the language level, but I'm sure that people would eventually want to hook into stuff per-project-ish (e.g., callgraph, or perhaps some gobject-specific analysis). I haven't decided how that squeezes in here yet.
* Should there be cross-tree defaults? If I have a set of mozilla trees for DXR, I probably want to have just one place where I can change stuff for most stuff at once. I'm not sure how to handle this.
* The <step>[.<n>].<option> isn't the best syntax, but it's the best I can think up on short notice. Perhaps it may be possible to name specific steps for use as setting further options....
* Currently, everything that's not a recognized section (i.e., DXR and Web) is a tree. That's pretty poor forward-compatibility. I see two main choices:
  1. Only stuff with sourcedir options form a tree.
  2. Have a trees option somewhere in the tree.
* The format of what ConfigParser accepts is actually poorly documented. Adding some documentation here for a DXR release would be helpful.
* The languages option may be a solution in search of a problem. For now, I'm pretty much ignoring it anyways.
* Standard set of per-step options: More, less?
  * cwd (directory for process execution)
  * arguments (extra arguments at end)
  * environment (extra environments)
  * outlog (where to send output)
  * errlog (ditto)
  * infile (what to enter for input)
Oh yeah, I forgot to mention what I see as the basic steps at this point:

CustomShell: A custom shell script
  .sh: Text to run as shell script
  .shell: Shell to use
  .sh-file: File to use as shell script (overrides .sh)
CustomMake: Runs a custom make command
  .file: make -f <file>
  .targets: whitespace-separated list of targets
Clobber: Clobbers the object directory
  .style: rm uses rm -rf to remove the directory (overrides targets).
          Anything else is an error
  .targets: Similar to targets of CustomMake. clean by default
{Hg,Svn,Cvs,Git}Update: Updates the source tree
  ... I don't know yet ...
StandardConfigure: Runs $(srcdir)/configure in objdir
  Options: beyond standard, I don't know
CmakeConfigure: Runs cmake $(srcdir) in objdir
  Options: beyond standard, I don't know
StandardMake: Runs make in objdir
  Options: beyond standard, I don't know
PatchSource: Patches source files
  Options: (in order of checking, first found is used)
  .series: A file containing a list of patches in the spirit of quilt or mq's series file.
  .patch-dir: A directory that contains patches. If the file "series" or "00list" is found in that directory, use it as the patch list. Otherwise, use only files that end in .diff or .patch if more than one such file exists. Otherwise, use any files in that directory. Last two processed in ASCII order of filenames.
  .patches: A list of patches to use.
  Other options for patching:
  .strip: -pdepth
  .guards: see hg help qguard to understand what this does.

I used <> to figure out how Debian's patching system worked.
Joshua, I'm trying to use DXR for a home project, and I probably could give you a hand as part of setting this up.

I do regret that doesn't mention the instruction it contains are outdated, and one should do what's described in the README of the hg repository of dxr instead.

What about first hosting a really maintained (maybe synchronized with the README) how to setup DXR page on mdc ?
I do have a hacked version of DXR for another project that works. Indeed, that was what I brought to the table in the first IRC discussion about how to generify the process.

At this point, I have a pretty good idea of what the config file would look like; what I'm lacking right now is the python API itself. My starting point is <>; but I haven't had time to poke at it too much recently due to other obligations.
Attached patch The start (obsolete) — Splinter Review
The basic idea here is that all of the dxr stuff gets organized in a nice python directory with namespaces. The eventual goal is to eradicate the xref-scripts directory.

What I've done is moved the building and first awk collection out of turns out that stuffing everything in a python set is faster than awk. Apparently, not having causes the sql insertion to go haywire. I'm not quite sure what all is going on in the SQL stuff, so I'm leaving that there for now.

There is more work needed on setting environment variables and all that, though...
Assignee: nobody → Pidgeot18
Attached patch First step (obsolete) — Splinter Review
What this patch does do:
* Reduces build steps from custom mozconfig step to specifying a series of steps.
* Requires a list of trees in the dxr config instead of assuming all non-known sections are some tree are another
* Should run on python 2.6 now.
* Moves some of the script processing into the dxr python script. Not all of the converted stuff worked, so I'm leaving those separate from now.
* Moves xref-scripts/ into (as well as some other python stuff).
* Creates a dxr/ directory for python modules for DXR.

What would need to be done in the future
* Generic language support. cpp2html and idl2html would be moved out of topdir as a result. This needs serious design work...
* Add support for stdin, stdout, and stderr processing in the runCommand stuff. Both as that and as redirected to output
* Add support for patching builds and updating source trees. The latter two would be tied into continuous indexing
* The configure step may be helpful as well...
* Move more of the SQL post-processing into the python file.
* Callgraph support!
* Macro support!!
* More stuff should probably be moved into common DXR library routines.
* Per-tree python module loading (for extensions).

With this patch, it should be possible to do an index of any C++ program with minimal source code changes and only editing a config file for DXR. Once the language stuff is cleaned up, it should be possible to make DXR packages for Linux distros.
Attachment #446644 - Attachment is obsolete: true
Attachment #457752 - Flags: review?
Attached patch Updated patch (obsolete) — Splinter Review
This patch should apply without problems to tip dxr.

I'm not wholly certain this is the full correct mozconfig (I tested in another repo copy), you may want to replace the paths with the correct values and build mozilla-central for yourself.
Attachment #457752 - Attachment is obsolete: true
Attachment #468200 - Flags: review?
Attachment #457752 - Flags: review?
Attachment #468200 - Flags: review? → review?(david.humphrey)
Per various conversations on IRC, we will be rearchitecting DXR so as to remove the idea of the build process from DXR. In essence, it will be sufficient to set CC/CXX/etc. as appropriate for your build repository.
OS: Windows ME → All
Summary: Generify the DXR build process → Separate the build process from everything else in DXR
Attachment #468200 - Attachment is obsolete: true
Attachment #468200 - Flags: review?(david.humphrey)
Essentially fixed now.
Closed: 9 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.