1030309 - Framework that indexes into ES

Status: RESOLVED FIXED

(Webtools Graveyard :: DXR, defect)

Description

[Erik Rose [:erik][:erikrose]]

Write the framework code that extracts information from the plugins and sticks it into ES. This also includes porting the static-HTML-building bits to the new plugin architecture, unless it turns out to be easier to transition immediately to request-time rendering.

Comment 1

[Erik Rose [:erik][:erikrose]]

Questions that will need to be ironed out:

Figure out how to build indices independently:
    Figure out where to put the registry of indices:
        config file?
        another index?
        Do we use ES aliases?
    Who chooses the index name?
        Does it get passed into dxr-index?

Comment 2

[Erik Rose [:erik][:erikrose]]

Answers:

It gets complicated keeping on-disk folders and ES indices synched when there are format versions and parallel building in play. Let's just stick with serial builds until we have request-time rendering, at which time we can do the full-on https://wiki.mozilla.org/DXR_Parallel_Tree_Indexing.

We'll build in series, lay down a config.py file that can keep holding all the tree metadata, and treat format versions just as we do now, with numbered symlinks pointing to instance folders. The main change will be the addition of an ES alias name to the value side of the TREES OrderedDict.

There will be a little race period between the FS move and the alias change, but the effects will be negligible: a duration of <= 1s (ES refresh time) and no interesting effects (the search results might be older than the rendered markup for 1s).

dxr-build.py should make up the index names based on the alias names in the config file. This fits with my policy of relieving the build system of requirements. The build script is not in the DXR source tree and is fairly Mozilla-specific. Thus, anything we require of it is a burden on each community deployer.

Comment 3

[[github robot]]

Commit pushed to es at https://github.com/mozilla/dxr

https://github.com/mozilla/dxr/commit/9dc89dac25130a99bf2cc1ba6d5303ee725a440f
Port the build pipeline and test harness to ES. Fix bug 1030309. Close #308.

Comment 4

[[github robot]]

Commit pushed to master at https://github.com/mozilla/dxr

https://github.com/mozilla/dxr/commit/9dc89dac25130a99bf2cc1ba6d5303ee725a440f
Port the build pipeline and test harness to ES. Fix bug 1030309. Close #308.