1074938 - More scalable, delegatable indexing process

Status: RESOLVED WONTFIX

(Webtools Graveyard :: DXR, defect)

Description

[hwine]

As the number of trees deployed grows, it would be great if folks who know the tree could provide the needed scripts to build the tree for dxr purposes.

One approach would be to support a conf directory (ala apache2, logrotate, cron) where scripts could be added & removed.

Bonus points if the driver learns to skip/ignore failed builders, and thus not let one bad apple spoil the whole bunch baby (sorry, gratuitous ancient R&B/Pop reference)

Comment 1

[Erik Rose [:erik][:erikrose]]

Fubar and I were thinking about Jenkins. It's already designed to check things out of source control, build them, run scripts on them, and deal with failures. We'd get a nice web UI (with ACLs?) for free, along with logging and history. Thoughts?

Comment 2

[hwine]

My primary interest is in having the "tree owner" be responsible for "tree build". Sounds like the Jenkins sidecar would handle that.

A secondary interest is in simplifying (and eventually automating) the admin work to add a new tree. I tend to stop short of "self serve", as I'd like to avoid "tragedy of the commons" abuse of dxr. e.g. jobs added faster than disk can be ordered.

I don't know Jenkins well enough to know if it would easily fit into that sort of workflow, and happy to defer to y'all's recommendation.

Comment 3

[Erik Rose [:erik][:erikrose]]

We should chat in Portland (and here, if you like). I'm going to have some kind of DXR talk and/or open space.

It used to be that releng was responsible for running the DXR indexing jobs. We quit that for the past year or so, as DXR started changing faster than the build setups. However, as we add more trees, the balance will shift the other way, and it may make sense to re-delegate build stewardship back to releng or whomever. (Also, the warranty is up on our build box, so it's decision time.) I'm not clear on how releng is organized these days and what projects they have under their wing. It'd be swell if you could fill me in. :-)

Comment 4

[Sylvestre Ledru [:Sylvestre]]

DXR is no longer available. Searchfox is now replacing it.
See meta bug 1669906 & https://groups.google.com/g/mozilla.dev.platform/c/jDRjrq3l-CY for more details.