1175886 - [tracker] build system updates

Status: RESOLVED WONTFIX

(Webtools Graveyard :: DXR, defect)

Description

[Kendall Libby [:fubar] (he/him)]

tracking bug so I don't lose them between the cracks

jenkins is up and running independent builds on a timer, but there's still some things to do:

1) uncomplicate docker images and move the SCM bits to jenkins to handle, gets us builds on code updates, rather than timed
2) documentation
3) have jenkins build DXR (at least as upstream build for), docker images
4) move jenkins off the admin node to some place publicly accessible (but with auth, probably a la ci.m.o)
5) move dxr-docker repo to mozilla GH org (or even roll into dxr)
6) semi-related, build docker images for ES and web head for testing purposes (see also jamon's docker work)

Comment 1

[Kendall Libby [:fubar] (he/him)]

1) is nearly done - ran into some dxr issues in testing (bug 1181189). hgrc fix should be easy since we control the jenkins user creation on the build nodes.

need to clean up the docker build, preferably stealing from the mozilla-central docker setup for TC. also, since docker can't pull in files above the build dir, need to drop dxr.config files there when built (or come up with a clever way to pull from a repo, but I think we run into chicken-egg problems there; documenting a workflow will help).

Comment 2

[Kendall Libby [:fubar] (he/him)]

have new docker images (quay.io/fubar) based on the docker stuff in mozilla-central/testing/docker. specifically, building upon ubuntu-build, since the centos images were failing m-c builds all over the place. currently trying to get around an issue with a timeout to the ES cluster at the end of the build. :-(  once that's sorted, will merge to master and get the existing trees rebuilding with the new image and SCM polling.

Comment 3

[Erik Rose [:erik][:erikrose]]

When does it hang? During "Refreshing Index"?

Comment 4

[Kendall Libby [:fubar] (he/him)]

00:27:01 rust-dxr post_build
00:27:01  - Processing files - first pass
00:27:01  - Processing files - second pass
00:27:01  - Updating references
00:27:01  - Generating inheritance graph
00:27:01  - Generating crate info
00:27:01  - Generating qualnames
00:27:01 A worker failed while indexing /builds/dxr-build-env/src/mozilla-central-test/obj-x86_64-unknown-linux-gnu/all-tests.json:
00:27:01 Traceback (most recent call last):
00:27:01   File "/builds/dxr-build-env/venv/local/lib/python2.7/site-packages/dxr-0.1-py2.7.egg/dxr/build.py", line 579, in index_chunk
00:27:01     index_file(tree, tree_indexers, path, es, index)
00:27:01   File "/builds/dxr-build-env/venv/local/lib/python2.7/site-packages/dxr-0.1-py2.7.egg/dxr/build.py", line 550, in index_file
00:27:01     es.bulk(chunk, index=index, doc_type=LINE)
00:27:01   File "/builds/dxr-build-env/venv/local/lib/python2.7/site-packages/pyelasticsearch/client.py", line 93, in decorate
00:27:01     return func(*args, query_params=query_params, **kwargs)
00:27:01   File "/builds/dxr-build-env/venv/local/lib/python2.7/site-packages/pyelasticsearch/client.py", line 448, in bulk
00:27:01     query_params=query_params)
00:27:01   File "/builds/dxr-build-env/venv/local/lib/python2.7/site-packages/pyelasticsearch/client.py", line 281, in send_request
00:27:01     raise exc.info
00:27:01 ReadTimeoutError: HTTPConnectionPool(host='node46.bunker.scl3.mozilla.com', port=9200): Read timed out. (read timeout=60)
00:27:01 

(I'm not sure why that says rust-dxr at the top; it's an m-c build)

Comment 5

[Erik Rose [:erik][:erikrose]]

It must be because you have the rust plugin enabled. It shouldn't hang, but you can certainly get past this for now by disabling it. (Eventually, we'll want to analyze the little bit of rust code that's in m-c.)

Comment 6

[Kendall Libby [:fubar] (he/him)]

tested indexing w/ pypy and oomkiller went to town on it. will have to try again later with a larger VM. back to testing builds with cpython.

also, have moved docker images off to quay.io/fubar, and will hook it up to the GH repo to automatically build docker images. doesn't get us new builds when DXR is updated, but it cuts out a step of the build process.

also also, have a probable workaround for getting cross-tree searching on the build repos. to be tested once the above m-c build actually works. :-P

Comment 7

[Sylvestre Ledru [:Sylvestre]]

DXR is no longer available. Searchfox is now replacing it.
See meta bug 1669906 & https://groups.google.com/g/mozilla.dev.platform/c/jDRjrq3l-CY for more details.