tracking bug so I don't lose them between the cracks jenkins is up and running independent builds on a timer, but there's still some things to do: 1) uncomplicate docker images and move the SCM bits to jenkins to handle, gets us builds on code updates, rather than timed 2) documentation 3) have jenkins build DXR (at least as upstream build for), docker images 4) move jenkins off the admin node to some place publicly accessible (but with auth, probably a la ci.m.o) 5) move dxr-docker repo to mozilla GH org (or even roll into dxr) 6) semi-related, build docker images for ES and web head for testing purposes (see also jamon's docker work)
1) is nearly done - ran into some dxr issues in testing (bug 1181189). hgrc fix should be easy since we control the jenkins user creation on the build nodes. need to clean up the docker build, preferably stealing from the mozilla-central docker setup for TC. also, since docker can't pull in files above the build dir, need to drop dxr.config files there when built (or come up with a clever way to pull from a repo, but I think we run into chicken-egg problems there; documenting a workflow will help).
have new docker images (quay.io/fubar) based on the docker stuff in mozilla-central/testing/docker. specifically, building upon ubuntu-build, since the centos images were failing m-c builds all over the place. currently trying to get around an issue with a timeout to the ES cluster at the end of the build. :-( once that's sorted, will merge to master and get the existing trees rebuilding with the new image and SCM polling.
When does it hang? During "Refreshing Index"?
00:27:01 rust-dxr post_build 00:27:01 - Processing files - first pass 00:27:01 - Processing files - second pass 00:27:01 - Updating references 00:27:01 - Generating inheritance graph 00:27:01 - Generating crate info 00:27:01 - Generating qualnames 00:27:01 A worker failed while indexing /builds/dxr-build-env/src/mozilla-central-test/obj-x86_64-unknown-linux-gnu/all-tests.json: 00:27:01 Traceback (most recent call last): 00:27:01 File "/builds/dxr-build-env/venv/local/lib/python2.7/site-packages/dxr-0.1-py2.7.egg/dxr/build.py", line 579, in index_chunk 00:27:01 index_file(tree, tree_indexers, path, es, index) 00:27:01 File "/builds/dxr-build-env/venv/local/lib/python2.7/site-packages/dxr-0.1-py2.7.egg/dxr/build.py", line 550, in index_file 00:27:01 es.bulk(chunk, index=index, doc_type=LINE) 00:27:01 File "/builds/dxr-build-env/venv/local/lib/python2.7/site-packages/pyelasticsearch/client.py", line 93, in decorate 00:27:01 return func(*args, query_params=query_params, **kwargs) 00:27:01 File "/builds/dxr-build-env/venv/local/lib/python2.7/site-packages/pyelasticsearch/client.py", line 448, in bulk 00:27:01 query_params=query_params) 00:27:01 File "/builds/dxr-build-env/venv/local/lib/python2.7/site-packages/pyelasticsearch/client.py", line 281, in send_request 00:27:01 raise exc.info 00:27:01 ReadTimeoutError: HTTPConnectionPool(host='node46.bunker.scl3.mozilla.com', port=9200): Read timed out. (read timeout=60) 00:27:01 (I'm not sure why that says rust-dxr at the top; it's an m-c build)
It must be because you have the rust plugin enabled. It shouldn't hang, but you can certainly get past this for now by disabling it. (Eventually, we'll want to analyze the little bit of rust code that's in m-c.)
tested indexing w/ pypy and oomkiller went to town on it. will have to try again later with a larger VM. back to testing builds with cpython. also, have moved docker images off to quay.io/fubar, and will hook it up to the GH repo to automatically build docker images. doesn't get us new builds when DXR is updated, but it cuts out a step of the build process. also also, have a probable workaround for getting cross-tree searching on the build repos. to be tested once the above m-c build actually works. :-P
You need to log in before you can comment on or make changes to this bug.