1402319 - indexing jobs using incredible amounts of memory/swap

Status: RESOLVED WONTFIX

(Webtools Graveyard :: DXR, enhancement)

Description

[Kendall Libby [:fubar] (he/him)]

I think since Stylo was added to the firefox trees, we've been running into much more frequent oom_killer events. We just had 8gb of memory added to the four indexing VMs and comm-central STILL trips it:

Sep 22 12:29:43 dxr-processor1.dmz.scl3.mozilla.com kernel: Out of memory: Kill process 30888 (dxr) score 651 or sacrifice child
Sep 22 12:29:43 dxr-processor1.dmz.scl3.mozilla.com kernel: Killed process 30888, UID 5507, (dxr) total-vm:17633080kB, anon-rss:15571852kB, file-rss:20kB

I had a quick look just before it fired and 2gb of swap was in use, yet 7gb of memory remained "free" (which still means 16gb was in use!).

  PID USER      PR  NI  VIRT  RES  SHR S %CPU    TIME+  %MEM SWAP COMMAND
30887 jenkins   20   0 4840m 2.8g 2264 S  0.0 123:45.13 11.8 1.7g dxr
30886 jenkins   20   0 4835m 2.8g 2308 S  0.0 134:25.15 11.8 1.7g dxr
 5079 jenkins   20   0 2753m 2.2g  956 S  0.0 231:03.34  9.6 163m dxr
 2954 jenkins   20   0 9386m  61m 1480 S  0.7  15:23.47  0.3  92m java
 2263 root      20   0  977m  12m 1200 S  0.0   1:09.36  0.1  11m docker

I think something is wonky on the OS side, but I also feel like DXR is not at all playing nicely.

Comment 1

[Erik Rose [:erik][:erikrose]]

My guess from looking at https://github.com/mozilla-platform-ops/dxr-infra/blob/master/config.yml is that it's the rust plugin. Stlyo means a lot more Rust code, and the rust plugin was always a very rough start. Try disabling it, and I bet things start using a lot less RAM.

Comment 2

[Sylvestre Ledru [:Sylvestre]]

DXR is no longer available. Searchfox is now replacing it.
See meta bug 1669906 & https://groups.google.com/g/mozilla.dev.platform/c/jDRjrq3l-CY for more details.