1904373 - Determine pipeline-server ulimit based on size of indexed data that will be memory mapped

Status: ASSIGNED

(Webtools :: Searchfox, enhancement)

Description

[Andrew Sutherland [:asuth] (he/him)]

In https://github.com/mozsearch/mozsearch/commit/3d4116cb7ba5f209d130d6e3c537c4cc48c836d0 I introduced a hard-coded ulimit for pipeline-server that was based around config1, but as observed in bug 1901117 this limit could prevent tree crossrefs from being mmap-ed in cases like config4 where we (now) have ~37G of data to mmap and that clearly is more than the 24G limit.

I've translated the previous rationale into shell scripting based on use of du and tested it on config4 and it seemed to work fine, so I'm going to put that up for review. The key thing to know is that $(( ... )) provides an arithmetic context in bash.