1165336 - Why is maxclient set to 64 on webheads ?

Status: RESOLVED WORKSFORME

(Developer Services :: Mercurial: hg.mozilla.org, defect)

Description

[Ludovic Hirlimann [:Usul]]

It seems low to me. Can we make sure we have the proper value there ?

Comment 1

[Gregory Szorc [:gps]]

The value was intentionally set after experimentation. I'd rather have servers fast 503 then hold onto requests for longer only to eventually time out or to take significantly longer to process requests.

We never come close to the worker limit under normal operation. This only becomes an issue during DoS-like events. And increasing the worker limit provides little benefit in these scenarios. Failing fast and expecting clients to honor 503 and back off is the best way to curtail a load event. Of course, this assumes clients are properly behaving, which it appears they aren't. But that's their bug.

Comment 2

[Gregory Szorc [:gps]]

Also, many requests to hg.mozilla.org take dozens of seconds of CPU time. At the point the worker pool is saturated, CPU is almost certainly at 100%. Processing more requests at that point just adds context switching and bogs down the server further.

If our expected request time was <1s or if we isolated long-running requests to their own process pool, I may have a different opinion about increasing the worker pool size. As it is, I don't want the server to bend too far past its breaking point.