1495741 - Multiple fixes for issues related to memory usage + configuration options for managing memory usage

Status: RESOLVED FIXED

(bugzilla.mozilla.org :: General, enhancement, P1)

Description

[Dylan Hardison [:dylan] (he/him)]

07:07 <ckolos> the bugzilla memory issue is causing the service to die in such a way that the health check from the elb *isn't* failing
07:08 <ckolos> or at least failing in a way that's triggering autoscaling.

My hunch is that this is the /rest/product API.
On -dev, we observed this using 2G of memory (and subsequently being killed by the oom-killer). Looking in the logs, it seems that (probably recently) this API has seen more use, up to 250 times per day.

Comment 1

[Dylan Hardison [:dylan] (he/him)]

Attachment: Cache flag data representation

There's only 600ish flags. For each flag we turn them into a hash.
This becomes a combinatorial explosion and it's the reason this request uses so much RAM.

This patch caches the hash per flag-type ID and saves about 200-300M

Comment 2

[Dylan Hardison [:dylan] (he/him)]

Attachment: Avoid encoding and decoding the JSON responses multiple times

This patch adds a Bugzilla::WebService::JSON class.
This class responds to encode/decode like JSON::MaybeXS but its encode() method returns a Bugzilla::WebService::JSON::Box object,
and its decode() method when passed a "Box" will simply return the box's value.
This avoids round-tripping the JSON value multiple times and doesn't require any changes to the WebService API methods.

Note that one copy is likely to remain -- generating an etag() will call Storable::freeze(). I'll address that in another patch.

Comment 3

[Dylan Hardison [:dylan] (he/him)]

Attachment: Disable etag generation for /rest/product

Generating etags requires cloning the entire perl data structure into a Storable string. So let's not do that.

Comment 4

[Dylan Hardison [:dylan] (he/him)]

Attachment: add memory size limit configuration options

add size limit bits and obsolete backed out patch

Comment 5

[Kohei Yoshino [:kohei]]

All merged to master.