Multiple fixes for issues related to memory usage + configuration options for managing memory usage

RESOLVED FIXED

Status

()

P1
normal
RESOLVED FIXED
6 months ago
3 months ago

People

(Reporter: dylan, Assigned: dylan)

Tracking

(Blocks: 1 bug)

Production

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(3 attachments, 1 obsolete attachment)

(Assignee)

Description

6 months ago
07:07 <ckolos> the bugzilla memory issue is causing the service to die in such a way that the health check from the elb *isn't* failing
07:08 <ckolos> or at least failing in a way that's triggering autoscaling.

My hunch is that this is the /rest/product API.
On -dev, we observed this using 2G of memory (and subsequently being killed by the oom-killer). Looking in the logs, it seems that (probably recently) this API has seen more use, up to 250 times per day.
(Assignee)

Updated

6 months ago
Component: Infrastructure → General
Priority: -- → P1
QA Contact: mcote
(Assignee)

Comment 1

6 months ago
There's only 600ish flags. For each flag we turn them into a hash.
This becomes a combinatorial explosion and it's the reason this request uses so much RAM.

This patch caches the hash per flag-type ID and saves about 200-300M
(Assignee)

Comment 2

5 months ago
This patch adds a Bugzilla::WebService::JSON class.
This class responds to encode/decode like JSON::MaybeXS but its encode() method returns a Bugzilla::WebService::JSON::Box object,
and its decode() method when passed a "Box" will simply return the box's value.
This avoids round-tripping the JSON value multiple times and doesn't require any changes to the WebService API methods.

Note that one copy is likely to remain -- generating an etag() will call Storable::freeze(). I'll address that in another patch.
(Assignee)

Comment 3

5 months ago
Generating etags requires cloning the entire perl data structure into a Storable string. So let's not do that.
(Assignee)

Comment 4

5 months ago
add size limit bits and obsolete backed out patch
Attachment #9015896 - Attachment is obsolete: true
(Assignee)

Updated

5 months ago
Summary: Recent memory issues → memory issues
(Assignee)

Updated

5 months ago
Summary: memory issues → Multiple fixes for issues related to memory usage + configuration options for managing memory usage
All merged to master.
Status: NEW → RESOLVED
Last Resolved: 5 months ago
Resolution: --- → FIXED
(Assignee)

Updated

3 months ago
Blocks: 1513921
You need to log in before you can comment on or make changes to this bug.