Closed Bug 1512815 Opened 6 years ago Closed 6 years ago

Optimize Bugzilla->active_custom_fields() for CPU and memory usage


( :: Extensions, defect, P1)






(Reporter: jdm, Assigned: dylan)



(Keywords: leave-open)


(5 files)

When performing the following query, I am receiving consistent 502 Bad Gateway responses.[secret]&target_milestone=Firefox+45&target_milestone=mozilla45&include_fields=id%2Cassigned_to

This query worked fine six weeks ago; it's one of many that I run when preparing the reports of new contributors to Firefox releases.

More specifically, I receive the following behaviour consistently when running my script that fetches data for releases 10 -> 65:
10->16: 200 OK
17: 502 Bad Gateway

This is preventing me from creating the reports that link to from the Firefox release notes.

Assignee: nobody → dylan

Something about the bad queries is using a huge amount of memory.
Worse, the allocations are coming from the mysqlclient C code.

I'm working out a fix.

I have found where the memory grows -- it's not in the mysqlclient after all, but is related to custom fields.

Each call to Bugzilla->active_custom_fields() uses between 1600 kb to 1800 kb of memory per call. This memory is not leaked -- it is returned when the request cache is cleaned up, but it grows to the number of bugs returned in the request.

Further investigation shows it isn't active_custom_fields() itself, but rather the hook that adds tracking flags.

Immediate-ish remediation for this would be bumping the max memory limit up a bit -- 3G instead of 2G may be sufficient.
Quick remediation would passing skip_extensions to Bugzilla->active_custom_fields() when include_fields doesn't include any tracking flags.

The most correction remediation would be making Tracking Flags extension not cache so aggressively.

Severity: normal → major
Component: API → Extensions: TrackingFlags
Priority: -- → P1


Do you have an immediate need for those APIs to work? If that is the case I can direct ops to drastically increase the memory limits until I can get a code fix out.

Flags: needinfo?(josh)

dkl: Any idea how we could leverage filter_wants() to tell if the user actually wants tracking flags, without first loading the list of tracking flags? Perhaps a regex?

Flags: needinfo?(dkl)

I will need the APIs to work in two weeks, but I can wait until then.

Flags: needinfo?(josh)

Ok, it will definitely work by then. I hope it works by monday.

Summary: Rest API queries are yielding 502 Bad Gateway → Tracking flags active_custom_fields() hook allocates 1700kb per call leading to OOM errors for some APIs
Attached file

script I used for collecting information.

A few runs of the attached script:

  DB<7> p main::cache_size()
filter_wants = 68
Bugzilla::Component,9 = 545
Bugzilla::User,302720 = 665
Bugzilla::Bug,_,_ = 2003
  DB<8> c
291:      my ($self, $args) = @_;
  DB<8> p main::cache_size()
Bugzilla::Component,149 = 662
Bugzilla::User,269762 = 665
Bugzilla::Bug,_,_ = 2120

it seems like this might be the problem

Here's some more debugging info, counting the number and size of objects in memory over time.

Need to think on this some more.

Well, there's the problem. We have 37,053 Bugzilla::Extension::TrackingFlags::Flag::Value objects allocated after the a few requests, and there are only 3,861 in the database.

Is there a chance this will be fixed this week? The next release is on the 28th.

Flags: needinfo?(dylan)

Absolutely, the two patches fix everything

Flags: needinfo?(dylan)
Attachment #9037704 - Attachment description: Ignore tracking flags if they're excluded from the results anyway → Store the list of all tracking flags in memcached.
Summary: Tracking flags active_custom_fields() hook allocates 1700kb per call leading to OOM errors for some APIs → Optimize Bugzilla->active_custom_fields() for CPU and memory usage
Keywords: leave-open
Version: Staging → Production
Flags: needinfo?(dkl)
Closed: 6 years ago
Resolution: --- → FIXED
Blocks: 1523536
Blocks: 1529362
Component: Extensions: TrackingFlags → Extensions
You need to log in before you can comment on or make changes to this bug.