1512815 - Optimize Bugzilla->active_custom_fields() for CPU and memory usage

Status: RESOLVED FIXED

Description

[Josh Matthews [:jdm]]

When performing the following query, I am receiving consistent 502 Bad Gateway responses.

https://bugzilla.mozilla.org/rest/bug?username=josh%40joshmatthews.net&status=RESOLVED&apikey=[secret]&target_milestone=Firefox+45&target_milestone=mozilla45&include_fields=id%2Cassigned_to

This query worked fine six weeks ago; it's one of many that I run when preparing the reports of new contributors to Firefox releases.

Comment 1

[Josh Matthews [:jdm]]

More specifically, I receive the following behaviour consistently when running my script that fetches data for releases 10 -> 65:
10->16: 200 OK
17: 502 Bad Gateway

This is preventing me from creating the reports that link to from the Firefox release notes.

Comment 2

[Josh Matthews [:jdm]]

Works: https://bugzilla.mozilla.org/rest/bug?username=josh%40joshmatthews.net&status=RESOLVED&apikey=[secret]&target_milestone=Firefox+16&target_milestone=mozilla16&include_fields=id%2Cassigned_to
Broken: https://bugzilla.mozilla.org/rest/bug?username=josh%40joshmatthews.net&status=RESOLVED&apikey=[secret]&target_milestone=Firefox+17&target_milestone=mozilla17&include_fields=id%2Cassigned_to

Comment 3

[Dylan Hardison [:dylan] (he/him)]

Something about the bad queries is using a huge amount of memory.
Worse, the allocations are coming from the mysqlclient C code.

I'm working out a fix.

Comment 4

[Dylan Hardison [:dylan] (he/him)]

I have found where the memory grows -- it's not in the mysqlclient after all, but is related to custom fields.

Comment 5

[Dylan Hardison [:dylan] (he/him)]

Each call to Bugzilla->active_custom_fields() uses between 1600 kb to 1800 kb of memory per call. This memory is not leaked -- it is returned when the request cache is cleaned up, but it grows to the number of bugs returned in the request.

Further investigation shows it isn't active_custom_fields() itself, but rather the hook that adds tracking flags.

Immediate-ish remediation for this would be bumping the max memory limit up a bit -- 3G instead of 2G may be sufficient.
Quick remediation would passing skip_extensions to Bugzilla->active_custom_fields() when include_fields doesn't include any tracking flags.

The most correction remediation would be making Tracking Flags extension not cache so aggressively.

Comment 6

[Dylan Hardison [:dylan] (he/him)]

josh:

Do you have an immediate need for those APIs to work? If that is the case I can direct ops to drastically increase the memory limits until I can get a code fix out.

Comment 7

[Dylan Hardison [:dylan] (he/him)]

dkl: Any idea how we could leverage filter_wants() to tell if the user actually wants tracking flags, without first loading the list of tracking flags? Perhaps a regex?

Comment 8

[Josh Matthews [:jdm]]

I will need the APIs to work in two weeks, but I can wait until then.

Comment 9

[Dylan Hardison [:dylan] (he/him)]

Ok, it will definitely work by then. I hope it works by monday.

Comment 10

[Dylan Hardison [:dylan] (he/him)]

Created attachment 9037361 [details]
debug.pl

script I used for collecting information.

Comment 11

[Dylan Hardison [:dylan] (he/him)]

A few runs of the attached script:

  DB<7> p main::cache_size()
filter_wants = 68
Bugzilla::Component,9 = 545
Bugzilla::User,302720 = 665
Bugzilla::Bug,_,_ = 2003
  DB<8> c
Bugzilla::Extension::TrackingFlags::active_custom_fields(/app/extensions/TrackingFlags/Extension.pm:291):
291:      my ($self, $args) = @_;
  DB<8> p main::cache_size()
Bugzilla::Component,149 = 662
Bugzilla::User,269762 = 665
Bugzilla::Bug,_,_ = 2120

it seems like this might be the problem

Comment 12

[Dylan Hardison [:dylan] (he/him)]

Here's some more debugging info, counting the number and size of objects in memory over time.
https://gist.github.com/dylanwh/ae8c37963cc146643e24ac0eb95c10e9

Need to think on this some more.

Comment 13

[Dylan Hardison [:dylan] (he/him)]

Well, there's the problem. We have 37,053 Bugzilla::Extension::TrackingFlags::Flag::Value objects allocated after the a few requests, and there are only 3,861 in the database.

Comment 14

[Dylan Hardison [:dylan] (he/him)]

Created attachment 9037648 [details] [review]
Skip preloading tracking flags

Comment 15

[Dylan Hardison [:dylan] (he/him)]

Created attachment 9037704 [details] [review]
Store the list of all tracking flags in memcached.

Comment 16

[Josh Matthews [:jdm]]

Is there a chance this will be fixed this week? The next release is on the 28th.

Comment 17

[Dylan Hardison [:dylan] (he/him)]

Absolutely, the two patches fix everything

Comment 18

[Dylan Hardison [:dylan] (he/him)]

Created attachment 9038289 [details] [review]
Add object that encapsulates the include/exclude logic

Comment 19

[Dylan Hardison [:dylan] (he/him)]

Created attachment 9039667 [details] [review]
Make Bugzilla->active_custom_wants() 10x faster when using specific include_fields