jobqueue.pl slowly leaks memory over time due to Email::MIME::walk_parts

RESOLVED FIXED in Bugzilla 4.0

Status

()

defect
RESOLVED FIXED
10 years ago
9 years ago

People

(Reporter: webmaster, Assigned: mkanat)

Tracking

3.4.1
Bugzilla 4.0
Dependency tree / graph
Bug Flags:
approval +
approval4.0 +
blocking4.0 +
blocking3.6.1 -
blocking3.6.3 -
blocking3.4.7 -

Details

Attachments

(3 attachments, 1 obsolete attachment)

Reporter

Description

10 years ago
User-Agent:       Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.2) Gecko/20090803 Fedora/3.5.2-2.fc11 Firefox/3.5.2
Build Identifier: Bugzilla 3.4.1

I noticed that jobqueue.pl was consuming the most memory on one of my bugzilla servers:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 3910 wwwrun    16   0  256m 225m 2372 R   14  5.6  40:30.92 jobqueue.pl


After restarting it:
 6003 wwwrun    16   0 65752  33m 2324 S    0  0.8   0:00.48 jobqueue.pl 

After a few days, it is now up to 65m:
 6003 wwwrun    16   0 95612  63m 2336 S    0  1.6   3:09.49 jobqueue.pl

Reproducible: Always

Steps to Reproduce:
1. Start jobqueue.pl as a non-root user
2. Use 'top' sorted by memory usage
3. Wait a week or two
4. Goto 2.

Updated

10 years ago
Version: unspecified → 3.4.1
Assignee

Comment 1

10 years ago
  Cool, thanks for filing. Could you attach (as an attachment) the current output of your checksetup.pl --check-modules?
Reporter

Comment 2

10 years ago
Assignee

Comment 3

10 years ago
Do you have any idea how fast it's growing? I'd like to get some idea of how many email-sending iterations I have to run to see the same growth you're seeing.
Reporter

Comment 4

10 years ago
> Do you have any idea how fast it's growing?

Eclipse's Bugzilla sends out about 7000-10000 emails/day during the week, and according to comment 0, jobqueue.pl doubled in size after a few days.
We are currently restarting each day to as our jobqueue.pl is growing quite large
after 24 hours of use even with no jobs left to process:

Some info

After a restart
top - 14:54:26 up 4 days, 20:34,  5 users,  load average: 1.53, 1.55, 1.50
28825 root      15   0  193m  41m 2900 S  0.0  0.5   1:03.52 jobqueue.pl

After running for 24 hours
top - 08:44:07 up 5 days, 14:24,  2 users,  load average: 1.44, 1.72, 1.99
28825 root      15   0 2351m 2.1g 2900 S  0.0 27.5   8:39.13 jobqueue.pl

Here is some more info.

[bhowmick@deathstar ~]$ cat jobqueue.txt 
After a restart
top - 14:54:26 up 4 days, 20:34,  5 users,  load average: 1.53, 1.55, 1.50
28825 root      15   0  193m  41m 2900 S  0.0  0.5   1:03.52 jobqueue.pl

After running for 24 hours
top - 08:44:07 up 5 days, 14:24,  2 users,  load average: 1.44, 1.72, 1.99
28825 root      15   0 2351m 2.1g 2900 S  0.0 27.5   8:39.13 jobqueue.pl

top - 08:19:41 up 6 days, 13:59,  2 users,  load average: 1.74, 1.76, 1.79
18365 root      15   0 2354m 2.2g 2924 S  0.0 27.6   7:29.40 /var/www/html/bugzilla/jobqueue.pl

top - 08:19:54 up 42 days, 15:51,  1 user,  load average: 2.21, 2.11, 2.17
30641 root      15   0 1180m 1.0g 2880 S  0.0 12.9   7:07.45 /var/www/html/bugzilla/jobqueue.pl
Assignee

Comment 6

9 years ago
Putting this on our radar for a fix, or at least an investigation.
Flags: blocking3.4.6+
Target Milestone: --- → Bugzilla 3.4
Assignee

Comment 7

9 years ago
Okay, I can confirm that each message sent through jobqueue.pl increases its memory size by a very small amount--maybe 4K or 8K. I'll investigate the cause of the problem.
Status: UNCONFIRMED → NEW
Ever confirmed: true
Assignee

Comment 8

9 years ago
  It's not each message, it seems somewhat random; the process increases by 4K or 8K after a seemingly random number of messages are sent.
Assignee

Updated

9 years ago
Flags: blocking3.4.6+ → blocking3.4.7+
Assignee

Comment 9

9 years ago
  My investigations didn't get me anywhere. If anybody wants to help find the memory leak here, I'd be really grateful.
Keywords: helpwanted
Assignee

Updated

9 years ago
Flags: blocking3.6.1+
Flags: blocking3.4.7-
Flags: blocking3.4.7+
Bugzilla 3.4 is now restricted to security bugs. We will retarget this bug (to 3.6 or later) when it's fixed.
Target Milestone: Bugzilla 3.4 → ---
Assignee

Updated

9 years ago
Target Milestone: --- → Bugzilla 3.6
Not a hard blocker. We will take it when it's ready.
Flags: blocking3.6.1+ → blocking3.6.1-
Assignee

Comment 12

9 years ago
I have found it!!! $email->walk_parts leaks.
Assignee

Comment 13

9 years ago
For those curious, here is now I instrumented jobqueue.pl to detect the memory leak. Most of my time was spent polishing up Bugzilla::Leak to work on arrays--after that. Once Bugzilla::Leak actually worked, the leaking object was somewhat obvious.

I also had to modify TheSchwartz itself to sleep if the job's $class->is_sleeping returned true, or it would kill the job before it was done dumping, and try to do other work.
Assignee: email-notifications → mkanat
Status: NEW → ASSIGNED
Assignee

Comment 14

9 years ago
  I've filed an upstream bug here:

  https://rt.cpan.org/Ticket/Display.html?id=59581
Assignee

Updated

9 years ago
Flags: blocking4.0+
Excellent news Max. So it is a matter of waiting til a newer fixed version is available from CPAN or is it possible to subclass Email::MIME and fix the issue
on Bugzilla end? We will volunteer to help by re-enablingthis feature when a fix is available and report back.

Dave
Assignee

Comment 16

9 years ago
Well, rjbs (the maintainer of Email::MIME) said that he'd get right on it. So I'm going to wait to see if he can fix it relatively soon, and then we can just up our requirement.
Assignee

Updated

9 years ago
Severity: minor → normal
Flags: blocking3.6.3+
Keywords: helpwanted
OS: Linux → All
Hardware: x86_64 → All
Summary: jobqueue.pl: possible memory leak? → jobqueue.pl slowly leaks memory over time due to Email::MIME::walk_parts
Assignee

Comment 17

9 years ago
Okay, so Email::MIME 1.904 fixes this problem. Anybody upgrading to that version will immediately have this issue fixed for them.

Since this is such a significant upgrade to, essentially, a cutting-edge version of a module, I don't want to check it in on the 3.6 branch--I don't like changing requirements that drastically for stable branches. However, we can tell anybody who's hit by this that all they have to do is update Email::MIME to 1.904 and they're good to go.
Flags: blocking3.6.3+ → blocking3.6.3-
Target Milestone: Bugzilla 3.6 → Bugzilla 4.0
Assignee

Comment 18

9 years ago
Posted patch v1 (obsolete) — Splinter Review
We should wait a few days to check this in, because 1.904 won't show up on CPAN mirrors for a little while.
Attachment #472227 - Flags: review?(LpSolit)
Comment on attachment 472227 [details] [diff] [review]
v1

You also have to remove Email::MIME::Encodings and Email::MIME::Modifier from docs/en/xml/installation.xml around line 360. Otherwise looks good. This will also fix bug 65477 comment 83.
Attachment #472227 - Flags: review?(LpSolit) → review-

Updated

9 years ago
Blocks: 65477
Assignee

Comment 20

9 years ago
Posted patch v2Splinter Review
Ah, right, thanks for catching that. Here's a fixed patch.
Attachment #472227 - Attachment is obsolete: true
Attachment #472577 - Flags: review?(LpSolit)
Comment on attachment 472577 [details] [diff] [review]
v2

r=LpSolit
Attachment #472577 - Flags: review?(LpSolit) → review+

Updated

9 years ago
Flags: approval4.0+
Flags: approval+
Assignee

Comment 22

9 years ago
Committing to: bzr+ssh://bzr.mozilla.org/bugzilla/trunk/                       
modified Bugzilla/Mailer.pm
modified Bugzilla/Install/Requirements.pm
modified docs/en/xml/installation.xml
Committed revision 7464.

Committing to: bzr+ssh://bzr.mozilla.org/bugzilla/4.0/                         
modified Bugzilla/Mailer.pm
modified Bugzilla/Install/Requirements.pm
modified docs/en/xml/installation.xml
Committed revision 7395.
Status: ASSIGNED → RESOLVED
Last Resolved: 9 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.