Closed Bug 234159 Opened 21 years ago Closed 9 years ago

Multiple notifications aggregation - BugMail doesn't finish sending

Categories

(Bugzilla :: Email Notifications, defect)

Product:
Bugzilla
Component:
Email Notifications
Version:
2.17.6
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

()

Status:
RESOLVED WORKSFORME

People

(Reporter: mnyromyr, Unassigned)

References

Details

(Whiteboard: Affects b.m.o)

Attachments

(2 files)

An example of an "aggregate" message sent by b.m.o
5.73 KB, message/rfc822
Details
Less than 5 hours' worth of bug 105547 bugmail
33.72 KB, text/plain
Details
Bugzilla aggregates multiple comments/notifications into one bugmail. This bugmail contains several old comments/notifications *one has already gotten* and the new one. For instance: Bugmails for http://bugzilla.mozilla.org/show_bug.cgi?id=18574 contain all changes since about 2004-02-08, bugmail size 7 KB and growing. Bugmails for http://bugzilla.mozilla.org/show_bug.cgi?id=215243 contain all changes since about 2004-02-06, bugmail size 19 KB and growing. Since bug 215243 contains only relatively few addressees, this seems not to be the problem.
OS: Windows 2000 → All
Hardware: PC → All
Whiteboard: Affects b.m.o
Example Message: -- http://bugzilla.mozilla.org/show_bug.cgi?id=18574 opi@gmx.at changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #9985 [details] [diff] is|0 |1 obsolete| | Attachment #10226 [details] [diff]|0 |1 is obsolete| | Attachment #124876 [details] [diff]|0 |1 is obsolete| | Attachment #134523 [details] [diff]|0 |1 is obsolete| | Attachment #134552 [details] [diff]|0 |1 is obsolete| | glennrp@imagemagick.org changed: What |Removed |Added ---------------------------------------------------------------------------- AssignedTo|glennrp@imagemagick.org |opi@gmx.at opi@gmx.at changed: What |Removed |Added ---------------------------------------------------------------------------- OtherBugsDependingO| |121927 nThis| | freso@livejournal.com changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |freso@livejournal.com glennrp@imagemagick.org changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #134552 [details] [diff]|Updated removal reversal |Updated removal reversal description|patch (trunk) |patch (mozilla-1.6) Attachment #134552 [details] [diff]|1 |0 is obsolete| | ------- Additional Comments From opi@gmx.at 2004-02-08 17:53 PST ------- Created an attachment (id=140905) --> (http://bugzilla.mozilla.org/attachment.cgi?id=140905&action=view) Full MNG (1.0.5) patch (trunk) bzip2 This is the full patch. The missing files are included in this patch, it is against trunk (after landing SVG). MNG doesn't compile by default with this patch, you need: --enable-image-decoders=default,mng to build MNG in your patched tree. @Glenn Can I assign this bug to me? @Pavlov I would Maintain the MNG clue code, I will have it back in trunk without enabled by default. What must I do? ------- Additional Comments From glennrp@imagemagick.org 2004-02-08 18:00 PST ------- Sure, take it. Glenn ------- Additional Comments From zfraction@yahoo.com 2004-02-08 20:00 PST ------- A Free program (for windows) called SlowView, will view, make, and batch convert MNG images. http://www.slowview.at/ it is at RC2 right now (and has been for months) for anyone interested in a really good MNG maker program. it is not opensource, but it is freeware. it is less than a megabyte (not including required software (http://www.slowview.at/?page=requirements) ------- Additional Comments From gwalla_mozilla@hotmail.com 2004-02-11 17:15 PST ------- (In reply to comment #376) > A Free program (for windows) called SlowView, will view, make, and batch convert > MNG images. How is this relevant? Don't spam bug reports. ------- Additional Comments From bernd.rinnerthaler@web.de 2004-02-11 17:27 PST ------- Alexander, maybe you should ask Pavlov for review/superreview/approval of your patch. I think that the best thing would be to contact him personally and to make sure that you are heading in the right direction. Maybe this could be a feature that can be enabled/disabled in the installer (like Domi and Venkman can already be switched on/off in the Firefox windows installer), or as you are already aiming at, make it an optional compile switch. In any way, either contact Pavlov personally (I am not sure if he is still looking at this spammed "thread") or set the review flag for your patch. TNX for offering to maintain the glue code. ------- Additional Comments From glennrp@imagemagick.org 2004-02-12 06:26 PST ------- Thanks, Alexander. The patch id=140905 wfm on FreeBSD 5.0 Be sure to use the "-p0" patch option, and remember to run autoconf afterwards. Glenn ------- Additional Comments From stmoebius@gmx.de 2004-02-13 01:00 PST ------- Sorry for the spam, but: I keep getting mails on this bug (which is fine) containing all changes since Feb 08 (which is not). Does anyone else experience this? It only happens with this bug. Stefan ------- Additional Comments From patrickhendriks@hotmail.com 2004-02-13 01:53 PST ------- (In reply to comment #380) > Sorry for the spam, but: > I keep getting mails on this bug (which is fine) containing all changes since > Feb 08 (which is not). Does anyone else experience this? > It only happens with this bug. > Stefan same here ------- Additional Comments From freso@livejournal.com 2004-02-13 03:55 PST ------- Hate to do this, but, well: "Me too." (re: two above comments) ------- Additional Comments From patrickhendriks@hotmail.com 2004-02-13 04:16 PST ------- Sorry for the spam: Per comment 380 and further: this is bugzilla bug 234159. Please file further comments on repeated text in bugmail in that bug. ------- Additional Comments From glennrp@imagemagick.org 2004-02-13 04:58 PST ------- (From update of attachment 134552 [details] [diff] [review]) Removed "obsolete" flag because this patch is still good for updating mozilla-1.6. -- Configure bugmail: http://bugzilla.mozilla.org/userprefs.cgi?tab=email
>Since bug #215243 contains only relatively few addressees, this seems not to be >the problem. Like bug #18574, it has a lot of votes, therefore a lot of implicit addresses. Bug #94035 is doing it too, since February 8th, and also has lots of votes.
*** Bug 234161 has been marked as a duplicate of this bug. ***
Yeah, the problem is that each of those bugs have over 600 emails being sent every time someone touches it. Apache times out (or users get impatient and hit the Stop button) before it finishes sending them all, so it never gets marked as sent. We've already investigated the mail sending speed, and compared to a mid-sized ISP, sending out 600 emails in 5 minutes is actually decently speedy for a single non-threaded process, so the speed of sending the mail probably isn't going to be able to be improved much. About the only option that looks workable at the moment is to simply record who's supposed to get the mail when a bug changes instead of sending it right then, and run a background daemon to actually do the mail delivery, which can take as long as it needs to since the user won't be sitting there waiting for it. JayPee: this logic may actually make your megapatch easier because it won't have the templates interfering with each other :)
Depends on: 84876
Does the script is canceled if Apache/User ends transfer? (In php you can act on that cancelation and you can clean up) And why this happens on both bugs since the same day?
> so the speed of sending the mail probably isn't > going to be able to be improved much. I'm suspicious of this statement. This has only started going wrong since the upgrade, AFAICS. And also, don't we turn off the thing where a user closing the connection stops the script? So that shouldn't be an issue. Could this be connected with filling up the error log with Perl errors? That's the cause of the mass change slowdown (patch awaiting review in bug 233645). Gerv
What changed in the upgrade? Bugzilla used to spawn off a separate process to send the email. Bugzilla itself might have gotten killed when Apache timed out, but the child process didn't, so it continued. However, this was a memory and performance sink because it required initializing a new perl interpreter for each bug processed (which was felt heavily during edit-multiple). The mail process now ocurrs in the same threadspace with the CGI itself, which (except for the warning problem you mention) was a tremendous speed boost to change-multiple. When I hack sanitycheck.cgi to be able to be run from the command-line to process the emails, no warnings are generated while the emails are being sent. I do know we supposedly have protection in place against Apache killing us off while we're doing things, though, so yes, the fact that the process is even getting killed is a problem.
Couldn't a solution be to have a single separate process that sends out emails which is used whenever mails are being sent, rather then spawning a new process every time mails are to be sent. Another alternative is to rather then sending out the mail syncronously just save out "something" to a separate table and then have separate process that every so often goes through that table and sends out emails. This process would then be compleatly disconnected from apache. This way users also wouldn't have to wait while emails are being sent out, which at least isn't something that i am interested in.
Jonas: that's more or less what I said in comment 5 :)
Blocks: bmo-regressions-old
Summary: Multiple notifications aggregation → Multiple notifications aggregation - BugMail doesn't finish sending
I think we should begin by investigating why the process gets killed even when we are ignoring signals (or should be.) If we can fix that, we're sorted. Gerv
Yeah, top priority is to figure out why Apache is killing us. Next priority is to start invoking sendmail once per session rather than once per mail. It's -ks option puts it into SMTP mode, which lets us send multiple messages. We should take advantage of this. We'll have to send SMTP envelope information and encode messages, though, like so: $msg =~ s/^\./../mg; $msg = join("\r\n", split(/[\r\n]+/, $msg)) . "\r\n.\r\n";
or just use Net::SMTP, which does all of that for you. :) For the quick hack version, see the Win32 section of the installation instructions in the Bugzilla Guide. The long-term "correct' fix will be included with bug 84876.
The "bugmail daemon" is also something lots of people seem to want... I started looking at that briefly before most of the work for bug 84876 was done.
*** Bug 234230 has been marked as a duplicate of this bug. ***
If we can get 84876 finished in time for the next b.m.o. update, we could try using Net::SMTP or another mailer module, to see if performance improved... Gerv
(In reply to comment #16) > If we can get 84876 finished in time for the next b.m.o. update, we could try > using Net::SMTP or another mailer module, to see if performance improved... When is that currently scheduled for?
The next b.m.o. update is currently scheduled for "when we've fixed all the stuff we want to fix and Dave and Myk think it's a good idea." 84876 is currently scheduled for "as soon as possible, if not sooner". ;-) Gerv
If the problem used to be memory consumption when spawning another process, you could spare that for bugs with exceptionaly long cc lists. There should not be to many bug that would trigger an extra thread. This might be a ugly solution compared to a background deamon but the advantage is that all neccesary code should already be done, you just have to put it back with a condition.
Hard to determine just what is actually new since the last one.
Just FYI, the cron job is currently broken, too... one of the bugs being affected by this has something in it that's making the bugmail stuff crash when it hits that bug. I'll attempt to track this down tonight.
(In reply to comment #21) > Just FYI, the cron job is currently broken, too... one of the bugs being > affected by this has something in it that's making the bugmail stuff crash when > it hits that bug. I'll attempt to track this down tonight. The cron job has been fixed. Someone changed their email address after adding themselves to the CC list, but before the cron job ran to complete the sending of the email which didn't go out initially because of this bug. The BugMail code was choking on the fact that a user listed as being added to the CC in the activity log didn't exist. I've hacked the bugmail code to silently ignore any such users instead of choking on it. (The likelihood of that scenario happening again isn't very big).
*** Bug 235565 has been marked as a duplicate of this bug. ***
*** Bug 249662 has been marked as a duplicate of this bug. ***
(regarding my recent dupe bug 249662) Actually, as a bugzilla mail recipient, I don't mind the aggregation, my problem is only the case in which I get another message with the exact same content, no new comments at all.
(In reply to comment #25) > (regarding my recent dupe bug 249662) > Actually, as a bugzilla mail recipient, I don't mind the aggregation, my problem > is only the case in which I get another message with the exact same content, no > new comments at all. Yeah, that happens if you're one of the first 2 or 3 hundred people it tried to send mail to before it got killed. When the cron job runs overnight to clean up after it, it doesn't know who got it and who didn't, so it just mails everyone again.
*** Bug 258700 has been marked as a duplicate of this bug. ***
Blocks: 84876
No longer depends on: 84876
*** Bug 276157 has been marked as a duplicate of this bug. ***
(In reply to comment #26) >When the cron job runs overnight to clean up after it, it doesn't >know who got it and who didn't, so it just mails everyone again. Apparently including people who excluded it the first time around :-(
OK, I just partially alleviated this on bmo... I moved the define(`confDELIVERY_MODE',`b')dnl line below the FEATURE(`msp', `[127.0.0.1]')dnl line in submit.mc. Apparently when FEATURE('msp') runs, it overrides confDELIVERY_MODE back to interactive instead of background.
Assignee: preed → email-notifications
QA Contact: mattyt-bugzilla → default-qa
This is far less of a problem now that we have jobqueue.pl.
Severity: major → normal
Not a regression.
Keywords: regression
jobqueue.pl fixes this issue, as already suggested in comment 5 and comment 9.
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: