If you think a bug might affect users in the 57 release, please set the correct tracking and status flags for Release Management.

Multiple notifications aggregation - BugMail doesn't finish sending

RESOLVED WORKSFORME

Status

()

Bugzilla
Email Notifications
RESOLVED WORKSFORME
14 years ago
a year ago

People

(Reporter: Karsten Düsterloh, Unassigned)

Tracking

Dependency tree / graph

Details

(Whiteboard: Affects b.m.o)

Attachments

(2 attachments)

(Reporter)

Description

14 years ago
Bugzilla aggregates multiple comments/notifications into one bugmail. This
bugmail contains several old comments/notifications *one has already gotten* and
the new one.

For instance:
Bugmails for http://bugzilla.mozilla.org/show_bug.cgi?id=18574 contain all
changes since about 2004-02-08, bugmail size 7 KB and growing.
Bugmails for http://bugzilla.mozilla.org/show_bug.cgi?id=215243 contain all
changes since about 2004-02-06, bugmail size 19 KB and growing.

Since bug 215243 contains only relatively few addressees, this seems not to be
the problem.

Updated

14 years ago
OS: Windows 2000 → All
Hardware: PC → All
Whiteboard: Affects b.m.o

Comment 1

14 years ago
Created attachment 141318 [details]
An example of an "aggregate" message sent by b.m.o

Comment 2

14 years ago
Example Message:
-- 

http://bugzilla.mozilla.org/show_bug.cgi?id=18574


opi@gmx.at changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
Attachment #9985 [details] [diff] is|0                           |1
           obsolete|                            |
  Attachment #10226 [details] [diff]|0                           |1
        is obsolete|                            |
 Attachment #124876 [details] [diff]|0                           |1
        is obsolete|                            |
 Attachment #134523 [details] [diff]|0                           |1
        is obsolete|                            |
 Attachment #134552 [details] [diff]|0                           |1
        is obsolete|                            |

glennrp@imagemagick.org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         AssignedTo|glennrp@imagemagick.org     |opi@gmx.at

opi@gmx.at changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
OtherBugsDependingO|                            |121927
              nThis|                            |

freso@livejournal.com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |freso@livejournal.com

glennrp@imagemagick.org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
 Attachment #134552 [details] [diff]|Updated removal reversal    |Updated removal reversal
        description|patch (trunk)               |patch (mozilla-1.6)
 Attachment #134552 [details] [diff]|1                           |0
        is obsolete|                            |




------- Additional Comments From opi@gmx.at  2004-02-08 17:53 PST -------
Created an attachment (id=140905)
 --> (http://bugzilla.mozilla.org/attachment.cgi?id=140905&action=view)
Full MNG (1.0.5) patch (trunk) bzip2

This is the full patch. The missing files are included in this patch, it is
against trunk (after landing SVG).

MNG doesn't compile by default with this patch, you need:

--enable-image-decoders=default,mng

to build MNG in your patched tree.



@Glenn

Can I assign this bug to me?

@Pavlov

I would Maintain the MNG clue code, I will have it back in trunk without
enabled by default. What must I do?


------- Additional Comments From glennrp@imagemagick.org  2004-02-08 18:00 PST
-------
Sure, take it.  Glenn

------- Additional Comments From zfraction@yahoo.com  2004-02-08 20:00 PST -------
A Free program (for windows) called SlowView, will view, make, and batch convert
 MNG images.
http://www.slowview.at/
it is at RC2 right now (and has been for months)
for anyone interested in a really good MNG maker program.
it is not opensource, but it is freeware.
it is less than a megabyte (not including required software
(http://www.slowview.at/?page=requirements)


------- Additional Comments From gwalla_mozilla@hotmail.com  2004-02-11 17:15
PST -------
(In reply to comment #376)
> A Free program (for windows) called SlowView, will view, make, and batch convert
>  MNG images.

How is this relevant? Don't spam bug reports.

------- Additional Comments From bernd.rinnerthaler@web.de  2004-02-11 17:27 PST
-------
Alexander, maybe you should ask Pavlov for review/superreview/approval of your
patch.

I think that the best thing would be to contact him personally and to make sure
that you are heading in the right direction.

Maybe this could be a feature that can be enabled/disabled in the installer
(like Domi and Venkman can already be switched on/off in the Firefox windows
installer),

or as you are already aiming at, make it an optional compile switch.

In any way, either contact Pavlov personally (I am not sure if he is still
looking at this spammed "thread") or set the review flag for your patch.

TNX for offering to maintain the glue code.

------- Additional Comments From glennrp@imagemagick.org  2004-02-12 06:26 PST
-------
Thanks, Alexander.  The patch id=140905 wfm on FreeBSD 5.0

Be sure to use the "-p0" patch option, and remember to run autoconf
afterwards.

Glenn

------- Additional Comments From stmoebius@gmx.de  2004-02-13 01:00 PST -------
Sorry for the spam, but:
I keep getting mails on this bug (which is fine) containing all changes since
Feb 08 (which is not). Does anyone else experience this?
It only happens with this bug.

Stefan

------- Additional Comments From patrickhendriks@hotmail.com  2004-02-13 01:53
PST -------
(In reply to comment #380)
> Sorry for the spam, but:
> I keep getting mails on this bug (which is fine) containing all changes since
> Feb 08 (which is not). Does anyone else experience this?
> It only happens with this bug.
> Stefan

same here

------- Additional Comments From freso@livejournal.com  2004-02-13 03:55 PST -------
Hate to do this, but, well: "Me too." (re: two above comments)

------- Additional Comments From patrickhendriks@hotmail.com  2004-02-13 04:16
PST -------
Sorry for the spam:
Per comment 380 and further: this is bugzilla bug 234159. Please file further
comments on repeated text in bugmail in that bug.

------- Additional Comments From glennrp@imagemagick.org  2004-02-13 04:58 PST
-------
(From update of attachment 134552 [details] [diff] [review])
Removed "obsolete" flag because this patch is still good for updating
mozilla-1.6.



-- 
Configure bugmail: http://bugzilla.mozilla.org/userprefs.cgi?tab=email

Comment 3

14 years ago
>Since bug #215243 contains only relatively few addressees, this seems not to be
>the problem.

Like bug #18574, it has a lot of votes, therefore a lot of implicit addresses.

Bug #94035 is doing it too, since February 8th, and also has lots of votes.

Comment 4

14 years ago
*** Bug 234161 has been marked as a duplicate of this bug. ***
Yeah, the problem is that each of those bugs have over 600 emails being sent
every time someone touches it.  Apache times out (or users get impatient and hit
the Stop button) before it finishes sending them all, so it never gets marked as
sent.

We've already investigated the mail sending speed, and compared to a mid-sized
ISP, sending out 600 emails in 5 minutes is actually decently speedy for a
single non-threaded process, so the speed of sending the mail probably isn't
going to be able to be improved much.  About the only option that looks workable
at the moment is to simply record who's supposed to get the mail when a bug
changes instead of sending it right then, and run a background daemon to
actually do the mail delivery, which can take as  long as it needs to since the
user won't be sitting there waiting for it.

JayPee: this logic may actually make your megapatch easier because it won't have
the templates interfering with each other :)
Depends on: 84876

Comment 6

14 years ago
Does the script is canceled if Apache/User ends transfer? (In php you can act on
that cancelation and you can clean up)

And why this happens on both bugs since the same day?
> so the speed of sending the mail probably isn't
> going to be able to be improved much.

I'm suspicious of this statement. This has only started going wrong since the
upgrade, AFAICS. And also, don't we turn off the thing where a user closing the
connection stops the script? So that shouldn't be an issue.

Could this be connected with filling up the error log with Perl errors? That's
the cause of the mass change slowdown (patch awaiting review in bug 233645).

Gerv
What changed in the upgrade?  Bugzilla used to spawn off a separate process to
send the email.  Bugzilla itself might have gotten killed when Apache timed out,
but the child process didn't, so it continued.  However, this was a memory and
performance sink because it required initializing a new perl interpreter for
each bug processed (which was felt heavily during edit-multiple).  The mail
process now ocurrs in the same threadspace with the CGI itself, which (except
for the warning problem you mention) was a tremendous speed boost to
change-multiple.

When I hack sanitycheck.cgi to be able to be run from the command-line to
process the emails, no warnings are generated while the emails are being sent.

I do know we supposedly have protection in place against Apache killing us off
while we're doing things, though, so yes, the fact that the process is even
getting killed is a problem.
Couldn't a solution be to have a single separate process that sends out emails
which is used whenever mails are being sent, rather then spawning a new process
every time mails are to be sent.

Another alternative is to rather then sending out the mail syncronously just
save out "something" to a separate table and then have separate process that
every so often goes through that table and sends out emails. This process would
then be compleatly disconnected from apache. This way users also wouldn't have
to wait while emails are being sent out, which at least isn't something that i
am interested in.
Jonas: that's more or less what I said in comment 5 :)
Blocks: 231927
Summary: Multiple notifications aggregation → Multiple notifications aggregation - BugMail doesn't finish sending
I think we should begin by investigating why the process gets killed even when
we are ignoring signals (or should be.) If we can fix that, we're sorted.

Gerv
Yeah, top priority is to figure out why Apache is killing us.  Next priority is
to start invoking sendmail once per session rather than once per mail.  It's -ks
option puts it into SMTP mode, which lets us send multiple messages.  We should
take advantage of this.  We'll have to send SMTP envelope information and encode
messages, though, like so:

        $msg =~ s/^\./../mg;
        $msg = join("\r\n", split(/[\r\n]+/, $msg)) . "\r\n.\r\n";
or just use Net::SMTP, which does all of that for you. :)

For the quick hack version, see the Win32 section of the installation
instructions in the Bugzilla Guide.  The long-term "correct' fix will be
included with bug 84876.
The "bugmail daemon" is also something lots of people seem to want... I started
looking at that briefly before most of the work for bug 84876 was done.
*** Bug 234230 has been marked as a duplicate of this bug. ***
If we can get 84876 finished in time for the next b.m.o. update, we could try
using Net::SMTP or another mailer module, to see if performance improved...

Gerv
(In reply to comment #16)
> If we can get 84876 finished in time for the next b.m.o. update, we could try
> using Net::SMTP or another mailer module, to see if performance improved...

When is that currently scheduled for?
The next b.m.o. update is currently scheduled for "when we've fixed all the
stuff we want to fix and Dave and Myk think it's a good idea."

84876 is currently scheduled for "as soon as possible, if not sooner". ;-)

Gerv

Comment 19

14 years ago
If the problem used to be memory consumption when spawning another process, you
could spare that for bugs with exceptionaly long cc lists.
There should not be to many bug that would trigger an extra thread.

This might be a ugly solution compared to a background deamon but the advantage
is that all neccesary code should already be done, you just have to put it back
with a condition.

Comment 20

14 years ago
Created attachment 143007 [details]
Less than 5 hours' worth of bug 105547 bugmail

Hard to determine just what is actually new since the last one.
Just FYI, the cron job is currently broken, too... one of the bugs being
affected by this has something in it that's making the bugmail stuff crash when
it hits that bug.  I'll attempt to track this down tonight.
(In reply to comment #21)
> Just FYI, the cron job is currently broken, too... one of the bugs being
> affected by this has something in it that's making the bugmail stuff crash when
> it hits that bug.  I'll attempt to track this down tonight.

The cron job has been fixed.  Someone changed their email address after adding
themselves to the CC list, but before the cron job ran to complete the sending
of the email which didn't go out initially because of this bug.

The BugMail code was choking on the fact that a user listed as being added to
the CC in the activity log didn't exist.  I've hacked the bugmail code to
silently ignore any such users instead of choking on it.  (The likelihood of
that scenario happening again isn't very big).
*** Bug 235565 has been marked as a duplicate of this bug. ***
*** Bug 249662 has been marked as a duplicate of this bug. ***

Comment 25

13 years ago
(regarding my recent dupe bug 249662)
Actually, as a bugzilla mail recipient, I don't mind the aggregation, my problem
is only the case in which I get another message with the exact same content, no
new comments at all.
(In reply to comment #25)
> (regarding my recent dupe bug 249662)
> Actually, as a bugzilla mail recipient, I don't mind the aggregation, my problem
> is only the case in which I get another message with the exact same content, no
> new comments at all.

Yeah, that happens if you're one of the first 2 or 3 hundred people it tried to
send mail to before it got killed.  When the cron job runs overnight to clean up
after it, it doesn't know who got it and who didn't, so it just mails everyone
again.
*** Bug 258700 has been marked as a duplicate of this bug. ***

Updated

13 years ago
Blocks: 84876
No longer depends on: 84876
*** Bug 276157 has been marked as a duplicate of this bug. ***

Comment 29

13 years ago
(In reply to comment #26)
>When the cron job runs overnight to clean up after it, it doesn't
>know who got it and who didn't, so it just mails everyone again.
Apparently including people who excluded it the first time around :-(
OK, I just partially alleviated this on bmo...

I moved the
define(`confDELIVERY_MODE',`b')dnl
line below the
FEATURE(`msp', `[127.0.0.1]')dnl
line in submit.mc.

Apparently when FEATURE('msp') runs, it overrides confDELIVERY_MODE back to
interactive instead of background.
Assignee: preed → email-notifications
QA Contact: mattyt-bugzilla → default-qa

Updated

11 years ago
Duplicate of this bug: 367722

Comment 32

8 years ago
This is far less of a problem now that we have jobqueue.pl.
Severity: major → normal

Comment 33

8 years ago
Not a regression.
Keywords: regression

Comment 34

a year ago
jobqueue.pl fixes this issue, as already suggested in comment 5 and comment 9.
Status: NEW → RESOLVED
Last Resolved: a year ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.