234159 - Multiple notifications aggregation - BugMail doesn't finish sending

Status: RESOLVED WORKSFORME

(Bugzilla :: Email Notifications, defect)

Description

[Karsten Düsterloh]

Bugzilla aggregates multiple comments/notifications into one bugmail. This
bugmail contains several old comments/notifications *one has already gotten* and
the new one.

For instance:
Bugmails for http://bugzilla.mozilla.org/show_bug.cgi?id=18574 contain all
changes since about 2004-02-08, bugmail size 7 KB and growing.
Bugmails for http://bugzilla.mozilla.org/show_bug.cgi?id=215243 contain all
changes since about 2004-02-06, bugmail size 19 KB and growing.

Since bug 215243 contains only relatively few addressees, this seems not to be
the problem.

Comment 1

[Aleksey Nogin]

Attachment: An example of an "aggregate" message sent by b.m.o

Comment 2

[Alexander Opitz]

Example Message:
-- 

http://bugzilla.mozilla.org/show_bug.cgi?id=18574


[e-mail] changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
Attachment #9985 [details] [diff] is|0                           |1
           obsolete|                            |
  Attachment #10226 [details] [diff]|0                           |1
        is obsolete|                            |
 Attachment #124876 [details] [diff]|0                           |1
        is obsolete|                            |
 Attachment #134523 [details] [diff]|0                           |1
        is obsolete|                            |
 Attachment #134552 [details] [diff]|0                           |1
        is obsolete|                            |

[e-mail] changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         AssignedTo|[e-mail]     |[e-mail]

[e-mail] changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
OtherBugsDependingO|                            |121927
              nThis|                            |

[e-mail] changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[e-mail]

[e-mail] changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
 Attachment #134552 [details] [diff]|Updated removal reversal    |Updated removal reversal
        description|patch (trunk)               |patch (mozilla-1.6)
 Attachment #134552 [details] [diff]|1                           |0
        is obsolete|                            |




------- Additional Comments From [e-mail]  2004-02-08 17:53 PST -------
Created an attachment (id=140905)
 --> (http://bugzilla.mozilla.org/attachment.cgi?id=140905&action=view)
Full MNG (1.0.5) patch (trunk) bzip2

This is the full patch. The missing files are included in this patch, it is
against trunk (after landing SVG).

MNG doesn't compile by default with this patch, you need:

--enable-image-decoders=default,mng

to build MNG in your patched tree.



@Glenn

Can I assign this bug to me?

@Pavlov

I would Maintain the MNG clue code, I will have it back in trunk without
enabled by default. What must I do?


------- Additional Comments From [e-mail]  2004-02-08 18:00 PST
-------
Sure, take it.  Glenn

------- Additional Comments From [e-mail]  2004-02-08 20:00 PST -------
A Free program (for windows) called SlowView, will view, make, and batch convert
 MNG images.
http://www.slowview.at/
it is at RC2 right now (and has been for months)
for anyone interested in a really good MNG maker program.
it is not opensource, but it is freeware.
it is less than a megabyte (not including required software
(http://www.slowview.at/?page=requirements)


------- Additional Comments From [e-mail]  2004-02-11 17:15
PST -------
(In reply to comment #376)
> A Free program (for windows) called SlowView, will view, make, and batch convert
>  MNG images.

How is this relevant? Don't spam bug reports.

------- Additional Comments From [e-mail]  2004-02-11 17:27 PST
-------
Alexander, maybe you should ask Pavlov for review/superreview/approval of your
patch.

I think that the best thing would be to contact him personally and to make sure
that you are heading in the right direction.

Maybe this could be a feature that can be enabled/disabled in the installer
(like Domi and Venkman can already be switched on/off in the Firefox windows
installer),

or as you are already aiming at, make it an optional compile switch.

In any way, either contact Pavlov personally (I am not sure if he is still
looking at this spammed "thread") or set the review flag for your patch.

TNX for offering to maintain the glue code.

------- Additional Comments From [e-mail]  2004-02-12 06:26 PST
-------
Thanks, Alexander.  The patch id=140905 wfm on FreeBSD 5.0

Be sure to use the "-p0" patch option, and remember to run autoconf
afterwards.

Glenn

------- Additional Comments From [e-mail]  2004-02-13 01:00 PST -------
Sorry for the spam, but:
I keep getting mails on this bug (which is fine) containing all changes since
Feb 08 (which is not). Does anyone else experience this?
It only happens with this bug.

Stefan

------- Additional Comments From [e-mail]  2004-02-13 01:53
PST -------
(In reply to comment #380)
> Sorry for the spam, but:
> I keep getting mails on this bug (which is fine) containing all changes since
> Feb 08 (which is not). Does anyone else experience this?
> It only happens with this bug.
> Stefan

same here

------- Additional Comments From [e-mail]  2004-02-13 03:55 PST -------
Hate to do this, but, well: "Me too." (re: two above comments)

------- Additional Comments From [e-mail]  2004-02-13 04:16
PST -------
Sorry for the spam:
Per comment 380 and further: this is bugzilla bug 234159. Please file further
comments on repeated text in bugmail in that bug.

------- Additional Comments From [e-mail]  2004-02-13 04:58 PST
-------
(From update of attachment 134552 [details] [diff] [review])
Removed "obsolete" flag because this patch is still good for updating
mozilla-1.6.



-- 
Configure bugmail: http://bugzilla.mozilla.org/userprefs.cgi?tab=email

Comment 3

[Glenn Randers-Pehrson]

>Since bug #215243 contains only relatively few addressees, this seems not to be
>the problem.

Like bug #18574, it has a lot of votes, therefore a lot of implicit addresses.

Bug #94035 is doing it too, since February 8th, and also has lots of votes.

Comment 4

[Max Alekseyev]

*** Bug 234161 has been marked as a duplicate of this bug. ***

Comment 5

[Dave Miller [:justdave]]

Yeah, the problem is that each of those bugs have over 600 emails being sent
every time someone touches it.  Apache times out (or users get impatient and hit
the Stop button) before it finishes sending them all, so it never gets marked as
sent.

We've already investigated the mail sending speed, and compared to a mid-sized
ISP, sending out 600 emails in 5 minutes is actually decently speedy for a
single non-threaded process, so the speed of sending the mail probably isn't
going to be able to be improved much.  About the only option that looks workable
at the moment is to simply record who's supposed to get the mail when a bug
changes instead of sending it right then, and run a background daemon to
actually do the mail delivery, which can take as  long as it needs to since the
user won't be sitting there waiting for it.

JayPee: this logic may actually make your megapatch easier because it won't have
the templates interfering with each other :)

Comment 6

[Alexander Opitz]

Does the script is canceled if Apache/User ends transfer? (In php you can act on
that cancelation and you can clean up)

And why this happens on both bugs since the same day?

Comment 7

[Gervase Markham [:gerv]]

> so the speed of sending the mail probably isn't
> going to be able to be improved much.

I'm suspicious of this statement. This has only started going wrong since the
upgrade, AFAICS. And also, don't we turn off the thing where a user closing the
connection stops the script? So that shouldn't be an issue.

Could this be connected with filling up the error log with Perl errors? That's
the cause of the mass change slowdown (patch awaiting review in bug 233645).

Gerv

Comment 8

[Dave Miller [:justdave]]

What changed in the upgrade?  Bugzilla used to spawn off a separate process to
send the email.  Bugzilla itself might have gotten killed when Apache timed out,
but the child process didn't, so it continued.  However, this was a memory and
performance sink because it required initializing a new perl interpreter for
each bug processed (which was felt heavily during edit-multiple).  The mail
process now ocurrs in the same threadspace with the CGI itself, which (except
for the warning problem you mention) was a tremendous speed boost to
change-multiple.

When I hack sanitycheck.cgi to be able to be run from the command-line to
process the emails, no warnings are generated while the emails are being sent.

I do know we supposedly have protection in place against Apache killing us off
while we're doing things, though, so yes, the fact that the process is even
getting killed is a problem.

Comment 9

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

Couldn't a solution be to have a single separate process that sends out emails
which is used whenever mails are being sent, rather then spawning a new process
every time mails are to be sent.

Another alternative is to rather then sending out the mail syncronously just
save out "something" to a separate table and then have separate process that
every so often goes through that table and sends out emails. This process would
then be compleatly disconnected from apache. This way users also wouldn't have
to wait while emails are being sent out, which at least isn't something that i
am interested in.

Comment 10

[Dave Miller [:justdave]]

Jonas: that's more or less what I said in comment 5 :)

Comment 11

[Gervase Markham [:gerv]]

I think we should begin by investigating why the process gets killed even when
we are ignoring signals (or should be.) If we can fix that, we're sorted.

Gerv

Comment 12

[Myk Melez [:myk] [@mykmelez]]

Yeah, top priority is to figure out why Apache is killing us.  Next priority is
to start invoking sendmail once per session rather than once per mail.  It's -ks
option puts it into SMTP mode, which lets us send multiple messages.  We should
take advantage of this.  We'll have to send SMTP envelope information and encode
messages, though, like so:

        $msg =~ s/^\./../mg;
        $msg = join("\r\n", split(/[\r\n]+/, $msg)) . "\r\n.\r\n";

Comment 13

[Dave Miller [:justdave]]

or just use Net::SMTP, which does all of that for you. :)

For the quick hack version, see the Win32 section of the installation
instructions in the Bugzilla Guide.  The long-term "correct' fix will be
included with bug 84876.

Comment 14

[J. Paul Reed [:preed]]

The "bugmail daemon" is also something lots of people seem to want... I started
looking at that briefly before most of the work for bug 84876 was done.

Comment 15

[Dave Miller [:justdave]]

*** Bug 234230 has been marked as a duplicate of this bug. ***

Comment 16

[Gervase Markham [:gerv]]

If we can get 84876 finished in time for the next b.m.o. update, we could try
using Net::SMTP or another mailer module, to see if performance improved...

Gerv

Comment 17

[J. Paul Reed [:preed]]

(In reply to comment #16)
> If we can get 84876 finished in time for the next b.m.o. update, we could try
> using Net::SMTP or another mailer module, to see if performance improved...

When is that currently scheduled for?

Comment 18

[Gervase Markham [:gerv]]

The next b.m.o. update is currently scheduled for "when we've fixed all the
stuff we want to fix and Dave and Myk think it's a good idea."

84876 is currently scheduled for "as soon as possible, if not sooner". ;-)

Gerv

Comment 19

[David Mårtensson]

If the problem used to be memory consumption when spawning another process, you
could spare that for bugs with exceptionaly long cc lists.
There should not be to many bug that would trigger an extra thread.

This might be a ugly solution compared to a background deamon but the advantage
is that all neccesary code should already be done, you just have to put it back
with a condition.

Comment 20

[Felix Miata]

Attachment: Less than 5 hours' worth of bug 105547 bugmail

Hard to determine just what is actually new since the last one.

Comment 21

[Dave Miller [:justdave]]

Just FYI, the cron job is currently broken, too... one of the bugs being
affected by this has something in it that's making the bugmail stuff crash when
it hits that bug.  I'll attempt to track this down tonight.

Comment 22

[Dave Miller [:justdave]]

(In reply to comment #21)
> Just FYI, the cron job is currently broken, too... one of the bugs being
> affected by this has something in it that's making the bugmail stuff crash when
> it hits that bug.  I'll attempt to track this down tonight.

The cron job has been fixed.  Someone changed their email address after adding
themselves to the CC list, but before the cron job ran to complete the sending
of the email which didn't go out initially because of this bug.

The BugMail code was choking on the fact that a user listed as being added to
the CC in the activity log didn't exist.  I've hacked the bugmail code to
silently ignore any such users instead of choking on it.  (The likelihood of
that scenario happening again isn't very big).

Comment 23

[Dave Miller [:justdave]]

*** Bug 235565 has been marked as a duplicate of this bug. ***

Comment 24

[Dave Miller [:justdave]]

*** Bug 249662 has been marked as a duplicate of this bug. ***

Comment 25

[Eyal Rozenberg]

(regarding my recent dupe bug 249662)
Actually, as a bugzilla mail recipient, I don't mind the aggregation, my problem
is only the case in which I get another message with the exact same content, no
new comments at all.

Comment 26

[Dave Miller [:justdave]]

(In reply to comment #25)
> (regarding my recent dupe bug 249662)
> Actually, as a bugzilla mail recipient, I don't mind the aggregation, my problem
> is only the case in which I get another message with the exact same content, no
> new comments at all.

Yeah, that happens if you're one of the first 2 or 3 hundred people it tried to
send mail to before it got killed.  When the cron job runs overnight to clean up
after it, it doesn't know who got it and who didn't, so it just mails everyone
again.

Comment 27

[Dave Miller [:justdave]]

*** Bug 258700 has been marked as a duplicate of this bug. ***

Comment 28

[Dave Miller [:justdave]]

*** Bug 276157 has been marked as a duplicate of this bug. ***

Comment 29

[[e-mail]]

(In reply to comment #26)
>When the cron job runs overnight to clean up after it, it doesn't
>know who got it and who didn't, so it just mails everyone again.
Apparently including people who excluded it the first time around :-(

Comment 30

[Dave Miller [:justdave]]

OK, I just partially alleviated this on bmo...

I moved the
define(`confDELIVERY_MODE',`b')dnl
line below the
FEATURE(`msp', `[127.0.0.1]')dnl
line in submit.mc.

Apparently when FEATURE('msp') runs, it overrides confDELIVERY_MODE back to
interactive instead of background.

Comment 32

[Max Kanat-Alexander]

This is far less of a problem now that we have jobqueue.pl.

Comment 33

[Frédéric Buclin]

Not a regression.

Comment 34

[Frédéric Buclin]

jobqueue.pl fixes this issue, as already suggested in comment 5 and comment 9.