1182004 - Charset in email notifications seems to get removed for sub parts of multipart/alternative

Status: RESOLVED DUPLICATE of bug 714724

(Bugzilla :: Email Notifications, defect)

Description

[Thorsten Schöning]

Yesterday I upgraded my 4.4.9 installation to 5.0.0 and today recognized that my mail notifications are not working properly: If I add comments with german umlauts to bugs, they appear properly encoded/shown/readable in the web ui, but not in the mails. Comparing older with current mails revealed that the charset statement is missing for all sub parts of the mail:

4.4.9:

> --1436201497.Ac75B0.10541
> Date: Mon, 6 Jul 2015 18:51:37 +0200
> MIME-Version: 1.0
> Content-Type: text/plain; charset="UTF-8"
> Content-Transfer-Encoding: quoted-printable

5.0.0:

> --1436431608.80DC01E0.18380
> Date: Thu, 9 Jul 2015 10:46:48 +0200
> MIME-Version: 1.0
> Content-Type: text/plain

The main content type looks ok for both:

4.4.9:

> Content-Type: multipart/alternative; boundary="1436201497.Ac75B0.10541"; charset="UTF-8"

5.0.0:

> Content-Type: multipart/alternative; boundary="1436431608.80DC01E0.18380";
>  charset="UTF-8"

The newline in the quote above is already there in the mail, if that matters.

The mail client I'm using is The Bat! 6.8.8, which had problems with charset definitions in the past, as can be read in bug 777685. You've added a workaround to fix that issue and it may be that the produced headers by 5.0.0 are perfectly valid and the charset of the subparts should automatically be inherited or such from the main content type. So this might be a bug in my mail client not properly recognizing this as well, I don't know...

But what I know is that Bugzilla::Mailer::MessageToMTA is recognizing missing charsets and setting them manually in the call to walk_parts. I've debugged that call and verified that it is working and adding the missing charsets and those added charsets are later REMOVED before actually sending the mail. I've verified this by enabling smtp_debug in Bugzilla and the log shows all the sub parts without the added charset from walk_parts, like if that call never happened. This looks wrong to me, so I guess this time there's a bug somewhere in Bugzilla or the used libs instead of my mail client. I have this behaviour with SMTP using some external mail server as well with using SENDMAIL with a local postfix. Therefore I don't think this is something the local postfix is doing on it's own.

My added debug statements:

>     $email->walk_parts(sub {
>         my ($part) = @_;
>         return if $part->parts > 1; # Top-level
>         my $content_type = $part->content_type || '';
>         $content_type =~ /charset=['"](.+)['"]/;
>         # If no charset is defined or is the default us-ascii,
>         # then we encode the email to UTF-8 if Bugzilla has utf8 enabled.
>         # XXX - This is a hack to workaround bug 723944.
> warn("$content_type: $1");
>         if (!$1 || $1 eq 'us-ascii') {
> warn('1');
>             my $body = $part->body;
>             if (Bugzilla->params->{'utf8'}) {
> warn('2');
>                 $part->charset_set('UTF-8');
>                 # encoding_set works only with bytes, not with utf8 strings.
>                 my $raw = $part->body_raw;
>                 if (utf8::is_utf8($raw)) {
> warn('3');
>                     utf8::encode($raw);
>                     $part->body_set($raw);
>                 }
>             }
> warn('4') if !is_7bit_clean($body);
>             $part->encoding_set('quoted-printable') if !is_7bit_clean($body);
>         }
> warn($part->content_type);
>     });
> warn($email->debug_structure());
>     if ($method eq "Test") {

...and a corresponding log:

> process_bug.cgi: Use of uninitialized value $1 in concatenation (.) or string at Bugzilla/Mailer.pm line 154., referer:
> process_bug.cgi: text/plain:  at Bugzilla/Mailer.pm line 154., referer:
> process_bug.cgi: 1 at Bugzilla/Mailer.pm line 156., referer:
> process_bug.cgi: 2 at Bugzilla/Mailer.pm line 159., referer:
> process_bug.cgi: 3 at Bugzilla/Mailer.pm line 164., referer:
> process_bug.cgi: 4 at Bugzilla/Mailer.pm line 169., referer:
> process_bug.cgi: text/plain; charset="UTF-8" at Bugzilla/Mailer.pm line 172., referer:
> process_bug.cgi: Use of uninitialized value $1 in concatenation (.) or string at Bugzilla/Mailer.pm line 154., referer:
> process_bug.cgi: text/html:  at Bugzilla/Mailer.pm line 154., referer:
> process_bug.cgi: 1 at Bugzilla/Mailer.pm line 156., referer:
> process_bug.cgi: 2 at Bugzilla/Mailer.pm line 159., referer:
> process_bug.cgi: 3 at Bugzilla/Mailer.pm line 164., referer:
> process_bug.cgi: 4 at Bugzilla/Mailer.pm line 169., referer:
> process_bug.cgi: text/html; charset="UTF-8" at Bugzilla/Mailer.pm line 172., referer:
> process_bug.cgi: + multipart/alternative; boundary="1436433767.7a285e0.19230"; charset="UTF-8", referer:
> process_bug.cgi:      + text/plain; charset="UTF-8", referer:
> process_bug.cgi:      + text/html; charset="UTF-8", referer:
> Net::SMTP>>> Net::SMTP(2.31), referer:
> Net::SMTP>>>   Net::Cmd(2.29), referer:
> Net::SMTP>>>     Exporter(5.68), referer:
> Net::SMTP>>>   IO::Socket::INET(1.33), referer:
> Net::SMTP>>>     IO::Socket(1.36), referer:
> Net::SMTP>>>       IO::Handle(1.34), referer:
> Net::SMTP=GLOB(0x548a5c8)<<< 220 mail.example.org ESMTP Postfix, referer:
> Net::SMTP=GLOB(0x548a5c8)>>> EHLO bugzilla.example.org\r, referer:
> Net::SMTP=GLOB(0x548a5c8)<<< 250-mail.example.org, referer:
> Net::SMTP=GLOB(0x548a5c8)<<< 250-PIPELINING, referer:
> Net::SMTP=GLOB(0x548a5c8)<<< 250-SIZE, referer:
> Net::SMTP=GLOB(0x548a5c8)<<< 250-ETRN, referer:
> Net::SMTP=GLOB(0x548a5c8)<<< 250-STARTTLS, referer:
> Net::SMTP=GLOB(0x548a5c8)<<< 250-AUTH LOGIN, referer:
> Net::SMTP=GLOB(0x548a5c8)<<< 250 8BITMIME, referer:
> Net::SMTP=GLOB(0x548a5c8)>>> AUTH LOGIN\r, referer:
[...]
> Net::SMTP=GLOB(0x548a5c8)<<< 235 Authentication successful, referer:
> Net::SMTP=GLOB(0x548a5c8)>>> MAIL FROM:<[e-mail]>\r, referer:
> Net::SMTP=GLOB(0x548a5c8)<<< 250 Ok, referer:
> Net::SMTP=GLOB(0x548a5c8)>>> RCPT TO:<[e-mail]>\r, referer:
> Net::SMTP=GLOB(0x548a5c8)<<< 250 Ok, referer:
> Net::SMTP=GLOB(0x548a5c8)>>> DATA\r, referer:
> Net::SMTP=GLOB(0x548a5c8)<<< 354 End data with <CR><LF>.<CR><LF>, referer:
> Net::SMTP=GLOB(0x548a5c8)>>> From: [e-mail], referer:
> Net::SMTP=GLOB(0x548a5c8)>>> To: [e-mail], referer:
> Net::SMTP=GLOB(0x548a5c8)>>> Subject: [Bug 243] Testfehler, referer:
> Net::SMTP=GLOB(0x548a5c8)>>> Date: Thu, 09 Jul 2015 09:22:47 +0000, referer:
> Net::SMTP=GLOB(0x548a5c8)>>> X-Bugzilla-Reason: QAcontact AssignedTo Reporter, referer:
> Net::SMTP=GLOB(0x548a5c8)>>> X-Bugzilla-Type: changed, referer:
> Net::SMTP=GLOB(0x548a5c8)>>> X-Bugzilla-Watch-Reason: None, refererhttp://bugzilla.example.org/show_bug.cgi?id=243
> Net::SMTP=GLOB(0x548a5c8)>>> X-Bugzilla-Product: Testprodukt, referer:
> Net::SMTP=GLOB(0x548a5c8)>>> X-Bugzilla-Component: Allgemein, referer:
> Net::SMTP=GLOB(0x548a5c8)>>> X-Bugzilla-Version: trunk, referer:
> Net::SMTP=GLOB(0x548a5c8)>>> X-Bugzilla-Keywords: , referer:
> Net::SMTP=GLOB(0x548a5c8)>>> X-Bugzilla-Severity: normal, referer:
> Net::SMTP=GLOB(0x548a5c8)>>> X-Bugzilla-Who: [e-mail], referer:
> Net::SMTP=GLOB(0x548a5c8)>>> X-Bugzilla-Status: ASSIGNED, referer:
> Net::SMTP=GLOB(0x548a5c8)>>> X-Bugzilla-Resolution: , referer:
> Net::SMTP=GLOB(0x548a5c8)>>> X-Bugzilla-Priority: P3, referer:
> Net::SMTP=GLOB(0x548a5c8)>>> X-Bugzilla-Assigned-To: [e-mail], referer:
> Net::SMTP=GLOB(0x548a5c8)>>> X-Bugzilla-Flags:, referer:
> Net::SMTP=GLOB(0x548a5c8)>>> X-Bugzilla-Changed-Fields: , referer:
> Net::SMTP=GLOB(0x548a5c8)>>> Message-ID: <[e-mail]/>, referer:
> Net::SMTP=GLOB(0x548a5c8)>>> In-Reply-To: <[e-mail]/>, referer:
> Net::SMTP=GLOB(0x548a5c8)>>> References: <[e-mail]/>, referer:
> Net::SMTP=GLOB(0x548a5c8)>>> Content-Type: multipart/alternative; boundary="1436433767.7a285e0.19230";, referer:
> Net::SMTP=GLOB(0x548a5c8)>>>  charset="UTF-8", referer:
> Net::SMTP=GLOB(0x548a5c8)>>> X-Bugzilla-URL: http://bugzilla.example.org/, referer:
> Net::SMTP=GLOB(0x548a5c8)>>> Auto-Submitted: auto-generated, referer:
> Net::SMTP=GLOB(0x548a5c8)>>> MIME-Version: 1.0, referer:
> Net::SMTP=GLOB(0x548a5c8)>>> , referer:
> Net::SMTP=GLOB(0x548a5c8)>>> , referer:
> Net::SMTP=GLOB(0x548a5c8)>>> --1436433767.7a285e0.19230, referer:
> Net::SMTP=GLOB(0x548a5c8)>>> Date: Thu, 9 Jul 2015 11:22:47 +0200\r, referer:
> Net::SMTP=GLOB(0x548a5c8)>>> MIME-Version: 1.0\r, referer:
> Net::SMTP=GLOB(0x548a5c8)>>> Content-Type: text/plain\r, referer:
> Net::SMTP=GLOB(0x548a5c8)>>> \r, referer:
> Net::SMTP=GLOB(0x548a5c8)>>> http://bugzilla.example.org/show_bug.cgi?id=243, referer:
> Net::SMTP=GLOB(0x548a5c8)>>> , referer:
> Net::SMTP=GLOB(0x548a5c8)>>> --- Comment #232 from Thorsten Sch\xc3\xb6ning <[e-mail]> ---, referer:
> Net::SMTP=GLOB(0x548a5c8)>>> m\xc3\xb6\xc3\xb6p, referer:
> Net::SMTP=GLOB(0x548a5c8)>>> , referer:
> Net::SMTP=GLOB(0x548a5c8)>>> -- , referer:
> Net::SMTP=GLOB(0x548a5c8)>>> You are receiving this mail because:, referer:
> Net::SMTP=GLOB(0x548a5c8)>>> You are the QA Contact for the bug., referer:
> Net::SMTP=GLOB(0x548a5c8)>>> You are the assignee for the bug., referer:
> Net::SMTP=GLOB(0x548a5c8)>>> You reported the bug., referer:
> Net::SMTP=GLOB(0x548a5c8)>>> --1436433767.7a285e0.19230, referer:
> Net::SMTP=GLOB(0x548a5c8)>>> Date: Thu, 9 Jul 2015 11:22:47 +0200\r, referer:
> Net::SMTP=GLOB(0x548a5c8)>>> MIME-Version: 1.0\r, referer:
> Net::SMTP=GLOB(0x548a5c8)>>> Content-Type: text/html\r, referer:
> Net::SMTP=GLOB(0x548a5c8)>>> \r, referer:
> Net::SMTP=GLOB(0x548a5c8)>>> <html>, referer:
> Net::SMTP=GLOB(0x548a5c8)>>>     <head>, referer:
> Net::SMTP=GLOB(0x548a5c8)>>>       <base href="http://bugzilla.example.org/">, referer:
> Net::SMTP=GLOB(0x548a5c8)>>>     </head>, referer:
> Net::SMTP=GLOB(0x548a5c8)>>>     <body>, referer:
[...]

./checksetup.pl --check-modules:

> Checking for         Email-Sender (v1.300011) ok: found v1.300018
> Checking for           Email-MIME (v1.904)    ok: found v1.925

Any ideas? Thanks!

Comment 1

[Thorsten Schöning]

Any ideas yet? Is there some info I can provide or things I need to test?

Comment 2

[:glob ✱]

i suspect my patch on bug 714724 would address this.

Comment 3

[Frédéric Buclin]

(In reply to Byron Jones ‹:glob› from comment #2)
> i suspect my patch on bug 714724 would address this.

Yes, I think so too.

Thorsten, there is a patch there. Could you apply it to your 5.0 installation and report in bug 714724 if it fixes the problem for you?