1116134 - Encoding problems for html-notifications under linux

Status: RESOLVED DUPLICATE of bug 714724

(Bugzilla :: Email Notifications, defect)

Description

[Matthias Ludwig]

User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; MDDRJS; rv:11.0) like Gecko

Steps to reproduce:

With Thorsten Schöning (thanks!) I've worked out the following Problem : see https://groups.google.com/forum/#!topic/mozilla.support.bugzilla/g9NYvJrzjiE


Short story:
- Notification-Mails from Bugzilla-Installation under Linux depicts german umlauts in two cryptographic characters in MS-Outlook. All characters are depicted correctly in the Bugzilla web-GUI of the same server.

Probable Reason:
Mail Notification is send as mime-multipart message. The mime headers for the mime content parts (Content-Type: "text/mail" and Content-Type: "text/html") are not set. The mime-header "Content-Transfer-Encoding" is not set. The content is plain UTF-8 without encoding to quoted-printable. The bytes are the correct UTF-8 encoded form of the characters. 

I switched the mail parameter from SMTP to Testmail in Bugzillas Mail Administration parameters: Attached the file "mail.testfile". 

More information:
(1) The same version of Bugzilla installed under windows does correctly set the mime headers. This mail is correctly depicted in MS-Outlook.
(2) Bugzillas UTF-8 Parameter is enabled.
(3) I suppose the headers should be set in the email->walk_parts hook.
(4) I've modified the walk_parts hook in Mailer.pm, results below:
This prooves that UTF-8 parameter is really set and that the characters are correctly UTF-8.

$email->walk_parts(sub {
	my ($part) = @_;
	return if $part->parts > 1; # Top-level
	my $content_type = $part->content_type || '';
	$content_type =~ /charset=['"](.+)['"]/;
	# If no charset is defined or is the default us-ascii,
	# then we encode the email to UTF-8 if Bugzilla has utf8 enabled.
	# XXX - This is a hack to workaround bug 723944.
	if(!$1){
		warn "walk_parts part: $(part->parts), regexp group is empty";
	}else{
	   warn "walk_parts part: $(part->parts), regexp group: $1";
	}
	
	if (!$1 || $1 eq 'us-ascii') {
		my $body = $part->body;
		if (Bugzilla->params->{'utf8'}) {
			warn "if condition true: Bugzilla->params->{'utf8'}";
			$part->charset_set('UTF-8');
			# encoding_set works only with bytes, not with utf8 strings.
			my $raw = $part->body_raw;
			if (utf8::is_utf8($raw)) {
				warn "if condition true: utf8::is_utf8(raw)";
				utf8::encode($raw);
				$part->body_set($raw);
			}
		}
		if(!is_7bit_clean($body)){
			warn "if condition true: !is_7bit_clean(body)";
		}
		$part->encoding_set('quoted-printable') if !is_7bit_clean($body);
	}
}); 



[Sat Dec 27 15:14:01 2014] process_bug.cgi: walk_parts part: 33 33part->parts), regexp group is empty at Bugzilla/Mailer.pm line 147.
[Sat Dec 27 15:14:01 2014] process_bug.cgi: if condition true: Bugzilla->params->{'utf8'} at Bugzilla/Mailer.pm line 155.
[Sat Dec 27 15:14:01 2014] process_bug.cgi: if condition true: utf8::is_utf8(raw) at Bugzilla/Mailer.pm line 160.
[Sat Dec 27 15:14:01 2014] process_bug.cgi: if condition true: !is_7bit_clean(body) at Bugzilla/Mailer.pm line 166.
[Sat Dec 27 15:14:01 2014] process_bug.cgi: walk_parts part: 33 33part->parts), regexp group is empty at Bugzilla/Mailer.pm line 147.
[Sat Dec 27 15:14:01 2014] process_bug.cgi: if condition true: Bugzilla->params->{'utf8'} at Bugzilla/Mailer.pm line 155.
[Sat Dec 27 15:14:01 2014] process_bug.cgi: if condition true: utf8::is_utf8(raw) at Bugzilla/Mailer.pm line 160.
[Sat Dec 27 15:14:01 2014] process_bug.cgi: if condition true: !is_7bit_clean(body) at Bugzilla/Mailer.pm line 166.

(5) The following hack below helps. After that the header is set, outlook displays the mail properly.

Date: Mon, 29 Dec 2014 11:47:41 +0100
MIME-Version: 1.0
Content-Type: text/html; charset="UTF-8"
                         ****************


The hack: I've inserted the two lines in  _generate_bugmail (BugMail.pm):
charset => "UTF-8",

sub _generate_bugmail {
    my ($user, $vars) = @_;
    my $template = Bugzilla->template_inner($user->setting('lang'));
    my ($msg_text, $msg_html, $msg_header);
  
    $template->process("email/bugmail-header.txt.tmpl", $vars, \$msg_header)
        || ThrowTemplateError($template->error());
    $template->process("email/bugmail.txt.tmpl", $vars, \$msg_text)
        || ThrowTemplateError($template->error());

    my @parts = (
        Email::MIME->create(
            attributes => {
                content_type => "text/plain",
                charset => "UTF-8",
            },
            body => $msg_text,
        )
    );
    if ($user->setting('email_format') eq 'html') {
        $template->process("email/bugmail.html.tmpl", $vars, \$msg_html)
            || ThrowTemplateError($template->error());
        push @parts, Email::MIME->create(
            attributes => {
                content_type => "text/html",         
                charset => "UTF-8",
            },
            body => $msg_html,
        );
    }

(6) This observation may rely to the problem:
Content-Type of the main multipart mime message part looks different. There is an extra line break on the linux mail.

windows
Content-Type: multipart/alternative; boundary="1419790906.D60b1A0.5972"; charset="UTF-8"

linux
Content-Type: multipart/alternative; boundary="1419784785.1C11d1b0.2200";
 charset="UTF-8"

(7) This observation may rely to the problem:

There is a mix of unix/win line endings in the mail.testfile of the linux server. The mime header lines ends with <CR><LF> (windows), the rest ends just with <LF> (linux)

(8) This observation may rely to the problem:
On the Windows Server the perl module "File-MimeInfo" is missing.


(9)

Version information:

Windows 7 / bugzilla 4.4.6 
parameter UTF-8=on
Bugzilla-de Version 4.4.6-1 
Missing perl modules: Encode-Detect,TheSchwartz,Daemon-Generic,File-Slurp,Apache-SizeLimit,File-MimeInfo

Ubuntu 14.04 / buzilla 4.4.6 (parameter UTF-8=on)
parameter UTF-8=on
Bugzilla-de Version 4.4.6-1 
Missing perl modules: None

Comment 1

[Matthias Ludwig]

Attachment: linux-mailer.testfile

The test-mailer file generated in Test-Mode under Linux
(not correctly presented in MS-Outlook)

Comment 2

[Matthias Ludwig]

Attachment: windows-mailer.testfile

The test-mailer file generated in Test-Mode under Windows

Comment 3

[Matthias Ludwig]

Attachment: windows-checksetup.txt

checksetup Output of the Windows machine

Comment 4

[Matthias Ludwig]

Attachment: linux-checksetup.txt

checksetup Output of the Linux machine

Comment 5

[Matthias Ludwig]

Attachment: linux-after-hack-mailer.testfile

The mailer.testfile after my hack (5)

Comment 6

[Matthias Ludwig]

(10) This observation may rely to the problem:
On the Windows Server the perl module "Encode-Detect" is missing.

Comment 7

[Carl-Johan Schenström]

Changing Mailer.pm:96 to the following changes nothing:

$email->walk_parts(sub {
    my ($part) = @_;
    $part->charset_set('UTF-8');
    return;
}

Seems the bug is in Email::MIME? I haven't updated past 1.926, though.

Comment 8

[Frédéric Buclin]

Looks like a dupe of bug 714724. Reopen if I'm wrong.