1259942 - HTML email formatting issue with AWS failover cluster (text was fine)

Status: RESOLVED WORKSFORME

(bugzilla.mozilla.org :: Infrastructure, defect)

Description

[David Lawrence [:dkl]]

Some characters seem to be double encoded during delivery. We can try to verify that the emails look correct before delivery by switching over to Test mode on the webheads and view the emails in data/mailer.testfile. I will attach a sample email or two to this bug.

Comment 1

[David Lawrence [:dkl]]

Attachment: [Bug 1259939] test bug for bugzilla dr failover.eml

Comment 2

[Philippe M. Chiasson (:gozer)]

Attachment: MIME-tools-5.502 -> MIME-tools-5.507 diff

Comment 3

[Philippe M. Chiasson (:gozer)]

I did a quick cursory look at Perl module differences in the /mail|mime/i range, and most are all the same, one notable difference I found was MIME-tools @5.502 in SCL3 and @5.507 in AWS.

I've attached the diff between these releases, but I haven't deeply dug into it.

To help get things as much in sync as possible, I've adjusted the AWS side to pin MIME-Tools@5.502 to
reduce one more source of variability.

<https://github.com/gozer/bugzilla.mozilla.org/commit/4916e83f769261ad485e2619a08376a9a380b006>

But haven't had time to try and test emails with that.

Comment 4

[Philippe M. Chiasson (:gozer)]

Typo fix : <https://github.com/gozer/bugzilla.mozilla.org/commit/631527620973be5b4e90d42cc0907df470113fdc>

Comment 5

[Philippe M. Chiasson (:gozer)]

Attachment: mailer.testfile

Attached is a current mailer.testfile generated from latest AMIs.

Encoding looks good to me.

Comment 6

[Richard Weiss [:r2]]

Gozer, was that output directly to a file or did it go through SES?

Comment 7

[David Lawrence [:dkl]]

(In reply to Richard Weiss [:r2] from comment #6)
> Gozer, was that output directly to a file or did it go through SES?

Thats would be mailed to the file and not actually left the server. Gozer, the output is all text formatted emails, so you also need to enable HTML email for the user you are testing the email with. Go to userprefs.cgi?tab=settings and change email format to html.

I would then try another test to mailer.testfile, observe if encoding is broken or not. Then update data/params to be

'mail_delivery_method' => 'Sendmail'
'use_mailer_queue => 1

and then see if SES is changing anything once it leaves AWS.

Thanks
dkl

Comment 8

[Philippe M. Chiasson (:gozer)]

(In reply to David Lawrence [:dkl] from comment #7)
> (In reply to Richard Weiss [:r2] from comment #6)
> > Gozer, was that output directly to a file or did it go through SES?
> 
> Thats would be mailed to the file and not actually left the server.

That's correct.

> Gozer,
> the output is all text formatted emails, so you also need to enable HTML
> email for the user you are testing the email with. Go to
> userprefs.cgi?tab=settings and change email format to html.


Yeah, I noticed and I've gone ahead and tried that.
 
> I would then try another test to mailer.testfile, observe if encoding is
> broken or not.

Encoding looked 100% quoted-printable fine.

> Then update data/params to be
> 
> 'mail_delivery_method' => 'Sendmail'
> 'use_mailer_queue => 1
> 
> and then see if SES is changing anything once it leaves AWS.

Haven't done that yet, but I've just allowed good old SMTP out to the world, bypassing SES and what I got in my inbox was a beautiful, non-munged HTML email.

Going to give SES a spin next.

Comment 9

[Philippe M. Chiasson (:gozer)]

Okay, SES is munging the e-mails, that's confirmed 100%

Comment 10

[Philippe M. Chiasson (:gozer)]

Attachment: Recieved bugmail directly from Postfix

Comment 11

[Philippe M. Chiasson (:gozer)]

Attachment: Recieved bugmail via SES

Comment 12

[Philippe M. Chiasson (:gozer)]

I've attached 2 e-mails recieved into my inbox, only difference is Postifx delivering directly to my inbox, or fowardring via SES...

And SES clearly has reformatted/double-escaped the content somehow.

Comment 13

[Philippe M. Chiasson (:gozer)]

Filed support request with Amazon:

<https://console.aws.amazon.com/support/home?region=us-east-1#/case/?displayId=1746641871>

Comment 14

[Philippe M. Chiasson (:gozer)]

Okay, did some RFC822/RFC2046 Spelunkeling, and I found something that might be a good hint at what's going on.

I've fed both the mail we generate and what SES ends up forwarding with:

https://tools.ietf.org/tools/msglint/

For Postfix:

MIME Lint v1.04 2011-02-25 is a strict syntax validator for Internet messages 
including MIME, RFC 822, DSN (RFC 1891) and MDN (RFC 2298) elements which has 
be run on the attached message.  The result follows:

-----------
UNKNOWN: unknown header 'X-Pobox-Filter-Version' at line 7
UNKNOWN: unknown header 'X-Pobox-Loop-ID' at line 8
UNKNOWN: unknown header 'Delivered-To' at line 9
UNKNOWN: unknown header 'X-Pobox-Delivery-ID' at line 10
WARNING: line too long in header 'Received:' at line 11
UNKNOWN: unknown header 'X-Bugzilla-Reason' at line 20
UNKNOWN: unknown header 'X-Bugzilla-Type' at line 21
UNKNOWN: unknown header 'X-Bugzilla-Watch-Reason' at line 22
UNKNOWN: unknown header 'X-Bugzilla-ID' at line 23
UNKNOWN: unknown header 'X-Bugzilla-Product' at line 24
UNKNOWN: unknown header 'X-Bugzilla-Component' at line 25
UNKNOWN: unknown header 'X-Bugzilla-Version' at line 26
UNKNOWN: unknown header 'X-Bugzilla-Keywords' at line 27
UNKNOWN: unknown header 'X-Bugzilla-Severity' at line 28
UNKNOWN: unknown header 'X-Bugzilla-Who' at line 29
UNKNOWN: unknown header 'X-Bugzilla-Status' at line 30
UNKNOWN: unknown header 'X-Bugzilla-Resolution' at line 31
UNKNOWN: unknown header 'X-Bugzilla-Priority' at line 32
UNKNOWN: unknown header 'X-Bugzilla-Assigned-To' at line 33
UNKNOWN: unknown header 'X-Bugzilla-Target-Milestone' at line 34
UNKNOWN: unknown header 'X-Bugzilla-Flags' at line 35
UNKNOWN: unknown header 'X-Bugzilla-OS' at line 36
UNKNOWN: unknown header 'X-Bugzilla-Changed-Fields' at line 37
UNKNOWN: unknown header 'X-Bugzilla-Changed-Field-Names' at line 38
WARNING: unexpected domain character '/' found in token 'com/' in header 
         'Message-ID' at lines 39-40
WARNING: unexpected domain character '/' found in token 'com/' in header 
         'In-Reply-To' at lines 41-42
WARNING: unexpected domain character '/' found in token 'com/' in header 
         'References' at line 43
UNKNOWN: unknown header 'X-Generated-By' at line 44
WARNING: line too long in header 'Content-Type:' at line 45
WARNING: Unexpected parameter 'charset' in header 'Content-Type' at line 45
UNKNOWN: unknown header 'X-Bugzilla-URL' at line 46
UNKNOWN: unknown header 'Auto-Submitted' at line 47
UNKNOWN: unknown header 'X-Pobox-Client-Address' at line 49
UNKNOWN: unknown header 'X-Pobox-Client-Name' at line 50
UNKNOWN: unknown header 'X-Pobox-Client-HELO' at line 51
UNKNOWN: unknown header 'X-Pobox-Original-Sender' at line 52
UNKNOWN: unknown header 'X-ICG-Account-ID' at line 53
ERROR: missing mandatory header 'return-path' lines 1-53
OK: found part multipart/alternative line 55
OK: preamble 55: 
OK: preamble 56: 
WARNING: MIME headers should only be 'Content-*'. No meaning will apply to 
         header 'Date' at line 58
WARNING: MIME headers should only be 'Content-*'. No meaning will apply to 
         header 'MIME-Version' at line 59
OK: found part text/plain line 63
WARNING: decoded line 68 too long (86 chars); text/plain shouldn't need 
         folding (RFC 2046-4.1.1)
WARNING: decoded line 73 too long (99 chars); text/plain shouldn't need 
         folding (RFC 2046-4.1.1)
WARNING: Character set mislabelled as 'UTF-8' when 'us-ascii' suffices, body 
         part ending line 88
WARNING: MIME headers should only be 'Content-*'. No meaning will apply to 
         header 'Date' at line 89
WARNING: MIME headers should only be 'Content-*'. No meaning will apply to 
         header 'MIME-Version' at line 90
OK: found part text/html line 94
WARNING: Character set mislabelled as 'UTF-8' when 'us-ascii' suffices, body 
         part ending line 171
-----------

For SES:

-----------
UNKNOWN: unknown header 'X-Pobox-Filter-Version' at line 7
UNKNOWN: unknown header 'X-Pobox-Loop-ID' at line 8
UNKNOWN: unknown header 'Delivered-To' at line 9
UNKNOWN: unknown header 'X-Pobox-Delivery-ID' at line 10
WARNING: line too long in header 'Received:' at line 11
WARNING: line too long in header 'DKIM-Signature:' at line 16
UNKNOWN: unknown header 'DKIM-Signature' at lines 14-20
UNKNOWN: unknown header 'X-Bugzilla-Reason' at line 25
UNKNOWN: unknown header 'X-Bugzilla-Type' at line 26
UNKNOWN: unknown header 'X-Bugzilla-Watch-Reason' at line 27
UNKNOWN: unknown header 'X-Bugzilla-ID' at line 28
UNKNOWN: unknown header 'X-Bugzilla-Product' at line 29
UNKNOWN: unknown header 'X-Bugzilla-Component' at line 30
UNKNOWN: unknown header 'X-Bugzilla-Version' at line 31
UNKNOWN: unknown header 'X-Bugzilla-Keywords' at line 32
UNKNOWN: unknown header 'X-Bugzilla-Severity' at line 33
UNKNOWN: unknown header 'X-Bugzilla-Who' at line 34
UNKNOWN: unknown header 'X-Bugzilla-Status' at line 35
UNKNOWN: unknown header 'X-Bugzilla-Resolution' at line 36
UNKNOWN: unknown header 'X-Bugzilla-Priority' at line 37
UNKNOWN: unknown header 'X-Bugzilla-Assigned-To' at line 38
UNKNOWN: unknown header 'X-Bugzilla-Target-Milestone' at line 39
UNKNOWN: unknown header 'X-Bugzilla-Flags' at line 40
UNKNOWN: unknown header 'X-Bugzilla-OS' at line 41
UNKNOWN: unknown header 'X-Bugzilla-Changed-Fields' at line 42
UNKNOWN: unknown header 'X-Bugzilla-Changed-Field-Names' at line 43
WARNING: line too long in header 'Message-ID:' at line 44
WARNING: unexpected domain character '/' found in token 'com/' in header 
         'In-Reply-To' at lines 45-46
WARNING: unexpected domain character '/' found in token 'com/' in header 
         'References' at line 47
UNKNOWN: unknown header 'X-Generated-By' at line 48
WARNING: line too long in header 'Content-Type:' at line 49
WARNING: Unexpected parameter 'charset' in header 'Content-Type' at line 49
UNKNOWN: unknown header 'X-Bugzilla-URL' at line 50
UNKNOWN: unknown header 'Auto-Submitted' at line 51
UNKNOWN: unknown header 'X-SES-Outgoing' at line 53
UNKNOWN: unknown header 'Feedback-ID' at line 54
UNKNOWN: unknown header 'X-Pobox-Client-Address' at line 55
UNKNOWN: unknown header 'X-Pobox-Client-Name' at line 56
UNKNOWN: unknown header 'X-Pobox-Client-HELO' at line 57
WARNING: line too long in header 'X-Pobox-Original-Sender:' at line 59
UNKNOWN: unknown header 'X-Pobox-Original-Sender' at lines 58-59
UNKNOWN: unknown header 'X-ICG-Account-ID' at line 60
ERROR: missing mandatory header 'return-path' lines 1-60
OK: found part multipart/alternative line 62
OK: preamble 62: 
OK: preamble 63: 
WARNING: MIME headers should only be 'Content-*'. No meaning will apply to 
         header 'Date' at line 65
WARNING: MIME headers should only be 'Content-*'. No meaning will apply to 
         header 'MIME-Version' at line 66
OK: found part text/plain line 70
WARNING: Character set mislabelled as 'UTF-8' when 'us-ascii' suffices, body 
         part ending line 115
WARNING: MIME headers should only be 'Content-*'. No meaning will apply to 
         header 'Date' at line 116
WARNING: MIME headers should only be 'Content-*'. No meaning will apply to 
         header 'MIME-Version' at line 117
OK: found part text/html line 121
WARNING: Character set mislabelled as 'UTF-8' when 'us-ascii' suffices, body 
         part ending line 270
--------

Especially intriguing to me are the warnings about:
WARNING: decoded line 68 too long (86 chars); text/plain shouldn't need 
         folding (RFC 2046-4.1.1)
WARNING: decoded line 73 too long (99 chars); text/plain shouldn't need 
         folding (RFC 2046-4.1.1)

(Note, this tool seems to mislabel line numbers just a bit, so the lines in question are specifically in this example:

--- Comment #21 from Philippe Chiasson <[e-mail]> 2016-05-06 10:28=
:18 PDT ---

and

Configure bugmail: http://ec2-52-39-123-254.us-west-2.compute.amazonaws.com=
/userprefs.cgi?tab=3Demail

I believe.

Comment 15

[Philippe M. Chiasson (:gozer)]

Smells more and more to me like the e-mails generated by Bugzilla are on the edge of the spec and that's what is tripping SES.

I am afraid I am not well versed enough in the RFCs to be able to tell at a glance what could be wrong.

But I suspect the mail linting warnings are a good start at where to look.

Comment 16

[Philippe M. Chiasson (:gozer)]

Attachment: Vca01I4363dM269019-postfix.eml

Comment 17

[Philippe M. Chiasson (:gozer)]

Attachment: Vca01I4363dM269019-ses.eml

Comment 18

[Philippe M. Chiasson (:gozer)]

Attachment: RFC822 patch

Here is a verified code patch that fixes the issue for me in testing against SES

Comment 19

[:glob ✱]

(In reply to Philippe M. Chiasson (:gozer) from comment #18)
> Created attachment 8775692 [details] [diff] [review]
> RFC822 patch
> 
> Here is a verified code patch that fixes the issue for me in testing against
> SES

nice!  i wonder if this patch will fix bug 737725 too.

ps.

- my $email = new Email::MIME("$msg_header");
+ my $email = new Email::MIME($msg_header);

Comment 20

[Philippe M. Chiasson (:gozer)]

(In reply to Byron Jones ‹:glob› from comment #19)
> (In reply to Philippe M. Chiasson (:gozer) from comment #18)
> > Created attachment 8775692 [details] [diff] [review]
> > RFC822 patch
> > 
> > Here is a verified code patch that fixes the issue for me in testing against
> > SES
> 
> nice!  i wonder if this patch will fix bug 737725 too.

Quite possible, smells like another side-effect of mixed line endings indeed.

> ps.
> 
> - my $email = new Email::MIME("$msg_header");
> + my $email = new Email::MIME($msg_header);

Ah, good catch there.

Comment 21

[Philippe M. Chiasson (:gozer)]

Monkey-patching in Nubis-land until this lands upstream

See: <https://github.com/mozilla-bteam/bmo-nubis/commit/c52c5d4ffc1cfe18d4d4e4ed97b470a566b3553b>

Comment 22

[David Lawrence [:dkl]]

Comment on attachment 8775692 [details] [diff] [review]
RFC822 patch

Review of attachment 8775692 [details] [diff] [review]:
-----------------------------------------------------------------

r=dkl

Comment 23

[Dylan Hardison [:dylan] (he/him)]

If we're sure this won't hurt prod, it should be commited.

I realize we can't test emails in dev/stage... 

dkl: do you believe this to be safe?

Comment 24

[David Lawrence [:dkl]]

(In reply to Dylan Hardison [:dylan] from comment #23)
> If we're sure this won't hurt prod, it should be commited.
> 
> I realize we can't test emails in dev/stage... 
> 
> dkl: do you believe this to be safe?

I do not see how it could hurt IMO.

dkl