1880288 - Noncharacters in comments cause bugmails sending to fail

Status: RESOLVED FIXED

(Bugzilla :: Email Notifications, defect)

Description

[Frédéric Buclin]

In Bugzilla 5.0.4 and 5.3.2 (?!), if you write a comment containing e.g. the unicode non-character \x{fdd2}, Bugzilla is then unable to send bugmails, causing errors like:

"\x{fdd2}" does not map to UTF-8 at /usr/share/perl5/vendor_perl/Email/MIME.pm line 278.

Then all subsequent comments make Bugzilla crash, because it's unable to clear the email queue and Bugzilla tries to send the problematic comment again and again. This happened to Mageia a few weeks ago, see https://bugs.mageia.org/show_bug.cgi?id=32609, and I can reproduce the problem locally.

I'm restricting this bug to the security group so that people do not start playing with it. If I understand correctly, Bugzilla 5.0.6 will become Bugzilla 5.2, but I don't know what Bugzilla 5.3.2 will become. 5.4? I don't know if Harmony is affected.

Comment 1

[Frédéric Buclin]

Attachment: Patch for 5.0.4, v1

By default, Email::MIME::new() falls back to FB_CROAK if 'encode_check' is undefined, meaning that it immediately dies if there are unexpected characters. Here, we now pass FB_DEFAULT, which replaces unexpected characters by the substitution character:

https://metacpan.org/pod/Encode#Handling-Malformed-Data

The data in the bug itself is unaltered. Only bugmails are.

Comment 3

[Kirill Bogotoba]

Workaround. The 'unless $text;' check has been removed from the Bugzilla/Template.pm file. line 149

I hope this doesn't break the service

Comment 4

[Dave Miller [:justdave]]

Confirmed this affects 5.2 and 5.3.3 as well.
Jury is still out on 4.4 and Harmony, the mail code there is completely different (and ironically, similar to each other).

Comment 5

[BMO Github Automation]

Attachment: [bugzilla/bugzilla-security] [5.0.4] Bug 1880288: Noncharacters in comments cause bugmail sending to fail (#10)

Comment 6

[BMO Github Automation]

Attachment: [bugzilla/bugzilla-security] [5.2] Bug 1880288: Noncharacters in comments cause bugmail sending to fail (#11)

Comment 7

[Dave Miller [:justdave]]

There's some awfully-similar-looking code to what you changed in Bugzilla/BugMail.pm in Bugzilla/Mailer.pm. Do we need to update it there, too?

Comment 8

[Dave Miller [:justdave]]

I'm guessing yes. 5.3 branch actually only has that code in Bugzilla/Mailer.pm and the spot in Bugzilla/BugMail.pm that used to have it calls the copy in Bugzilla/Mailer.pm now (consolidated the duplicate code).

Comment 9

[BMO Github Automation]

Attachment: [bugzilla/bugzilla-security] [5.3] Bug 1880288: Noncharacters in comments cause bugmail sending to fail (#12)

Comment 10

[Dave Miller [:justdave]]

I can't reproduce this in 4.4, I get a warning in the log from Template Toolkit that show_bug.cgi: Unicode non-character U+FDD2 is not recommended for open interchange in print at /usr/lib/x86_64-linux-gnu/perl5/5.34/Template.pm line 167. but the email sends. Looking at the code, I think this is because we're not trying to force-re-encode the data to utf8 (which the other branches are doing). It looks like 4.4 is just doing a straight up quoted-printable encode on the binary byte values.

I can reproduce this on Harmony though, even though it's using almost the same mail code as 4.4. But it knows how to do Unicode. :-)

Comment 11

[BMO Github Automation]

Attachment: [bugzilla/bugzilla-security] [harmony] Bug 1880288: Noncharacters in comments cause bugmail sending to fail (#13)

Comment 12

[Dave Miller [:justdave]]

The conditions which started this error happening appear to have been introduced in bug 714724 which landed in 5.0.2 and 5.1.2

Comment 13

[Dave Miller [:justdave]]

The final patches for these fixes can be found in the associated commits as listed below:

5.0.4: https://github.com/bugzilla/bugzilla/commit/66db7a430586474f8a68007ac07546f49e9ab4df
5.2: https://github.com/bugzilla/bugzilla/commit/ee9cf0026b35df50a8070dff0aea609c35213910
5.3: https://github.com/bugzilla/bugzilla/commit/2a91862177cd008e2963b4e349e99f55f1851ea3
harmony: https://github.com/bugzilla/harmony/commit/22202c561487481e21456a396b1f386b402eec3c

Comment 14

[Dave Miller [:justdave]]

One additional commit on this for the 5.3 branch, the patch for that had a missing semicolon :-/
This is broken in the 5.3.3 tarball. Running git pull after unpacking it should pull this fix.
https://github.com/bugzilla/bugzilla/commit/d4313ce6f61c0b35fb364526445a5e7d460eedd1

Comment 15

[Dave Miller [:justdave]]

Comment on attachment 9380189 [details] [diff] [review]
Patch for 5.0.4, v1

Forgot to clear the review flag on here, review got taken care of on the GitHub PR.