785309 - Profanivore is throwing "ascii "\xB4" does not map to Unicode" errors

Status: RESOLVED FIXED

(bugzilla.mozilla.org :: Extensions, defect)

Description

[:glob ✱]

when trying to send the "new bug" bugmail for bug 785004, profanivore throws an error:

job failed: ascii "\xB4" does not map to Unicode at lib/x86_64-linux-thread-multi/Encode.pm line 174.
 at lib/x86_64-linux-thread-multi/Encode.pm line 174
 Encode::decode('us-ascii', 'https://bugzilla.mozilla.org/show_bug.cgi?id=785004...', 1) called at lib/Email/MIME.pm line 340
 Email::MIME::body_str('Email::MIME=HASH(0x912e0d0)') called at ./extensions/Profanivore/Extension.pm line 75
 Bugzilla::Extension::Profanivore::mailer_before_send(..) called at Bugzilla/Hook.pm line 32
 Bugzilla::Hook::process('mailer_before_send', ..) called at Bugzilla/Mailer.pm line 155
 Bugzilla::Mailer::MessageToMTA('Email::MIME=HASH(0x912e0d0)', 1) called at Bugzilla/Job/Mailer.pm line 47
 eval {...} called at Bugzilla/Job/Mailer.pm line 47

Comment 1

[:glob ✱]

i've pushed a change which stops it from filtering bugmail:

Committing to: bzr+ssh://[e-mail]/bmo/4.0/
modified extensions/Profanivore/Extension.pm
Committed revision 8291.

Comment 2

[:glob ✱]

once we've figured out the unicode issue, i suspect the right way to process the html bugmail would be to use html::tree to modify just the text nodes.

Comment 3

[David Lawrence [:dkl]]

Attachment: Patch to make profanivore work with multipart emails (v1)

This makes with Profanivore work with either single or multipart emails as well as hopefully fix the unicode character issues which I test successfully.

dkl

Comment 4

[:glob ✱]

Comment on attachment 655148 [details] [diff] [review]
Patch to make profanivore work with multipart emails (v1)

creating or updating a bug without adding a comment results in:
> Can't call method "look_down" on an undefined value at ./extensions/Profanivore/Extension.pm line 128.
>  at ./extensions/Profanivore/Extension.pm line 128

adding a comment with linked text, such as:
> this is bug 785309
results in:
> this is HTML::Element=HASH(0x55a52f8).


>+                    $part = _fix_encoding($part);

as $part is an object, _fix_encoding() doesn't need to return it.

>+                    if ($part->content_type =~ /text\/html/) {

only match text/html at the start of the content_type.

>+                    else {
>+                        $body = _filter_text($body);
>+                    }

only filter if it's text/plain content_type.


it should only touch parts where the parent part is multipart/alternative.  other parts (such as an attached patch, or an inline image) shouldn't be filtered by profanivore.

Comment 5

[David Lawrence [:dkl]]

Attachment: Patch to make profanivore work with multipart emails (v2)

New patch that addresses the suggested changes. Had to parse the HTML recursively to preserve any other HTML links inside the comment such as bug links but still filter any plain text. I have tested pretty extensively but please add more offensive language yourself to see if it breaks.

dkl

Comment 6

[David Lawrence [:dkl]]

Attachment: Patch to make profanivore work with multipart emails (v3)

New patch that fixed issue with clearing out of HTML when no comment made :(
dkl

Comment 7

[:glob ✱]

Comment on attachment 655891 [details] [diff] [review]
Patch to make profanivore work with multipart emails (v3)

r=glob

>+sub _filter_html_node {
>+    my $node = shift;
>+    my $content = [ $node->content_list ];
>+    for (my $i = 0; $i < @$content; ++$i) {
>+        if (ref $content->[$i]) {
>+            # It's a child node, process it recursively:
>+            _filter_html_node($content->[$i]);
>+        }
>+        else {
>+            # It's text:
>+            $node->splice_content($i, 1, _filter_text($content->[$i]));
>+        }
>+    }

this would be cleaner as:

    foreach my $item_r ($node->content_refs_list) {
        if (ref $$item_r) {
            _filter_html_node($$item_r);
        } else {
            $$item_r = _filter_text($$item_r);
        }
    }

Comment 8

[David Lawrence [:dkl]]

Committing to: bzr+ssh://[e-mail]/bmo/4.0
modified extensions/Profanivore/Config.pm
modified extensions/Profanivore/Extension.pm
Committed revision 8296.

Committing to: bzr+ssh://[e-mail]/bmo/4.2
modified extensions/Profanivore/Config.pm
modified extensions/Profanivore/Extension.pm
Committed revision 8321.