1556478 - Port Bug 1543077 part 2+3 - Follow changes to Japenese charset detection

Status: RESOLVED FIXED

(MailNews Core :: General, task)

Description

[Jorg K (CEST = GMT+2)]

Quite a bit of bustage there.

Comment 1

[Jorg K (CEST = GMT+2)]

Attachment: Bustage fix patch

This compiles.

Henri, you took away the Japanese detection here:
https://hg.mozilla.org/mozilla-central/rev/f6d04ade73b5#l4.67

So how does Japanese detection work now? We have a test for it here which fail now:
https://searchfox.org/comm-central/rev/ba9e556dd988305051cd54ba1c6c13176e797f61/mailnews/compose/test/unit/test_detectAttachmentCharset.js#40
And maybe this will also fail:
https://searchfox.org/comm-central/rev/ba9e556dd988305051cd54ba1c6c13176e797f61/mail/test/mozmill/composition/test-cp932-display.js#6
EDIT: No, this one still passes.

Comment 2

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/03606755e033
Port bug 1543077, part 2+3: Follow changes to Japanese charset detection. rs=bustage-fix
https://hg.mozilla.org/comm-central/rev/e551caf6775a
disable subtests testShiftJIS and testISO2022JP of test_detectAttachmentCharset.js. rs=bustage-fix

Comment 3

[Jorg K (CEST = GMT+2)]

Henri, can we use the new mozilla::JapaneseDetector to detect the various charsets. As you know, without specialised detection, ISO-2022-JP is detected as UTF-8 since it's 7bit :-( - If so, could you provide a code snippet for us.

Comment 4

[Jorg K (CEST = GMT+2)]

Wow, we have another SHift_JIS test which fails now :-(
TEST-UNEXPECTED-FAIL | comm/mailnews/import/test/unit/test_shiftjis_csv.js
https://searchfox.org/comm-central/rev/ba9e556dd988305051cd54ba1c6c13176e797f61/mailnews/import/test/unit/test_shiftjis_csv.js#4

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/70e2ac37e753
disable failing test_shiftjis_csv.js. rs=bustage-fix DONTBUILD

Comment 6

[Henri Sivonen (:hsivonen)]

(In reply to Jorg K (GMT+2) from comment #3)

Henri, can we use the new mozilla::JapaneseDetector to detect the various charsets. As you know, without specialised detection, ISO-2022-JP is detected as UTF-8 since it's 7bit :-( - If so, could you provide a code snippet for us.

In https://searchfox.org/comm-central/source/mailnews/base/util/nsMsgUtils.cpp#1839 in place of

  // Use detector.
  nsCOMPtr<nsICharsetDetector> detector;
  nsAutoCString detectorName;
  Preferences::GetLocalizedCString("intl.charset.detector", detectorName);
  if (!detectorName.IsEmpty()) {
    // We recognize one of the three magic strings for the following languages.
    if (detectorName.EqualsLiteral("ruprob")) {
      detector = new nsRUProbDetector();
    } else if (detectorName.EqualsLiteral("ukprob")) {
      detector = new nsUKProbDetector();
    } else if (detectorName.EqualsLiteral("ja_parallel_state_machine")) {
      detector = new nsJAPSMDetector();
    }
  }

  if (detector) {
    RefPtr<CharsetDetectionObserver> observer = new CharsetDetectionObserver();

    rv = detector->Init(observer);
    NS_ENSURE_SUCCESS(rv, rv);

    char buffer[1024];
    uint32_t numRead = 0;
    bool dontFeed = false;
    while (NS_SUCCEEDED(inputStream->Read(buffer, sizeof(buffer), &numRead))) {
      // XXX: We need to break early here to work around a problem in Shift-JIS
      // detection. If we call `DoIt()` with any empty buffer, Shift-JIS is not
      // detected, however ISO-2022-JP is detected.
      if (numRead == 0) break;
      detector->DoIt(buffer, numRead, &dontFeed);
      NS_ENSURE_SUCCESS(rv, rv);
      if (dontFeed)  // XXX: We should really break here with:
                     // if (dontFeed || numRead == 0).
        break;
    }
    rv = detector->Done();
    NS_ENSURE_SUCCESS(rv, rv);

    observer->GetDetectedCharset(aCharset);
  }

Instead something along the lines of:

  // Use detector.
  nsCOMPtr<nsICharsetDetector> detector;
  mozilla::UniquePtr<mozilla::JapaneseDetector> japaneseDetector;
  nsAutoCString detectorName;
  Preferences::GetLocalizedCString("intl.charset.detector", detectorName);
  if (!detectorName.IsEmpty()) {
    // We recognize one of the three magic strings for the following languages.
    if (detectorName.EqualsLiteral("ruprob")) {
      detector = new nsRUProbDetector();
    } else if (detectorName.EqualsLiteral("ukprob")) {
      detector = new nsUKProbDetector();
    } else if (detectorName.EqualsLiteral("ja_parallel_state_machine")) {
      japaneseDetector = mozilla::JapaneseDetector::Create(true);
    }
  }

  if (detector) {
    RefPtr<CharsetDetectionObserver> observer = new CharsetDetectionObserver();

    rv = detector->Init(observer);
    NS_ENSURE_SUCCESS(rv, rv);

    char buffer[1024];
    uint32_t numRead = 0;
    bool dontFeed = false;
    while (NS_SUCCEEDED(inputStream->Read(buffer, sizeof(buffer), &numRead))) {
      // XXX: We need to break early here to work around a problem in Shift-JIS
      // detection. If we call `DoIt()` with any empty buffer, Shift-JIS is not
      // detected, however ISO-2022-JP is detected.
      if (numRead == 0) break;
      detector->DoIt(buffer, numRead, &dontFeed);
      NS_ENSURE_SUCCESS(rv, rv);
      if (dontFeed)  // XXX: We should really break here with:
                     // if (dontFeed || numRead == 0).
        break;
    }
    rv = detector->Done();
    NS_ENSURE_SUCCESS(rv, rv);

    observer->GetDetectedCharset(aCharset);
  } else if (japaneseDetector) {
    char buffer[1024];
    uint32_t numRead = 0;
    bool dontFeed = false;
    while (NS_SUCCEEDED(inputStream->Read(buffer, sizeof(buffer), &numRead))) {
      mozilla::Span src = mozilla::AsBytes(mozilla::MakeSpan(buffer, numRead));
      auto encoding = japaneseDetector->Feed(src, (numRead == 0));
      if (encoding) {
        encoding->Name(aCharset);
        break;
      }
      if (numRead == 0) {
        break;
      } 
  }

In https://searchfox.org/comm-central/source/mailnews/mime/src/comi18n.cpp#69 if you want to keep detecting UTF-8 in the Japanese case, you need to add a UTF-8 check but let ISO-2022-JP take precedence.

Comment 7

[Henri Sivonen (:hsivonen)]

Comment on attachment 9069432 [details] [diff] [review]
Bustage fix patch

Review of attachment 9069432 [details] [diff] [review]:
-----------------------------------------------------------------

r+ as a bustage fix, but see previous bug comment about an actual fix.

Comment 8

[Jorg K (CEST = GMT+2)]

Attachment: 1556478-fix-jap-detection.patch

Like this? The auto encoding = japaneseDetector->Feed(src, false); is correct with the false, right?

Both tests pass again, even without the hunk in comi18n.cpp.

Comment 9

[Jorg K (CEST = GMT+2)]

Attachment: 1556478-fix-jap-detection.patch (alternative version)

How about this alternative version, it adds UTF-8 detection to MIME_detect_charset. That makes sense, no?

EDIT: https://bugzilla.mozilla.org/attachment.cgi?oldid=9069591&action=interdiff&newid=9069610&headers=1

Comment 10

[Henri Sivonen (:hsivonen)]

Comment on attachment 9069591 [details] [diff] [review]
1556478-fix-jap-detection.patch

Review of attachment 9069591 [details] [diff] [review]:
-----------------------------------------------------------------

Thanks.

::: mailnews/base/util/nsMsgUtils.cpp
@@ +1879,5 @@
> +      if (encoding) {
> +        encoding->Name(aCharset);
> +        break;
> +      }
> +      if (numRead == 0) break;

I'd prefer braces around `break;`.

::: mailnews/mime/src/comi18n.cpp
@@ +105,5 @@
> +      // If ISO-2022-JP return, since being a 7bit charset, it would be
> +      // detected as UTF-8.
> +      if (aCharset.EqualsLiteral("ISO-2022-JP")) return NS_OK;
> +
> +      if (IsUTF8(mozilla::MakeStringSpan(aBuf))) aCharset.AssignLiteral("UTF-8");

Instead of `mozilla::MakeStringSpan(aBuf)`, please use `mozilla::MakeSpan(aBuf, aLength)`. (I'd prefer to have braces around `aCharset.AssignLiteral("UTF-8");`)

Comment 11

[Henri Sivonen (:hsivonen)]

Comment on attachment 9069610 [details] [diff] [review]
1556478-fix-jap-detection.patch (alternative version)

Review of attachment 9069610 [details] [diff] [review]:
-----------------------------------------------------------------

::: mailnews/mime/src/comi18n.cpp
@@ +109,3 @@
>    }
> +
> +  if (IsUTF8(mozilla::MakeStringSpan(aBuf))) {

Same thing here about `mozilla::MakeSpan(aBuf, aLength)`.

Comment 12

[Henri Sivonen (:hsivonen)]

(In reply to Jorg K (GMT+2) from comment #9)

How about this alternative version, it adds UTF-8 detection to MIME_detect_charset. That makes sense, no?

Yeah, the second patch looks better.

Comment 13

[Jorg K (CEST = GMT+2)]

Thanks, Henri. I never posted the comment about my try run ... Try for the second version that changes the behaviour of MIME_detect_charset:
https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=a915ebf3bb68afca2d08f850fe452a18963f404b
It came out green, so let's take the second version.

Comment 14

[Jorg K (CEST = GMT+2)]

Attachment: 1556478-fix-jap-detection.patch (alternative version, v2)

I added the braces and switched to MakeSpan(). Good to go.

Comment 15

[Henri Sivonen (:hsivonen)]

(In reply to Jorg K (GMT+2) from comment #8)

Like this? The auto encoding = japaneseDetector->Feed(src, false); is correct with the false, right?

I missed this question earlier, sorry.

Passing false unconditionally is technically incorrect if the entire stream rather than just a prefix is examined. The only case that I can think of where it might make a difference is if there's a three-byte EUC-JP sequence so that the EUC-JP and Shift_JIS no longer agree on which pairs of bytes map to characters.

Comment 16

[Jorg K (CEST = GMT+2)]

This is not examining a stream, but a single string passed in. So it's a single call and aLast should be passed in as true right?

Comment 17

[Henri Sivonen (:hsivonen)]

(In reply to Jorg K (GMT+2) from comment #16)

This is not examining a stream, but a single string passed in. So it's a single call and aLast should be passed in as true right?

Yes.

Comment 18

[Jorg K (CEST = GMT+2)]

Attachment: 1556478-fix-jap-detection.patch (alternative version, v2b)

With true.

Comment 19

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/dda6d487c69a
Fix Japanese detection and re-enable tests. r=hsivonen