2073409 - Non-ASCII tags easily hit max-length limitation of IMAP servers

Status: NEW

(Thunderbird :: General, defect)

Description

[YUKI "Piro" Hiroshi]

After the bug 650623 was fixed on Thunderbird 153, newly created non-ASCII tags are sent to an IMAP server as quoted-printable strings. However, string length of tags are quite bloated by quoted-pritable encoding, and it hits the limitation of max length of tags too easily.

For example, six characters Japanese text "ああああああ" is encoded as a 55 length string "=e3=81=82=e3=81=82=e3=81=82=e3=81=82=e3=81=82=e3=81=82". Dovecot's default max length is 50, so such a tag is rejected if the server is configured with less customization.
Here is a part of MOZ_LOG=IMAP:5 logs indicating the problem (masked):

2026-09-18 07:42:07.439000 UTC - [Parent 10868: IMAP]: I/IMAP 1f8d2726600:example.com:S-INBOX:ProcessCurrentURL:imap://[e-mail]:993/customKeywords%3EUID%3E.INBOX%3E1824:1827%3E=e3=81=82=e3=81=82=e3=81=82=e3=81=82=e3=81=82=e3=81=82%3E:  = currentUrl
2026-09-18 07:42:07.440000 UTC - [Parent 10868: IMAP]: D/IMAP ProcessSelectedStateURL [this=1f8d2726600], m_imapAction = 0x10000037
2026-09-18 07:42:07.440000 UTC - [Parent 10868: IMAP]: I/IMAP 1f8d2726600:example.com:S-INBOX:SendData: 36 uid store 1824:1827 +FLAGS (=e3=81=82=e3=81=82=e3=81=82=e3=81=82=e3=81=82=e3=81=82)

2026-09-18 07:42:07.461000 UTC - [Parent 10868: IMAP]: V/IMAP ReadNextLine [rv=0x0 stream=1f8d126de50 nb=62 needmore=0]
2026-09-18 07:42:07.461000 UTC - [Parent 10868: IMAP]: I/IMAP 1f8d2726600:example.com:S-INBOX:CreateNewLineFromSocket: 36 NO [CANNOT] Keyword length too long (0.001 + 0.000 secs).

Dovecot has an option to extend the max length of a tag and we self-host Dovecot so I can avoid this problem, but some mail server may have no such option, and such a workaround is very hard for non-admin users.
I think Thunderbird should use more safely short version of tags internally, e.g. MD5 hash string.

I've confirmed this with Thunderbird 156.0 build ID: 20260910221217.

Comment 1

[YUKI "Piro" Hiroshi]

Here is a workaround for people who encountered same situation.
This script converts internal name of tags longer than 32 characters to 32 characters MD5 hash string.
It also tries to update tags locally stored in messages and message filters.
You just need to copy this and paste to the error console (Ctrl-Shift-J), then run it.

(async () => {
  const { MailServices } =
    ChromeUtils.importESModule("resource:///modules/MailServices.sys.mjs");
  const tags = MailServices.tags;

  async function md5(s) {
    const hash = Cc["@mozilla.org/security/hash;1"]
      .createInstance(Ci.nsICryptoHash);
    hash.init(Ci.nsICryptoHash.MD5);
    const bytes = new TextEncoder().encode(s);
    hash.update(bytes, bytes.length);
    const binary = hash.finish(false);
    return Array.from(binary, c =>
      c.charCodeAt(0).toString(16).padStart(2, "0")
    ).join("");
  }

  function updateFilters(oldKey, newKey) {
    for (const account of MailServices.accounts.accounts) {
      const server = account.incomingServer;
      if (!server) continue;
      let list;
      try {
        list = server.getFilterList(null);
      } catch (_) {
        continue;
      }
      if (!list) continue;
      for (let i = 0; i < list.filterCount; i++) {
        const filter = list.getFilterAt(i);
        let changed = false;
        for (const action of filter.sortedActionList) {
          if (action.type == Ci.nsMsgFilterAction.AddTag &&
              action.strValue == oldKey) {
            action.strValue = newKey;
            changed = true;
            console.log(
              `filter "${filter.filterName}" action: ` +
              `${oldKey} -> ${newKey}`
            );
          }
        }
        for (const term of filter.searchTerms) {
          if (term.attrib == Ci.nsMsgSearchAttrib.Keywords &&
              term.value.str == oldKey) {
            const value = term.value;
            value.str = newKey;
            term.value = value;
            changed = true;
            console.log(
              `filter "${filter.filterName}" condition: ` +
              `${oldKey} -> ${newKey}`
            );
          }
        }
        if (changed)
          list.saveToDefaultFile();
      }
    }
  }
  for (const t of tags.getAllTags()) {
    if (t.key.length < 32 ||
        !/^(?:=[0-9a-f]{2})+$/i.test(t.key))
      continue;
    const newKey = await md5(t.tag);
    console.log(`${t.tag}: ${t.key} -> ${newKey}`);
    tags.addTagForKey(newKey, t.tag, t.color, t.ordinal);
    updateFilters(t.key, newKey);
    for (const account of MailServices.accounts.accounts) {
      const root = account.incomingServer?.rootFolder;
      if (!root) continue;
      for (const folder of root.descendants) {
        try {
          const msgs = [];
          for (const e of folder.msgDatabase.enumerateMessages()) {
            const h = e.QueryInterface(Ci.nsIMsgDBHdr);
            if (h.getStringProperty("keywords")
                .split(/\s+/).includes(t.key))
              msgs.push(h);
          }
          if (msgs.length) {
            folder.addKeywordsToMessages(msgs, newKey);
            folder.removeKeywordsFromMessages(msgs, t.key);
          }
        } catch (_) {}
      }
    }
    tags.deleteKey(t.key);
  }
})();

Comment 2

[Magnus Melin [:mkmelin]]

Right, using QP for that doesn't seem all great.

Comment 3

[Hartmut Welpmann [:welpy-cw]]

Attachment: Bug 2073409 - Bound generated tag keys with a hashed fallback. r=#thunderbird-reviewers

Keep readable encoding for short tag keys. For longer keys, encode the first 128 bits of SHA-256 as lowercase Base32hex under a reserved prefix, keeping generated keys within 32 characters. Salt collisions and leave existing keys unchanged.

Comment 4

[Hartmut Welpmann [:welpy-cw]]

Attachment: Bug 2073409 - Limit generated tag keys to 50 characters. r=#thunderbird-reviewers

Treat automatically generated tag keys as persistent, server-visible compatibility state and cap every newly generated key at 50 characters as a Thunderbird portability policy.

Before deriving a key, look up the exact stored tag string and reuse its existing mapping. This preserves case-sensitive, non-normalized tag identity and prevents legacy tags with readable keys longer than the new limit from being duplicated under compact keys.

Keep the existing modified quoted-printable-style encoding unchanged as the first tier. Continue using readable _N collision suffixes while the complete candidate fits within 50 characters, and fall through instead of creating an over-limit suffixed key.

For readable encodings that do not fit, encode the exact UTF-8 bytes as lowercase unpadded RFC 4648 Base32hex under the versioned &b1 prefix when the complete representation fits. Ampersand is legal in the supported IMAP keyword path and is always escaped by the readable encoder, unlike the preferred greater-than marker, which Thunderbird rejects in its IMAP URL handling.

When the reversible representation is too long, hash the domain-separated byte sequence thunderbird-tag-key-v1, a NUL separator, and the exact UTF-8 tag bytes with SHA-256. Encode the first 200 digest bits as lowercase unpadded Base32hex under &h1, producing a deterministic 43-character key through NS_NewCryptoHash.

Do not normalize, case-fold, salt, randomize, or consult profile occupancy when deriving compact keys. If an explicitly assigned key occupies a generated Base32 or hash key for a different tag, log the exceptional collision to native diagnostics and the mailnews JavaScript console, return NS_ERROR_UNEXPECTED, and leave both mappings unchanged rather than deriving a profile-specific alternate.

Expand nsIMsgTagService coverage for unchanged short keys, readable collision suffixing, 49/50/51-character boundaries, suffix overflow, independently calculated Base32hex and SHA-256 vectors, exact UTF-8 round trips, NFC/NFD distinction, deterministic recreation and creation order, reserved-prefix escaping, compact-key collision failure, distinct long inputs, and idempotent reuse of over-limit legacy mappings. Update the interface documentation to describe bounded generated keys as persistent compatibility state.