2057187 - IMAP: Assigning/removing message tags (keywords) never sends STORE to server — action is decoded then silently dropped, tags revert on next refresh

Status: RESOLVED FIXED

(MailNews Core :: Networking: IMAP, defect)

Description

[dott]

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:152.0) Gecko/20100101 Firefox/152.0

Steps to reproduce:

1. Set up a Gmail IMAP account (custom keywords supported — confirmed via PERMANENTFLAGS ending in *).
2. Launch Thunderbird from a terminal with MOZ_LOG=IMAP:5,timestamp and MOZ_LOG_FILE=<path> set, to capture wire-level IMAP traffic.
3. Open the IMAP folder (INBOX) and let it finish syncing.
4. Right-click a message → Tag → assign a tag. Wait 2-3 seconds. Repeat several times, alternating assign/remove on the same message.
5. Click "Get Messages", or restart Thunderbird / reopen the folder.

Actual results:

Every tag change from step 4 reverts after step 5 — assigned tags disappear, removed tags reappear.

MOZ_LOG shows the click correctly enqueues an IMAP action URL:
imap://<user>@<host>:993/customKeywords>UID>/INBOX><uid>>>

This is picked up by ProcessCurrentURL, and ProcessSelectedStateURL logs m_imapAction = 0x10000037 for it. Immediately afterward the log shows "ImapThreadMainLoop: idlePending set", and the connection re-enters IDLE — no UID STORE ... FLAGS (...) command, or any command at all, is ever sent to the server for this action.

This identical sequence occurred 6 times for 6 separate tag clicks on the same UID within one session (10:00:01–10:00:15 UTC in the attached log) — every single one silently dropped before reaching the wire.

For comparison, a normal "select" action in the same log (m_imapAction = 0x10000002, triggered by "Get Messages") correctly proceeds to send noop, getquotaroot, and UID fetch ... (FLAGS) — so the IMAP command dispatch pipeline works for other action types; only the keyword-change action appears to be affected.

Server-side cause ruled out: tag changes made from a second Thunderbird instance on another machine, same account, sync correctly to this instance — confirming the IMAP server accepts and persists custom keyword flags normally.

Account's PERMANENTFLAGS reports existing keywords in uppercase ($LABEL1...$LABEL5) — flagging for visibility, though the STORE command never gets far enough here to confirm any relation.

Full MOZ_LOG capture (IMAP:5,timestamp) of a clean reproduction is attached (tb2.log.moz_log).

Expected results:

Assigning/removing a tag should send UID STORE <uid> +FLAGS ($labelN) / -FLAGS ($labelN) (or custom keyword name) to the server, and the change should persist across refresh/restart.

Comment 1

[dott]

Attachment: tb2.log.moz_log.zip

Comment 2

[Max Emig [:maxe]]

I fear this is a regression caused by your bug 355205. Would you like to take care of that or should someone else? I am not sure whether backout would still be a viable option.

Comment 3

[Hartmut Welpmann [:welpy-cw]]

Thanks for the pointer, I have already a suspicion (ampersand backwards compatibility foot gun in the new sanitize function) and will take a closer look tomorrow.

Comment 4

[gene smith]

I tried this on 3 different imap servers (2 supporting IDLE and one was gmail) and don't see a problem (setting or clearing a tag, built-in or custom, generates the appropriate imap STORE command). I'm running with self-built daily pulled last on Jun 14 and it has both of Hartmut's commits from bug 355205.
Maybe there has to be a special not-7bit ASCII char in the tag name?

Comment 5

[Max Emig [:maxe]]

I think I found something, UTF-7 among other circumstances.

Comment 6

[dott]

Attached a screenshot of about:config filtered on mailnews.tags. for my profile, which supports this theory but with a nuance.

All my custom tags with Cyrillic display names are stored under Modified UTF-7 encoded keys, e.g.:

mailnews.tags.&bbaepqq0beaenqq5-.tag = Андрей
mailnews.tags.&bb0eoaq6bdgeqgqw-.tag = Никита
mailnews.tags.&bb4eogrbbdaepqqw-.tag = Оксана

However, I also have a pure-ASCII custom tag named ++, and it's stored the same way — MUTF-7 encoded (mailnews.tags.&aa.tag = ++) rather than as a plain key. That's presumably because + is the shift character in UTF-7 and gets escaped regardless of whether the rest of the name is Latin.

So the condition might not be "non-7bit ASCII" specifically, but rather any tag name that goes through MUTF-7 encoding at all, which would include ASCII names containing characters like +. I'll do an A/B test (STORE via MOZ_LOG) comparing this ++ tag against a plain alphanumeric ASCII tag to narrow this down further and report back.

One more relevant data point: this used to work completely fine for me — it broke specifically after updating to 153.0. So whatever changed in the MUTF-7 keyword handling in this release looks like the regression point, not a pre-existing issue that just wasn't noticed before.

Practically, this makes the bug quite disruptive: my tagging workflow relies entirely on these custom keyword tags, and the only workaround right now would be manually renaming/recreating tags to avoid anything that triggers MUTF-7 encoding — across multiple mail clients that already sync against the existing (encoded) keywords. That's not really viable as a workaround for something that was working correctly before this update.

Comment 7

[dott]

Attachment: Screenshot 2026-07-24 at 08.48.11.png

Comment 8

[Max Emig [:maxe]]

Attachment: Bug 2057187 - Allow legacy modified UTF-7 tag keys in IMAP keyword sanitization. r=#thunderbird-back-end-reviewers

Comment 9

[Max Emig [:maxe]]

Thanks @dott for the final confirmation, all boiled down to & not being whitelisted to go on the wire.

Could you possibly try the build from my proposed patch? https://treeherder.mozilla.org/jobs?repo=try-comm-central&revision=fd2984f6ef624e8e88a2a949f3c3b8b0c5547cbc

I also have added a regression test for the intended behaviour of bug 355205.

Comment 10

[dott]

Confirmed — tag persists after refresh with the patch build.

Comment 13

[mac.mitura]

Hi,

I believe I am experiencing the same issue in a different profile. The affected legacy tag keys are:

mailnews.tags.przej&ark-te
mailnews.tags.zam&apm-wienie

The symptoms are identical. Would it be possible for me to test a Windows x64 build containing your patch? I'd be happy to verify whether it also fixes my profile and report the results.

Comment 14

[Max Emig [:maxe]]

(In reply to mac.mitura from comment #13)

Hi,

I believe I am experiencing the same issue in a different profile. The affected legacy tag keys are:

mailnews.tags.przej&ark-te
mailnews.tags.zam&apm-wienie

The symptoms are identical. Would it be possible for me to test a Windows x64 build containing your patch? I'd be happy to verify whether it also fixes my profile and report the results.

Thank you for confirming.

You can, keep in mind that it will use a seperate profile from your main: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/FbX0HEIsR66UiPFMkaqc-w/runs/0/artifacts/public/build/setup.exe

However, many reviewers are on vacation, I do think it will pass without much issue.

Comment 15

[mac.mitura]

Thanks!

Frankly speaking, I was hoping I could start using my legacy tags again from today with the fix already in place.

Unfortunately, I am very careful about anything related to my email. I cannot afford any risk, including making a mistake while copying or migrating my profile to a test installation. I'm not a developer, just a regular user, maybe a bit more advanced than average.

So, to keep my profile safe, I think I'll wait until the fix is included in an official Thunderbird release.

Comment 16

[Max Emig [:maxe]]

(In reply to mac.mitura from comment #15)

Thanks!

Frankly speaking, I was hoping I could start using my legacy tags again from today with the fix already in place.

Unfortunately, I am very careful about anything related to my email. I cannot afford any risk, including making a mistake while copying or migrating my profile to a test installation. I'm not a developer, just a regular user, maybe a bit more advanced than average.

So, to keep my profile safe, I think I'll wait until the fix is included in an official Thunderbird release.

In that case I would only suggest to start using when it enters Beta, if all goes well this could happen within a month or less.

Comment 18

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/79bc15a37c33
Allow legacy modified UTF-7 tag keys in IMAP keyword sanitization. r=welpy-cw,tobyp

Comment 20

[Magnus Melin [:mkmelin]]

Please request 153esr uplift for this