Closed Bug 1892385 Opened 10 months ago Closed 5 months ago

Messages are untagged when moved from a folder to a shared folder using Courier IMAP server with filesystem permissions-based sharing.

Categories

(MailNews Core :: Networking: IMAP, defect)

Product:
MailNews Core
Component:
Networking: IMAP
Version:
Thunderbird 115
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED INVALID

People

(Reporter: eduardp, Unassigned)

References

(
URL
)

Details

Attachments

(1 file)

log_file.moz_log
39.55 KB, text/plain
Details

User Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:124.0) Gecko/20100101 Firefox/124.0

Steps to reproduce:

Tag/Label a message in Inbox as important/later/.... or mark it as favorite.
Move the message to a folder.

Actual results:

The message looses its Tag/Label. If you want it labeled, you must go to the folder, find it, and label it again. That breaks all personal workflow.

Expected results:

I would expect labeled messages remain labeled while changing folders (that applies also to favorite marked ones).
I don't see any reason why someone would want the message to be "unlabeled" when moving it from one folder to another.
If messages in folders can be labeled, then I understand, there is a way for Thunderbird to move it AND THEN go to that folder, find it, and label it.

I don't see why, but just in case there is some reason to forget the tagging while moving, there could be a modal asking if you are willing to keep tagging once you try to move a labeled message.

@Eduard: Thanks for your report. You are talking about an IMAP account, right ?

Flags: needinfo?(eduardp)

Yes, I am talking about IMAP

Flags: needinfo?(eduardp)
Type: enhancement → defect
Component: Untriaged → Networking: IMAP
OS: Unspecified → Linux
Product: Thunderbird → MailNews Core
Hardware: Unspecified → x86_64
See Also: → 1901509, 729732, 760856, 1504993, 1511362, 1863741
Summary: Message Labels/Tags should remain while changing folders → IMAP: messages are unlabeled/untagged when moved from one folder to another

Reporter eduard,
I haven't been able to duplicate this with my primary imap account. Can you tell me what kind of imap server your account is using?
Also, if possible, please record and attach (using "Attach new file" button above) an IMAP:4 log that records what is happening when you move a tagged message to another folder. See https://wiki.mozilla.org/MailNews:Logging
Also, I might ask if you are moving the message within the same imap account or maybe between accounts. Anyhow, that should preserve the tag too, I think.

Hi,
I tested it and found that, as you say, the tags are preserved between folders for the same user.
The flags are lost when you move an email to a Shared Folder (using Courier MTA).
I understand that is an edge case, then.
Maybe related to Courier IMAP instead of thunderbird.
Sorry to bother you. Thanks for checking.

Are you still seeing this in version 128 or newer?

Flags: needinfo?(eduardp)

Yes, I tested it with 128.2.0esr from Ubuntu 24.04 snap and the problem remains. Moving the message to a Shared Folder erases it's color/flag.

Flags: needinfo?(eduardp)

I've tested this with copying/moving tagged messages to folders in "public" and "other" namespaces and the tag and its color is preserved. This is on dovecot server. I don't have a setup to try this on Courier or any other server with non-private namespaces.
When you say "Shared Folder" do you actually mean you are using the folders in the public or other-user namespaces (as shown on the advance server setting screen of tb) or do you have another way of sharing folders?

Sorry for the delay, I needed to look into it and couldn't find time. A loong time has passed since I first configured those Shared folders. Now I keep passing them from one server to the next one and they work.

First:
What I mean for shared folder is a kind of folder in Courier MTU where you can have different users push or get messages from (through IMAP) and every user can see the messages from all of them.

There are two ways to make this kind of folders in Courier-mta:

https://www.courier-mta.org/imap/README.sharedfolders.html

For what I can remember and I see, I use the second one: Filesystem permissions-based shared folders

I have an script that creates the physical folder. Do not remember how but this folder gets automagically shared with all users (I think that there is some symlinks for every user mail tree:

// COMMAND: createSharedFolder NewFolderName 

maildirmake -s write -f $1 /var/qmail/mailnames/<DOMAIN.COM>/<SHARED_FOLDER_NAME>/Maildir/
chown -R popuser.popuser /var/qmail/mailnames/<DOMAIN.COM>/<SHARED_FOLDER_NAME>/Maildir/.$1

I think I already said it but when I use thunderbird to reflag the message as Important AFTER I moved it to the Courier MTU Shared Folder (as moving it makes the flag disappear)... the flag remains in the IMAP Shared Folder, so I see it when I connect to the same IMAP Shared Folder from another thunderbird (say, my desktop computer or my laptop one, for example).
So as I understand it, even if the error is with Courier MTU folder based shared folders, it could be solved if Thunderbird tests the email after being moved and reflag it in case the flag has been lost.

Ok, Courier doesn't use the standard imap namespace method to share folders. I think Zimbra imap server uses a similar technique.
One question I still have: does the problem also occur when you just copy, instead of move, a message to a shared folder?
FYI, Courier doesn't support imap MOVE extension so only supports COPY. So TB has to "simulate" the move by copying to destination, marking the source message as deleted and then expunging (permanently deleting) the source message.
If you could produce an IMAP:4 log (see comment 3) and edit it down to the part after where you see imap COPY command occur to the end. Be sure to try to select the message without the tag in the shared folder before shutting down. The log might show better what the problem is.
Here's my log doing a "move" of a message tagged Important (label = $label1) using a remote Courier server (not mine) that I have a test account on (the time/data and thread info is clipped off):

fwmail.tana.it:S-INBOX:SendData: 87 uid copy 17328 "INBOX.Archives"
fwmail.tana.it:S-INBOX:CreateNewLineFromSocket: 87 OK [COPYUID 711083945 17328 1] COPY completed.
fwmail.tana.it:S-INBOX:SendData: 88 uid store 17328 +FLAGS (\Deleted \Seen)
fwmail.tana.it:S-INBOX:CreateNewLineFromSocket: * 1 FETCH (UID 17328 FLAGS (\Seen \Deleted $label1 nonjunk))
fwmail.tana.it:S-INBOX:CreateNewLineFromSocket: 88 OK STORE completed.
fwmail.tana.it:S-INBOX:SendData: 89 uid expunge 17328
fwmail.tana.it:S-INBOX:CreateNewLineFromSocket: * 1 EXPUNGE
fwmail.tana.it:S-INBOX:CreateNewLineFromSocket: * 7 EXISTS
fwmail.tana.it:S-INBOX:CreateNewLineFromSocket: * 0 RECENT
fwmail.tana.it:S-INBOX:CreateNewLineFromSocket: * FLAGS (nonjunk \Draft \Answered \Flagged \Deleted \Seen \Recent)
fwmail.tana.it:S-INBOX:CreateNewLineFromSocket: * OK [PERMANENTFLAGS (nonjunk * \Draft \Answered \Flagged \Deleted \Seen)] Limited
fwmail.tana.it:S-INBOX:CreateNewLineFromSocket: 89 OK EXPUNGE completed

Then a bit later in the log where I select folder "Archives" and return the flags for the new message and then the header for the new message. Both report that label $label1 is present:

fwmail.tana.it:S-INBOX.Archives:SendData: 81 UID fetch 1:* (FLAGS)
fwmail.tana.it:S-INBOX.Archives:CreateNewLineFromSocket: * 1 FETCH (UID 1 FLAGS (\Seen $label1 nonjunk))
fwmail.tana.it:S-INBOX.Archives:CreateNewLineFromSocket: 81 OK FETCH completed.
[Parent 2598312: Main Thread]: D/IMAP_CS UpdateImapMailboxInfo(): Store highest MODSEQ=0 for folder=INBOX.Archives
fwmail.tana.it:S-INBOX.Archives:SendData: 82 UID fetch 1 (UID RFC822.SIZE FLAGS BODY.PEEK[HEADER.FIELDS (From To Cc Bcc Subject Date Message-ID Priority X-Priority References Newsgroups In-Reply-To Content-Type Reply-To)])
fwmail.tana.it:S-INBOX.Archives:CreateNewLineFromSocket: * 1 FETCH (UID 1 RFC822.SIZE 6694 FLAGS (\Seen $label1 nonjunk) BODY[HEADER.FIELDS ("From" "To" "Cc" "Bcc" "Subject" "Date" "Message-ID" "Priority" "X-Priority" "References" "Newsgroups" "In-Reply-To" "Content-Type" "Reply-To")] {219}
fwmail.tana.it:S-INBOX.Archives:STREAM:OPEN Size: 6694: Begin Message Download Stream

So if you could show me similar info from an IMAP:4 log that you record, we can make sure the problem is Courier not reporting the tag flag (e.g., $label1) in the fetch responses.

Hi Alessandro,
I still have the tana.it test account you provided me several years ago, thank you. I only use it occasionally for TB testing, of course. However, the issue of this bug involves Courier imap server that you may know something about.
The bug reporter is attempting to move a "tagged" message from a normal folder to a "shared" folder on Courier and the tag seems to be removed at the destination shared folder. I don't see a problem when moving between not-shared "normal" folders.
The test account you provided doesn't have any folders in the "shared" or "#shared" namespaces. So I'm wondering if it it's possible to add a shared folder to the account that I can test this with?
Or maybe you know if this issue is valid from you own use of Courier server with "shared" folders?
I've tested the issue with shared folders on Dovecot server and don't see a problem, so doesn't appear to be a TB bug.
Thanks!

Flags: needinfo?(vesely)

Hi Gene,

I can create a shared folder when I'll get to the server, if needed.

I tried and reproduce that bug but couldn't. The tag remains there. However, I have a virtual-user setup of Courier, that is all files and folders are owned by courier:courier, including shared folders. Perhaps the problem with file permissions lays there. IIRC Courier may use a subdirectory to store IMAP flags and a permission mismatch could fail the operation.

@Eduard: in the COMMAND above I see a chown -R popuser.popuser, which makes me wonder how shared folders can be based on filesystem permissions. Can you please revise your setup and check folders' permissions on the server?

Flags: needinfo?(vesely) → needinfo?(eduardp)

Hi,
Sorry, but It toke a while to be able to debug it. I tried to activate thunderbird logging to console (with mailnews.imap.jsmodule=true ) but it does not work for me and emails seem lost in translation (yet they are moved/¿copied+deleted?).
So I ended up debugging with

export MOZ_LOG="IMAP:4,timestamp"
export MOZ_LOG_FILE="/home/eduardp/tmp/log_file"
thunderbird

It works, but the log jumps to 22K lines in less than a minute... as I have hundreds of folders in the IMAP structure.

Ok, I flaged a couple of messages/spam I had in the inbox and moved them, one to a "Not Shared" IMAP folder and another to a "Shared" IMAP Folder.

As I was expecting, the error remains, and I can see the one that goes to a Not Shared folder flagged and the second one not flagged.

Here you have the relevant code I found in the log file.

2024-09-23 07:44:01.938935 UTC - [Parent 1995790: IMAP]: I/IMAP 7f220e70d600:bredax.com:S-INBOX:SendData: 37 uid copy 53167 "INBOX._COSES"
2024-09-23 07:44:01.998261 UTC - [Parent 1995790: IMAP]: I/IMAP 7f220e70d600:bredax.com:S-INBOX:CreateNewLineFromSocket: 37 OK [COPYUID 1229606122 53167 6419] COPY completed.
2024-09-23 07:44:02.001504 UTC - [Parent 1995790: IMAP]: I/IMAP 7f220e70d600:bredax.com:S-INBOX:SendData: 38 uid store 53167 +FLAGS (\Deleted \Seen)
2024-09-23 07:44:02.030022 UTC - [Parent 1995790: IMAP]: I/IMAP 7f220e70d600:bredax.com:S-INBOX:CreateNewLineFromSocket: * 153 FETCH (UID 53167 FLAGS (\Seen \Deleted $label1 NonJunk))
2024-09-23 07:44:02.036139 UTC - [Parent 1995790: IMAP]: I/IMAP 7f220e70d600:bredax.com:S-INBOX:CreateNewLineFromSocket: 38 OK STORE completed.
2024-09-23 07:44:02.036152 UTC - [Parent 1995790: IMAP]: I/IMAP 7f220e70d600:bredax.com:S-INBOX:SendData: 39 uid expunge 53167
2024-09-23 07:44:02.073790 UTC - [Parent 1995790: IMAP]: I/IMAP 7f220e70d600:bredax.com:S-INBOX:CreateNewLineFromSocket: * 153 EXPUNGE
2024-09-23 07:44:02.073807 UTC - [Parent 1995790: IMAP]: I/IMAP 7f220e70d600:bredax.com:S-INBOX:CreateNewLineFromSocket: * 153 EXISTS
2024-09-23 07:44:02.073809 UTC - [Parent 1995790: IMAP]: I/IMAP 7f220e70d600:bredax.com:S-INBOX:CreateNewLineFromSocket: * 0 RECENT
2024-09-23 07:44:02.073811 UTC - [Parent 1995790: IMAP]: I/IMAP 7f220e70d600:bredax.com:S-INBOX:CreateNewLineFromSocket: 39 OK EXPUNGE completed
2024-09-23 07:44:02.073928 UTC - [Parent 1995790: Main Thread]: I/IMAP offline imap url succeeded :imap://eduardp@bredax.com@bredax.com:993/onlinemove>UID>.INBOX>53167>.INBOX._COSES
2024-09-23 07:44:02.075140 UTC - [Parent 1995790: Main Thread]: D/IMAP proposed url = INBOX folder for connection INBOX._COSES.2024 has To Wait = false can run = false
2024-09-23 07:44:02.075156 UTC - [Parent 1995790: Main Thread]: D/IMAP proposed url = INBOX folder for connection INBOX has To Wait = true can run = false
2024-09-23 07:44:02.075159 UTC - [Parent 1995790: Main Thread]: I/IMAP queuing url:imap://eduardp@bredax.com@bredax.com:993/select>.INBOX
2024-09-23 07:44:02.075162 UTC - [Parent 1995790: Main Thread]: I/IMAP considering playing queued url:imap://eduardp@bredax.com@bredax.com:993/select>.INBOX
2024-09-23 07:44:02.075164 UTC - [Parent 1995790: Main Thread]: I/IMAP creating protocol instance to play queued url:imap://eduardp@bredax.com@bredax.com:993/select>.INBOX
2024-09-23 07:44:02.075290 UTC - [Parent 1995790: Main Thread]: D/IMAP proposed url = INBOX folder for connection INBOX._COSES.2024 has To Wait = false can run = false
2024-09-23 07:44:02.075300 UTC - [Parent 1995790: Main Thread]: D/IMAP proposed url = INBOX folder for connection INBOX has To Wait = true can run = false
2024-09-23 07:44:02.075302 UTC - [Parent 1995790: Main Thread]: I/IMAP failed creating protocol instance to play queued url:imap://eduardp@bredax.com@bredax.com:993/select>.INBOX
2024-09-23 07:44:02.076104 UTC - [Parent 1995790: Main Thread]: I/IMAP considering playing queued url:imap://eduardp@bredax.com@bredax.com:993/select>.INBOX
2024-09-23 07:44:02.076108 UTC - [Parent 1995790: Main Thread]: I/IMAP creating protocol instance to play queued url:imap://eduardp@bredax.com@bredax.com:993/select>.INBOX
2024-09-23 07:44:02.076131 UTC - [Parent 1995790: Main Thread]: D/IMAP proposed url = INBOX folder for connection INBOX._COSES.2024 has To Wait = false can run = false
2024-09-23 07:44:02.076141 UTC - [Parent 1995790: Main Thread]: D/IMAP proposed url = INBOX folder for connection INBOX has To Wait = false can run = true
2024-09-23 07:44:02.076144 UTC - [Parent 1995790: Main Thread]: I/IMAP playing queued url:imap://eduardp@bredax.com@bredax.com:993/select>.INBOX
2024-09-23 07:44:02.076236 UTC - [Parent 1995790: Main Thread]: I/IMAP 7f220e70d600:bredax.com:S-INBOX:SetupSinkProxy: got m_imapMailFolderSink
2024-09-23 07:44:02.076314 UTC - [Parent 1995790: IMAP]: I/IMAP 7f220e70d600:bredax.com:S-INBOX:ProcessCurrentURL: entering
2024-09-23 07:44:02.076464 UTC - [Parent 1995790: Main Thread]: I/IMAP 7f220e70d600:bredax.com:S-INBOX:SetupSinkProxy: got m_imapMailFolderSink
2024-09-23 07:44:02.076478 UTC - [Parent 1995790: IMAP]: I/IMAP 7f220e70d600:bredax.com:S-INBOX:ProcessCurrentURL:imap://eduardp%40bredax.com@bredax.com:993/select%3E.INBOX:  = currentUrl
2024-09-23 07:44:02.076545 UTC - [Parent 1995790: IMAP]: I/IMAP 7f220e70d600:bredax.com:S-INBOX:SendData: 40 noop
2024-09-23 07:44:02.107829 UTC - [Parent 1995790: IMAP]: I/IMAP 7f220e70d600:bredax.com:S-INBOX:CreateNewLineFromSocket: 40 OK NOOP completed
2024-09-23 07:44:02.107924 UTC - [Parent 1995790: IMAP]: I/IMAP 7f220e70d600:bredax.com:S-INBOX:SendData: 41 getquotaroot "INBOX"
2024-09-23 07:44:02.136569 UTC - [Parent 1995790: IMAP]: I/IMAP 7f220e70d600:bredax.com:S-INBOX:CreateNewLineFromSocket: * QUOTAROOT "INBOX" "ROOT"
2024-09-23 07:44:02.136619 UTC - [Parent 1995790: IMAP]: I/IMAP 7f220e70d600:bredax.com:S-INBOX:CreateNewLineFromSocket: * QUOTA "ROOT"
2024-09-23 07:44:02.136620 UTC - [Parent 1995790: IMAP]: I/IMAP 7f220e70d600:bredax.com:S-INBOX:CreateNewLineFromSocket: 41 OK GETQUOTAROOT Ok.
2024-09-23 07:44:02.136634 UTC - [Parent 1995790: IMAP]: I/IMAP 7f220e70d600:bredax.com:S-INBOX:SendData: 42 UID fetch 53169:* (FLAGS)
2024-09-23 07:44:02.165197 UTC - [Parent 1995790: IMAP]: I/IMAP 7f220e70d600:bredax.com:S-INBOX:CreateNewLineFromSocket: 42 OK FETCH completed.

and

2024-09-23 07:44:14.957642 UTC - [Parent 1995790: IMAP]: I/IMAP 7f220e70d600:bredax.com:S-INBOX:CreateNewLineFromSocket: 51 OK [COPYUID 1229508979 53158 706] COPY completed.
2024-09-23 07:44:14.970271 UTC - [Parent 1995790: IMAP]: I/IMAP 7f220e70d600:bredax.com:S-INBOX:SendData: 52 uid store 53158 +FLAGS (\Deleted \Seen)
2024-09-23 07:44:14.999223 UTC - [Parent 1995790: IMAP]: I/IMAP 7f220e70d600:bredax.com:S-INBOX:CreateNewLineFromSocket: * 148 FETCH (UID 53158 FLAGS (\Seen \Deleted $label1 NonJunk))
2024-09-23 07:44:15.005242 UTC - [Parent 1995790: Main Thread]: D/IMAP proposed url = shared.ClientsGerard._ANTICS.Frescota.LaRueda folder for connection INBOX._COSES.2024 has To Wait = false can run = false
2024-09-23 07:44:15.005259 UTC - [Parent 1995790: Main Thread]: D/IMAP proposed url = shared.ClientsGerard._ANTICS.Frescota.LaRueda folder for connection INBOX has To Wait = false can run = false
2024-09-23 07:44:15.005807 UTC - [Parent 1995790: Main Thread]: I/IMAP 7f2205979b00:bredax.com:A:SetupSinkProxy: got m_imapMailFolderSink
2024-09-23 07:44:15.005874 UTC - [Parent 1995790: IMAP]: I/IMAP 7f2205979b00:bredax.com:A:ProcessCurrentURL: entering
2024-09-23 07:44:15.005881 UTC - [Parent 1995790: IMAP]: I/IMAP 7f2205979b00:bredax.com:A:ProcessCurrentURL:imap://eduardp%40bredax.com@bredax.com:993/folderstatus%3E.shared.ClientsGerard._ANTICS.Frescota.LaRueda:  = currentUrl
2024-09-23 07:44:15.005960 UTC - [Parent 1995790: IMAP]: I/IMAP 7f220e70d600:bredax.com:S-INBOX:CreateNewLineFromSocket: 52 OK STORE completed.
2024-09-23 07:44:15.005973 UTC - [Parent 1995790: IMAP]: I/IMAP 7f220e70d600:bredax.com:S-INBOX:SendData: 53 check
2024-09-23 07:44:15.006674 UTC - [Parent 1995790: IMAP]: I/IMAP 7f2205979b00:bredax.com:A:SendData: 80 STATUS "shared.ClientsGerard._ANTICS.Frescota.LaRueda" (UIDNEXT MESSAGES UNSEEN RECENT)
2024-09-23 07:44:15.045880 UTC - [Parent 1995790: IMAP]: I/IMAP 7f220e70d600:bredax.com:S-INBOX:CreateNewLineFromSocket: 53 OK CHECK completed
2024-09-23 07:44:15.045891 UTC - [Parent 1995790: IMAP]: I/IMAP 7f2205979b00:bredax.com:A:CreateNewLineFromSocket: * STATUS "shared.ClientsGerard._ANTICS.Frescota.LaRueda" (MESSAGES 3 RECENT 0 UIDNEXT 4 UNSEEN 3)
2024-09-23 07:44:15.045897 UTC - [Parent 1995790: IMAP]: I/IMAP 7f220e70d600:bredax.com:S-INBOX:SendData: 54 uid expunge 53158
2024-09-23 07:44:15.045905 UTC - [Parent 1995790: IMAP]: I/IMAP 7f2205979b00:bredax.com:A:CreateNewLineFromSocket: 80 OK STATUS Completed.
2024-09-23 07:44:15.090002 UTC - [Parent 1995790: IMAP]: I/IMAP 7f220e70d600:bredax.com:S-INBOX:CreateNewLineFromSocket: * 148 EXPUNGE
2024-09-23 07:44:15.090051 UTC - [Parent 1995790: IMAP]: I/IMAP 7f220e70d600:bredax.com:S-INBOX:CreateNewLineFromSocket: * 152 EXISTS
2024-09-23 07:44:15.090056 UTC - [Parent 1995790: IMAP]: I/IMAP 7f220e70d600:bredax.com:S-INBOX:CreateNewLineFromSocket: * 0 RECENT
2024-09-23 07:44:15.090061 UTC - [Parent 1995790: IMAP]: I/IMAP 7f220e70d600:bredax.com:S-INBOX:CreateNewLineFromSocket: 54 OK EXPUNGE completed
2024-09-23 07:44:15.110300 UTC - [Parent 1995790: Main Thread]: I/IMAP offline imap url succeeded :imap://eduardp@bredax.com@bredax.com:993/onlinemove>UID>.INBOX>53158>.shared.Clients.Acatec
2024-09-23 07:44:15.111309 UTC - [Parent 1995790: Main Thread]: D/IMAP proposed url = INBOX folder for connection INBOX._COSES.2024 has To Wait = false can run = false
2024-09-23 07:44:15.111321 UTC - [Parent 1995790: Main Thread]: D/IMAP proposed url = INBOX folder for connection INBOX has To Wait = true can run = false
2024-09-23 07:44:15.111323 UTC - [Parent 1995790: Main Thread]: I/IMAP queuing url:imap://eduardp@bredax.com@bredax.com:993/select>.INBOX
2024-09-23 07:44:15.111324 UTC - [Parent 1995790: Main Thread]: I/IMAP considering playing queued url:imap://eduardp@bredax.com@bredax.com:993/select>.INBOXcompleted.
Flags: needinfo?(eduardp)

Starting thunderbird, going to a shared folder, flaging and leaving... I think this is the relevand log (from 200K lines...)...

IMAP]: I/IMAP 749c8a2faf00:bredax.com:S-shared.Clients.Acatec:SendData: 14 uid store 705 +FLAGS ($label1)
2024-09-23 08:12:30.912031 UTC - [Parent 2004807: IMAP]: I/IMAP 749c8a2faf00:bredax.com:S-shared.Clients.Acatec:CreateNewLineFromSocket: * 705 FETCH (UID 705 FLAGS (\Seen $label1))
2024-09-23 08:12:30.922471 UTC - [Parent 2004807: IMAP]: I/IMAP 749c8a2faf00:bredax.com:S-shared.Clients.Acatec:CreateNewLineFromSocket: 14 OK STORE completed.

In this case, the flagging seems to work, as it remains between thunderbird instances.

From comment 13:

export MOZ_LOG="IMAP:4,timestamp"
export MOZ_LOG_FILE="/home/eduardp/tmp/log_file"
thunderbird

Yes, that is the correct way to record the log.

From comment 13:

As I was expecting, the error remains, and I can see the one that goes to a Not Shared folder flagged and the second one not flagged.

I don't see where the "second one not flagged"? Which log line are you seeing this on?

From comment 14:

In this case, the flagging seems to work, as it remains between thunderbird instances.

Here it looks like you are setting the flag on the message in the shared folder, so I would expect it to stay set.

I think what we need is to select Inbox, shutdown TB and then restart TB while recording the log.
Then choose any tagged message in inbox and copy (no need to move) it to "shared.ClientsGerard._ANTICS.Frescota.LaRueda" or any other shared folder.
Now click on the destination shared folder and see the message list with the just copied message.
Is the tag coloring now not present? No need to open the message.
Shutdown tb.

Open the recorded log and search for
SendData: <a number> SELECT "shared.ClientsGerard._ANTICS.Frescota.LaRueda"

Where <a number> is can be anything (random integer). (I use gvim editor so I search for this with "SendData.*SELECT")

Show me the log from this select command to the end of file. You can attach it as a file using the "Attach new file" button above.

From comment 12 (Alessandro):

@Eduard: in the COMMAND above I see a chown -R popuser.popuser, which makes me wonder how shared folders can be based on filesystem permissions. Can you please revise your setup and check folders' permissions on the server?

I don't know anything about the server log produced by Courier, but maybe there is some indication in that log pointing to a permission problem when the copy/move occurs?
Edit: If you do see an error in the Courier log, probably no need for the IMAP:4 log requested in comment 15.

(In reply to gene smith from comment #16)

From comment 12 (Alessandro):

@Eduard: in the COMMAND above I see a chown -R popuser.popuser, which makes me wonder how shared folders can be based on filesystem permissions. Can you please revise your setup and check folders' permissions on the server?

I don't know anything about the server log produced by Courier, but maybe there is some indication in that log pointing to a permission problem when the copy/move occurs?
Edit: If you do see an error in the Courier log, probably no need for the IMAP:4 log requested in comment 15.

Cannot see anything related to an error in /var/log/maillog or /var/log/mail.err (lots of authentication errors but none related to my IMAP user, and no error moving emails around).
By the while... I had to use chown because maildirmake is called from cmd by an admin user and there were problems then with courier and permissions. I checked and, after so many years and server switching, as long as I see it, popuser keeps being Courier MTU user in plesk. I created some local folders from thunderbird and they get created on the server as dirs with popuser.popuser owner.

Attached file log_file.moz_log —
> Show me the log from this select command to the end of file. You can attach it as a file using the "Attach new file" button above.

Show me the log from this select command to the end of file. You can attach it as a file using the "Attach new file" button above.

I think I attached it yet i am not sure...

For what I see, courier mtu saves the flags in a dir named courierimapkeywords.
In the case of moving to shared folders, this file is not created.
Yet, when you flag the files AFTERWARDS the file gets created.
So I bet an "easy" hack would be to move the email to the folder and THEN, in case the server is Courier and the folder is Shared, set the flags to the newly created message.
I understand that is easier said that done, simply because I don't even know if the moving call gives you back the new email id, so you can add the flags to that email.

Courier is quite tight on logging errors. It too can log the whole IMAP dialog, but I don't think that would reveal more than TB logs.

Courier uses a courierimapkeywords directory inside the Maildir home to store a :list of permanent flags, and temporary files for quickly manage flags of specific messages. These files are created in the tmp directory and then moved in place. Permission problems on either of those directories would fail the operation, of course. However, I don't understand why then they later succeed, unless TB logs in as a different user when accessing the shared folder.

Something like find .ClientsGerard._ANTICS.Frescota.LaRueda -type d -ls would show permissions, if shell access to the server is allowed.

(In reply to eduardp@bredax.com from comment #19)

Show me the log from this select command to the end of file. You can attach it as a file using the "Attach new file" button above.

I think I attached it yet i am not sure...

Yes, it's attached. Thanks!
It shows that the server is losing the $label1 flag. It only has the \recent flag (meaning it's a new message in folder):
S-shared.Clients.Acatec:CreateNewLineFromSocket: * 703 FETCH (UID 707 RFC822.SIZE 436 FLAGS (\Recent) ...

Yet, when you flag the files AFTERWARDS the file gets created.
So I bet an "easy" hack would be to move the email to the folder and THEN, in case the server is Courier and the folder is Shared, set the flags to the newly created message.
I understand that is easier said that done, simply because I don't even know if the moving call gives you back the new email id, so you can add the flags to that email.

Yes, probably not so easy. Also, putting in code specific to Courier probably would never be approved.

Alessandro wrote:

However, I don't understand why then they later succeed, unless TB logs in as a different user when accessing the shared folder.

TB just just logs into the server with the username put into the Server settings config screen. Always uses the same name and password.

Yes, probably not so easy. Also, putting in code specific to Courier probably would never be approved.
If you can see that the flag is lost, then, I assume that means you can track the new created message.
If that is the case, then I imagine you can force back the flags without having to make it specific to Courier.

$emailFlags=getFlagsFromEmail($email);
$email2=copyEmailToFolder($email,$folder);
deleteEmail($email);
$email2Flags=getFlagsFromEmail($email2);
if ($emailFlags<>$email2Flags) setFlagsToEmail($email2, $emailFlags):

(Sorry for the ugly stupid code, just an example of the idea.

Alessandro wrote:

However, I don't understand why then they later succeed, unless TB logs in as a different user when accessing the shared folder.

TB just just logs into the server with the username put into the Server settings config screen. Always uses the same name and password.

Right, but one can have more accounts and a folder should be shared among them.

This bug-or-misconfig should be reported on Courier mailing list (or on GitHub, albeit with possibly less outcome.)

Right, but one can have more accounts and a folder should be shared among them.

So I guess a question for Eduard would be can the flag be manually set by all users sharing the folder?

Sorry for the ugly stupid code, just an example of the idea.

Ok, thanks. Hopefully this won't be necessary if you can find the problem in Courier setup.
But, actually, there is some code in TB that is specific to Courier. It was put in years ago to work around some problems that are probably no longer relevant.

Every user can set its flags. The way it works (at folder level, for what I see)...
Every user that shares a folder has its own copy of the folder.

/var/qmail/mailnames/<DOMAIN>/<USER>/Maildir/shared-folders/<SHARED_FOLDERS_GROUP_NAME>/<SHARED_FOLDER>

in this folder, everything belongs to the user, except shared that is a soft link to

/var/qmail/mailnames/<DOMAIN>/<SHARED_FOLDERS_GROUP_NAME>//.<SHARED_FOLDER>

and then in 'new' 'cur' 'tmp' dirs, instead of holding the emails they contain soft links to them.

lrwxrwxrwx 1 popuser popuser  144 Sep 24 13:36 '172<SOME_NUMBERS_A>51.M94<SOME_NUMBERS_B>126V0000000000000902I0000000002<SOME_NUMBERS_C>4_0.<DOMAIN>,S=447:2,S' -> '/var/qmail/mailnames/<DOMAIN>/Clients/Maildir//.Acatec/cur/172<SOME_NUMBERS_A>51.M94<SOME_NUMBERS_B>126V0000000000000902I0000000002<SOME_NUMBERS_C>4_0.<DOMAIN>,S=447:2,S

You have to ACTIVELY subscribe to a folder to be able to read its messages.

Now... I don't know if Courier MTU updates all the user subscription folders every time something is added up to the shared folder OR (as I assume) it syncs the subscription ONLY WHEN THE USER REFRESHES THE SUBSCRIBED FOLDER.

I would guess the subscription is only refreshed once the user asks for it. That would explain why the label is not moved when copied.

I mean... when one user move the flagged email to a shared folder, it can move the email, but it CANNOT move the flags BECAUSE the flags belong to the user AND the user EMAIL STRUCTURE has not been created yet, as it will be ONCE THE NEXT EMAIL CLIENT ASKS Courier MTU to refresh that shared folder.

If that is the case, I see no other way to solve the problem but to save the flags, move the email, REFRESH the folder, and THEN copy the flags to the message.

Now, I guess that has to be done by TB as Courier MTU might not able to solve this problem.
Not sure Courier MTU shared folder is not aware of who are the users that are subscribed to it. So, once an email is added to that folder, it cannot create user side flags because It does not know who the users are at a given moment.

I mean... when one user move the flagged email to a shared folder, it can move the email, but it CANNOT move the flags BECAUSE the flags belong to the user AND the user EMAIL STRUCTURE has not been created yet, as it will be ONCE THE NEXT EMAIL CLIENT ASKS Courier MTU to refresh that shared folder.

Courier's shared folders management is complicated enough for me to avoid trying to understand it.

Please, report this issue on the Courier mailing list.

If that works the way I think it works, Courier cannot make anything to solve the problem, as the structure to hold the flags has not been created and will not be until that user refreshes the list of messages in that same folder. So, if TB don't refresh the list and then sets the flags, I don't think Courier can do anything.
Anyway... I sent a message to the list. Thanks a lot for your time.

I'd be surprised to learn that Courier loses keywords by design...

Please resend the message to the list, after three hours it's not arrived yet...

I sent it to courier-imap@lists.sourceforge.net and I got a reply just now, so someone got the message.
I don't know if you think I must subscribe to another list.
You tell me.
Thanks.

I sent it to courier-imap@lists.sourceforge.net

Oh, I see; sorry for misunderstanding. I'm not subscribed to that list, so I read Sam's response in the web archive. It confirms that the folder has to be refreshed. Let me quote a paragraph, where Sam blames the parallel connection:

If the most recent EXISTS message told the IMAP client that there are 10
messages in the folder, there are still ten messages in this folder, even if
another IMAP connection COPYed a new message into this folder. There are
going to be just 10 messages in the folder until it gets another EXISTS (or
until it gets an EXPUNGE but that's immaterial for the purpose of this
discussion). That's just how IMAP works.

Yet, in virtual-user mode, it works...

As you can understand, once the problem is detected, I cannot do anything more to solve it.

I am with Sam in this case. I think TB should refresh (always?) the destination folder AFTER copying, to make sure the user side of the message structure exists, and BEFORE it asks the IMAP Server to change flags or state to that message.

Yet I cannot make anything. I am on your hands.

Anyway... one last thing regarding to this problem....
... not only FLAGS are not updated BUT the ALREADY READ status is not updated neither so every message that is sent to a shared folder get market as TO BE READ regardless of the status of the message or if the user moved the message manually after opening it.
Very annoying.

Sorry for delay in getting back to this. Was offline some due to storms.
FWIW, I tried to "need info" Sam V. on this bug but his BMO account is disabled.
Seems like his main point is that maybe TB is accessing the new UID of the copied message in the destination folder before getting a new untagged EXISTS count. I don't think TB is doing that but I'll look into it.

I think TB should refresh (always?) the destination folder AFTER copying, to make sure the user side of the message structure exists, and BEFORE it asks the IMAP Server to change flags or state to that message.

Not sure I understand what you (or Sam) means by "refresh" the destination folder. Currently TB assumes the flags, tags, keywords or whatever are copied along with message into the destination shared folder. So there should be no reason for TB to have to "change flags or state" of the message in the destination folder (like I think you suggest in comment 23).

Eduard,
Could you try this experiment?:

Before moving or copying the message to the shared folder, first select the shared folder (click on it). This will cause the folder to become imap SELECTed. Since Courier supports imap IDLE and if you have the server setting "Allow immediate server notifications when new messages arrive" the server should send a * <number of messages> EXISTS to TB informing that the count of messages has changed as soon as you do the move/copy. Then go back to your source folder and do the move/copy. Does this cause the flags to be preserved at the shared destination folder?

I have my doubts that this will make a difference, but it might.

Finally, I don't see TB accessing the newly copied messages until after an EXISTS response occur when the shared destination folder is imap SELECTed. TB doesn't make use of the destination folder UID returned in the COPYUID response.

(In reply to gene smith from comment #34)

I have my doubts that this will make a difference, but it might.
No difference (hope I got your instructions right): I read the folder, immediately after i go to Inbox, move a flagged message to the folder and refresh the folder again. The message appears, unread and with no flags.

Finally, I don't see TB accessing the newly copied messages until after an EXISTS response occur when the shared destination folder is imap SELECTed. TB doesn't make use of the destination folder UID returned in the COPYUID response.

I don't have any idea on the code, not Courier's not TB's, but I would bet that:
a) Courier's Shared folders (by folders) implementation do split clearly the "user view" and the "shared view", so much that "shared" side actions do not know anything about the user or the user structure.
b) In this sense, when you ask for an email to be copied to a folder, it takes the email from the user and cp's it to the SHARED folder. Job done. It does not care WHICH user has asked for the cp and what flags that user has. I bet, in fact, that that side of the code is not able to create the user's structure on that folder (different libraries of functions, maybe?).
c) Now... I bet that shared emails remain in the Shared folder BUT the part of the message structure that belongs to the user (flags, read/not read,...) has not been created.
d) Once the user refreshes (EXIST?) its folder to see its content, a process looks into the Shared folder, looking for messages that have been copied to the folder but are not included in the user structure and for everyone of them it creates the User specific structure. In fact, It has to do it this way anyway, because it maybe that a new user is subscribed to an existing folder and the User side of the messages has to be recreated long after the messages have been moved to the folder. So... why make things twice?

So, again, If I am right, there are two ways to solve the problem:

a) Make CourierMTA copy process "User aware", so it copies the message, then, internally, call for the refresh/Exists process, and then set the flags of that process.
b) Make TB change a little bit its behaviour and after copying the message, refresh/EXISTS the destination folder and apply the flags to the message.

I understand the second option is easier to do, and less impacting if an error occurs in the new code, so I would bet for doing the second option.

My understanding was based on the general description of the issue: that something wasn't working right when TB, presumably, copied message flags to the message that was copied to another folder; and skimming the logs that's what stood out to me.

I also want to point out that RFC 3501 does not require any particular behavior with respect to flags of a copied message. Preservation of message flags is a SHOULD, and not a MUST:

https://www.rfc-editor.org/rfc/rfc3501.html#page-59

  The COPY command copies the specified message(s) to the end of the
  specified destination mailbox.  The flags and internal date of the
  message(s) SHOULD be preserved, and the Recent flag SHOULD be set,
  in the copy.

If it's imperative for message flags to be preserved, then this needs to be done separately, and that's what I thought was happening: that TB was trying to set the message flags after COPYing the message to another folder, but something wasn't working right.

P.S. Bugzilla disabled my account supposedly because it thought that my mail server was bouncing its mail. The presented documentation was an error message from some AWS server that I never heard of, and which has nothing to do with my domain. I dutifully sent an inquiry to the mods, I haven't heard anything back.

Sam V.,
Thanks for the response. Yes, the rfc 3501 does say that and the updated/proposed imap rfc 9051 says the same thing.
So is it correct that Courier preserves the flags when copying between private folders but not when copying from private to shared (using Eduard's setup)?
Also, from Alessandro's comment 31 he says

Yet, in virtual-user mode, it works...

meaning flags are preserved when set-up like that too.

Note: I haven't attempted to setup a local Courier-mta to test this so exactly how both Eduard and Alessandro have setup Courier is not clear to me. However, I do have a locally setup Dovecot (just for testing TB) and it does preserve the imap flags/tags/keywords when copying from private to private, shared and public folders.

The shared folders in question are folders shared between different system accounts, different system userids. This is done by making them publicly readable. Sharing is based on ordinary filesystem permissions. By carefully setting up each account's userid and groupid it's also possible to implement some measure of control, I suppose, based on group permissions, instead of global permissions, and control access to shared folders that way.

In this situation, other accounts have access to the messages, but they obviously don't write write access to the mailbox. That would require a globally writable mailbox, which, obviously, is a non-starter proposition.

This is the reason why these shared folders work that way. The messages themselves are publicly readable, but each subscriber to the shared folder then has to maintain their own metadata, for each message, and sync their own copy of the shared folder. Each subscriber to the shared folder maintains their own, individual flags for each message in the shared folder.

It's a pain, I will admit. But given that:

  • you have different mailboxes that are owned by a different system userid and groupid

  • you want to have some mechanism for sharing folders between them, that's functional and works with POSIX filesystem permissions

That's how I ended up making that work.

With virtual folders, on the other hand, they all use the same userid and groupid, and POSIX filesystem permissions are no longer a factor.

Now you have IMAP ACLs that control each individual mailbox's access to a shared folder.

Everyone is just one big happy family, so now each folder is no different than any other folder. They all can read and write a single set of metadata for each message, in each folder.

Hi Sam,
Thanks for your explanation.
Yet, there is a thing...
When one user with its userid moves (copies) through IMAP a message from one other folder to a shared one in this setup... I get that It cannot act in name of all other users because it cannot represent them... but as I understand... he should be able to be able to create its own flags, doesn't he?

I mean... there is a way for him to create the user side structure for himself, even when he cannot create others, but at least he should be able to create his own structure.

In this case it is just what I need (and I think it is what should be done). If I move a message to a folder, flagged as Important and Read, it is Important and Read for me, not for the others that read into the folder. So It is enough that the user can move the flags for himself. He don't need to recreate the flags for others...

Am I right? Does it make sense? Is it possible to be changed?

Copying messages in folders in sharable maildirs is no different than a regular maildir. Are you saying that the sharable maildir is set up with itself as one of the shared maildirs? You've copied a message into a regular folder, then go and look at that folder in the shared namespace?

No special behavior is done for a shared folder that comes from the same sharable maildir. It is handled like a shared folder from any other shared maildir. It's true that this situation would allow for some additional flexibility, but Courier expects no more, and no less, from one shared folder than any other shared folder. I have no immediate plans to do any additional work, in this area.

You should be able to see the original flags in the original folder that's sharable. If that's not the case, this is worth investigating, but this shouldn't have anything to do with the fact that the folder is in a sharable maildir.

Or you may wish to look into reconfiguring your system with virtual mail accounts, and virtual shared folders. A brief search finds a Thundebird add-on for IMAP ACLs, which sounds promising.

Hi Sam,
I am here just to report an error I found as a User of Thunderbird against Courier MTA. Nothing more. Given that I am a computer's scientist myself, I try to help determining what is causing the problem, but I work on LAMP so it is far away from my field.
Once determined the problem and given that it has two solutions possible:
a) making TB refresh the folder before copying the flags again
OR
b) making Courier MTA copy the flags to the user's side of the shared folder on COPY from given User.

I understand that no one of the projects will lead on the solution and the error will keep unresolved.

I know I could move to ACLs. In fact... I think I am the only user of most of those shared folders! But that is not the point.

I think moving to the "virtual" setup would be the best solution. If this requires adjusting the imap ACL settings for each shared folder it's not too hard just using telnet or openssl to connect to the server and do imap command SETACL. So no addon required. I had to do this when I was testing shared folders with dovecot which does preserve the flags after copy to a shared destination mailbox.

I'm reluctant to add the re-copy/refresh of flags to TB, even as an optional activity. Not sure that at the protocol level the code even knows when the copy involves "shared" mailboxes. (Maybe could look at the "prefix" name.) It would also add at least an extra UID STORE imap operation after each copy or move action, so more processing and network activity.

I'm not clear whether Courier does an attempt at copying flags and fails because of permissions, or just skips it because it won't work. In the former case, perhaps it can be fixed setting ACLs, without moving to a virtual setup.

In the virtual setup, everything works fine without tinkering with ACLs.

Filesystem permission-based shared folders in Courier effectively share just the messages, and each subscriber to a sharable folder, itself, syncs the messages and maintains its own metadata for each message. Filesystem permissions govern who has, or does not have, access to each shared folder.

URL: https://www.courier-mta.org/imap/READ...
Status: UNCONFIRMED → RESOLVED
Closed: 5 months ago
OS: Linux → All
Hardware: x86_64 → All
Resolution: --- → INVALID
Summary: IMAP: messages are unlabeled/untagged when moved from one folder to another → Messages are untagged when moved from a folder to a shared folder using Courier IMAP server with filesystem permissions-based sharing.
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: