2009013 - EWS account does not synchronize inbox with external changes (emails that have been deleted via IMAP still show up)

Status: RESOLVED FIXED

(MailNews Core :: Networking: Exchange, defect)

Description

[Orion Poplawski]

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:140.0) Gecko/20100101 Firefox/140.0

Steps to reproduce:

I'm testing out a EWS connection for my MS365 account alongside my current IMAP configuration. There are a number of emails that have been deleted via the IMAP account that still show up in the inbox message list in the EWS despite many syncs/days passing.

This is with 148.0a1 (2026-01-07) (64-bit)

Actual results:

Clicking on one of the deleted message results in a blank message window, but no other error and it is still present in the list.

Expected results:

Messages read in IMAP should start to show as read in EWS. Messages deleted in IMAP should get removed from the inbox list.

Comment 1

[Orion Poplawski]

I did a repair folder operation and that appears to have helped. I'll keep an eye on it today and see if it does manage to sync up again eventually.

Comment 2

[Justin Tracey [:jtracey]]

Do you see any warnings or errors in the latest nightly?

Comment 3

[Orion Poplawski]

So after running the folder repair things seem to be syncing well now. So not sure what triggered it to get into the bad state initially.

Starting up today I did see:

[ERROR ews_xpcom::safe_xpcom] an error occurred when performing operation SyncFolderItems: Ews(Deserialize(Error { path: Path { segments: [Map { key: "Body" }, Map { key: "SyncFolderItemsResponse" }, Map { key: "ResponseMessages" }, Map { key: "$value" }, Seq { index: 0 }, Map { key: "Changes" }, Map { key: "$value" }, Seq { index: 33 }, Enum { variant: "Create" }, Map { key: "$value" }] }, original: Custom("unknown variant Contact, expected one of CalendarItem, Message, MeetingMessage, MeetingRequest, MeetingResponse, MeetingCancellation") }))

Comment 4

[Justin Tracey [:jtracey]]

Hm, looks like we missed some variants of item type (in this case, Contact). I could see that causing issues if those were in the update. Though I also don't know what it means for a folder to have created a contact, we should at least add the missing variants so it doesn't fail to parse.

Comment 5

[Justin Tracey [:jtracey]]

Attachment: Bug 2009013 - Pull latest fixes from ews-rs. r=#thunderbird-reviewers

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/51c40a723700
Pull latest fixes from ews-rs. r=babolivier

Comment 7

[Orion Poplawski]

So I hadn't done anything with the EWS / thunderbird daily account for a while. Today I downloaded a new daily and fired it up. It seemed to eventually mostly sync up, but there is still one marked unread message from Jan 13th that shows up and is not present in the IMAP account. When I click on it I get:

[ERROR protocol_shared::safe_xpcom] an error occurred when performing operation GetItem: ResponseError(ResponseError { message_text: "The specified object was not found in the store., The process failed to get the correct properties.", response_code: ErrorItemNotFound, message_xml: None })
[ERROR protocol_shared::safe_xpcom] an error occurred when performing operation change read status: Processing { message: "response contained 1 errors; the first error (at index 0) was: ResponseError(ResponseError { message_text: "The specified object was not found in the store., The process failed to get the correct properties.", response_code: ErrorItemNotFound, message_xml: None })" }

but it does not go away.

Total message counts don't line up either - 2955 in EWS, 2934 in IMAP. Perhaps it only goes back a short period for syncing?

Comment 8

[Eleanor Dicharry]

That is the expected server response if TB tries to fetch something from Exchange and it has been deleted, so the issue here is why the deletion is not getting picked up in the folder sync. I'll look into this more.

Comment 10

[Eleanor Dicharry]

I managed to reproduce this.

STR:

1. Connect to an Exchange account from two different Thunderbird profiles, one using EWS and one using IMAP.
2. Hard delete (shift-delete) a message from the IMAP instance.
3. Resync the EWS instance and note that the deleted message stays.

This appears only to be the case when using the account concurrently over EWS and IMAP. Hard delete from the web interface behaves as I would expect in TB (message disappears on next sync).

Dominik and Orion - Can you confirm this is what you're seeing? Specifically that it only occurs on hard delete and only when you delete from an IMAP-connected instance of Thunderbird.

Comment 11

[Eleanor Dicharry]

... Actually this is not a bug in the EWS implementation, it's a bug in the IMAP implementation. Hard-delete is not making messages disappear from OWA either.

Comment 12

[Orion Poplawski]

I certainly do a lot of hard-deletes, and I'm using IMAP for my other access, so that checks out.

Comment 13

[Dominik Seelow]

Well, when I delete a message with Thunderbird/IMAP, it is no longer shown in Outlook/EWS. So I would rather not think that it depends on the IMAP implementation, but I'm not familiar with either protocol.

Test results:
147:
deleted with Outlook/EWS -> message still in the list, but no longer accessible
deleted with Thunderbird/MAP -> message still in the list, but no longer accessible

149.0a1:
deleted with Outlook/EWS -> message gone
deleted with Thunderbird/MAP -> message gone

Seems to have been fixed - thank you!

Comment 14

[Ben Campbell]

(This may be covered by the "Hard Delete" comments above, and I'm probably not adding anything you don't already know, but hey. I don't know if shift-delete in TB does an expunge or not...)

Does it still happen if you expunge the folder after deleting the message? (In thunderbird, expunge is annoyingly tied to "compact folder", so compact the folder after deleting the message).

IMAP just adds the \Deleted flag, but the message isn't actually removed on the server until the expunge.

I'm guessing delete on an exchange server is an actual hard delete. So I could easily imagine that accessing exchange via EWS just ignores the IMAP \Deleted flag (however that's represented on the server! e.g. maybe it's just stored as part of the state of the IMAP gateway, and that the main exchange database doesn't ever even know about it!).

I'm guessing web mail just does a direct hard delete, without going through IMAP at all.

Comment 15

[Dominik Seelow]

Does it still happen if you expunge the folder after deleting the message? (In thunderbird, expunge is annoyingly tied to "compact folder", so compact the folder after deleting the message).

There is no need to compact the IMAP folder - Outlook/EWS will notice the deletion immediately. But this is happening on the mail server and I don't have the faintest idea how it is configured.

Comment 16

[Ben Bucksch (:BenB)]

(In reply to Justin Tracey [:jtracey] from comment #4)

Hm, looks like we missed some variants of item type (in this case, Contact). I could see that causing issues if those were in the update. Though I also don't know what it means for a folder to have created a contact, we should at least add the missing variants so it doesn't fail to parse.

In Exchange, everything is a folder, not only mail folders. A calendar is a "folder" of calendar events, and an addressbook is a "folder" of contacts, etc.
EWS sync include changes for the entire account, not only all mail folders, but also calendar, contacts and all other data types. This is why you see a Contact in a "folder" during the sync. If you bail on unknown types, the sync will stop at this point, causing this bug. So, you hit the nail on its head.

It is critical to not bail on unknown data types, but to ignore them.

Reproduction

From what you describe, a likely reproduction of the bug would be to change a contact in your personal addressbook in OWA, then sync the account in Thunderbird.

Comment 17

[Ben Bucksch (:BenB)]

(In reply to Ben Campbell from comment #14)

IMAP just adds the \Deleted flag, but the message isn't actually removed on the server until the expunge.
I could easily imagine that accessing exchange via EWS just ignores the IMAP \Deleted flag

If this was the reason, then the user wouldn't get "The specified object was not found in the store". If you get this error, that means the email was indeed deleted on the server. But you still have it in the message list, so when the user clicks on it and you try to download it, or you try to update its flags, you'd get this error.

So, the problem is in the EWS sync of deletions. It may be a fallout of the problem I explained in my last comment, or may be a separate bug.

Comment 18

[Eleanor Dicharry]

I think we're conflating two different issues here:

1. The original issue, and Dominik's issue, in which an unknown message type blocks updates on folders. This was fixed by the patch attached to this issue.
2. An additional issue discovered once the first issue was addressed, in which items that are hard-deleted by an IMAP connection to the exchange account don't immediately delete in an EWS connection (or in OWA).

I came back to test this today and here's what I found:

1. When I create a new profile and connect to the Exchange account with IMAP, the previously "deleted" message shows back up.
2. I hard delete it again, and it again disappears from the IMAP-connected Thunderbird.
3. It remains in the EWS-connected thunderbird and in OWA.
4. I can still access the message content from the store. (This behavior differs from what Orion observed)
5. Running a compact folder from the imap connected instances causes the delete to propagate to EWS and OWA.

I think Ben C is saying at least this is by design in the IMAP implementation: We only set the /Deleted flag. Since neither EWS or OWA are using that IMAP flag, the message still shows. It is actually deleted once we expunge (by compacting the folder).

I'm going to close this issue and open a separate issue to track what we want to do when an item is deleted via IMAP. Also, I understand that the messages Orion received about the message not being in the store are not consistent with this behavior, so there is probably something else going on there.

Comment 19

[Eleanor Dicharry]

Actually, I went to file that issue, and it appears the IMAP behavior is as-designed. We've had a number of similar issues reported in the past about what Thunderbird is showing for IMAP-deleted messages vs. what other clients are showing, so I'm not sure if I should open another issue for that one.

Comment 20

[Magnus Melin [:mkmelin]]

For hard-delete, on IMAP we're supposed to purge the message after. See https://searchfox.org/comm-central/search?q=ExpungeAfterDelet&path=

Comment 21

[Eleanor Dicharry]

It looks like mail.imap.expunge_after_delete defaults to false. Should that default to true?

Comment 22

[Magnus Melin [:mkmelin]]

Probably not. Especially if the server doesn't support UIDPLUS I'd expect people not to like that.

Comment 23

[Eleanor Dicharry]

OK, so I think with the delete message behavior, this is all as-designed still.

Comment 24

[Eleanor Dicharry]

I set mail.imap.expunge_after_delete to true and hard-deleted a message, and the message still doesn't disappear from OWA until I manually expunge, so I think there is an issue with the IMAP implementation here. It doesn't appear that there's any way to hard delete a message without following the delete operation with a folder compaction.

Comment 25

[gene smith]

I tested on a dovecot server with UIDPLUS (allows expunges on individual messages) and per IMAP log, the shift-deleted message is getting marked deleted and then expunged. Does the EWS server support UIDPLUS for imap?

If UIDPLUS not supported (all popular imap servers support it from what I have seen) tb should expunge the folder so manual compact not needed. But i haven't tested this.

Comment 26

[Eleanor Dicharry]

Exchange's IMAP implementation is supposed to support UIDPLUS, but information on the Internet is all over the place about this. I'll turn on IMAP logging against our M365 account to see if it's being reported as supported.

Comment 27

[Eleanor Dicharry]

It looks like UIDPLUS should be supported:

[Parent 37649: IMAP]: I/IMAP 14aeb6800:outlook.office365.com:A:SendData: 35 capability
[Parent 37649: IMAP]: V/IMAP ReadNextLine [rv=0x0 stream=168c90220 nb=211 needmore=0]
[Parent 37649: IMAP]: I/IMAP 14aeb6800:outlook.office365.com:A:CreateNewLineFromSocket: * CAPABILITY IMAP4 IMAP4rev1 AUTH=PLAIN AUTH=XOAUTH2 SASL-IR UIDPLUS MOVE ID UNSELECT CLIENTACCESSRULES CLIENTNETWORKPRESENCELOCATION BACKENDAUTHENTICATE BACKENDAUTHENTICATE-IR CHILDREN IDLE NAMESPACE LITERAL+
[Parent 37649: IMAP]: V/IMAP SetConnectionStatus(0x0)
[Parent 37649: IMAP]: V/IMAP ReadNextLine [rv=0x0 stream=168c90220 nb=29 needmore=0]
[Parent 37649: IMAP]: I/IMAP 14aeb6800:outlook.office365.com:A:CreateNewLineFromSocket: 35 OK CAPABILITY completed.

So either something is going wrong in TB specifically when we're talking to Exchange over IMAP, or Exchange's implementation of UIDPLUS is different than we are expecting.

Comment 28

[gene smith]

When you shift-del you should see in log a "uid store \deleted" followed by a "uid expunge <message UID>" with expunge_after_delete set true. With it false you will only see a "uid store \deleted".
FWIW, logging with IMAP:4 gives good info.

Comment 29

[Eleanor Dicharry]

I see all of that, and weirdly it just started working. Messages are now disappearing from OWA. AFAIK, the only thing I changed between when it wasn't working and now was starting with logging enabled.

Comment 30

[Eleanor Dicharry]

...oooh unless...is a restart or reconnect required after setting expunge_after_delete?

Comment 31

[gene smith]

(In reply to Eleanor Dicharry from comment #30)

...oooh unless...is a restart or reconnect required after setting expunge_after_delete?

Maybe. I usually restart when I change a "hidden pref" just to be sure it's seen.
Edit: Actually, yes restart is needed for expunge_after_delete. The pref is obtained only once at start of the thread.