2010331 - Allow more than 1 concurrent runner in the EWS operation queue

Status: RESOLVED FIXED

(MailNews Core :: Networking: Exchange, task, P1)

Description

[Brendan Abolivier [:babolivier]]

The EWS operation queue currently only allows one runner, meaning each request is blocked behind the previous one. This isn't ideal since it makes the traffic slower than it could be.

With more than one runner, we'd want to avoid a case where several (or all) of the runners are retrying a failed authentication or waiting after being rate-limited. It should be possible to use channels to allow a runner to signal to others that it's doing one of these and other runners should wait till it's done before moving ahead with their next request.

Comment 1

[Chris Lamb]

Good day all - thanks for fixing my previous issue (1998978), now that it's downloaded my folders it seems to find everything except the messages in Inbox or Inbox sub-folders. I can see it looking for new messages in a random set of my folders and then gets to 'Looking for new messages in Inbox' and nothing seems to happen. I deleted my other 2 accounts so they weren't tying up resources. I've waited about 15 minutes etc.

I tried: C:\Users\clamb> .\thunderbird.exe -MOZ_LOG=ews_xpcom::*:4 -MOZ_LOG_FILE=tb_ews.log -P but I can't find any logs.

Running the daily 149.0a1 (2026-01-21)

I don't think it's just an issue related to this bug because it has found thousands of other messages, including 3300 in one folder from last year. It seems to be an issue with Inbox specifically. There should be about 500 messages there, I've been on the road all week and haven't clear it out yet.

Comment 2

[Orion Poplawski]

Chris - you might want to check out bug #2009013 - I've had issues with inbox synchronization as well. I've found that doing a "repair folder" operation helps.

Comment 3

[Chris Lamb]

That didn't seem to help, maybe I was impatient. Also repaired all the sub folders and nothing loaded.

So, restarted Thunderbird and Inbox was gone.

So, deleted the account and re-created it... and finally seems to be working! I'll use it tonight and we'll see what happens.

I suspect there's still a bug out there and it will come back once there are massive number of users, but it's resolved at least for now.

Thanks!
Chris

Comment 4

[Chris Lamb]

Initial impression, it almost works well.
If reading and deleting messages and it goes and tries to load anything other than the one you're currently reading you can't necessarily move on and you definitely can't delete the message you just read. It may take a minute or two to finish the other things it was doing and get back to me, that's pretty impractical. Maybe once it's done sync'ing it will be usable, but it means fetching new messages or syncing old ones may cause to respond sporadically.

I had to go back to the Outlook web as it just stopped responding to what I was doing.

Comment 5

[Chris Lamb]

It's still really struggling to sync messages and is notifying me of Auth errors while offline, that seems like a disconnect.

I have set it to Offline a few times and used the Download/Sync now to try to force all messages to be downloaded. I went away for over an hour and it had gone offline but clearly not downloaded all messages. If I prompt it to go offline, it is now only scanning inbox, not syncing other folders. If I pick a random folder, it clearly hasn't downloaded all messages.

So, single threaded makes it nearly impossible to use if much is going on and the Offline/Sync feature is not syncing all folders.

Comment 6

[Brendan Abolivier [:babolivier]]

Sorry to hear you're having troubles with using your Exchange account in Thunderbird Chris!

Just a heads up, I would recommend creating new bugs for the kind of issue you initially reported. This bug is tracking work for a specific upcoming internal performance improvement rather than any specific problem with the current queuing system (or Exchange support in Thunderbird in general). Opening a new bug helps getting new issues to the attention of the team so they can be investigated and addressed.

Comment 7

[Chris Lamb]

(In reply to Brendan Abolivier [:babolivier] from comment #6)

Sorry to hear you're having troubles with using your Exchange account in Thunderbird Chris!

Just a heads up, I would recommend creating new bugs for the kind of issue you initially reported. This bug is tracking work for a specific upcoming internal performance improvement rather than any specific problem with the current queuing system (or Exchange support in Thunderbird in general). Opening a new bug helps getting new issues to the attention of the team so they can be investigated and addressed.

Understood, I think many of the issues are artifacts of this issue. I will open specific bugs as it becomes clearer what they are. It's currently unusable as is, it's completely indeterminate when new messages will be loaded, even if the headers are shown, it appears they are not downloaded because it's off doing something else.

Comment 8

[Eleanor Dicharry]

Brendan - I mentioned in another unrelated issue that this might be the cause of some other behavior some users were seeing. I think as we're hitting larger mailboxes with lots of history, we have a lot of contention for the single connection as messages are being downloaded in the background and users are attempting to interact with more recent messages. Since we're not prioritizing, I think this is manifesting as operations looking like they're not completing when really they're just waiting on the limited connection resource. I'm going to bump up the priority of this one and I think we should move on to it after we finish the current round of in progress issues. No need to interrupt, but I think this is a higher priority than we originally thought.

Comment 9

[Brendan Abolivier [:babolivier]]

Makes sense - I agree we need to get to it sooner than later.

I think it's also worth noting another avenue for improving this would be batching - we query each folder and message in a separate request where the API would allow us to do several at one.

Comment 10

[Brendan Abolivier [:babolivier]]

Attachment: Bug 2010331 - Synchronize error handling between EWS queue runners. r=#thunderbird-back-end-reviewers

Comment 11

[Chris Lamb]

Good to see some progress. The latest update or two have fixed a few problems it seems (unable to save sent messages for example). However, I'm not getting new messages w/o restarting. Should I open a new bug report?

Comment 12

[Kai Engert [:KaiE:]]

FYI, lint makes the following change to your code, I'll apply that at checkin time:

diff --git a/rust/ews_xpcom/src/line_token.rs b/rust/ews_xpcom/src/line_token.rs
--- a/rust/ews_xpcom/src/line_token.rs
+++ b/rust/ews_xpcom/src/line_token.rs
@@ -22,11 +22,11 @@
 //!
 //! [`OperationQueue`]: crate::operation_queue::OperationQueue
 
 use std::cell::RefCell;
 
-use futures::{FutureExt, future::Shared};
+use futures::{future::Shared, FutureExt};
 use oneshot::{Receiver, Sender};
 
 /// A oneshot channel used internally by a [`Line`] that's been acquired to
 /// communicate that the token has been dropped and the line was released.
 ///

Comment 13

[Justin Tracey [:jtracey]]

The original order is what my machine generates, and passed the cargo fmt check in the try run. What version of Rust do you have installed? (For reference, current MSRV is 1.90.)

Comment 14

[Brendan Abolivier [:babolivier]]

(In reply to Chris Lamb from comment #11)

Good to see some progress. The latest update or two have fixed a few problems it seems (unable to save sent messages for example). However, I'm not getting new messages w/o restarting. Should I open a new bug report?

I'd say wait for the next Daily that includes this fix, and if the problem remains then opening a new report would be preferable.

Comment 15

[Chris Lamb]

Will do. I am often seeing failures saving to Sent - earlier today it seemed good, now bad. Looking forward to the next release!

Comment 16

[Chris Lamb]

And it has now lost my inbox, no sign of it anywhere. Sorry for the spam, it was usable this morning, so now I'm running into all the other issues.

I tried .\thunderbird.exe -MOZ_LOG=ews_xpcom::*:4 -MOZ_LOG_FILE=e:_Documents\tb_ews.log -P and no logs are created or errors shown

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/cc1770540d4a
Synchronize error handling between EWS queue runners. r=jtracey

Comment 18

[Eleanor Dicharry]

(In reply to Chris Lamb from comment #16)

And it has now lost my inbox, no sign of it anywhere. Sorry for the spam, it was usable this morning, so now I'm running into all the other issues.

I tried .\thunderbird.exe -MOZ_LOG=ews_xpcom::*:4 -MOZ_LOG_FILE=e:_Documents\tb_ews.log -P and no logs are created or errors shown

When you say "lost your inbox", are you saying your inbox folder disappeared from Thunderbird? If so, that might be another instance of an issue we discussed with a user on SUMO that we thought was related to this, but if you're seeing it with the concurrent connections fix, then I think we need to take another look.

Comment 19

[Chris Lamb]

(In reply to Eleanor Dicharry from comment #18)

(In reply to Chris Lamb from comment #16)

And it has now lost my inbox, no sign of it anywhere. Sorry for the spam, it was usable this morning, so now I'm running into all the other issues.

I tried .\thunderbird.exe -MOZ_LOG=ews_xpcom::*:4 -MOZ_LOG_FILE=e:_Documents\tb_ews.log -P and no logs are created or errors shown

When you say "lost your inbox", are you saying your inbox folder disappeared from Thunderbird? If so, that might be another instance of an issue we discussed with a user on SUMO that we thought was related to this, but if you're seeing it with the concurrent connections fix, then I think we need to take another look.

• Yes, it has disappeared, despite restarting TB several times, even waiting for hours to see if it was some kind of sync issue (as well as some inbox subfolders)
• Before that it was often showing only messages from the last few hours in Sent/Inbox/inbox subfolders. Repairing the folder would eventually fix that.
• I read the Sumo thread, I can do some more logging or try a new profile if needed, but probably not until tomorrow