1850289 - onNewMailReceived listener unreliable

Status: REOPENED

(Thunderbird :: Add-Ons: Extensions API, defect, P2)

Description

[Jonathan Kamens]

I add a listener with messenger.messages.onNewMailReceived.addListener in my add-on when it loads. Then after adding the listener I scan all the messages currently in the user's inbox to catch any that were there before I added the listener. Nonetheless, I subsequently discovered that there are messages in the user's inbox that my add-on never saw, because they didn't show up in the scan of all messages in the folder and my listener was never called about them.

Jury's still out on whether the listener is unreliable in general or just during this startup period.

Comment 1

[Jonathan Kamens]

More data about this. I received an email at 17:57 tonight. It went into my inbox. Remote Content By Folder (my add-on) should have received a listener call about it. It never did.

My guess is that the message came in while my laptop was sleeping and for some reason the listener isn't being called on messages that come in when Thunderbird is "catching up" after being offline for a while. This may be the same reason why the listener isn't being called for messages that come in right when Thunderbird starts up.

Comment 2

[Mark Banner (:standard8)]

(In reply to Jonathan Kamens from comment #1)

My guess is that the message came in while my laptop was sleeping and for some reason the listener isn't being called on messages that come in when Thunderbird is "catching up" after being offline for a while. This may be the same reason why the listener isn't being called for messages that come in right when Thunderbird starts up.

This could be related to bug 1850679.

Comment 3

[John Bieling]

@Jonathan: Are you using an OAUTH IMAP account (like GMAIL)? Are you able to reproduce this with a normal non-OAUTH IMAP account?

Comment 4

[Jonathan Kamens]

This is happening with a regular username/password IMAP account. On a Cyrus IMAPd server, specifically.

Every workaround I've tried for this problem has failed. It doesn't seem to be restricted to startup and offline -> online transitions. For whatever reason, I am frequently not getting notifications for all new messages that come into my inbox. The next theory I'm going to test when I have a few minutes to code it is that sometimes when multiple emails come in at the same time the notification doesn't include all of them. If that's the case then the workaround is going to be to rescan at least my entire inbox every time I get a notification.

Comment 5

[John Bieling]

Any news on solid steps to reproduce this issue? Is it also happening when the laptop was not sleeping?

Comment 6

[Jonathan Kamens]

I don't know anything more about why this is happening or how to reproduce it. All I can tell you is it seems to happen pretty often. I eventually gave up on relying on the events and coded my extension to scan folders regularly looking for messages it didn't get notified about.

Comment 7

[rste79]

As far as I can tell, the event is fired only if messages land in the Inbox folder, i.e. when messages aren't moved to other folders by filters.
According to the api docs this is not the expected behaviour

https://webextension-api.thunderbird.net/en/latest/messages.html#onnewmailreceived

Fired when a new message is received, and has been through junk classification and message filters.

Comment 8

[Jonathan Kamens]

As far as I can tell, the event is fired only if messages land in the Inbox folder

That's bug 1848787, which is different from this bug.

Comment 9

[rste79]

(In reply to Jonathan Kamens from comment #8)

As far as I can tell, the event is fired only if messages land in the Inbox folder

That's bug 1848787, which is different from this bug.

Hi Jonathan, no it isn't. That bug is about special folders other than Inbox. My complaint is about the reliability of onNewMailReceived for normal folders (i.e. subfolders of Inbox).
Why is the event not triggered if an email is received and a filter moves it to a subfolder of Inbox?

Comment 10

[Jonathan Kamens]

Well, if you don't think it's bug 1848787, fine, but it's not this bug. This bug is specifically about emails that absolutely should have been notified about, that don't fall into the filter problem you're mentioning, but notifications still weren't received.

If you're reliably seeing messages moved by filters not getting notified about, that's a different bug. I suggest you open a new bug for it.

Comment 11

[rste79]

(In reply to Jonathan Kamens from comment #10)

Well, if you don't think it's bug 1848787, fine, but it's not this bug. This bug is specifically about emails that absolutely should have been notified about, that don't fall into the filter problem you're mentioning, but notifications still weren't received.

If you're reliably seeing messages moved by filters not getting notified about, that's a different bug. I suggest you open a new bug for it.

Jonathan I see your point. I have followed your advice and opened a new bug 1864870.

Comment 12

[Mic]

I tested this with 128.6.1esr and 135.0b4.
It seems that the problem occurs when the new mail notification is not displayed.
I identified two cases, thought there may be others (such as the one mentioned in comment 1):

1. The "Show an alert" option in "When new messages arrive" is unchecked.
2. Clicking on the "Get new messages" button

I was able to reproduce the issue consistently.

You can find the full thread in the add-on dev room.

Comment 13

[John Bieling]

Attachment: Bug 1850289 - Change implementation of messages.onNewMailReceived event to no longer use BiffState. r=mkmelin

The BiffState approach turned out to be cumbersome, specifically if
filters are involved. Instead of using the BiffState property, we now
simply report new messages which have the new flag set.

We keep the original behavior to bunch up multiple new message events
per folder.

Comment 14

[John Bieling]

Attachment: onNewMailReceived.xpi

This add-on includes an Experiment which uses the same mechanism as the new implementation introduced in D237936. It can be installed in Thunderbird 128 ESR to try out if any messages are still missed, without having to wait for the patch to be backported from Thunderbird Daily to Thunderbird 128 ESR.

Testing is appreciated, thanks!

The add-on will print an entry for each received message to the console.

Comment 15

[Jonathan Kamens]

I attempted to plug the experiment into Remote Content By Folder but it appears that the event listener is getting passed an array of message subjects instead of a MessageList? Given that I don't think I can test this in real-world conditions.

Comment 16

[John Bieling]

I wanted to go the extra mile and provide a self-contained technology demonstrator, which cannot do any harm and can be easily tested. I cannot provide a drop-in replacement, as that needs extensive testing and may introduce unwanted side effects if used in real add-ons.

To allow real-world testing with your add-on, I started a try-run where you could pull executables of the patched build:
https://treeherder.mozilla.org/jobs?repo=try-comm-central&revision=d9eef29dbe3f543118c8b493ceab2126f68dd04a

Comment 17

[John Bieling]

Removing the dependency from Bug 1850679, as we no longer depend on BiffState information: A BiffState event only gives us the original folder, "inbox" for example, but if the message is moved by filters into a different folders (not being a sub-folder of the original folder), before junk classification, the current system does not know where the message is and cannot report it.

Comment 18

[Jonathan Kamens]

(In reply to John Bieling (:TbSync) from comment #16)

I wanted to go the extra mile and provide a self-contained technology demonstrator, which cannot do any harm and can be easily tested. I cannot provide a drop-in replacement, as that needs extensive testing and may introduce unwanted side effects if used in real add-ons.

To allow real-world testing with your add-on, I started a try-run where you could pull executables of the patched build:
https://treeherder.mozilla.org/jobs?repo=try-comm-central&revision=d9eef29dbe3f543118c8b493ceab2126f68dd04a

Thanks! I will test for a week and report back.

Comment 19

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/73ba995def58
Change implementation of messages.onNewMailReceived event to no longer use BiffState. r=mkmelin

Comment 20

[Jonathan Kamens]

Reopening because I am still not receiving notifications for all new messages.

For example, this morning when I launched Thunderbird (the try version you gave me to test) I received two emails one right after the other, as indicated by the IMAP Order Received column. I have two different extensions installed that use onNewMailReceived. As far as I can tell, both of these extensions were only notified about one of those two messages.

Comment 21

[Jonathan Kamens]

Ping

Comment 22

[Wayne Mery (:wsmwk)]

(In reply to Jonathan Kamens from comment #21)

Ping

It will be more reliable to needinfo the person you wish to respond.

Comment 23

[Jim DeLaHunt]

(In reply to Wayne Mery (:wsmwk) from comment #22)

It will be more reliable to needinfo the person you wish to respond.

If I understand you correctly, you are offering advice on how to use Bugzilla within this project more effectively. Rather than just posting "ping", someone who wants attention will perhaps get better results by going to the People tab at the top of this report, going to the NeedInfo From: menu, and selecting the entry for a specific person from whom you want a reply.

(I write this because my first interpretation of this reply was that it was suggestion to use some needinfo() API rather than the messenger.messages.onNewMailReceived.addListener API mentioned in the Description.

Comment 24

[Jonathan Kamens]

For the record, I can confirm that this still is not fixed.

I spent many hours over the past day or so heavily instrumenting my Remote Content By Folder extension and rewriting major parts of it so that I can be absolutely certain I am accurately detecting whether I am getting all of the notifications I should be.

I am not.

The problem is intermittent.

I am using Thunderbird 142.0b4 on Debian linux.

I am adding the listener like this:

  messenger.messages.onNewMailReceived.addListener(checkNewMessages, true);

My extension just detected and notified me about the fact that I never received an event about a message deposited into my Sent folder by an actor external to Thunderbird nearly a minute prior.

However, when another message was deposited into the Sent folder a minute or two later, I did receive an event about that one.

Comment 25

[Jonathan Kamens]

I don't know if this is the only failure mode, but one failure mode I am seeing consistently is that if I have an IMAP account that isn't set up to synchronize messages locally (don't know if that's relevant) and isn't set up to check messages periodically in the background (probably relevant), when I open my Inbox in that account and Thunderbird suddenly discovers and loads the headers for the big batch of new messages that have arrived since the last time I opened the inbox, I don't get notifications for all of those newly discovered messages.

I don't know if this is possible, but I wonder if there is a race condition, i.e., if a new message is discovered by Thunderbird in parallel with the code for generating the new mail notification running, it gets missed. I know that JavaScript is single-threaded, but it seems to me this might be possible both because at least some of this code is probably running in C++ or Rust rather than in JavaScript, and because of async JavaScript, i.e., two different JavaScript workloads—process new incoming mail and construct new mail event—can be passing the single thread back and forth between themselves because of async functions.

Comment 26

[nanananto]

I'm seeing what appears to be the same issue, and I'd like to add some observations that may help narrow down the conditions under which it occurs.

Environment:

• Thunderbird 151 and 152 (also observed in 150, but less frequently)
• Windows 11
• Gmail account configured over IMAP

Observations:
This is not limited to my own setup — three of my colleagues have independently reported the same behavior, and they also find it occurs more readily on Thunderbird 151 and 152.
In my case, the failure of messages.onNewMailReceived to fire seems to correlate with binary attachments. It is most reproducible when unread messages with binary attachments have accumulated on the IMAP server and Thunderbird is then started and fetches them. I've reproduced this with both .jpeg and .zip attachments.
That said, the behavior is intermittent rather than fully deterministic. In one session, receiving messages with binary attachments triggered the failure, whereas subsequently receiving messages without attachments did not reproduce it for a while. I haven't been able to confirm this reliably, so I can't say for certain whether binary attachments are actually a contributing factor, or whether this just reflects the overall instability of the event firing.
Compared to Thunderbird 150 and earlier, the failure rate seems noticeably higher in 151 and 152.