Closed Bug 1925631 Opened 1 year ago Closed 1 year ago

Firefox showing old notifications over and over again

Categories

(Core :: DOM: Push Subscriptions, defect, P2)

Product:
Core
Component:
DOM: Push Subscriptions
Version:
Firefox 131
Platform:
Desktop
All
Type:
defect

Tracking

()

Status:
RESOLVED WORKSFORME

People

(Reporter: u761629, Assigned: saschanaz)

References

(Depends on 2 open bugs,
URL
)

Details

(Keywords: regression, regressionwindow-wanted)

Attachments

(5 files)

serviceworker.js
3.70 KB, text/plain
Details
serviceworker.js
3.70 KB, text/plain
Details
sw.js
3.70 KB, text/plain
Details
console-export-2024-10-28_9-13-5.txt
2.38 MB, text/plain
Details
console-export-2024-10-28_13-3-27_redacted.txt
9.78 KB, text/plain
Details

Steps to reproduce:

Multiple users are reporting that Firefox keeps showing old notifications when it starts running. Disabling and re-enabling notifications in about:config didn't fix this.

I don't personally use notifications. Please reach out to these users if you need further info: https://old.reddit.com/r/firefox/comments/1g5u9od/firefox_13103_windows_sends_old_and_repeated/ . They've refused to create Bugzilla accounts when told to come here.

Component: Untriaged → DOM: Notifications
OS: Unspecified → All
Product: Firefox → Core
Hardware: Unspecified → Desktop
Severity: -- → S3
Priority: -- → P2

Sounds like it started very recently.
Kagami, could you, please, take a look?

I set to S3 since it does not block critical functionality (although, it might be annoying)

Flags: needinfo?(krosylight)

Interesting. Moving to Push Subscriptions as this seems push thing.

:jrconlin, any relevant change in autopush side? I think on startup we get the push messages from the server, which then should go away from the server, right?

Component: DOM: Notifications → DOM: Push Subscriptions
Flags: needinfo?(krosylight) → needinfo?(jrconlin)
Duplicate of this bug: 1925904

There's been no change to the push server since last month, and we've not seen or heard of similar problems, so I'm a bit curious.

FWIW, the push server will "retry" sending old notifications to a desktop user agent if the user agent fails to send an "ack" of the message. I don't believe there's been any changes to the User Agent or server recently that should be causing this.

If I may, can I ask you to provide a bit more info?

In about:config can you go to dom.push.loglevel and set it to Debug. Then open up the
≡ > More Tools > Browser Console
you can filter on PushServiceWebSocket and make sure that all of the Errors Warnings Logs Info Debug fields are active.

When you get a Push Notification (If you want to try one right away, you can use something like the Web Push Data Encryption Test Page) you should see a few messages in the Console, including

PushServiceWebSocket: handleNotificationReply()                                    PushServiceWebSocket.sys.mjs:780
PushServiceWebSocket: sendAck()                                                    PushServiceWebSocket.sys.mjs:854

This should be followed by a wsSendMessage: Sending message {"messageType": "ack", .... with no error afterward.
Don't paste those messages here, but I'd be curious to see if you see those messages in the Browser Console and if there might be any errors being returned around them.

Flags: needinfo?(jrconlin)

I assume that's a request for the reporter. (I can also try, although I've never seen this issue)

Flags: needinfo?(krosylight)
Flags: needinfo?(fpmkdqugh)

Ok, looking into this more...

Our metrics started showing a spike in NACKs around October 14th. These NACKS indicate (code: 301 & 302) that there was a problem with the UserAgent sending the Push Message content to the ServiceWorker. (Which is a little bit of Javascript that the site installs which figures out what to do with the incoming message, usually just popping up that "Notification" DOM element.)

Firefox has no insight into what might be causing the serviceworker to fail, but it is a bit odd that it started the day after 131.0.3 rolled out.

(For folk that want to play along at home:

https://searchfox.org/mozilla-central/source/dom/serviceworkers/ServiceWorkerOp.cpp#193
handles the service worker response which goes to
https://searchfox.org/mozilla-central/source/dom/serviceworkers/ServiceWorkerOp.cpp#137
https://searchfox.org/mozilla-central/source/dom/serviceworkers/ServiceWorkerOp.cpp#727
which gets translated to
https://searchfox.org/mozilla-central/source/dom/push/PushServiceWebSocket.sys.mjs#40
which gets send as part of the NACK.)

That's about the best I can do.

:saschanaz(In reply to Kagami Rosylight [:saschanaz] (they/them) from comment #5)

I assume that's a request for the reporter. (I can also try, although I've never seen this issue)

I mean, you can try, although I'm not sure it would work for you. I'm wondering if there might be some weird bug in some library serviceworker that may be failing due to a fix introduced in the latest release? I'm really not sure what might have triggered that.

If the original reporter can offer the origin site for the Push Notification they're seeing repeat, we might be able to look into the serviceworker code and see what it might be doing.

Update:
From the video, it looks like folk are seeing problems from sofascore.com and instagram.com.
I don't have accounts on either of those, so I can't test things myself, but I'm guessing others may.

I created an instagram account using my previously created test Facebook account on nightly but I haven't been able to get the website to actually request permissions to send pushes. Looking at the pretty printed source of the JS for the main instagram page, there is a ton of telemetry gathered along these paths but it's not immediately obvious how to get to look at it; it seems like the data may be posted to /ajax/bz as multipart with the actual payload being deflate or snappy-compressed. And if I set breakpoints on the pretty printed source where it might be probing for the ability to send pushes, the breakpoints don't get hit.

Youtube is also generating duplicates for me on 131.01 desktop. Not the original reporter but seeing the same thing - every [period of time] a seemingly increasing amount of past dismissed notifications are being spammed all at once.

Thanks for reporting about youtube!

A potential intervention we might make for the Receiving a Push Message algorithm is treat use of the notifications API from a ServiceWorker as indicating any active "push" message functional events should be acked. This would be somewhat like the FetchEvent where respondWith() can be resolved earlier than the functional event's lifetime extension promises (one of which is automatically the promise provided to respondWith()). I always found that algorithm a bit handwavey.

Additional note: Occurs once every hour.
Something I'm noticing is on the push spam they are not incrementing the pushCount nor the lastPush in the debug log.
So leading up I see pushCount 1745, 1746 with the last push incrementing (not by 1's though) for normal push notifications. Then the repeat notification spam happens and they are all that the same pushCount and lastPush #'s.

At this point i'm getting 28 pushes when it happens with only 6-7 being visible in the windows notification log.

I was experiencing this problem for the last couple of days but now the problem seems to be gone

I wonder there's anything changed for websocket connection in 131. Randell, has there been relevant pref change or something?

And I wonder whether the affected people can use websocket properly; in this demo https://libwebsockets.org/testserver/, can one of you check whether "reset counter" actually resets counter?

Flags: needinfo?(krosylight) → needinfo?(rjesup)

Uhm, but comment #6 implies maybe the problem is elsewhere. Let me think more.

Flags: needinfo?(rjesup)

Assuming this is a Firefox regression.

Keywords: regression, regressionwindow-wanted

(In reply to reisei1 from comment #12)

I was experiencing this problem for the last couple of days but now the problem seems to be gone

nevermind, it's back. just in small number of notifications

(In reply to Kagami Rosylight [:saschanaz] (they/them) from comment #13)

I wonder there's anything changed for websocket connection in 131. Randell, has there been relevant pref change or something?

And I wonder whether the affected people can use websocket properly; in this demo https://libwebsockets.org/testserver/, can one of you check whether "reset counter" actually resets counter?

Reset counter does reset the counter. Opening two tabs each has their own individual counter that can be reset. Send junk closes the websocket connection.

Flags: needinfo?(fpmkdqugh)

Raising the severity for now as it seems to have higher impact than initially expected.

Severity: S3 → S2

I'm also seeing this in Nightly (Fx133) with push notifications from chess.com. I'll see if the steps in comment 4 yield anything (will post a follow up). Let me know if there is anything else I can do.

To add a bit more to all of this:

PushService will open a websocket connection between the UA and the server at launch. This websocket connection is the only way that desktop UAs will talk to the push service. This websocket connection is fairly long lived. I would guess if a websocket connection was failing, that would lead to other problems (such as no updates going through at all), or frequent connects/disconnects (which we're not seeing in our metrics).

I can't say with any certainty, but I feel the problem is that somewhere in the serviceworker.js file that these sites are using there is some exception that is happening, which is being captured by the callback, and triggering the "Rejected" case, which is later resulting in the NACK. (I'm hesitant to say that this is an invalid case, though, since if the serviceworker failed for some reason we may want to resend. The problem is that we don't know what is triggering the exception or if the serviceworker fired the DOM Notification to the user.)

What I'd be more curious about is what are these serviceworkers doing that might be triggering an exception?

from #10, youtube is displaying duplicates, so I grabbed the serviceworker URL: https://www.youtube.com/sw.js, which pulls in https://www.youtube.com/s/desktop/dd166493/jsbin/serviceworker-kevlar-appshell.vflset/serviceworker-kevlar-appshell.js (which is DEEPLY obfuscated, and appears to be doing all sorts of things including hash generation and index_db calls, but I'm just a back-end server guy so I'm just guessing about a lot of this.) I would guess that it would have to be some step occurring after the Notification() call is made but I have no clue where that's happening in any of this. I think the notification pop occurs in function xm() which then calls zm() (since I presume no action occurred).

QA Whiteboard: [qa-regression-triage]

Is anyone not being on Windows getting this issue? i.e. is this Windows specific?

Is anyone not being on Windows getting this issue?

See the comments on the page I've added to the URL field of this bug. There are reports from Linux users too.

i have this error since 131.0.3, i am on Windows 10 Pro 22H2, and my main source is sofascore.com with all the goals and stuff, every day i turn on the computer and get a ton of old notifications over and over again, and every time i open firefox

glad to see there is a captured bug for this, was driving me nuts

I failed to reproduce this on Instagram unfortunately (through DM). For other websites like chess.com and sofascore.com, it would be very helpful if someone can spell out how to subscribe the push notifications there and how to trigger the notification (or the condition to get a new notification).

I see this issue in Windows 10 (but not Windows 11) since version 131.0.3 on two websites I manage. Can I help in any way?

(In reply to [:maciek-szn] from comment #27)

I see this issue in Windows 10 (but not Windows 11) since version 131.0.3

Interesting, it implies it's profile specific at least (as the question of platform-specific has been answered - with no).

on two websites I manage. Can I help in any way?

Sharing the steps mentioned in comment #26 would be greatly helpful. If your service worker script is obfuscated, a deobfucated version would also be great help.

(In reply to Kagami Rosylight [:saschanaz] (they/them) from comment #28)

Sharing the steps mentioned in comment #26 would be greatly helpful. If your service worker script is obfuscated, a deobfucated version would also be great help.

The website is https://www.zditm.szczecin.pl/pl (you will probably have to wait until Monday for new notifications to appear). Click the bell icon in the header of the page. A modal window will appear, in which select all the checkboxes and press the button below them to enable notifications.

Here is the deobfuscated version of service worker

Thanks a lot! From that script it seems the only way that PushEvent.waitUntil can reject is to get exception from showNotification(), given this code:

            event.waitUntil(
                this.sendNotification(event.data.json())
            )

    sendNotification(data) {
        return self.registration.showNotification(data.title, data)
    }

But given we currently resolve showNotification() incorrectly too early before even contacting parent process to show notification (bug 1891705), there's no possible way for showNotification to show a notification and then reject. So the possibility of rejecting push event mentioned in comment #7 is gone.

As we are getting some notification, it means push event itself is running correctly, would it be possible that waitUntil() somehow is working improperly and getting internal error, and that's somehow translated as NACK? Something to check further.

(Just made the dupe comments hidden, as deleting comments needs special admin permission)

One thing that would also be helpful is that whether you can see reportDeliveryError() for any repeating push notifications when you follow the debug steps in comment #4. That can confirm whether the affected profiles are reporting NACK as we are assuming here.

(Enough internal and external reports to mark this confirmed)

Status: UNCONFIRMED → NEW
Ever confirmed: true
Assignee: nobody → krosylight

(In reply to Kagami Rosylight [:saschanaz] (they/them) from comment #35)

One thing that would also be helpful is that whether you can see reportDeliveryError() for any repeating push notifications when you follow the debug steps in comment #4. That can confirm whether the affected profiles are reporting NACK as we are assuming here.

Here's the console output. I've redacted it a bit, but if there's any important information missing, please let me know.

Error notifying app Error: No record for key ID d88aa044-12bc-4f77-a121-f29af8e9d2a1

That's interesting.... I think I've seen that error while adding test in bug 1874394. I deferred the investigation as the test worked as expected anyway. Maybe I shouldn't have done that.

But that error means it couldn't call service worker at all, meaning no actual notification could happen. Which contradicts the behavior here. Now I'm not sure how the test even passes with that error. Perhaps something happening in parellel? Should be checked further.

(BTW the log is saying the nonstandard aesgcm instead of aes128gcm... What push server library are you using? Not that it's related here, just interested.)

(In reply to Kagami Rosylight [:saschanaz] (they/them) from comment #39)

(BTW the log is saying the nonstandard aesgcm instead of aes128gcm... What push server library are you using? Not that it's related here, just interested.)

I always get aesgcm encoding for the endpoints https://updates.push.services.mozilla.com/wpush/v2/...

This is the library I use to handle notifications.

I think I have an explanation for the aesgcm value. I have this line in my code:

const contentEncoding = (PushManager.supportedContentEncodings || ['aesgcm'])[0];

According to MDN supportedContentEncodings isn't supported by Firefox, so aesgcm is the default value. I don't know if it's correct (I got this code from somewhere), but it works.

Hmm, actually that's wrong, Firefox supports aes128gcm and aesgcm both. It works as you say, the latter is just nonstandard. I guess I just got one strong reason to ship that attribute 😅 (bug 1497430)

This started happening again this morning for me on chess.com. I had the debug pref set, so hopefully it captured something useful (apologies for the large log file, I wasn't sure where it was safe to prune, probably best to jump to the bottom and work backwards).

per #42:
aesgcm is the older, pre-RFC standard that is still VERY MUCH used (it accounts for 50-60% of incoming Subscription updates.)

I tried really hard a while ago to get library authors to switch the default encoder to be aes128gcm, and a lot of packages did update, but apparently there are a lot of Subscription Publishing sites that have not updated their backend packages in over 4 years.

(Makes quiet, horrified noises thinking about all the unaddressed CVEs and errors. Please, if you have software you run on a computer that is connected to the internet, keep it up-to-date.)

(In reply to JR Conlin [:jrconlin,:jconlin] from comment #44)

I tried really hard a while ago to get library authors to switch the default encoder to be aes128gcm, and a lot of packages did update, but apparently there are a lot of Subscription Publishing sites that have not updated their backend packages in over 4 years.

Even in the latest version of the web-push-php library, aesgcm is the default value. So it's not a problem of outdated packages.

This just happened for me again with a Google Calendar entry from Wednesday (so this was exactly 5 days later). I had turned on the logging from comment #4 this morning, and here's the console log (with the UID X'd out after the first few characters; not sure if it's sensitive or not). Let me know if there's something else helpful I can do!

Comment #43 shows sendAck() and queueRequest() but unfortunately the value of updates is hidden behind [...]. Does updates.code ever get anything other than 100 for you two? (I don't see any actual push receive in comment #46 btw, it only shows initial handshake with the push server.)

Edit: You should only be able to check it in browser toolbox.

Flags: needinfo?(mconca)
Flags: needinfo?(gstoll)

Actually I can see "code":102 from wsSendMessage.

Flags: needinfo?(mconca)
Flags: needinfo?(gstoll)

So from comment #43 I can see two channels: one starting with 4f2 and another with 3f3. 4f2 always ends up being code 102 because there's no matching record in PushDB, and ends up doing nothing. 3f3 always ends up being code 100 and this one actually hits notifyApp(). Given it acks with code 100 I'm not sure why it's repeating.

Is there any possibility that the server somehow gets confused with two channels and thinks 3f3 got code 102?

Flags: needinfo?(lina)
Flags: needinfo?(jrconlin)

Greg, do you have more log that includes wsSendMessage?

Flags: needinfo?(gstoll)

Unfortunately I had to close Firefox. I'll keep an eye out for it happening again.

Flags: needinfo?(gstoll)

os : windows 11
firefox version : 131.0.3 and 132.0
platform that i'm using is x.com(twitter) with push notification enabled

Unfortunately I still haven't been able to reproduce, even though we have a few internal reports too. Given the metrics say it started at Oct 14 (the 131.0.3 release data) and it included some network patches, I wonder there's some network side issue that is causing this. But I can't imagine how network can cause this issue yet. (And that patch is done to fix some other issue, so we can't easily back it out.)

For people who get a lot of pain with this issue, you may be able to suppress it by setting the value of dom.push.maxRecentMessageIDsPerSubscription to something high as 100.

See Also: → 1928124

This may be a false lead, but perhaps the occurrence of the issue depends on the age of the Firefox profile. I have the problem on a very old profile (Windows 10), but not on a relatively new profile (about 2 years old, Windows 11).

See Also: → 1928152

We prepared test builds without the patch which we think might be the potential cause of this. https://treeherder.mozilla.org/jobs?repo=try&revision=55f8ab02c1eda3cd22e64d40e42152e7c042ce62&selectedTaskRun=aW8ERXgNT2KgFr-IgOmPkA.0

How to test:

  1. In the current Firefox window, nagivate to about:profiles and find "This is the profile in use and it cannot be deleted.". Remember the name of the profile.
  2. Open the treeherder link and select the right platform for you (Click B next to the matching platform)
  3. In "Artifacts and Debugging Tools", download the binary. The file name is different for each platform:
    • For Linux it's target.tar.bz2.
    • For macOS it's target.dmg.
    • For Windows it's target.zip.
  4. Unzip the file and run the application.
  5. Again in about:profiles, select the profile you found in step 1 and click "Launch profile in new browser".
  6. Use it for a while (or let it run in the background about 1 hour) and see if you still get repeated notifications after that.

Note that you should not open the main Firefox while doing this, so please do this only when you have free time. (Edit: And given this is a debug build, it's significant slower than the original Firefox. Use it only for test.)

(In reply to Kagami Rosylight [:saschanaz] (they/them) from comment #57)

  1. Use it for a while (or let it run in the background about 1 hour) and see if you still get repeated notifications after that.

Unfortunately, the issue also occurs in this build (Windows 10).

(In reply to [:maciek-szn] from comment #58)

(In reply to Kagami Rosylight [:saschanaz] (they/them) from comment #57)

  1. Use it for a while (or let it run in the background about 1 hour) and see if you still get repeated notifications after that.

Unfortunately, the issue also occurs in this build (Windows 10).

Thanks! To confirm whether this happens in an existing profile but not in a new one, can you:

  1. In your existing Firefox installation, open about:profiles and click "Create a new profile"
  2. On the new profile, click "Launch profile in new browser"
  3. Make push subscriptions and wait for some push notifications
  4. Either:
    • Use it for a while (~1hr), or
    • Close the browser and repeat step 2, or
    • Flip dom.push.connection.enabled in about:config a few times
  5. Check if repeated notifications happen

I'm also curious to know whether the same issue happens in https://mozilla-services.github.io/WebPushDataTestPage/:

  1. Click "Get subscription info" and allow the permission
  2. Scroll down a bit and click "Encrypt the data"
  3. Scroll down more and click "Send the Data" and confirm the notification opens
  4. Either:
    • Use it for a while (~1hr), or
    • Restart the browser, or
    • Flip dom.push.connection.enabled in about:config a few times
  5. Check if repeated notifications happen

(In reply to Kagami Rosylight [:saschanaz] (they/them) from comment #50)

So from comment #43 I can see two channels: one starting with 4f2 and another with 3f3. 4f2 always ends up being code 102 because there's no matching record in PushDB, and ends up doing nothing. 3f3 always ends up being code 100 and this one actually hits notifyApp(). Given it acks with code 100 I'm not sure why it's repeating.

Is there any possibility that the server somehow gets confused with two channels and thinks 3f3 got code 102?

Right, so I think I may have identified a problem with the server code. (Bear with me, it's annoying.)

Because of cost considerations when we first built the server, we had to limit the number of read/write operations that were happening. To do that, we used a "cursor" value to indicate where we were in the data set and only updated that cursor once, when the UA ACK'd all of the messages. There were a few assumptions that went into that, including the fact that the UA would ACK all the messages eventually. (FWIW, our NACK is currently a "NO-OP", where we just log some metrics, partly based on that.)

What I think is happening is that there's some Bad Notification that is triggering a NAK that cannot be resolved in the batch of messages we're sending, even one will do it, so the server retries them. Then the UA disconnects, and things reset to the batch of pending messages that have already been sent.

I think that's why the UA may see Notifications that have already been ACK'd (at least, that's my working theory).

Since we recently switched data stores, database writes are far cheaper, so we can drop the weird "cursor" cost saving we were doing.

One thing I need to know is which of the NACK codes should the server use to destroy a pending message. (That may also solve problems.)

Flags: needinfo?(jrconlin) → needinfo?(krosylight)

Any of the current NACK should trigger retry per Push API spec. What the client is not doing properly is that it does not send ACK for repeated SW failure:

If the same push message has been delivered to a service worker registration multiple times unsuccessfully, acknowledge the receipt of the push message according to [RFC8030].

Although this is a client side spec, the server might also want to count the retry and halt it eventually.

It's still unclear what is causing this sudden increase of failures, but there certainly is an existing problem in the client side too.

Flags: needinfo?(krosylight)

Reading bug 1280639, it seems another core problem is that ACK and NACK is not mutually exclusive: NACK can and always happen after ACK. Meaning ACK alone cannot be the signal as an ack receipt. That sounds awkward. 😬

See Also: → 1280639

Sigh. (I feel like I should hold a wake for the neurons that died in the past 8 years.)

So, IIUC, the Client ACK is to indicate "Yes, I have received the message." There are degrees of "received", which are indicated by the various codes, but essentially the ACK is there to indicate to the server to not send this message again. Not sending an ACK (which includes sending a NAK) is essentially "Reply hazy, try again", in which case, the server will return unack'd messages before fetching out new messages. This basically leads to a storm of potentially previously "ACK"'d message which the client is responsible for filtering out.

There are a few additional observations I have:

  • Martin's main concerns were that altering from the RFC spec meant that we would not be able to fully comply with the RFC spec. I'm quite comfortable saying that will not happen in any case. (Also because one of the principle means of communication was an altered HTTP/2 PUSH packet, which was dropped, unofficially, but it was pretty poorly understood).
  • He also expressed concern that the NAK mechanism could be exploited, but if we have functional NAKs disguised as ACKs with codes, I think we basically hung a fig leaf over what a NAK is and it would still be exploited. I'm still not quite sure how it would be exploited, since the protocol is only between the trusted Push server and the trusted Client implying a MITM websocket attack (where websockets are VERY aggressive about closing connections for pretty much any reason). I would think that ensuring that the TLS channel remains secure would be outside of the scope of Push, otherwise we would send fully encrypted data between the client and server (TLS over TLS, essentially)

I know that the main reason for our server (at least) to potentially send duplicates was mostly around preventing costly database writes. We handle millions of messages per second so those write costs add up fast. Writes are now less of a concern so we should be able to mark ACK'd records for deletion. This should reduce the pool of potential duplicates. There is still a possibility that some previously ACK'd messages could be re-delivered due to the nature of distributed databases.

To that end. ACK should probably still be considered "I have received this message. Do not resend it."
We may wish to either consider adding a NAK with a specific "I have fully rejected this message. Do not resend it." in the case where a message is completely undeliverable for some core reason, or overload one of the existing ACK codes to indicate that.
The server will continue to resend messages that it has not been told to never send again. This change should not impact older client codes because the core behavior and existing protocol have not changed.

(In reply to Kagami Rosylight [:saschanaz] (they/them) from comment #59)

Thanks! To confirm whether this happens in an existing profile but not in a new one, can you [...]

I can confirm that the issue does not occur on the newly created profile.

I'm also curious to know whether the same issue happens in https://mozilla-services.github.io/WebPushDataTestPage/

In this case the problem does not occur on either the new or the old profile.

Thanks! Just to check, what's the current value of network.cookie.cookieBehavior.optInPartitioning in about:config in the affected profile?

(In reply to Kagami Rosylight [:saschanaz] (they/them) from comment #65)

Thanks! Just to check, what's the current value of network.cookie.cookieBehavior.optInPartitioning in about:config in the affected profile?

It's false.

Depends on: 1928868

From Reddit:

same issue on Win10. I found that at least on Instagram opening the chat and loading the messages can fix it but doesnt seem to work every time.

I thought it was fixed too, but I just received and youtube notification from 7 days ago along with a new one. Back to Vivaldi again.

Meaning the old ones are coming along with the new ones at exact same time, and doesn't come if there's no new one? Is this behavior what other's getting too?

(In reply to Kagami Rosylight [:saschanaz] (they/them) from comment #67)

Meaning the old ones are coming along with the new ones at exact same time, and doesn't come if there's no new one? Is this behavior what other's getting too?

In my case, new notifications appear as single ones. Then they are added to the "queue" and displayed with the others every hour (I think so because Windows 10 limits the number of notifications displayed to 20, and Firefox returns them from the oldest ones).

(In reply to [:maciek-szn] from comment #68)

(In reply to Kagami Rosylight [:saschanaz] (they/them) from comment #67)

Meaning the old ones are coming along with the new ones at exact same time, and doesn't come if there's no new one? Is this behavior what other's getting too?

In my case, new notifications appear as single ones. Then they are added to the "queue" and displayed with the others every hour (I think so because Windows 10 limits the number of notifications displayed to 20, and Firefox returns them from the oldest ones).

How many notifications do you get at each repeat right now? Have you adjusted dom.push.maxRecentMessageIDsPerSubscription to something higher than the default 10? (In bug 1928868 we are changing that to 100)

(In reply to Kagami Rosylight [:saschanaz] (they/them) from comment #69)

How many notifications do you get at each repeat right now? Have you adjusted dom.push.maxRecentMessageIDsPerSubscription to something higher than the default 10? (In bug 1928868 we are changing that to 100)

No, it's the default value of 10.

Can you change the value to 100 and see the issue persists?

(In reply to Kagami Rosylight [:saschanaz] (they/them) from comment #71)

Can you change the value to 100 and see the issue persists?

I changed this value and restarted Firefox. After the first restart the notifications showed up, but after the second they didn't. So this probably fixes the issue.

Thanks! Although it doesn't solve the core issue (repeated push messages may still hit Firefox) but with higher value there's higher chance that Firefox can detect duplicated notifications and stop showing it.

Given we mitigated the impact on users, can we downgrade severity to S3 now?

Flags: needinfo?(krosylight)

I think the incident severity can go S3 but given the impact on our push server (the usage is like 8x of usual right now) maybe it should remain as S2?

Flags: needinfo?(krosylight)
Depends on: 1929667
Depends on: 1929668

Hello people, first of all thank you for the bug report and all the activity here!

We are observing that the volume of the notification duplications significantly reduced down to almost zero since November 7, so I'm happily closing this issue. If you are still affected, please don't hesitate to comment here and we can open a new issue for that if needed.

Status: NEW → RESOLVED
Closed: 1 year ago
Resolution: --- → WORKSFORME
Flags: needinfo?(lina)
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: