1623400 - Firefox Nighly push notifications not working when sec-fetch- headers are enabled

Status: RESOLVED FIXED

(Core :: DOM: Push Subscriptions, defect, P1)

Description

[Mic Sparrow]

Attachment: bug.png

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:76.0) Gecko/20100101 Firefox/76.0

Steps to reproduce:

Update Firefox Nightly to the latest build (build 2020-03-18 in this case).

Actual results:

Dektop push notifications are no longer appearing as they used to for sites where they were previously enabled. For new sites nottifications cannot be switched on (see pic attached).

Expected results:

Notifications should appear as usual.

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

Since the status are different for nightly and release, what's the status for beta?
For more information, please visit auto_nag documentation.

Comment 2

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Do you get the prompt to allow desktop notifications if you click the button on https://mdn.mozillademos.org/en-US/docs/Web/API/Notifications_API/Using_the_Notifications_API$samples/Tag_example?revision=1606062 ?

Comment 3

[BugBot [:suhaib / :marco/ :calixte]]

Bugbug thinks this bug should belong to this component, but please revert this change in case of error.

Comment 4

[Mic Sparrow]

(In reply to Sebastian Hengst [:aryx] [limited availability until end of March](needinfo on intermittent or backout) from comment #2)

Do you get the prompt to allow desktop notifications if you click the button on https://mdn.mozillademos.org/en-US/docs/Web/API/Notifications_API/Using_the_Notifications_API$samples/Tag_example?revision=1606062 ?

Yes and notifications seem to work. Strange...

Comment 5

[Mic Sparrow]

For more info - I did some tests and several websites with push notifications seems to work.
Examples:
https://www.bennish.net/web-notifications.html
https://www.pushengage.com/demo
Not working:
https://gauntface.github.io/simple-push-demo/
and most of the sites where I use notifications like YouTube, Facebook etc.

Comment 6

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://gauntface.github.io/simple-push-demo/ also doesn't work with Firefox 74 or 68.6.0esr here.

Do you know a website where it doesn't work, preferably without an account (or where not multiple accounts are needed)?

Comment 7

[Mic Sparrow]

(In reply to Sebastian Hengst [:aryx] [limited availability until end of March](needinfo on intermittent or backout) from comment #6)

https://gauntface.github.io/simple-push-demo/ also doesn't work with Firefox 74 or 68.6.0esr here.

Do you know a website where it doesn't work, preferably without an account (or where not multiple accounts are needed)?

In Firefox 74 it is working for me.
For a website where notifications do not work right now - maybe a Polish website sportowefakty.wp.pl . After enabling notifications they should send a 'thank you' notification.

Comment 8

[Daniel Bodea [:danibodea]]

I have attempted to find this bug's regressor on Windows 10 and this are my results:

2020-03-19T11:23:20: INFO : Narrowed integration regression window from [23c4c99d, a0508ae6] (3 builds) to [f6b26481, a0508ae6] (2 builds) (~1 steps left)
2020-03-19T11:23:20: DEBUG : Starting merge handling...
2020-03-19T11:23:20: DEBUG : Using url: https://hg.mozilla.org/integration/autoland/json-pushes?changeset=a0508ae6c037928981ac2733860b6ec84d7069ec&full=1
2020-03-19T11:23:22: DEBUG : Found commit message:
Bug 1508292: Implement Sec-Fetch-*. r=baku
Differential Revision: https://phabricator.services.mozilla.com/D66283
2020-03-19T11:23:22: DEBUG : Did not find a branch, checking all integration branches
2020-03-19T11:23:22: INFO : The bisection is done.
2020-03-19T11:23:22: INFO : Stopped

NI me if it seems incorrect. Thank you.

Comment 9

[Christoph Kerschbaumer [:ckerschb]]

FWIW, Sec-Fetch-* headers are enabled in Nightly only - we added a pref 'dom.security.secFetch.enabled' which should allow to figure out if the Sec-Fetch Headers are really responsible for the problem described here.

Comment 10

[Jens Stutte [:jstutte]]

Hi Dnaiel, can you confirm that this goes away with the pref 'dom.security.secFetch.enabled' switched off?

Comment 11

[Mic Sparrow]

Any update on the progress of this bug?

Comment 12

[Daniel Bodea [:danibodea]]

Yes, I confirm that if the "dom.security.secFetch.enabled" pref is switched to false, the original issue does not occur anymore.

Comment 13

[Tom Tung [:tt, :ttung]]

(In reply to Bodea Daniel [:danibodea] from comment #12)

Yes, I confirm that if the "dom.security.secFetch.enabled" pref is switched to false, the original issue does not occur anymore.

Since it's related to the "dom.security.secFetch.enabled" pref, would you mind taking a look into this bug, Christoph?

Comment 14

[Christoph Kerschbaumer [:ckerschb]]

(In reply to Tom Tung [:tt, :ttung] from comment #13)

Since it's related to the "dom.security.secFetch.enabled" pref, would you mind taking a look into this bug, Christoph?

I am willing to help, but I guess it's not a Firefox bug. More precisely, when enabled Firefox sends the Sec-fetch-* HTTP request headers [1]. It seems the problem must be on the server side of the specific app; in the case reported within http://gauntface.github.io/simple-push-demo/

Using a different demo, e.g: https://pushalert.co/demo and pressing 'Send Notification' - push notifications work as expected!

[1] https://w3c.github.io/webappsec-fetch-metadata/

Comment 15

[Jens Stutte [:jstutte]]

Hi Mic, I am closing this then but feel free to come back in case. Thank you for reporting!

Comment 16

[Mic Sparrow]

Ok I understand but isn't that strange that notifications from sites like YouTube and Facebook are not appearing?

Comment 17

[Edouard Oger [:eoger]]

Bug 1508292 broke send tab and I would guess Push Notifications in general. See Push log from Firefox Nightly startup:

PushServiceWebSocket: onTimerFired: Reconnecting after backoff PushServiceWebSocket.jsm:182
PushServiceWebSocket: beginWSSetup() PushServiceWebSocket.jsm:508
PushServiceWebSocket: beginWSSetup: Connecting to wss://push.services.mozilla.com/ PushServiceWebSocket.jsm:532
PushServiceWebSocket: wsOnStop() PushServiceWebSocket.jsm:1081
PushServiceWebSocket: wsOnStop: Reconnecting after socket error 2152398861 PushServiceWebSocket.jsm:1084
PushServiceWebSocket: reconnect() PushServiceWebSocket.jsm:347
PushServiceWebSocket: shutdownWS() PushServiceWebSocket.jsm:353
PushServiceWebSocket: startBackoffTimer() PushServiceWebSocket.jsm:417
PushServiceWebSocket: startBackoffTimer: Retry in 40000 Try number 4 PushServiceWebSocket.jsm:426

And mozregression:

 6:58.96 INFO: Narrowed integration regression window from [23c4c99d, a0508ae6] (3 builds) to [f6b26481, a0508ae6] (2 builds) (~1 steps left)
 6:58.96 INFO: No more integration revisions, bisection finished.
 6:58.96 INFO: Last good revision: f6b264818bbfc385da9010ed49dfecf2efd436f3
 6:58.96 INFO: First bad revision: a0508ae6c037928981ac2733860b6ec84d7069ec
 6:58.96 INFO: Pushlog:
https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=f6b264818bbfc385da9010ed49dfecf2efd436f3&tochange=a0508ae6c037928981ac2733860b6ec84d7069ec

Comment 19

[Christoph Kerschbaumer [:ckerschb]]

Hey Edouard, I am not sure what I can do here to help. FWIW, adding Sec-Fetch-* headers (implemented in Bug 1508292 ) is behind the pref 'dom.security.secFetch.enabled' which is only enabled in Nightly and we will not let it ride the trains for a while. I added some more information about what I think the problem is within comment 14 - if there is anything else I can be helpful with, please let me know.

Comment 20

[JR Conlin [:jrconlin,:jconlin]]

:ckerschb I don't think this is a server bug.

Autopush (the mozilla push server), hasn't changed since October of last year.
I know for a fact that it does not use any of the Sec-Fetch-* headers since I just found out about them today while working with the client team on this issue. In addition, if it were a server side issue, sites like https://pushalert.co/demo would also fail. I, too, am confused why that site works yet older code like https://serviceworke.rs/push-simple_demo.html does not. I'm able to connect to the push servers using a python websocket client:

>pypy/bin/wsdump.py wss://push.services.mozilla.com/
Press Ctrl+C to quit
> {"messageType":"hello","use_webpush":true}
< {"messageType":"hello","uaid":"f916f2c314e54c579d4e598a6cb38bc1","status":200,"use_webpush":true,"broadcasts":{}}
>

Looking at the client code that appears correct and also hasn't changed in quite some time.

I'll admit that my expertise is in back end and not user agent code, so I'm at a loss to figure out why a connection under nightly is clearly failing.
I know that the server will reject any connection that initially sends empty or invalid data, but again, the code linked above does not appear to do that (unless there's some recent change inside of Ci.nsIWebSocketChannel)

I think we're just at a loss to determine what might be causing this sudden failure.

Comment 21

[Christoph Kerschbaumer [:ckerschb]]

(In reply to JR Conlin [:jrconlin,:jconlin] from comment #20)

Autopush (the mozilla push server), hasn't changed since October of last year.
I know for a fact that it does not use any of the Sec-Fetch-* headers since I just found out about them today while working with the client team on this issue.

I am not saying the server is making use of the sec-fetch-* headers, but maybe it fails to ignore them properly?

I think we're just at a loss to determine what might be causing this sudden failure.
I really can't imagine why the user agent is causing the problem here, we are really just setting new http request headers, that's it.

Maybe Dan can help us identify the problem - Dan, any suggestions?

Comment 22

[Daniel Veditz [:dveditz]]

Several of the "working" examples are using Notifications, not push. Others are sending pushes to their own sites -- are they setting it up in the backend? Do they later have their own communication when you're on the site that tells the site to send a Notification while you're on it? I didn't see any that sent me a "push" when I wasn't on the site itself.

The only "broken" one I've seen so far is the https://gauntface.github.io/simple-push-demo/ one from earlier. Unlike comment 6 it worked just fine in Firefox 74. It's also the only demo site I've seen making requests to wss://push.services.mozilla.com/. With the sec-fetch- headers that request doesn't get any response at all. With the sec-fetch headers turned off then I get the expected 101 response and the websocket set-up HTTP headers.

Is there a load-balancer or something in front of that service that's choking on the sec-fetch headers? Is the server itself looking for Sec- headers instead of sec-websocket- headers and getting confused that there are too many of them?

Comment 23

[JR Conlin [:jrconlin,:jconlin]]

The server is not looking for any Sec-* or sec-websocket-* headers. I'm quite positive about that because I'm one of the authors.

I can run some experiments, but it would be helpful to have candidate headers I can use to ping off of the server.

(As an aside, in the future it would be super useful if server impacting changes to the client like these could be be run by the Services Engineering group (services-engineering@/ slack#services-engineering). We can notify other services groups if need be, but it would give us a good start.)

Comment 24

[Christoph Kerschbaumer [:ckerschb]]

(In reply to JR Conlin [:jrconlin,:jconlin] from comment #23)

(As an aside, in the future it would be super useful if server impacting changes to the client like these could be be run by the Services Engineering group (services-engineering@/ slack#services-engineering). We can notify other services groups if need be, but it would give us a good start.)

I am really sorry about that. To emphasize, the pref dom.security.secFetch.enabled is currently only enabled in Nightly. If needed we could even flip it to false in Nightly. We are not planning to let it ride the trains in the super near future. Let me know!

Comment 25

[JR Conlin [:jrconlin,:jconlin]]

(In reply to Christoph Kerschbaumer [:ckerschb] from comment #24)
I am really sorry about that. To emphasize, the pref dom.security.secFetch.enabled is currently only enabled in Nightly. If needed we could even flip it to false in Nightly. We are not planning to let it ride the trains in the super near future. Let me know!

Totally understandable, and these sorts of things can and will happen. As I know you feel intimately, there's an endless tsunami of specifications, features and changes that don't always get propagated to where they should and not everyone always has the bandwidth to monitor every change coming down the pike. It can often be better to over communicate rather than assume "they'll find out, I suppose". Just offering a bucket you can yell into if and when you see something that might be a concern in the future.

Your humble server folk thank you.

Comment 26

[Daniel Veditz [:dveditz]]

Using the example at https://serviceworke.rs/push-simple_demo.html I see the same behavior. The headers I get when there's no response:

Accept: */*
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.5
Cache-Control: no-cache
Connection: keep-alive, Upgrade
Host: push.services.mozilla.com
Origin: wss://push.services.mozilla.com/
Pragma: no-cache
Sec-Fetch-Dest: websocket
Sec-Fetch-Mode: websocket
Sec-Fetch-Site: cross-site
Sec-WebSocket-Extensions: permessage-deflate
Sec-WebSocket-Key: eyGFCl0x3LtrfBzjknmISg==
Sec-WebSocket-Protocol: push-notification
Sec-WebSocket-Version: 13
Upgrade: websocket
User-Agent: Mozilla/5.0 (Macintosh; Intel …) Gecko/20100101 Firefox/76.0

If I turn off the Sec-Fetch- headers (pref dom.security.secFetch.enabled) it works fine and I get the 101 response with the upgrade response headers.

Comment 27

[JR Conlin [:jrconlin,:jconlin]]

Ok, so doing a few experiments using a python web client:

> pypy/bin/wsdump.py -vvv --headers="Sec-Fetch-Dest: websocket,Sec-Fetch-Mode: websocket,Sec-Fetch-Site: cross-site,Sec-WebSocket-Extensions: permessage-deflate,Sec-WebSocket-Protocol: push-notification" wss://push.services.mozilla.com
--- request header ---
GET / HTTP/1.1
Upgrade: websocket
Connection: Upgrade
Host: push.services.mozilla.com
Origin: http://push.services.mozilla.com
Sec-WebSocket-Key: EXTeF8flvgcy/dU5uWp0iQ==
Sec-WebSocket-Version: 13
Sec-Fetch-Dest: websocket
Sec-Fetch-Mode: websocket
Sec-Fetch-Site: cross-site
Sec-WebSocket-Extensions: permessage-deflate
Sec-WebSocket-Protocol: push-notification


-----------------------
--- response header ---
HTTP/1.1 101 Switching Protocols
Connection: Upgrade
Upgrade: websocket
Sec-WebSocket-Accept: fj9u9cHokh28aUGs4HJJj2PiHRM=
-----------------------
Press Ctrl+C to quit
> {"messageType":"hello","use_webpush":true}
send: '\x81\xaam\xedTN\x16\xcf9+\x1e\x9e5)\x08\xb9->\x08\xcfnl\x05\x888"\x02\xcfxl\x18\x9e1\x11\x1a\x886>\x18\x9e<lW\x99&;\x08\x90'
< text: {"messageType":"hello","uaid":"6db91383b70b49f8b06aec12b2e00bfb","status":200,"use_webpush":true,"broadcasts":{}}
>

This is a successful connection to our push servers that uses the example headers provided in the above comment. I don't believe that the server or anything in between is blocking the connection from working. (note that the send:... is the encrypted data being relayed over websocket.)

Please also note that the sending payload ({"messageType":"hello","use_webpush":true} must be entered within 2 seconds of connection otherwise the server will aggressively timeout the connection

Hopefully, this can also act as a connection proof for later integration testing verification.

Comment 28

[Daniel Veditz [:dveditz]]

The sec-fetch- code in the browser is very simple, just a pref check and a series of calls to aHTTPChannel->SetRequestHeader() which obviously succeed.

Is there an internet path issue, and it might be different from your machine to the server? If you try one of the known-broken push examples in Firefox do they work in your nightly? https://gauntface.github.io/simple-push-demo/ seems easier to use since it has a send button. In the servicework.rs example you have to reload the page for each attempt. Is the Browser Console lying about what was actually sent and received? I haven't looked at the actual traffic.

Comment 29

[JR Conlin [:jrconlin,:jconlin]]

I'm doing the tests from the same machine that I am running nightly on, so the pathing should be the same.

Attempting to establish a new push subscription to the test site fails when the GET wss://push.services.mozilla.com/ does not return a response within the browser.

Making the same call using the same header values outside of firefox nightly succeeds as

I don't have the ability to delve into the active User Agent Push internals but there are a few possible reasons for the failure:

1. The client is failing to send the push "hello" packet for some reason.
2. The client is failing to accept the push "hello" response for some reason.
3. Something in between the WebPush.jsm calls outlined in #20 and the actual websocket network dispatch is intercepting or preventing the data.

The push server is deeply, deeply indifferent about any additional header values. It ignores anything that is unknown or unexpected.

As noted, setting the dom.security.secFetch.enabled to false allows the socket data to be exchanged and the push service to resume. (I'll note that even in the case of a success, the browser console does not display the exchanged information, so I can only verify off of expected DOM event behaviors.) Therefore, I strongly assert that this is not a server based issue, and urge examining anything that may interfere with either websocket data transmission or reception, particularly around elements related to the dom.security.secFetch component.

Comment 30

[Christoph Kerschbaumer [:ckerschb]]

Attachment: Bug 1623400: Test SecFetch Headers and websockets.

Comment 31

[Christoph Kerschbaumer [:ckerschb]]

I did a lot of testing trying to narrow down the problem, here is what I found out:

a) I wrote a mochitest (see phabricator) which creates a websocket also attaching the three Sec-Fetch-Headers.
The details about the websocket:

URI: https://example.com/tests/dom/security/test/general/file_sec_fetch_websocket
Sec-Fetch-Dest: websocket
Sec-Fetch-Mode: websocket
Sec-Fetch-Site: cross-site

Running the mochitest -> works

b) I tested https://gauntface.github.io/simple-push-demo/ using all kinds of settings:
The details about the websocket:

URI: https://push.services.mozilla.com/
Sec-Fetch-Dest: websocket
Sec-Fetch-Mode: websocket
Sec-Fetch-Site: cross-site

I tested sending all three headers -> failure
I tested sending only the first two headers -> works
I tested sending only the third header -> works

Is it possible that there are some server side size constraints on request headers? Given that I also wrote a mochitest which works correctly, I still think the problem must be somewhere on the server.

Comment 32

[JR Conlin [:jrconlin,:jconlin]]

This is why I was using something other than firefox in order to test the server. In my case I was using a python based websocket client app that is part of the websocket_client package. There are other tools that can be used as well, such as websocat which runs on a wide variety of platforms. This enables me to test the server directly.

Using websocat from my windows platform:

> .\websocat_win64.exe -vvv --header="Sec-Fetch-Dest: websocket" --header="Sec-Fetch-Mode: websocket" --header="Sec-Fetch-Site: cross-site" --header="Sec-WebSocket-Extensions: permessage-deflate" --header="Sec-WebSocket-Protocol: push-notification" wss://push.services.mozilla.com
[INFO  websocat::lints] Auto-inserting the line mode
[DEBUG websocat] Done third phase of interpreting options.
[DEBUG websocat] Done fourth phase of interpreting options.
[DEBUG websocat] Preparation done. Now actually starting.
[INFO  websocat::sessionserve] Serving Line2Message(ThreadedStdio) to Message2Line(WsClient("wss://push.services.mozilla.com/")) with Options { websocket_text_mode: true, websocket_protocol: None, websocket_reply_protocol: None, udp_oneshot_mode: false, unidirectional: false, unidirectional_reverse: false, exit_on_eof: false, oneshot: false, unlink_unix_socket: false, exec_args: [], ws_c_uri: "ws://0.0.0.0/", linemode_strip_newlines: false, linemode_strict: false, origin: None, custom_headers: [("Sec-Fetch-Dest", [119, 101, 98, 115, 111, 99, 107, 101, 116]), ("Sec-Fetch-Mode", [119, 101, 98, 115, 111, 99, 107, 101, 116]), ("Sec-Fetch-Site", [99, 114, 111, 115, 115, 45, 115, 105, 116, 101]), ("Sec-WebSocket-Extensions", [112, 101, 114, 109, 101, 115, 115, 97, 103, 101, 45, 100, 101, 102, 108, 97, 116, 101]), ("Sec-WebSocket-Protocol", [112, 117, 115, 104, 45, 110, 111, 116, 105, 102, 105, 99, 97, 116, 105, 111, 110])], custom_reply_headers: [], websocket_version: None, websocket_dont_close: false, one_message: false, no_auto_linemode: false, buffer_size: 65536, broadcast_queue_len: 16, read_debt_handling: Warn, linemode_zero_terminated: false, restrict_uri: None, serve_static_files: [], exec_set_env: false, reuser_send_zero_msg_on_disconnect: false, process_zero_sighup: false, process_exit_sighup: false, socks_destination: None, auto_socks5: None, socks5_bind_script: None, tls_domain: None, tls_insecure: false, headers_to_env: [], max_parallel_conns: None, ws_ping_interval: None, ws_ping_timeout: None }
[INFO  websocat::stdio_threaded_peer] get_stdio_peer (threaded)
[DEBUG websocat::sessionserve] Underlying connection established
[INFO  websocat::ws_client_peer] get_ws_client_peer
[INFO  websocat::ws_client_peer] Connected to ws
{"messageType":"hello","use_webpush":true}
[DEBUG websocat::ws_peer] incoming text
{"messageType":"hello","uaid":"7c8e82bae21d4fc2bacce8ad63d7d9b4","status":200,"use_webpush":true,"broadcasts":{}}
^D
[DEBUG websocat::ws_peer] incoming close
[DEBUG websocat::my_copy] BrokenPipe: read_done
[DEBUG websocat::my_copy] done
[INFO  websocat::sessionserve] Reverse finished
[INFO  websocat::sessionserve] Reverse shutdown finished
[DEBUG websocat::my_copy] zero len
[DEBUG websocat::my_copy] read_done

While it's a bit hard to read through that output with the verbosity set high, note that I entered
{"messageType":"hello","use_webpush":true} (which is a proper WebPush "hello" message)
and received back:
{"messageType":"hello","uaid":"7c8e82bae21d4fc2bacce8ad63d7d9b4","status":200,"use_webpush":true,"broadcasts":{}}
which is the expected response.

Note also that I used the following --header arguments and that they were relayed in the websocat::sessionserve body:

• Sec-Fetch-Dest: websocket
• Sec-Fetch-Mode: websocket
• Sec-Fetch-Site: cross-site
• Sec-WebSocket-Extensions: permessage-deflate
• Sec-WebSocket-Protocol: push-notification

These include the headers you note. including Sec-Fetch-Dest: websocket which you report as not working.

(The sessionserve message transcribes the values of these strings into byte arrays, so some use of an ascii chart may be required to verify.)

What this means is that a simple client that specifies these headers and values is indeed able to send and receive data from the mozilla WebPush servers.

Look, I don't want to be contrarian or sound overly negative here, but I feel like we're arguing that the road must be broken because your car won't start after you changed the oil. I understand that firefox is a highly complex virtual machine. Far more complex than the push server. That's why I'm trying to provide as much tooling as possible so that you can solve this problem. It may not be with the code responsible for writing these headers (again, highly complex virtual machine), but it could be surfacing because of some inter-relationship around how the browser should act regarding handling these headers. I don't know, either.

In fact, if you like, you can run a fake Push Server locally, and alter dom.push.serverURL to point to it. (Websocket is an upgraded HTTP connection, so the server should listen on ports 80 for (ws://) or 443 (for wss://) just like a normal HTTP server.) The client should connect and send the "hello" message I've been using. (I'll remind that firefox tends to treat localhost specially, so it might be worth trying a different address as well.)

I am not sure how I can possibly proceed on this further. I completely agree that firefox 76.0a1 (2020-03-18+) is unable to properly communicate to the server in order to initiate a push connection. This is not the case for older versions of firefox, or independent clients. I have neither the tools nor experience in order to work out why the internal websocket channels within firefox in order to further diagnose or resolve this issue, but continue to be happy to assist where I can. I will gleefully fix and frankly be in rapture if this does turn out to be a server side issue but I am unable to replicate it using anything other than firefox 76.0a1 (2020-03-18+) which, again, leads me to believe that the problem is not with the server.

Comment 33

[Christoph Kerschbaumer [:ckerschb]]

Dragana, Valentin, I'll do more debugging as I have time. Anything you could think of why websockets would start failing because we are adding sec-fetch-* request headers? See comment 31, for details about testing. If there is anything you could think of, please let me know. I tried all the obvious places, if you have any pointers, please let me know.

Comment 34

[Dragana Damjanovic [:dragana]]

Do you know if we are sending anything from Firefox? Do you know where exactly it is failing, e.g. we do not send a request or we send a request but server is not responding?
The comment 29 is asking similar questions.

Comment 35

[Daniel Veditz [:dveditz]]

(In reply to JR Conlin [:jrconlin,:jconlin] from comment #32)

This is why I was using something other than firefox in order to test the server. In my case I was using a python based websocket client app that is part of the websocket_client package.

Your test in comment 27 did not send as many headers as the browser does in comment 26. If it's a size or # of packets issue, rather than sec-fetch- specifically, that could make a difference.

Comment 36

[JR Conlin [:jrconlin,:jconlin]]

(In reply to Daniel Veditz [:dveditz] from comment #35)

(In reply to JR Conlin [:jrconlin,:jconlin] from comment #32)

This is why I was using something other than firefox in order to test the server. In my case I was using a python based websocket client app that is part of the websocket_client package.

Your test in comment 27 did not send as many headers as the browser does in comment 26. If it's a size or # of packets issue, rather than sec-fetch- specifically, that could make a difference.

Ah, indeed! Awesome, I've got a reproducible error.

using the following script:

    --header="Accept: */*" \
    --header="Cache-Control: no-cache" \
    --header="Host: push.services.mozilla.com" \
    --header="Origin: wss://push.services.mozilla.com/" \
    --header="Pragma: no-cache" \
    --header="Sec-Fetch-Dest: websocket" \
    --header="Sec-Fetch-Mode: websocket" \
    --header="Sec-Fetch-Site: cross-site" \
    --header="Sec-WebSocket-Extensions: permessage-deflate" \
    --header="Sec-WebSocket-Protocol: push-notification" \
    --header="Accept-Language: en-US,en;q=0.5" \
    --header="Accept-Encoding: gzip, deflate, br" \
    wss://push.services.mozilla.com

succeeds, but adding an additional header

echo '{"messageType":"hello","use_webpush":true}' | /usr/bin/websocat -v \
    --header="Accept: */*" \
    --header="Cache-Control: no-cache" \
    --header="Host: push.services.mozilla.com" \
    --header="Origin: wss://push.services.mozilla.com/" \
    --header="Pragma: no-cache" \
    --header="Sec-Fetch-Dest: websocket" \
    --header="Sec-Fetch-Mode: websocket" \
    --header="Sec-Fetch-Site: cross-site" \
    --header="Sec-WebSocket-Extensions: permessage-deflate" \
    --header="Sec-WebSocket-Protocol: push-notification" \
    --header="Accept-Language: en-US,en;q=0.5" \
    --header="Accept-Encoding: gzip, deflate, br" \
    --header="Jabberwocky: T'was" \
    wss://push.services.mozilla.com

causes the server to fail the connection. In this case I'm using a nonce header in place of the User-Agent string.

@oremj are you aware of any ELB header restrictions which might cause these sorts of failures?

Comment 37

[JR Conlin [:jrconlin,:jconlin]]

Filed: https://github.com/mozilla-services/autopush-rs/issues/136

Comment 38

[JR Conlin [:jrconlin,:jconlin]]

So as a bit of follow up here while I enjoy this plate of crow:

In the rust application, we use httparse, which parses and reads the HTTP headers, but uses a user provided mut slice. (e.g.

use httparse;

let mut headers = [httparse::EMPTY_HEADER; 16];
let mut req = httparse::Request::new(&mut headers);

while 16 may have been plenty in the past, the new headers pushed us to 17. Literally one more than we expected, which triggered an out of bounds error that resulted in a too many headers debug message that got dropped because we don't log those.

The very short term fix is to just double the number of accepted headers (to 32).

Comment 39

[Christoph Kerschbaumer [:ckerschb]]

(In reply to JR Conlin [:jrconlin,:jconlin] from comment #38)

The very short term fix is to just double the number of accepted headers (to 32).

Really glad that we figured out the problem. I suppose you'll keep this bug to fix the number of accepted headers? If so, I'll file a new bug for the test to land, since we already have it. If not, please assign the bug to me and I'll land the test within this bug. I am fine either way. Thank you!

Comment 40

[JR Conlin [:jrconlin,:jconlin]]

We tend to use github issues for tracking service projects, so whether or not this bug remains open is not really a concern. If this bug is reassigned or closed, I would very much appreciate getting a crosslink to https://github.com/mozilla-services/autopush-rs/issues/136

Comment 41

[Jens Stutte [:jstutte]]

(In reply to Christoph Kerschbaumer [:ckerschb] from comment #39)

(In reply to JR Conlin [:jrconlin,:jconlin] from comment #38)

The very short term fix is to just double the number of accepted headers (to 32).

Really glad that we figured out the problem. I suppose you'll keep this bug to fix the number of accepted headers? If so, I'll file a new bug for the test to land, since we already have it. If not, please assign the bug to me and I'll land the test within this bug. I am fine either way. Thank you!

If I understand correctly, in any case for the DOM: Push Notifications component there is nothing left to do here, right? So if you keep the bug, please change the component accordingly. Thank you!

Comment 42

[Christoph Kerschbaumer [:ckerschb]]

(In reply to Jens Stutte [:jstutte] from comment #41)

If I understand correctly, in any case for the DOM: Push Notifications component there is nothing left to do here, right? So if you keep the bug, please change the component accordingly. Thank you!

I think that's a question for JR Conlin.

Comment 44

[JR Conlin [:jrconlin,:jconlin]]

Yep. The problem is with the server accepting a limited number of headers. A quick patch is included in

https://github.com/mozilla-services/autopush-rs/releases/tag/1.55.0
which is part of
https://bugzilla.mozilla.org/show_bug.cgi?id=1629973

Comment 45

[Jens Stutte [:jstutte]]

Hi JR, (as bug 1627794 seems to be a duplicate of this) is this a common problem with the default settings of (some) servers? Do we know with which? I assume that then many websites can be bitten by this. In this case we probably should understand, to which extent we are breaking the web with this feature and what we can do about it.

Comment 46

[JR Conlin [:jrconlin,:jconlin]]

Correct, this error is on the server and is triggered by more than 16 headers.

It has been addressed by https://github.com/mozilla-services/autopush-rs/releases/tag/1.55.0

In our case, we were using a rust library called "httparse" that for reasons takes a pre-allocated set of header slots and populates them. Since we had no idea how many headers we were going to be using, we used the demo code value of "16". Everything worked just fine for a long time, until the new set of headers pushed us over by literally one header. That's when things broke for us.

I've filed an issue with that packages authors to have them tweak the demo number up a significant chunk.

I don't think it's possible to say how end sites might be impacted, since there's not a common set of actions around header processing. I'm sure that many sites will simply ignore these headers and work fine. Others might hit the new header and raise warnings, and I'm sure that some will fail for similar reasons (or maybe the request will exceed whatever buffer size they originally allocated and it'll fail for that reason).

Honestly, the best we can do is to better advertise these headers out to server folk. I can tell you that we don't generally follow header changes with slavish devotion because we're trying to figure out why the foo-baz library 0.192.3 breaks code written with foo-baz library 0.192.2 and who the hell is using a library named foo-baz since all the documentation just says "TBD".

Comment 47

[Daniel Veditz [:dveditz]]

Nightly is now working with the following demo sites that weren't working before:

• https://gauntface.github.io/simple-push-demo/
• https://serviceworke.rs/push-simple_demo.html

Comment 48

[Daniel Veditz [:dveditz]]

(In reply to Jens Stutte [:jstutte] from comment #45)

(as bug 1627794 seems to be a duplicate of this) is this a common problem with the default settings of (some) servers? Do we know with which? I assume that then many websites can be bitten by this. In this case we probably should understand, to which extent we are breaking the web with this feature and what we can do about it.

Any site using the Push API in Firefox causes Firefox to connect to the Mozilla push server -- so they all fail the same way and are now all fixed.

Other sites not using Push also appear to have been broken by adding the sec-fetch- headers. It may well be the same kind of issue but would have to be addressed individually at each server. These are tracked as separate regressions from bug 1508292 and will need outreach.