1667214 - DuckDuckGo HTML browser search broken due to Origin: null

Status: RESOLVED WORKSFORME

(Firefox :: Search, defect, P1)

Description

[Kevin Locke]

DuckDuckGo HTML (i.e. non-JavaScript) search from the browser bar is broken in the current Nightly. To reproduce:

1. Visit https://html.duckduckgo.com
2. "Add Search Engine" from page actions menu.
3. Search for anything using the newly added "DuckDuckGo HTML".
4. The response is text/plain "forbidden" with code 403.

mozregression indicates the issue was introduced by Bug 1605305:

Last good revision: d7c68a3161242a0442d01f114ca9eebd9e200f8a
First bad revision: f12c895fc5e764414175efe737f8b2f256ea86a7
Pushlog: https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=d7c68a3161242a0442d01f114ca9eebd9e200f8a&tochange=f12c895fc5e764414175efe737f8b2f256ea86a7

To confirm the Origin header is the issue:

$ curl -isS -H 'Origin: null' -d q=test 'https://html.duckduckgo.com/html/' | head -n1
HTTP/2 403

$ curl -isS -d q=test 'https://html.duckduckgo.com/html/' | head -n1
HTTP/2 200

For reference, a similar issue about sending Origin in OpenSearch POST requests (for DuckDuckGo Lite) was addressed in Bug 1615901.

I reported the issue to the DuckDuckGo developers in https://redd.it/iz473c

Comment 1

[:Gijs (he/him)]

Are we expected to send Origin headers in this case, per the spec? We don't for "normal" https navigations that are not POST, as far as I can tell. Maybe we need an exception for system-principal triggered POST navigations?

Comment 2

[:Gijs (he/him)]

(Oh, and thanks for picking this up so quickly, Kevin!)

Comment 3

[Anne (:annevk)]

Fetch does require an Origin header for POST requests to help websites tackle CSRF. It does seem like the pragmatic way forward here is to not include it for system-principal requests, however. Bit on the fence whether that should be documented.

Comment 4

[Pascal Chevrel:pascalc]

Dale, could we get a priority and a severity set for this bug? Do we need to get it fixed before 83 ships? Thanks

Comment 5

[Sebastian Streich [:sstreich]]

The offending patch is currently backed out and there is no plan to land it until this is fixed :)

See https://bugzilla.mozilla.org/show_bug.cgi?id=1605305#c33

Comment 6

[Dale Harvey (:daleharvey)]

I think S1 since breaking duckduckgo is a pretty big deal? but marked P3 since the breakage is backed out

Comment 7

[Pascal Chevrel:pascalc]

Fixed by the backout of bug 1605305