1551315 - Large URL Parameter Causes 502 Bad Gateway Error on addons.thunderbird.net

Status: RESOLVED WONTFIX

(Thunderbird :: Untriaged, defect)

Description

[frontierpsychiatrist-bugzilla]

Attachment: URLs that display 502 Bad Gateway error after redirect

While investigating the URL parameter doubling in bug #1550398, I found that a large URL (parameter) causes a 502 Bad Gateway error on addons.thunderbird.net.

The URL parameter doubling causes the first URL (addons.mozilla.org, 8203 bytes, parameter 8157 bytes) in the attachment to redirect to addons.thunderbird.net, which displays a Cloudflare error page every time since the doubled parameter is too long. Adding a single byte to the first URL causes a 414 Request-URI Too Large error on the AMO Server.

The second URL in the attachment (addons.mozilla.org, 7936 bytes, parameter 7890 bytes) has the same issue, but removing a few bytes causes it to redirect successfully. The exact number of bytes you should remove seems to vary (depending on the total number of request bytes?).

A large request made directly to addons.thunderbird.net directly also has the same issue. However, I am having some trouble identifying the exact number of bytes needed, due to Cache-Control headers and other differences between subsequent requests.

Comment 1

[Greg G]

Thanks for the report frontierpsychiatrist-bugzilla! This is possibly interesting, but expected behavior. What's the security impact?

Comment 2

[frontierpsychiatrist-bugzilla]

I;m not sure what occurs server-side, but the expected behavior would be a 414 error instead of the actual 502 error. This made me think something unusual (e.g. memory corruption) was going on server-side. I obviously can't tell for sure and didn't think further testing on a production system was a good idea. My theory would be that crafted request data could lead to DoS or RCE.

Comment 3

[Greg G]

(In reply to frontierpsychiatrist-bugzilla from comment #2)

I;m not sure what occurs server-side, but the expected behavior would be a 414 error instead of the actual 502 error.

OK yep that makes sense the target servers should return 414s

This made me think something unusual (e.g. memory corruption) was going on server-side. I obviously can't tell for sure and didn't think further testing on a production system was a good idea. My theory would be that crafted request data could lead to DoS or RCE.

Oh yeah, I definitely appreciate your caution there. Feel free to test against the stage AMO server (allizom is mozilla backwards) at: https://addons.allizom.org/

Comment 4

[frontierpsychiatrist-bugzilla]

Attachment: Archive 19-05-14 00-31-31.har

HAR file for a request that triggers the error.

Comment 5

[frontierpsychiatrist-bugzilla]

ServerFault says that something on your backend is crashing that nginx proxies requests for, so a 502 error apppears. Without access to the nginx logs, I think there's no way for me to exploit this. It may still be exploitable with more knowledge of the backend service, though. You may wish to close this as not a security bug, or else reassign this to the appropriate backend team.

A (much) longer URL does cause a 414 error as expected. The bug to be fixed is whatever crashes for URLs shorter than the 414 limit. For Tor Browser, 15784 URL bytes are needed to trigger the error (for https://addons.thunderbird.net/en-US/thunderbird/? with 15734 byte parameter). Attached is a HAR file of a request that fails.

Please let me know if you have further questions, otherwise I'll let you close/reassign this bug as necessary.

Comment 6

[frontierpsychiatrist-bugzilla]

I just found the AMO addons-server repo on Github - is addons.thunderbird.net also running that Django app, and is it the same version? If yes, I can setup a copy locally and test on that.

To clarify, the bug occurs only on addons.thunderbird.net, not AMO or the stage AMO server. addons.allizom.net/en-US/thunderbird/ redirects to addons.thunderbird.net, so testing the live system is not possible (production server).

Comment 7

[Greg G]

(In reply to frontierpsychiatrist-bugzilla from comment #6)

I just found the AMO addons-server repo on Github - is addons.thunderbird.net also running that Django app, and is it the same version? If yes, I can setup a copy locally and test on that.

yep! https://github.com/mozilla/addons-server is in https://addons.thunderbird.net/contribute.json

However it looks they're running different versions. I'm seeing:

2019.05.16 in https://addons.mozilla.org/__version__ and no version in https://addons.thunderbird.net/__version__

To clarify, the bug occurs only on addons.thunderbird.net, not AMO or the stage AMO server. addons.allizom.net/en-US/thunderbird/ redirects to addons.thunderbird.net, so testing the live system is not possible (production server).

Yeah, I'm not sure how addons.thunderbird.net is configured.

Comment 8

[Greg G]

Moving this to the Thunderbird product, since I'm not sure who runs that site and I don't see a server component.

Comment 9

[Al Billings [:abillings - ex-MoCo]]

Minusing for bounty. Thunderbird sites are community maintained and not covered in the list of sites for the bounty program.

Comment 10

[Daniel Veditz [:dveditz]]

I;m not sure what occurs server-side, but the expected behavior would be a 414 error instead of the actual 502 error.

OK yep that makes sense the target servers should return 414s

Servers typically return 502 Bad Gateway when HTTP headers are unreasonably long, where "unreasonable" is defined by the server but typically in the 8K to 16K range. doesn't hurt the server, it's just the server rejecting things it doesn't want to deal with. Switching server software (e.g. from Apache to Node, or whatever it is) will only change the specific size it gets unhappy about but not the basic issue.

You can play with this on pretty much any site, by, say, adding a bunch of cookies and seeing when it stops working.

Comment 11

[Andrei Hajdukewycz [:sancus]]

Thanks to whomever CCed me on this. I maintain the Thunderbird sites currently, for future reference.

I have spent some time investigating this, and as far as I can tell, it's the Amazon ELB itself that returns a 502 on these crazy long URLs -- the request doesn't even show up in logs at all on the servers behind the ELB.

I don't think it is worth the time to go down the rabbit hole of why the ELB does this. Regardless, nothing untowards is going on server side, nothing is crashing, in fact the servers don't even know these weird requests are being made. So I don't really care what error number the ELB returns :)

Comment 12

[frontierpsychiatrist-bugzilla]

Thanks for following up. Can you please tell me what version of addons-server is running behind the ELB? I'd still like to install and test my own copy.

Comment 13

[Andrei Hajdukewycz [:sancus]]

https://github.com/thundernest/addons-server