1672185 - Issue with Firefox AES-GCM ECE WebCrypto

Status: RESOLVED DUPLICATE of bug 1673340

(Core :: Networking: WebSockets, defect)

Description

[Tim Visée]

User Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:83.0) Gecko/20100101 Firefox/83.0
Firefox for Android

Steps to reproduce:

It looks like I'm seeing cryptography errors in the latest Firefox Nightly release. I assume there to be an issue with AES-GCM ECE crypto.

Discovered this about a week back, when uploading through Send (uses client-side encryption) fails. Only Nightly fails, Firefox stable and other browsers work fine.

To reproduce:

• Open Firefox Nightly 83
• Go to: https://send.vis.ee/ (a public Send instance I'm hosting)
• Upload a file
• Attempt to download, it fails

My system:

• Ubuntu 20.04.1 LTS 64-bit, Linux
• Firefox Nightly 83.0a1 (2020-10-19) (64-bit)
• Using the firefox-trunk package from Mozilla's PPA.

More information on the crypto being used:
https://github.com/timvisee/send/blob/master/docs/encryption.md

Related issue on Send where I discovered this:
https://gitlab.com/timvisee/send/-/issues/1

Although this happens in Send, I'm quite sure the root cause is a Firefox issue, because it only occurs when uploading through Firefox Nightly. Sorry if this issue is vague, I'm not sure how to pinpoint it to a specific Firefox module, nor am I sure on how to debug this.

Actual results:

The upload seems to succeed. However, downloading the file (in any browser) fails because decryption fails.

Expected results:

The upload succeeds. Downloading it again in any browser should succeed.

Comment 1

[Tim Visée]

Some additional details:

• Using Firefox stable, or any other browser, succeeds
• Using Firefox nightly, it fails
• The upload using Firefox nightly seems to succeed, but when downloading (in any browser) decryption fails because the upload is corrupt
• I do not see (crytpo) errors in the developer tools console
• I do not see (crypto) errors in stdout/stderr
• I am only able to test this on Linux

Comment 2

[J.C. Jones [:jcj] (he/him)]

Thanks for reporting, Tim!

So we've tested Nightly ourselves and, like the other commenter at Gitlab, didn't run into any issues. We haven't changed anything related to our AES acceleration nor our WebCrypto implementation in several releases, so the first thing that comes to mind for us is that maybe your copy of Firefox is using an unexpected version of NSS? I know you said it is packaged by the Mozilla PPA, but on Ubuntu, Firefox can still use the OS-supplied copied of NSS under some circumstances. So, could you give us the "Library Versions" listed in about:support (or just copy the whole thing, if you'd rather) ?

Thanks!

Comment 3

[Tim Visée]

(In reply to J.C. Jones [:jcj] (he/him) [increased latency due to COVID-19] from comment #2)

We haven't changed anything related to our AES acceleration nor our WebCrypto implementation in several releases

That's useful to know, thanks.

Here are the library versions I'm using ([expected minimum] and in use).

On Firefox nightly which errors:
NSPR: [4.29] 4.29
NSS: [3.58] 3.58
NSSSMIME: [3.58] 3.58
NSSSSL: [3.58] 3.58
NSSUTIL: [3.58] 3.58

On Firefox stable (which works fine):
NSPR: [4.28] 4.28
NSS: [3.56] 3.56
NSSSMIME: [3.56] 3.56
NSSSSL: [3.56] 3.56
NSSUTIL: [3.56] 3.56

Here are some apt show firefox-trunk lines that might be of use:

Package: firefox-trunk
Version: 83.0~a1~hg20201019r553351-0ubuntu0.20.04.1~umd1
Maintainer: Ubuntu Mozilla Team <[e-mail]>
Installed-Size: 224 MB
Depends: lsb-release, libatk1.0-0 (>= 1.12.4), libc6 (>= 2.30), libcairo-gobject2 (>= 1.10.0), libcairo2 (>= 1.10.0), libdbus-1-3 (>= 1.9.14), libdbus-glib-1-2 (>= 0.78), libfontconfig1 (>= 2.12.6), libfreetype6 (>= 2.10.1), libgcc-s1 (>= 3.3), libgdk-pixbuf2.0-0 (>= 2.22.0), libglib2.0-0 (>= 2.37.3), libgtk-3-0 (>= 3.4), libharfbuzz0b (>= 0.6.0), libpango-1.0-0 (>= 1.14.0), libpangocairo-1.0-0 (>= 1.14.0), libpangoft2-1.0-0 (>= 1.14.0), libstdc++6 (>= 9), libx11-6, libx11-xcb1 (>= 2:1.6.9), libxcb-shm0, libxcb1, libxcomposite1 (>= 1:0.4.5), libxcursor1 (>> 1.1.2), libxdamage1 (>= 1:1.1), libxext6, libxfixes3, libxi6, libxrender1, libxt6
Xul-Appid: {ec8030f7-c20a-464f-9b0e-13a3a9e97384}
APT-Manual-Installed: yes
APT-Sources: http://ppa.launchpad.net/ubuntu-mozilla-daily/ppa/ubuntu focal/main amd64 Packages

Thanks for the prompt response, and thanks for helping me debug this.

Comment 4

[Jeremy]

Hello

I'm having the same issue, for me firefox 82 stable, firefox beta 83.0b4 all both FAILED.
Only firefox ESR works

Comment 5

[Tim Visée]

Based on my comment and Jeremy's report here it looks like this started breaking with NSS* 3.57.

Comment 6

[Kevin Jacobs [:kjacobs]]

I haven't been able to reproduce this. I've had a few "Something went wrong" messages appear on the upload page, but there's no indication of a client-side error.

What error are you seeing that points to the decryption failing? A screenshot or a specific message would be helpful.

If this is reliably reproducible for you, could you try running mozregression to find a regression range? Thanks!

Comment 7

[Jeremy]

Hello
I've done it, here result:

Found commit message:
Bug 1663718 - Don't put too much data in buffer when the data can't be written to socket r=dragana

The main reason we used too much memory is that we ignore the NS_BASE_STREAM_WOULD_BLOCK returned from socket output stream.
When the output stream is blocked, all the data is stored in the output queue and make the memory usage high.

Differential Revision: https://phabricator.services.mozilla.com/D89563

Putting screenshot in attachment

Comment 8

[Jeremy]

Attachment: mozregression.png

Comment 9

[Jeremy]

What error are you seeing that points to the decryption failing? A screenshot or a specific message would be helpful.

The process is: uploading using firefox to fsend website and downloading using what-do-want give you corrupted file at the end.

There is no real direct trace that the root cause is firefox, and it's why I opened at first the issue on a tier-ffsend client (https://gitlab.com/timvisee/ffsend/-/issues/105) then we saw that root cause what not this client but ffsend project (https://gitlab.com/timvisee/send/-/issues/1) and only occuring on Firefox 82+ so here we are :)

Comment 10

[Tim Visée]

(In reply to Kevin Jacobs [:kjacobs] from comment #6)

I haven't been able to reproduce this. I've had a few "Something went wrong" messages appear on the upload page, but there's no indication of a client-side error.

What error are you seeing that points to the decryption failing? A screenshot or a specific message would be helpful.

If this is reliably reproducible for you, could you try running mozregression to find a regression range? Thanks!

Yes, the 'Something went wrong' is currently the only indication of an error. And yes, I'm talking about an encryption (upload) issue here, even though this error is shown at decryption (download).

When uploading, the client encrypts and uploads the encrypted data. The ciphertext is somehow corrupted. The client doesn't receive an encryption error notification and thus continues. The server receiving and storing the ciphertext is not able to verify the ciphertext (because it has no way of encrypting without a key). When downloading, the client attempts to decrypt the ciphertext but fails because the ciphertext is corrupt. This is where the client receives a notification of a crypto issue. This is why an error is shown while downloading (the 2nd step), while the actual problem is likely in encryption during upload (the 1st step). I hope this explains it well.

Sadly, Send currently just shows 'Something went wrong' when it receives a decryption error notification. I'm not sure whether any useful crypto error information is dropped. I'm not too familiar with the Send codebase as it's originally Mozilla's project.

I'll try to dive into Send's codebase soon to see if I can catch any useful crypto error information. Both for uploading and downloading. I might attempt to set up a minimal proof of concept that shows this error as well.

Comment 11

[Kevin Jacobs [:kjacobs]]

(In reply to Jeremy from comment #7)

Hello
I've done it, here result:

Found commit message:
Bug 1663718 - Don't put too much data in buffer when the data can't be written to socket r=dragana

The main reason we used too much memory is that we ignore the NS_BASE_STREAM_WOULD_BLOCK returned from socket output stream.
When the output stream is blocked, all the data is stored in the output queue and make the memory usage high.

Differential Revision: https://phabricator.services.mozilla.com/D89563

Putting screenshot in attachment

Thanks! I see that patch was just backed out today for WebSocket problems in bug 1673340. The backout patch hasn't actually shipped in a Nightly release yet, so it would be very helpful to know if the behavior persists in the next build (about:buildconfig will tell you exactly which revision the build uses).

(In reply to Tim Visée from comment #10)

(In reply to Kevin Jacobs [:kjacobs] from comment #6)

Yes, the 'Something went wrong' is currently the only indication of an error. And yes, I'm talking about an encryption (upload) issue here, even though this error is shown at decryption (download).

When uploading, the client encrypts and uploads the encrypted data. The ciphertext is somehow corrupted. The client doesn't receive an encryption error notification and thus continues. The server receiving and storing the ciphertext is not able to verify the ciphertext (because it has no way of encrypting without a key).

Yeah, this is why seeing that error on upload was a bit confusing: download worked fine in all my tests. I'm hopeful the that the issue will be resolved in the next Nightly.

Comment 12

[Tim Visée]

Great work on finding bug 1673340. Uploads are done over a websocket. Definitely sounds plausible this is the issue, instead of cryptography. I'll give the new nightly a spin once it releases and will update this report.

Comment 13

[Tim Visée]

Just gave 84.0a1 (2020-10-28) (64-bit) a spin, and it seems to be working now. Fantastic!

Sorry for the vague report, and for accusing Firefox' crypto.

Thanks for the help! I hope this can be pushed through to stable soon.

Comment 14

[Kevin Jacobs [:kjacobs]]

Great! No worries at all, we appreciate the detailed report :).

Looks like this is fixed for release in 82.0.2: https://www.mozilla.org/en-US/firefox/82.0.2/releasenotes/