1782775 - Displaying encrypted mail with one 20 MiB (random content) attachment is too slow (15 s)

Status: NEW

(MailNews Core :: Security: OpenPGP, defect)

Description

[u617804]

Attachment: attachment list

Steps to reproduce:

• TB 102.1.0 Linux
• Have computer with CPU 2 core Intel(R) Celeron(R) CPU G540 @ 2.50GHz
• Receive an encrypted and signed mail with little text and four attachments (jpg images), each roughly 5 MB big. (see screenshot)
• Click on the mail in the inbox folder to see it.

Actual results:

• TB hangs for 26 seconds with "Message loading"
• I was confused if TB would have crashed because you do not know how big the mail is
• after long waiting the mail is finally displayed.
• Then double click on the first attachment (in this case IMG_20220802_163504.jpg, 6.5 MB)
• Only after 30 (thirty) seconds, which is longer than displaying the whole mail, the ristretto image viewer opens the one attached image file

Expected results:

Mail display and attachment image opening should happen each in < 5 s

Current situation is nearly unusable experience.

Comment 1

[u617804]

Just did right click on the above mentioned mail and selected "Open as new mail" and sent it away unsigned and unencrypted.
In the "sent" folder the mail is displayed at once (<1 s) and also double clicking on one image works at once (<1 s).

For the long time of the encrypted message loading, one CPU is at 100%.

Comment 2

[Kai Engert [:KaiE:]]

Are you sure the slowness isn't caused by slow internet connection?

Maybe Thunderbird is inefficient and downloads the message repeatedly?

Comment 3

[u617804]

This is for sure an encryption issue, as I have never before faced this slowness and have fast 10 mbit internet connection. As I have written in comment 1 if you open the same mail unencrypted it is fast as known. Also CPU is 100% as reported.
I just reproduecd it with TB 103.0
Just send an encrypted mail with a 20 MB or even 40 MB attachment to yourself and you will see opening the received mail is awfully slow (> 1 min and CPU at 100%) and finally opening the attachment again awfully slow.

Comment 4

[u617804]

Correction: TB 102.3
Also clicking on the mail in the sent folder is awfully slow.

Then right-click the message in the sent folder, select "edit as new" and send again, but this time unencrypted.
Now the upload is significantly faster, and on receive the unencrypted mail is opened fast (< 5s) and opening the attachment also fast (<5 s).

Clearly an encryption performance issue.

Comment 5

[Kai Engert [:KaiE:]]

Do you think this is a regression? It would be helpful to know the latest version that still was fast.

Comment 6

[u617804]

I do not know and do not have the capabibilities to downgrade to earlier versions to check.

Comment 7

[u617804]

Could just reproduce the issue with TB 115.0:

• Did send an encrypted + signed mail with 4 attached JPEG files, each roughly 5 MB size, to myself.
• Click on the "sent" folder and there click on the above sent mail.
• I have to wait 25 seconds until the mail is displayed, one CPU in that time is at 100%
• 1 second after the mail is displayed (e.g. subject, attachments etc.) and CPU is 0%, and you do nothing, the CPU jumps up again to 100% for 25 seconds. I do not know what TB does there.

Comment 8

[Mathias Schindler]

Greetings. This description matches the one I made when reporting bug #1752798 a while ago and the issue remains. The earliest version I encountered the problem was tb 91, to answer your question @:KaiE:, but I cannot rule out the issue wasn't there earlier because I did not use PGP extensively back then.

Mathias

Comment 10

[u617804]

Attachment: mail to reproduce

Could just reproduce the issue with TB 115.5.0 (64-Bit) Linux:

• My CPU (a little faster than in OP): 2 core Intel(R) Celeron(R) G5900T CPU @ 3.20GHz)
• Create attachment files with random content, each 5 MiB size:
  for number in 1 2 3 4 ; do dd if=/dev/urandom of="./attachment${number}.dat" bs=1M count=5 ; done
• Send an encrypted + signed mail with subject: "test", body: "test" and the four above created files attached, to yourself.
• Click on the "sent" folder and there click on the above sent mail.
• I have to wait 15 seconds until the mail is displayed, one CPU in that time is at 100%
• This is no longer the case: 1 second after the mail is displayed (e.g. subject, attachments etc.) and CPU is 0%, and you do nothing, the CPU jumps up again to 100% for 25 seconds.
• After you have received your own mail, you can also click on that mail in the inbox, it takes also 15 s until the mail is displayed

Comment 11

[Nickolay Olshevsky]

Could you please specify which algorithm and key size do you use? RNP was designed with performance in mind, however certain operations, like encryption/signing with large RSA key or unlocking of the key may take few seconds. And if that happens for each attachment separately (not familiar with OpenPGP MIME) this could sum up to 20-25 seconds.
Kai, could you please clarify how much key unlocking and encryption/signing operations would happen for 5 attachments?

Comment 12

[u617804]

Just checked: You can reproduce this bug also with only one 20 MiB attachment, created with
dd if=/dev/urandom of="./attachment.dat" bs=20M count=1
Looks like time until display is even a little more time (17 s) than with the four attachments each of 5 MiB size.

Used key is of type RSA with 3072 bit length (created in 2022, I am pretty sure that I created the key within TB with the default options), it has primary key to sign/verify and a sub key for encryption.

Comment 13

[u617804]

I did some more tests:

The display of the mail is only slow when the mail is encrypted. In that case it does not matter if it is also signed or not, or if the subject is also encrypted or not.

When the mail is only signed (and not encrypted, subject encryption is also not possible in that case), the mail is displayed relatively fast (approx. 0.7 s)

In other words, the reproducability can be shortened to:

• CPU: 2 core Intel(R) Celeron(R) G5900T CPU @ 3.20GHz
• Create an attachment file with random content of 20 MiB size:
  dd if=/dev/urandom of=./attachment.dat bs=20M count=1
• Send an encrypted mail (it does not matter if it is also signed or not, or if the subject is also encrypted or not) with subject: "test", body: "test" and the above created file attached, to yourself.
• Click on the "sent" folder and there click on the above sent mail.

Comment 14

[Nickolay Olshevsky]

Thanks for trying it out. Single RSA 3072 bit key operation + encryption of 20Mb of data by RNP would not take that much time normally. Just curious, what if you'll try 50-100Mb of data if that is possible? Would that change time linearly or even worse?

Comment 15

[u617804]

@Nickolay For this to test I would need a mail receiver address that accepts mail message size >100 MB (for 50 MB attachment due to encoding) respective >200 MB for 100 MB attachment. Unfortunately I could not make my local exim accept those, allthough I had set the message size limit to 200M. Other real internet providers are limited to 30MB attachment. But maybe someone knows a blackhole receiving address for testing purposes like this?

Comment 16

[u617804]

OK one can also reproduce this bug when instead of sending the mail, in the composer click the "save" button, so the mail is stored encrypted in the draft folder. (you need to have "save drafts encrypted" enabled in the account's OpenPGP setting)

With 40 MiB attachment it takes 30 s so that is proprtional to 15 s with 20 MiB attachment.

When I do this with 80 MiB attachment, and then click on the saved mail in the draft folder, the mail is not shown properly ever: 27 s after having clicked on the mail, the CPU usage declines from 100% to normal, and TB is responisve again (you can click on other areas of TB), but nothing else happens. Neither the subject is decrypted (still shows "...") nor the body text nor the attachment file list is displayed. Seems like another bug, due to that I cannot tell you how long it would take to see the 80 MiB mail.

Comment 17

[Nickolay Olshevsky]

Did some tests locally with 20Mb attachment, encrypted locally, and here are the results of Thunderbird run with time profiler:
rnp_op_verify_execute call (it is used to decrypt message): 1.33seconds (and some 100-200 milliseconds for other rnp-related calls).
Decryption processing time: ~10 seconds (please see the attachments).

Comment 18

[Nickolay Olshevsky]

Attachment: Screenshot 2023-11-28 at 13.53.49.png

Comment 19

[Nickolay Olshevsky]

Attachment: Screenshot 2023-11-28 at 13.54.14.png

Comment 20

[Kai Engert [:KaiE:]]

I've looked into it.

About 80% of the time is taken by inefficient JavaScript string operations, and converting the input and output into different datatypes.

It takes us some time to get the input queued into a JavaScript string.
Then we split the string into an array that stores the charcodes.
Then we convert that into the JS ctypes data type that we need for passing it into the RNP C library.
Then we take the result buffer from RNP and convert it back to a JavaScript string, appending character by character by converting the byte value to a character of that charcode.

Comment 21

[Kai Engert [:KaiE:]]

This isn't a regression.