1666005 - OpenPGP: Save draft or template, close window, subject is "...". If message pane is closed, subject is not restored.

Status: RESOLVED FIXED

(MailNews Core :: Security: OpenPGP, defect)

Description

[harry_cube]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:56.0) Gecko/20100101 Firefox/56.0 Waterfox/56.3

Steps to reproduce:

• OpenPGP has been set up for sending encrypted messages

• I save the message (i.e. manually by Strg-S) ;

• I close the compose window;

• I open the draft folder, click on this message in the list to open it again in a compose window;

• I send this message as is (without changing anything)

Actual results:

• when reopening this message in the draft folder, the subject line in the compose window has been replaced by 3 dots;
• when sending this message without replacing those 3 dots again (=manually filling in text), the subject line will read "...", i.e. this is also what the receiver of the message gets (same is also true for the automatic copy of this message in the 'sent'-folder)

Expected results:

My Comment:
The replacement by 3 dots is most probable caused by OpenPGP, i.e. replacing the subject line of a message is the standard behavior of OpenPGP;

Should happen:
These '3 dots' should be converted back into the original text when the message in the draft folder is re-opened.

I understand that encrypted messages have to be encrypted (and thus also their subject line), i.e. also when stored in the draft folder because in the case of an IMAP account, the draft messages will be uploaded to the server

Since Thunderbird now (as TB now always remembers the secret passphrase) always shows all encrypted messages in unencrypted form, i.e. messages in the inbox, I would even propose: all messages listed in the draft folder should always show their subject line in clear text (not show them as 3 dots).

Comment 1

[Christian Riechers]

Also see bug 1662408.

Comment 2

[harry_cube]

I found something else that might be helpful to know for solving this issue:

When

• I write a message (including a subject line), close the compose window
• I go the draft folder, and manually move the message (whose subject line has been replaced by 3 dots) to any other folder
• I go to this "any other folder" and look for the message

Result:

• the message still has 3 dots as the subject line before I open it;
• after I opened it, the 3 dots are changed back the original text of the subject line.

Note: this could be used as workaround for retrieving the lost subject line; maybe it could also be a useful information for the dev

Comment 3

[harry_cube]

Update: Probably since updating to TB version 78.3.2 (32bit) the behavior changed:
As of now, the subject line of the messages in the draft folder gets 'reconverted' back from three dots to the original text when opening the message. And stays like this, i.e. the subject line of the message is then in clear text when listed in the draft folder (except if one make new changes in the compose window).

Comment 4

[Lasana Murray]

Hello harry_cube, are you saying you no longer experience the problems you mentioned above?

Comment 5

[harry_cube]

Attachment: tb- three dots for subject line.jpg

Comment 6

[harry_cube]

Well actually, as of today the old behavior of TB is back - and I have no idea, why.

Anyway, as I stated here in my first post in the last paragraph, the encrypted messages in the draft folder should not show their respective subject line as three dots at all, which is the case for encrypted messages in all other folders.

I would advocate that the subject line of any encrypted message in any folder should be shown in clear text when this folder is being opened, which is currently and permanently never the case prior to opening the message.

Hence, the aspect that the subject line is converted back when opening the draft folder message (which was the case yesterday, but not any more today) means only that one part of the problem (temporarily) disappeared.

The (permanent) problem of the "three dot" subject line obviously is that one has a hard time to find messages in the draft folder.
To clarify, I have included an image of the message list in the draft folder (see attachment: "tb- three dots for subject line.jpg"; the German word "Betreff" in the image means "subject line", obviously).

Comment 7

[harry_cube]

(In reply to harry_cube from comment #6)

Well actually, as of today the old behavior of TB is back - and I have no idea, why.

Anyway, as I stated here in my first post in the last paragraph, the encrypted messages in the draft folder should not show their respective subject line as three dots at all, which is the case for encrypted messages in all other folders.

I would advocate that the subject line of any encrypted message in any folder should be shown in clear text when this folder is being opened, which is currently and permanently never the case prior to opening the message.

Hence, the aspect that the subject line is converted back when opening the draft folder message (which was the case yesterday, but not any more today) means only that one part of the problem (temporarily) disappeared.

The (permanent) problem of the "three dot" subject line obviously is that one has a hard time to find messages in the draft folder.
To clarify, I have included an image of the message list in the draft folder (see attachment: "tb- three dots for subject line.jpg"; the German word "Betreff" in the image means "subject line", obviously).

Comment 8

[harry_cube]

sorry for the double post - I wanted to edit my use of 'markdown' from "italic" to "bold" -

Comment 9

[Magnus Melin [:mkmelin]]

So what your report is about is that there are the three dots until the message is loaded, but after loading the message the message subject shows alright?

Comment 10

[harry_cube]

The behavior "three dots until the (any) message is loaded" is always the case (situation as shown in first image "tb- three dots for subject line.jpg").

The behavior "three dots after the message is loaded (re-opened)" changes from day to day, i.e. yesterday it was true, but not today. As I mentioned, if the subject line is "reconverted" to clear text, it stays this way, i.e. the subject line of this message is then also shown as clear text in the message list of the draft folder (see second image "tb - three dots are 'reconverted' if messages are re-opened.jpg"), i.e. when you later close the message without introducing any changes to this message.

Comment 11

[harry_cube]

Attachment: tb - three dots are 'reconverted' if messages are reopened.jpg

Comment 12

[Christian Riechers]

(In reply to Lasana Murray from comment #4)

Hello harry_cube, are you saying you no longer experience the problems you mentioned above?

It did seem to work for me too with TB78.3.2, but it's broken again in v78.3.3. I.e. the subject information is lost when opening an encrypted saved draft message again.

Comment 13

[Lasana Murray]

(In reply to Christian Riechers from comment #12)

(In reply to Lasana Murray from comment #4)

Hello harry_cube, are you saying you no longer experience the problems you mentioned above?

It did seem to work for me too with TB78.3.2, but it's broken again in v78.3.3. I.e. the subject information is lost when opening an encrypted saved draft message again.

I just tried on 78.3.3 and could not reproduce. The subject showed in the compose window as normal for both signed and unsigned encrypted messages.

Comment 14

[harry_cube]

Well, the problem is not at all about the compose windows.

When writing a new message in the compose window, I experience the same you describe in Comment #13: with or without encryption or signing selected for this message, the subject line is shown, letter by letter as I type it into the subject field (I hope I understood correctly that this is what you meant in your post).

Please read my initial report here for reproducing this bug. For this problem to happen, it doesn't matter if you have chosen to encrypt or sign the message or not. It is crucial, though, that the sender account that you use for composing the message is set up for using encryption. Please also have a look on the images that I posted to understand the problem.

Comment 15

[Magnus Melin [:mkmelin]]

But your initial report talks about composing. Is the bug report that in the message list of the draft folder, the message subject is listed as "..." until the message was loaded?

Comment 16

[harry_cube]

In reply to #comment 15
Yes, this is the problem, at least a part of the problem. Please read comment #10 for details.

Obviously, in order to get a message into the draft folder, you need to compose a message first, and to do this, you use the compose window.

A draft folder without subject lines (i.e. in this case: three dots) is not very convenient to work with, in case you are searching for messages that you composed earlier, didn't send yet, and wanted to complete at a later time. Obviously, this is only bothering you when you are using the draft folder at all, which I did for several purposes. I have the impression that many people don't use it at all, or do not even know that composed messages that haven't been sent yet can be found in the draft folder.

As I said in my original report here

I understand that encrypted messages have to be encrypted (and thus also their subject line), i.e. also when stored in the draft folder because > in the case of an IMAP account, the draft messages will be uploaded to the server

( comment: "encrypted messages" in the quote should be understood as "messages belonging to a sender account for which encryption is set up, even if encryption has not been selected when composing this message)

I would think that a solution to this bug could be really tricky, and that it can reasonably only be done by somebody who is directly involved in the coding effort to bring OpenPGP functionality from Enigmail into Thunderbird core.

Comment 17

[Christian Riechers]

It looks to me two separate issues are getting mixed up here.

1. As the bug title says, after saving a draft message, closing the compose window, and subsequently opening the saved draft message again in a Write window, the subject information gets lost, and only the three dots are left.
   This is pretty annoying, and the most pressing issue to me.

2. Whether the subject information should be encrypted in the first place or not at all for saved drafts, when the account is set up for encryption.
   This may be addressed in a separate bug.

Comment 18

[Magnus Melin [:mkmelin]]

For #1, please retest after bug 1661510 lands. But it doesn't seem like it happens normally, the only complaint fixed there was that the "Re:" could be lost.

Comment 19

[harry_cube]

In response to Comment #17:
I agree, it would probably make sense to split up the two issues in separate bugs. But I think they should explicitly reference each other.

Given that issue #1 disappears once in a while (at this time the problem is in effect on my TB), it may be true that it is easier to find a solution to issue #1 without solving #2.

Off-topic:
I am actually thinking about temporarily deleting all OpenPGP keys in my Thunderbird install to get rid these two issues since I need encryption only once in a while. But I am still a bit reluctant because re-importing the keys is a bit of a hassle. So it came to my mind that it would be great to have a preference setting that could be set to temporarily disable encryption. Like it was possible with the Enigmail add-on just by disabling it.

Comment 20

[Christian Riechers]

(In reply to Magnus Melin [:mkmelin] from comment #18)

For #1, please retest after bug 1661510 lands. But it doesn't seem like it happens normally, the only complaint fixed there was that the "Re:" could be lost.

Tested with TB83.0b2. The problem described in this bug is unchanged. Subject information is still lost when opening an encrypted previously saved draft message.

Comment 21

[Christian Riechers]

(In reply to Lasana Murray from comment #13)

I just tried on 78.3.3 and could not reproduce. The subject showed in the compose window as normal for both signed and unsigned encrypted messages.

This bug is about opening an encrypted previously saved draft message.

Comment 22

[Magnus Melin [:mkmelin]]

Well drafts are always encrypted when a key is in use for the identity, so in case it happens to you there's something special that we're missing.

Comment 23

[harry_cube]

there's something special that we're missing.

So, what you say is basically you can't reproduce this error in your system, i.e. when proceeding as described in comment 17, issue 1 or in the original report? I just tested again (TB v78.4.0), and I can confirm that the issue is still in effect.

As I said earlier, I only use encryption once in a while, but in contrast quite often pick up messages in the draft folder that I wrote earlier. And I concur with Christian that the issue is annoying. I got a hint by Patrick on tb-e2ee how to toggle encryption off an on, so this is what I use at the time.

(setting in TB about: config)

set mail.openpgp.enable to false. You'll need to restart Thunderbird for this to take effect.

Comment 24

[Christian Riechers]

(In reply to Magnus Melin [:mkmelin] from comment #22)

Well drafts are always encrypted when a key is in use for the identity, so in case it happens to you there's something special that we're missing.

May be I wasn't clear enough. 'Encrypted draft' means
a) there's a recipient email address in the To: field
b) the recipients key is present in the OpenPGP key manager
c) 'Require Encryption' is selected in the Write window

Only then subject encryption is triggered.

Comment 25

[Magnus Melin [:mkmelin]]

But that's not the case in Thunderbird: drafts are always encrypted if the sender identity is set up with an openpgp key. Subjects are always encrypted as well when the body is. What we do for drafts doesn't depend on Require Encryption or not.

Comment 26

[harry_cube]

I'd like to second Comment #25 here: it only depends on the fact whether sender identity is set up with an openpgp key.

I think I'll put up a question on tb-e2ee as to who is experiencing the issue or can reproduce the error.

Comment 27

[Christian Riechers]

(In reply to Magnus Melin [:mkmelin] from comment #25)

But that's not the case in Thunderbird: drafts are always encrypted if the sender identity is set up with an openpgp key. Subjects are always encrypted as well when the body is. What we do for drafts doesn't depend on Require Encryption or not.

OK, understood. In any case, that doesn't change the problem at hand.
I can reproduce the problem using TB78.4.0 on Linux with a new profile.
Created an account (IMAP).
Created an ECC key for the account.
Opened a new Write window, and put in my own email address into the To: field. Put in a subject and something into the body. Composing in plain text.
Saved the draft message to the Drafts folder on the server.
Opened the previously saved draft. Subject is gone, only "..." is shown.

It did briefly work with 78.3.3, but broke again with TB78.4.0 using my regular profile. I therefore don't believe that there is something special with my regular profile.

Comment 28

[Kai Engert [:KaiE:]]

(In reply to Christian Riechers from comment #27)

I can reproduce the problem using TB78.4.0 on Linux with a new profile.

I'm trying to reproduce using your steps.

Created an account (IMAP).
Created an ECC key for the account.
Opened a new Write window, and put in my own email address into the To: field. Put in a subject and something into the body. Composing in plain text.
Saved the draft message to the Drafts folder on the server.

I used CTRL+S to save to the drafts folder, then I clicked CTRL-W to close the composer window.

Opened the previously saved draft. Subject is gone, only "..." is shown.

If I look at the drafts folder, I see the drafts message I have just saved.
It's shown with ...

If I use a right click on the message, and select "edit draft message", then I can confirm your behavior. Subject is gone.

Is this how you opened the draft? With a right click menu?

I never saw this bug, because I always use the following approach:

I click the message shown in the drafts folder once. That "fixes" the draft. As soon as I click the draft message, the subject will be updated and shown correctly, with the text I had previously entered.

This fix is permanent for me. From that time on, editing the draft has a correct subject.

Do you get the same behavior?

Comment 29

[harry_cube]

I concur:
You only run into the error If you open the message by double clicking (or by clicking right mouse button and "edit message"); and one more provision, in case you use double clicking to open: the message pane must be closed because otherwise the issue is circumvented right before you can do the second click.

Sounds weird, I agree. Still is an issue, in my opinion, because when I work with the draft folder I never use the message pane, i.e my habit is to open the draft message in a new tab, which I do by double clicking.

Comment 30

[Christian Riechers]

(In reply to Kai Engert (:KaiE:) from comment #28)

If I look at the drafts folder, I see the drafts message I have just saved.
It's shown with ...

Same here.

If I use a right click on the message, and select "edit draft message", then I can confirm your behavior. Subject is gone.
Is this how you opened the draft? With a right click menu?

No, I just double-click to open the draft.
I can also confirm the the problem when using the right-click approach.

I never saw this bug, because I always use the following approach:
I click the message shown in the drafts folder once. That "fixes" the draft. As soon as I click the draft message, the subject will be updated and shown correctly, with the text I had previously entered.
This fix is permanent for me. From that time on, editing the draft has a correct subject.
Do you get the same behavior?

No, I don't see that. I can click as many times as I want on the draft in the thread pane, it doesn't change a thing. Just keep seeing "..." as subject.
The message pane is always disabled here.

I do get the same behavior as you described above with the message pane enabled, i.e. the subject is restored with a single click then.

Comment 31

[Kai Engert [:KaiE:]]

Thanks Harry and Christian for this important detail.
So, what happens is this:

• we save the draft in its encrypted form in the folder storage, with the subject encrypted inside the message, and the email header containing the "..." placeholder

• for messages stored on an imap server, Thunderbird uses local files to remember message status. Those files are inside the the profile directory, folder ImapMail/server/folder-name.msf

• if you click an encrypted message with a message pane enabled (regardless if its in a drafts folder or in some other folder), the message gets decrypted, and Thunderbird executes code that will store the decrypted subject in the .msf file

• when editing a draft, (currently) we seem to always use the stored subject, but will not dynamically decrypt it

So the current workaround is to enable the message pane, to display the draft message once, prior to editing the draft

In order to fix this bug, we need to do one of the following:

(Potential solution 1)
At the time we store a draft message, save the unencrypted subject to the .msf file.

(Potential solution 2)
At the time we open a draft message and decrypt it, use the decrypted message (don't use the information from the .msf file which might not yet contains the decrypted subject.

I'm guessing that (2) is easier. Someone needs to dive in to the respective code to develop a fix.

Comment 32

[Kai Engert [:KaiE:]]

Lasana, in comment 13 you said you couldn't reproduce.

With the information from comment 28 and later, you should be able to reproduce.

Comment 33

[harry_cube]

(In reply to Kai Engert (:KaiE:) from comment #31)

I have two more remarks :

• if you manually move the message from draft folder to any other folder, and you then open the message by double click, and this issue does not occur
  (my interpretation: TB obviously handles the draft folder in a different way than all other folders, i.e. when and if information is written to the .msf file at the time of opening the message)

for messages stored on an imap server, Thunderbird uses local files to remember message status.

• the issue is independent of weather or not you use a POP account or an IMAP account; issue is present in both cases;

Comment 34

[Lasana Murray]

Thanks for the detailed investigation Kai. I was able to reproduce with the additional information provided. I'll come back to this as soon as I can.

Comment 35

[franz.heinz]

Thanks a lot for the workaround with the message pane. Now at least I can see my subjects again by pressing F8.

I just wanted to add that this issue also affects the template folder.

Comment 36

[Christian Riechers]

This still hasn't been fixed by now. Even though there's a workaround, it is somewhat annoying. It also came up in a support topic recently.
https://support.mozilla.org/en-US/questions/1332547

Comment 37

[Lasana Murray]

I came back to this bug a couple of times and could not figure out how to implement Kai's recommendations. I'm not comfortable with the inner workings of the code involved. Seems to me that however the "Open Message In Tab"/"Open Message In New Window" etc loads the message, "Edit Draft Message" should do something similar. I'm not sure what or how to though.

Comment 38

[Wayne Mery (:wsmwk)]

What is the next step?

Sounds like this is a regression, when compared to enigmail?

Comment 39

[Magnus Melin [:mkmelin]]

Christian, I suppose you have the message pane turned off. With that it's reproducible.

I don't think this is a regression per se, Enigmail didn't encrypt subjects usually (IIRC).
I guess the underlying problem is that we correct the subject when the message is parsed and we know the information. When the message didn't yet get open there is no parsing so the subject is "..." - handling that is a bigger nut to crack...

I would think though that we could fix this in an easier way by a call to set the subject on the msg header directly after we finished storing the draft. Potentially for https://searchfox.org/comm-central/rev/5ab0af272bc5648e6b3452e228896344e48453a9/mail/components/compose/content/MsgComposeCommands.js#714 - grab the id, get the messages by id from the draft folder, set the subject on the header. Or something like that...

Lasana, can you take another look at that approach?

Comment 40

[Lasana Murray]

Question: Won't setting the decrypted subject on the message undermine the point of encryption?

Also, onStatus() does not seem to be called when I save a draft.

Comment 41

[Lasana Murray]

I think I better understand now, but some of those nsIMsgSendListener methods don't seem to be called by MessageSend.jsm or even the cpp version. I can try adding a callback to the interface so the MimeMessage can be modified elsewhere when needed.

Comment 42

[Magnus Melin [:mkmelin]]

(In reply to Lasana Murray from comment #40)

Question: Won't setting the decrypted subject on the message undermine the point of encryption?

Not on the message source. What I meant is to set it on the db (-ish) header as a property - this is what is done now when you read the message: the decrypted subject gets set on the msgHdr, and that header is where the UI gets its information.

I didn't not verify exactly where we'd need to hook in. In general saving as draft calls GenericSendMessage and related functions.

Comment 43

[Lasana Murray]

Attachment: Bug 1666005 - Restore encrypted drafts subject in local db. r=mkmelin

Comment 44

[Thomas D. (:thomas8)]

We also have a similar problem that the subject ellipsis gets printed:
Bug 1727209

Comment 45

[Christian Riechers]

Another complaint at SUMO:
https://support.mozilla.org/en-US/questions/1362653

Comment 46

[Adam]

I have found that with a PGP key enabled the subject line does not restore when saving (CTRL+S), and restoring the email by double-clicking or clicking edit but also the entire message contents and any attachments are not restored. Is this a seperate bug or a new one?

Comment 47

[[:Aureliano Buendía]]

This issue affects also 106.0b2 (64-bit) (IMAP, on Windows 11).

Comment 48

[Kai Engert [:KaiE:]]

Gene, do you happen to know, where is there code that stores email subjects in the .msf files?
I guessing it involves the use of type nsIMsgDBHdr.

For a quick background, for encrypted messages, we store "..." for the subject in the .msf file. If an encrypted message is decrypted the first time, we update the .msf file with the decrypted subject. I would like to find where this is done.

(Finding this code would allow me to trace from where it is called, and it could hopefully allow me to trigger the same action from a different scope.)

Comment 49

[Kai Engert [:KaiE:]]

Ok, I think I found the place in the OpenPGP code where we try to store it.

Comment 50

[gene smith]

Gene, do you happen to know, where is there code that stores email subjects in the .msf files?
I guessing it involves the use of type nsIMsgDBHdr.

Not something I've even dealt with. Maybe it occurs here: https://searchfox.org/comm-central/rev/0794fa43a80d162f0422953a484355ab1d01474c/mailnews/local/src/nsParseMailbox.cpp#1100
or somewhere in this file, not sure.

Ok, I think I found the place in the OpenPGP code where we try to store it.

Or maybe you already found what you need?

Comment 51

[Kai Engert [:KaiE:]]

(In reply to gene smith from comment #50)

Not something I've even dealt with. Maybe it occurs here: https://searchfox.org/comm-central/rev/0794fa43a80d162f0422953a484355ab1d01474c/mailnews/local/src/nsParseMailbox.cpp#1100
or somewhere in this file, not sure.

Thanks Gene, that's a helpful pointer.

Or maybe you already found what you need?

Yes!

Comment 52

[Kai Engert [:KaiE:]]

I have a fix, it seems!

I'm just not sure if that is a safe thing to do in all scenarios!
But what I do, I'm adding an URL to some display options structure, in case there's no (display) channel.

That allows the decryption code to know the original message URI, and the code that decrypts and learns the decrypted subject, is able to store that subject in the message db, which seems to fix it for me.

Comment 54

[Kai Engert [:KaiE:]]

Attachment: Bug 1666005 - Restore encrypted subject of draft/template if message pane is hidden. r=mkmelin

Comment 55

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/d1f91dd3b3cc
Restore encrypted subject of draft/template if message pane is hidden. r=mkmelin

Comment 56

[Magnus Melin [:mkmelin]]

This breaks comm/mail/test/browser/openpgp/browser_openPGPDrafts.js

Comment 57

[Pulsebot]

Backout by [e-mail]:
https://hg.mozilla.org/comm-central/rev/5c226bb4fd92
Backed out changeset d1f91dd3b3cc for test failures

Comment 58

[Kai Engert [:KaiE:]]

The following calls no longer work:
await BrowserTestUtils.waitForEvent(..., "focus", true);

It works for me, if I move the focus waiting into the initial waiting for the window.
I guess that means, the current place to wait for the even is too late?

I found another possible incorrectness with the test.
There's a loop for two messages, and in the second iteration, there are two messages in the folder, but the second iteration again clicks the first message. I think we must then click the second message.

Comment 59

[Kai Engert [:KaiE:]]

try build started:
https://treeherder.mozilla.org/jobs?repo=try-comm-central&revision=d626d74fff96c139fe74710f168f8d9a8c8adfb4

Comment 60

[b5]

Does this also fix bug 1788534?

Comment 61

[b5]

(In reply to Kai Engert (:KaiE:) from comment #48)

... where is there code that stores email subjects in the .msf files?

Just for the record:
https://searchfox.org/comm-central/search?q=hdr.*-%3EsetSubject&path=&case=false&regexp=true
Looks like https://searchfox.org/comm-central/rev/0794fa43a80d162f0422953a484355ab1d01474c/mailnews/local/src/nsParseMailbox.cpp#1093

Comment 62

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/47de9bd35c81
Restore encrypted subject of draft/template if message pane is hidden. r=mkmelin

Comment 63

[Kai Engert [:KaiE:]]

(In reply to b5 from comment #60)

Does this also fix bug 1788534?

no :(