Open Bug 188988 Opened 22 years ago Updated 10 months ago

Gloda doesn't find content in encrypted messages (S/MIME and PGP)

Categories

(MailNews Core :: Backend, defect)

Product:
MailNews Core
Component:
Backend
Type:
defect
Priority:
Not set
Severity:
major

Tracking

(Not tracked)

People

(Reporter: julien.pierre, Assigned: patrick)

References

(Blocks 1 open bug)

Details

(Whiteboard: [datalossy])

Attachments

(2 files, 1 obsolete file)

188988-trigger-folder-index.patch
1.06 KB, patch
Details | Diff | Splinter Review
188988-switch-on-indexing-for-encr-parts.patch
1.01 KB, patch
Details | Diff | Splinter Review 
When doing an "advanced" search of messages, ie. for example in the "body"
field, Mozilla does not search messages that have been encrypted.
This significantly reduces the usefulness of the mail program.

I would expect mozilla to be able to search messages when the user is logged in
to the necessary token containing the private key.

It might take a very long time to process however, in the case of a hardware
token, due to hardware access time. A partial solution would be for mozilla to
cache the symmetric keys for messages as long as the token hasn't been removed
or the user didn't log off from the token, so that one doesn't need to access
the token again. Ie. when you move back and forth between messages in mozilla,
it should only access the hw token the first time you view a given message, but
not again if you come back to it two seconds later.

Another solution to improve performance would involve storing the encrypted
messages differently. Perhaps they could be stored twice :
- once in their original format
- a second time decrypted and re-encrypted with a master symmetric key. This
would be for searching purposes only. All messages would be encrypted with the
same symmetric key; however you would still need access to the hardware token to
decrypt that key. But you could decrypt that key from any given encrypted
message stored. The benefit is that the process isn't limited to the software
security device, the messages aren't in the clear, and performance will be very
good due to only needing to access the hardware token once per search to get the
master symmetric key
Just some thoughts, I'd say duplicating the required storage should be avoided.
Having the ability at all, to search through archived encrypted messages, would
be a good first step, even it were slow.

This is such a serious issue.  The bug has been filed long ago, but is there
anyone actually working on this? 

I have been trying to convince my employer to switch to Thunderbird and Enigmail
from Outlook, but it is really diffcult to do if you cannot search in your e-mail
messages.  This basically makes the otherwise wonderful mail client and encryption
almost useless in a serious setting.

I know this is not an enigmail issue, but the mail client knows that enigmail
needs to be called in order to display the message.  Therefore, it should be
similarly possible to know that enigmail needs to be called in order to search
the message, no?

There will certainly be performance issues, but something is better than nothing
at all.  Just get that feature in, and performance improvements can be worked
on in the future.  Please!

Mozilla (thunderbird, firefox, sunbird, etc.) has great stuff, but not being
able to search your messages really makes it useless in many situations.  :-(

Thanks!

Tom

Tom,

You are preaching to the choir here. You might want to try to locate some people
who actually work on mozilla mail and add them to the cc list of this bug. At
the very least, it should be assigned to a current bugzilla e-mail address.
Unfortunately, I don't know who the person would be for mail.
Assignee: mscott → sspitzer
QA Contact: esther
Assignee: sspitzer → mscott
Product: MailNews → Core
This is not only a problem with Enigmal, but also with the integrated S/MIME
capability.
Severity: normal → major
do we know any cases currently where search fails for crypted messages?
Whiteboard: CLOSEME 09/18
(In reply to comment #5)
> do we know any cases currently where search fails for crypted messages?

Yes. I used this test case to avoid having to search through a lot of messages and waiting a long time:

- compose message yourself, in the body enter "test1234"
- send as encrypted message to yourself
- create folder "encrypted"
- move the message from yourself to folder encrypted
- in thunderbird, use edit/search/search for messages
- select "search in" folder encrypted
- select "message body" contains "test1234"
- hit search

actual results: nothing is found
Whiteboard: CLOSEME 09/18
Assignee: mscott → nobody
QA Contact: backend
In some things, the best is the enemy of the good.  In this case, I would be perfectly happy if all of my encrypted mails were stored in unencrypted form. They are actually in a folder that is protected by the Windows Encrypting file system in any event. My interest in encryption is in keeping ISPs and others who do not have access to my computer from reading my mail while it is in transit. Indeed, if mail were kept in decrypted form then email encryption could be almost completely transparent.

Others might want stronger encryption. When I first suggested this decades ago to the originators of PEM they said it was a bad idea. I disagree. Almost no one encrypts their email today and lack of transparency is one of the main reasons.

If Thunderbird is going to store encrypted email in decrypted format it will have to be a user interface option.
Tlauck,

That may be an acceptable option for you if you use POP and download the messages onto a file system you believe to be secure. Definitely it should not be the default.

And in the case of people using IMAP servers, we would never want to store decrypted copies of the messages onto the servers.
Product: Core → MailNews Core
Any update? This is critical problem if S/MIME is used. There's no use of an e-mail client if it can't find messages it's stored. It could force me to start looking on alternatives to very nice Thunderbird, which I like using very much.
I concur.  This problem makes the use of S/MIME useless for me, too.

I am appalled that this problem has been outstanding for so many years. It calls into question the viability of the entire Mozilla open source process.
gloda should be able to search encrypted e-mails, right, Andrew?
Yes, since gloda stores the message body plaintext in SQLite, gloda would be the easiest solution to the searching problem.

If someone were interested in implementing this, they would need to teach gloda how to decrypt encrypted messages (is it a flag we tell the streaming layer?), as well as provide an explicit opt-in UI that causes gloda to enter a mode of operation where it will trigger the decryption process.  We would not want to start storing the encrypted messages in a decrypted form without explicit approval.

Vladislav, tlauck; would you be interested in trying to implement this?  The open source process is driven by people solving problems they are interested in fixing.  As you can probably tell, there basically no active contributors trying to improve the state of S/MIME and encrypted e-mail inside Thunderbird, which means we need new contributors who are interested in the state of S/MIME and encrypted e-mail in Thunderbird!  That could be you!

(The programming effort required in this case would not be complicated, it's more a question of learning about the better-documented internals of Thunderbird and the mozilla platform.)
Before implementing a feature that copies the contents of user's encrypted emails to a non-encrypted search database, a policy decision should be made.

Is the above approach acceptable or not?

Or should, for security reasons, searching in encrypted emails be restricted to linear walking through mails?

Maybe the behaviour of "copy encrypted messages to plain serach index"
should be based on a user preference.

Maybe the default of that preference should be in the more secure setting.


And I agree with the previous comment.
This is an open source project.
If you want features, we need contributions.
Contribution includes the process to find compromises acceptable to the project peers and helping with patches.
If decrypted emails are stored in plaintext form then there won't be a problem with searching, either in terms of performance or uncertainty as to what information may be leaked by searches. 

Storing the plaintext as well as cyphertext seems the easier way to deal with the search problem, given that there are already reasons why some people want to store the plaintext of decrypted mail in folders.  (One such reason concerns the problem of expired private keys. Requiring these to be stored is itself a security problem.)

If agreement can be reached on a suitable user interface (including defaults the user can set up) then it doesn't seem like a major problem to implement this capability. A message would reside in a folder either cipher text only (as at present) or cipher text plus plaintext (when the new capability has been applied to the message).  Messages in the former form could not be searchable, but messages in the new form would be. (By analogy the mail is in a sealed envelope or the mail has been opened and the document removed from the envelope but the envelope is retained to permit resealing, for evidence, etc.)

In the past there appeared to be some controversy as to whether or not this capability is a good idea.  If there can be consensus that this should be an option, then the next step would be to entertain suggestions as to what a suitable user interface might be.
I don't think searching in encrypted emails stored locally should be restricted anyhow. I believe that purpose of S/MIME is only to protect messages during transfer and storage on intermediate mail servers. If they are in my local mail box, there's no point to protect them. At least, for me and people like me. If I wanted to have my mailbox encrypted, I'd rather use one of many products encrypting on the file system level, starting from encryptfs on Linux. Anyway, if you want your e-mails be encrypted, you'd definitely want to protect as well many other related things (address book, etc), where only file system level encryption can help you.

I believe problems of searching in S/MIME messages and storing or not them in decrypted form are quite orthogonal. If we start deciding to store decrypted messages, we would have to decide many related questions, like if to keep or not the encrypted form for future references, e.g. to check if messaged were transfered encrypted, with which keys, by which ciphers, ..., if not to store, then how and where to keep those reference data, etc.

If to store both encrypted and decrypted messages when encrypted only messages are not searchable, it would still keep the confusion that if I can with all the keys open see all my encrypted only messages by a mouse click, but a plain body search (not gloda) in them doesn't return anything. Such behavior is pretty disappointing and would distract people.

(Plus storing both encrypted and decrypted messages would be a big storage overuse.)

From other side the searching problem is simply about be able to find bodies of encrypted messages which users can read by a mouse click.

So, better to not confuse those 2 problems and solve the searching bug at first. Then later we can go ahead with the storing decrypted messages problem. I agree, better to store encrypted messages decrypted, but there's too many related problems to solve before making it possible.

I can look on implementing the search. But please somebody give me some step by step guide where to start looking to implement it, similarly as I wrote in http://scst.sourceforge.net/contributing.html#MEM_REG. Thunderbird and Mozilla are very big projects, so it would save a lot of time.
Thank you for your interest, Vladislav!

The general overview for the global database mechanism is here:
https://developer.mozilla.org/en/thunderbird/gloda

The general documentation for libmime, the low-level layer that performs the actual parsing of messages is at the top of this file:
http://mxr.mozilla.org/comm-central/source/mailnews/mime/src/mimei.h

The libmime encryption overview header file is here:
http://mxr.mozilla.org/comm-central/source/mailnews/mime/src/mimecryp.h
with the encryption guts here:
http://mxr.mozilla.org/comm-central/source/mailnews/mime/src/mimecryp.cpp

The global database messaging process initiates streaming of the message here:
http://mxr.mozilla.org/comm-central/source/mailnews/db/gloda/modules/index_msg.js#2831

Our JavaScript wrapper around libmime that does this exposes its API here:
http://mxr.mozilla.org/comm-central/source/mailnews/db/gloda/modules/mimemsg.js
but the actual representation building happens here:
http://mxr.mozilla.org/comm-central/source/mailnews/db/gloda/components/jsmimeemitter.js

libmime is aware of the implementation of the JS mime emitter and has logic that sets specific flags to enable special behavior:
http://mxr.mozilla.org/comm-central/source/mailnews/mime/src/mimei.cpp#1550

The unit test for the JS mime emitter lives here:
http://mxr.mozilla.org/comm-central/source/mailnews/db/gloda/test/unit/test_mime_emitter.js

It knows how to make fake/gibberish S/MIME messages and uses this test library to do that:
http://mxr.mozilla.org/comm-central/source/mailnews/test/resources/messageGenerator.js

You would run the test by building thunderbird, changing to the objdir/mailnews/db/gloda/test dir and running:
make SOLO_FILE=test_mime_emitter.js check-one

There are also other global database tests in that directory, you can run them all by doing the following from the same directory:
make xpcshell-tests

The general technical strategy I would propose as the first steps:
- Figure out how to cause the mime streaming process to cause decryption of messages, ideally exposing this as a flag that can be passed to mimemsg.js's streaming mechanism.
- Modify the indexing mechanism of the global database to pass that flag and attach it to a preference probably.

As Kai makes the excellent point, the defaults and such are something that do need some discussion and the appropriate venue for that is the tb-planning list:
https://wiki.mozilla.org/Thunderbird/tb-planning

I generally agree with your viewpoint that if you actually want your messages secured locally, you should be using an encrypted file-system for storage plus potentially encrypted swap.  Thunderbird makes zero guarantees about keeping decrypted message content safe once we decrypt it; it could easily end up in un-locked memory and swapped to disk, etc.

Please let me know if you have additional questions or need additional guidance outside of the actual S/MIME implementation.  I have basically zero knowledge about how that works, and Kai is the only person I am aware of with any familiarity with the implementation.  (And of course Enigmail/PGP is not directly supported in-tree at this time.)
And the documentation for building Thunderbird should be here:
https://developer.mozilla.org/En/Simple_Thunderbird_build

You may also want to drop into #maildev IRC channel or ask for feedback not specific to this bug in tb-planning.  Other documentation and info on those venues can be found here:
https://developer.mozilla.org/en/Thunderbird
Quote from Bug #260665:

"This limitation makes this otherwise wonderful mail-client and encryption module
BASICALLY UNUSABLE in a serious environment." (emphasis added)

That's exactly my feeling and experience. 

I will have to quit TB sooner or later for professional use if I won't be able to search my S/MIME and Enigmail encrypted mail. This feature (-> full text search in encrypted mails OR decrypted local mail storage) is so crucial that it single-handedly can make the difference between failure or success of my work (being a patent attorney).

Simply make it an option to STORE mail that has been TRANSFERRED (!) using encryption DECRYPTED locally.

Mail encryption of course is for TRANSFER (like a sealed envelope with paper mail) but not for local storage! Nobody is so stupid to re-seal and re-cut-open their paper letters every time they need to look up something, since this simply will KILL their productivity and thus their business! 

Rather and of course you keep your paper letters locked away in a safe cupboard (but with OPEN envelopes, or rather WITHOUT envelopes!) if they contain sensitive information.

I want to bang my head against the wall if I think about the ignorance at Mozilla that has been forcing people for almost a decade to keep (and re-seal every evening, like idiots) every single ******* envelope of every ******* letter that they store in their office cupboards.
This Bug SUCKS me so much EVERY DAY.

Know what? Every time I need to read those ca. two lines of text in one of my encrypted e-mails that also contain like 15MB of attachments: my triple monitor multi CPU multi SSD monster PC stalls at 100% CPU load for like 10 seconds before it displays those stupid two lines of text that I need to read.

This not only SUCKS incredibly because it stupidly and unnecessarily disrupts one's workflow. Additionally, it also takes like 50 additional Watts of CPU power times 10 seconds which is exactly 0.1389 Watt Hours. 

This 20 times a day times 10 Million users makes 1.1575 megawatts of wasted electrical power (365/24/7) because of the STUPIDITY and IGNORANCE of that guy at Mozilla who supposedly has decided that users should not be allowed to store encrypted mail in decrypted state locally, nor detach or delete attachmens from their encrypted mail.

http://www.wolframalpha.com/input/?i=0.1389+watt+hours+per+day+in+watts+times+10+million+times+20

This Bug SUCKS me so much EVERY DAY.
David, while it's entirely reasonable to be frustrated by aspects of Thunderbird, Bugzilla is a venue specifically for driving the technical aspects of bugs forward.

If you feel the need to advocate or vent, <http://getsatisfaction.com/mozilla_messaging/topics/> is a more suitable place for that.

Please review <https://bugzilla.mozilla.org/page.cgi?id=etiquette.html> before your next comment in Bugzilla so that you have a better understanding of what is and isn't appropriate here.  Thanks!
Blocks: glodafailtracker
Whiteboard: [datalossy]
Thanks. I however see no other way of making the point to PLEASE finally START WORKING on bugs like this WICH HAVE NOT BEING WORKED ON for a ******* decade!

Thank you all.
This is going to be long, so please bear with me, Dear Reader.

Vladislav, Andrew, and others who agree that S/MIME is for mail in transit, o n l y, please skip this post.

tlauk writes: In the past there appeared to be some controversy as to whether or not this capability is a good idea.

I would like to convince you that it is a good idea. 
That what we have here is not a bug, but a design flaw.

There is no blame, so it did not matter then, just as it does not matter now, nine years later, how this design flaw came to be in Thunderbird.

tlauck is right.  
In some things, the best is the enemy of the good.

In this case, the enemy is a misunderstanding of what is the best.

If you think about it, I believe you will agree that the question of confidentiality of data in transit is distinct from the question of confidentiality of data at rest.

The question of when and why and how to secure secure mail in transit is a different question from the question of of when and why and how to secure secure mail when in the recipient's control.

S/MIME is a solution for mail in transit.  It seems to do the job just fine. (The problems with keys and certificates and the whole (not required) PKI structure are not the fault of S/MIME.)

How to secure mail at rest?  

tlauk is right again.  
File system encryption is the solution for confidentiality of data at rest.

Forgive the choice of words, not offense intended, good hacks are excellent: 
One hack confidentiality of stored mail is to store the S/MIME attachment.

It is not a good hack, if it prevents searching messages.

Dear Reader, please take yourself back to the time that Julien Pierre posted this bug, six months before the release of Thunderbird 0.1, over eight years ago. Become an architect of Thunderbird. Thinking as a software architect, design an implementation of the confidentiality function for Thunderbird. 
[http://joaquin.net/ODP/Part3/15.html#15.6]

I except you will say to your self: We'll use S/MIME, for all the obvious reasons.  That will provide confidentiality in transit.  Now, ask yourself: What about confidentiality at rest, on the user's machine?

Vladislav Bolkhovitin is right.  S/MIME is to protect messages in transit.

I feel the correct conclusion is this: The job of the mail reader is to create, send, receive, store, and search mail.  Security at rest is the job of the file system the mail reader using.

"Wrong," you say.  And you are correct: I ask, "Why?" and you reply, "Because the file systems our users are stuck with do not provide the confidentiality function."  I find I have to agree: certainly Windows and Mac OS did not provide confidentiality in a satisfactory way.  And TrueCrypt 1.0 was a year in the future. 

You may may feel that we need to look out for the less sophisticated user.  They will not understand, may not even think about the issues. If we store the message in the clear, we may be doing them a disservice. 

Perhaps that is how the decision was made back in 2003.  
But that was then.  This is now.

We have an adequate-for-many-users, well integrated encrypting file system on Windows.  We have pretty good full disk encryption on Mac OS (finally!). We have TrueCrypt. 

What we also have is that Thunderbird users can have either the message searching function or the confidentiality in transit function, whichever they prefer.  

But not both.

I have co-workers who want to use Thunderbird.  And did.  But finally gave up on Thunderbird.  It is  e x t r e m e l y  frustrating to not be able to search e-mail.  Makes it  h a r d  to do one's job.

Thanks, Dear Reader, for sticking with me. I must abjectly apologize for the fact that I am incompetent to be a contributor on this.

Thanks, Vladislav for volunteering.  Thanks, Andrew for coming right back to help. 

Your, 
Joaquin


Footnote: 
Q: Why save the encrypted form of the message?
A: (in the form of questions) Is that required for authentication, non-repudiation, and integrity? 
[http://joaquin.net/ODP/Part3/15.html#15.4]
[http://joaquin.net/ODP/Part3/15.html#15.5]
[http://joaquin.net/ODP/Part3/15.html#15.7]

Is the signature calculated on the cleartext of the S/MIME attachment or on the encrypted text?

If so, then, until someone volunteers to provide the code to preserve these functions once only the cleartext is stored, the ciphertext will need to be stored, too.
As a technical update about the amount of work required if someone wants to move things forward technically, bug 527927 (targeted at Thunderbird 8) has recently taught the JS Mime Emitter about encrypted messages.  This would likely reduce or eliminate any libmime hacking I discussed in comment #17.

The mechanism allows the JS mime emitter to see encrypted parts and their body contents and they are specially indicated as such.  This occurs for S/MIME encrypted messages and, when using enigmail, PGP-mime messages, but not PGP ASCII-armored messages which enigmail has to magically fix-up somehow using the message reader.  I believe (but could be wrong) that we only get the decrypted part if the relevant crypto subsystem has the crypto keys available and we won't cause prompting.  As such, someone wishing to enable pervasive indexing of decrypted messages would need to also address the need to make sure the keys are available to the indexing process and the fallout if the user decides not to enter the password, etc.

It is important to note that the patch causes the global database to intentionally avoid indexing the contents of such messages for consistency with our current behaviour of not indexing encrypted messages.
I might be missing something, but it doesn't look like fixing 527927 will help anyhow fixing this bug.

We don't really want to search in encrypted e-mails. Instead, we want to have ALL SMIME/PGP/etc encrypted emails be stored locally decrypted in plain text and let lower FS level to handle encryption if necessary (see comments 20 and 24). Then we will get the search possibility automatically.

Storing the ciphertext should be optional, because for most cases storing only flags that authentication, integrity, etc. tests passed on the decryption phase + keys used (for reference) should be sufficient.

I'd love implementing it, but, unfortunately, it needs more concentration than I can afford at the moment :(
Vladislav,

I don't think you can rely on the local filesystem being properly encrypted. I would be very wary of any scheme that stored the e-mails in plain-text. Such a scheme would not be secure accross platforms and would not be a good solution, IMO.

The first time when building the index, you would have to actually decrypt all e-mails. However after that, you could store this index securely, without relying on platform specific capabilities. For S/MIME users, the index could be stored as a special message encrypted with the user's private key. For other users, it could be encrypted with a symmetric key such as the one that already protects passwords in Thunderbird/Firefox.

Note : I'm not involved with Thunderbird development in any way at this point. I just reported the bug against PSM in Seamonkey originally.
Julien,

SMIME/etc is a _transport_ level encryption and only it. Many users, including me, don't need local encryption (see coment 20).

Moreover, as soon as you indexed encrypted messages, this encryption gets useless, because the index will, basically, contain context of those messages. Or are you going to put the index under SMIME as well?

Hence, FS level encryption is the best way to go.

Let's not confuse fundamentally separate things and go the simplest way by making boys go right and girls - left.
Vladislav,

Yes, read comment 27, that's exactly what I suggested, put the index in an S/MIME message.

If you want something that decrypts things in plain text locally, I suspect you will have a hard time finding someone knowledgeable enough about security who would be willing to implement this, and an even harder time passing review. Unless perhaps the patch was able to detect the properties of the local FS and figure out it was encrypted and secure enough. However, that would be of limited usefulness, IMO, given that most users still don't use encrypted file systems.

Encrypting the index would be a more generic implementation that should be secure regardless of the file system.
Anyone who is competent in security understands the different threat models involved. Anyone who is competent in computer systems architecture understands the importance of locating functions in the correct place in the system. 

I've about given up on this issue.  I don't believe all the volunteers are incompetent. I think there are probably government agencies that are doing their best to delay the widespread use of encrypted email. One way of accomplishing this is to make common mail client software impractical to use.

In addition, one competent in security protocols would understand that retaining encryption keys for transmitted traffic is poor practice.  One wishes to provide as good an approximation to "perfect forward secrecy" as possible. Presently, it is necessary to retain encryption keys forever to decrypt old mail messages. This in itself is a known security risk.
Once again, don't confuse 2 fundamentally separate things and let boys and girls know their rooms. There's a lot of FS level encryption tools on both Linux and Windows as well as on other platforms whose trustworthy is not less than of the S/MIME, so no S/MIME encrypted messages should be stored locally. KISS, you know.
Encrypting the index smacks of unneeded complexity. The complexity invades the software development process, confuses the users, and will degrade the responsiveness of the software.  In addition, if there are any encrypted messages it will be necessary for the user to provide a password to unlock them for searching to work properly. 

This is not only inconvenient, it is potentially insecure. In particular, old keys need to be retained long after their expiration lest old messages become unreadable. Even if the old messages have been thoroughly deleted from every computer system in which they were ever stored, there is a possibility that an encrypted form of these messages had been captured during their original transmission. If a computer containing a key for these messages should happen to become compromised the secrecy of these messages will be lost, even if they have been deleted.  (This is a variant on one of NSA's formerly classified threat scenarios, i.e. the present design of SMIME is far from providing "perfect forward secrecy".)

Whether incoming encrypted messages are automatically decrypted and whether they are stored in encrypted or decrypted form should be a user option. I would find it perfectly acceptable if the default were to decrypt when specifically requested and store in encrypted form. If the user has a secure file system or otherwise trusts his computer system then he can change these settings. If he doesn't then I would leave things as they presently are, i.e. leave the still encrypted messages unsearchable. The user always has the option of installing file system encryption and decrypting existing encrypted messages at any time.

This also has the virtue of being simple to implement.  :-)  

BTW, my email and all Thunderbird data associated with it is presently stored in encrypted form using the Windows Encrypting File System. At least after log out, none of this data will be available to someone who has physically captured my computer unless they can guess my very strong password. This works well to protect stored email and other client information.
This is how I see this issue should be fixed.

TB would have 2 modes how to deal with encrypted and signed e-mails: "store decrypted" and legacy "stored encrypted", i.e. as it is at the moment.

Setting which mode to use would be available with the config editor. By default this settiong would not be set in any value, which means that TB seeing the first time an encrypted/signed message during any action (receiving, viewing, compacting, etc.) would ask user which mode to use offering by default new "store decrypted" mode.

In this mode TB would process encrypted/signed messages during any action (receiving, viewing, compacting, etc.) the same way. It would decrypt them and/or check signature, if the message is signed, then store it decrypted and set in its internal fields X-Mozilla-SMIME-Encrypt-Keys, X-Mozilla-SMIME-Sign-Keys and X-Mozilla-SMIME-Signature-Correct keys used to encrypt and/or sing this message as well as result of the signature verification (correct/incorrect) correspondingly. Then, if this message already saved, TB would delete the original.

This would make S/MIME messages full grade citizens and effectively make them fully searchable.
First, thanks very much to Vladislav and Joaquin for their insightful contributions.

Anyway, this needs to be solved ASAP. I can't wait any more decades before I am allowed to take care of what I need to encrypt and what not.

I bl**dy need to full-text search my (growing) giant amount of business e-mail (and I need to be able to detach those thousands of useless 20MB attachments from my encrypted e-mails, see also comment 20). 

Guys, THIS IS RIDICULOUS! THIS SOFTWARE IS NOT QUALIFIED FOR PROFESSIONAL USE, in its present state!

I suppose, what comes next is "well, then don't use it, sucker".

:-(

But thanks anyway for listening. 

PLEASE, someone who knows how do do this (I unfortunately don't) -- take some action and make a parser that crawls those local mbox files and decrypts them; and a patch that allows storage of new mails DEcrypted locally.

Or something :-(
Can please somebody summary the status here in two/three lines?  I'm willing to work on this now.  My everyday work is ruined by this bug.  Thanks.
Assignee: nobody → honzab.moz
(In reply to Honza Bambas (:mayhemer) from comment #35)
> Can please somebody summary the status here in two/three lines?  I'm willing
> to work on this now.  My everyday work is ruined by this bug.  Thanks.

Technical discussion on the bug has been primarily about teaching the global database to index encrypted messages.  Please see the posts by me for information on that; it should be fairly easy now.  If you are looking for the "advanced search" mechanism to search encrypted messages, I am unable to provide any input on the technical issues.

Kai Engert has also raised potential policy issues which have not been discussed (tb-planning would be the venue for that), but a patch to cause the global database to index encrypted messages only when a (hidden) preference is set should be minimal enough that no discussion is really required.  A unit test that verifies that we don't index encrypted e-mails by default and that we do when the preference is set would be required.  If you wanted to surface the preference in the UI, you would need UX signoff from Blake Winton (:bwinton).
Thank you.

I think I can start with a hidden pref.  UI can be solved in a different bug.

I will create the test as well.
Honza: Praise you for volunteering!

Andrew: I don't know what the "advanced search" mechanism is. 

Menu Edit Find SearchMessages brings up a useful search window.  All the header fields of encrypted messages that mail readers leave in the clear (To:, From:, Date:, Subject:, ...) can be searched just fine.

<all caps> some of us want to search the body of the message </all caps>

(I write "want to."  It means: can't do their work if they can't search the body.)

I trust the proposed fix enables us to do that.

.......

Kai: As to the policy question: I'm sure Mozilla procedures will take care of that.
I do hope everyone who participates in the policy discussions will agree to the base architectural principle: Security in transit and security in storage are separate items; architects need to keep them separate in their mind.

(There is little wrong with looking out for naive (best sense of the word) users by keeping encrypted what arrives encrypted; a reasonable hack (best sense of the word) for that may be to store the S/MIME attachment and not store the clear text body.
 
(If so: please fix Thunderbird so it can   a l s o  be used by  not-naive users who must encrypt in transit and must search in the body of their mail.  Those not-naive users will know if they need to encrypt in storage and how to accomplish that.)

Many thanks again to all.
(In reply to Joaquin Miller from comment #38)
> Andrew: I don't know what the "advanced search" mechanism is. 
> 
> Menu Edit Find SearchMessages brings up a useful search window.  All the
> header fields of encrypted messages that mail readers leave in the clear
> (To:, From:, Date:, Subject:, ...) can be searched just fine.

That's what I'm referring to as advanced search, although it's bad terminology since no part of the process has the word advanced in any captions, so I'll stop referring to it like that.  It's not hooked up to the global database mechanism in Thunderbird, although I think Eudora/Penelope has forked the dialog and there is an option to use the global search mechanism in their UI.

That mechanism can be made to work on encrypted messages too, I just can't speak as to why it doesn't work right now and so what would be required to make it work.  It's probably something small (which is not to say easy to find or entirely trivial to fix.)
Thanks for straightening me out, Andrew.

How shall I put this?

Darn.  A mail reader that won't let the user search her e-mail messages (the message part of the message: the body) ain't too good.
xref bug 280588
See Also: → 280588
(In reply to Honza Bambas (:mayhemer) from comment #37)
> Thank you.
> 
> I think I can start with a hidden pref.  UI can be solved in a different bug.
> 
> I will create the test as well.

May I ask about your progress? I can't help, but would be willing to test it.
(In reply to Kai Eckert from comment #42)
> May I ask about your progress?

None.  Releasing for now, I could get to it later (months).
Assignee: honzab.moz → nobody
I don't intend to work on this bug any more, my priorities has unfortunately changed.
I'm currently trying to organize a solution for bug 280588 which will solve this bug also. I'm not a programmer, but willing to organize a crowdfunding to finally remove this 10 year old trouble beast. Please input your comments in the other thread.
We started a crowdfunding for a related issue, which will be a workaround for at least some users. Your help is appreciated:
http://freedomsponsors.org/core/issue/434/encrypted-email-messages-should-be-stored-decrypted-in-the-local-folders
Hello Everybody,

I have read this post and a couple related or similar bug posts and I thought maybe I can work on this bug. Before I start to work, I just wanted to ask if there is any progress?
Definitely not from me.  Go ahead!
I'm not a programmer (although I plan to be one day!) but I wanted to simply add that this is, as everyone said, a kind if 'deal breaker' for most people to make the switch to a) Thunderbird and b) secure email with PGP.  I have about a year's worth of enrypted communications with my close friends, and one day I needed to search some keywords in those communications. I expected that I would be prompted for the PGP passphrase which would then allow the encrypted emails to be searched in addition to unencrypted.  Apparently this doesn't happen!  I could not find the content of the email I needed which caused me a good deal of concern.  If I can help in any way, let me know.
Thanks to Janosch Rux, Patrick Brunschwig and others a filter rule to decrypt messages permanently was added to Enigmail. Maybe that solves this bug here?

http://sourceforge.net/p/enigmail/bugs/1/
https://www.enigmail.net/download/changelog.php#enig1.8

Thank you so much!
Whoa can this be true, after more than a decade of this Bug?

However, there is still the problem that S/MIME encrypted messages are still stored encrypted and can't be searched.
I would also like to point out again that the issue is the same for OpenPGP/MIME signed messages. In this case I understand it even less that Thunderbird can't search the body of the message. Yes, it's perhaps a bit more hidden than the body of a standard message, but at least its still available as clear text, no?
Removing myslef on all the bugs I'm cced on. Please NI me if you need something on MailNews Core bugs from me.

I think we should prefer not decrypting mail permanently to require searching. I know that Thunderbird used an index for searching, called Gloda.

source: https://support.mozilla.org/en-US/kb/global-search

I would also caution against using unencrypted indexing for encrypted mail. Some pgp users may need the security.

I have never contributed to Thunderbird, but I know that you would need to modify the search system to either use two databases (one for encrypted e-mail, which is encrypted itself), one encrypted database for everything(which would be locked for people without the key), or to have encrypted entries in the database somehow. The latter I would not recommend, as that would require extra work both for the encryption and for "salting" the entries so that no known-plaintext or replay attack can be done. It would also require a lot of work modifiying existing systems to work with the new entries while still maintaining compatiblity with the older entries. There is a lot of potential for errors there, in my opinion, which can lead to security issues.

Right now the Gloda search system uses a single file (global-messages-db.sqlite). It would need to be able to use two databases, one which has to be stored encrypted. We could also encrypt Gloda by default for PGP users (and then just treat the e-mails as unencrypted while the user is logged in), but that might be hard for people who also use unencrypted mail regulary without wanting to use a password. I do not know the Thunderbird/enigmail code, but that might be less work.

I think indexing the cleartext of all emails in one database using stopword lists to reduce the index is the way to go.
Makes it smaller and central and then you can encrypt it using the Masterpassword. Which might need a bit of redesign with stronger protection.

(In reply to Rob van den Berg from comment #54)

I think we should prefer not decrypting mail permanently to require searching. I know that Thunderbird used an index for searching, called Gloda.

Not all searches go through Gloda. The Quick Filter and the Advanced Search just search through your messages one at a time (plus using the Mail Summary Files where possible, which are separate from Gloda). It should be possible to use the Quick Filter or Advanced Search on encrypted mail without storing any (more) data from encrypted mail in unencrypted form.

This would probably be a bit confusing ("why can I only find encrypted messages using some of the search tools?") but it shouldn't have any new privacy/security issues.

I'm a little surprised since Gloda does trigger the PGP Proxy which should lead to decryption and subsequent indexing of messages. BTW, I see this, perhaps it's related:
gloda.datastore ERROR got error in _asyncTrackerListener.handleError(): 19: constraint failed

Summary: encrypted emails are not searched → Gloda doesn't find content in encrypted messages (S/MIME and PGP)
Depends on: 280588

I'll take a look, given comment #57.

See Also: → 1562737

This does not depend on 280588. Sure, if you store messages unencrypted, the searches will "just work" (TM).

Assignee: nobody → jorgk-bmo
No longer depends on: 280588

Simple patch to trigger folder re-index by hijacking OpenMessageInNewTab().

Attached patch 188988-WIP.patch (obsolete) — — Splinter Review

With this patch, the request from Gloda is processed and the decrypted message is returned. Further investigation is necessary to see why the returned result is in fact not indexed by Gloda.

Looking further into the issue, it's basically a one-line change to enable this function, as there is already provision for it. Maybe there are further adjustments necessary in mimeDecrypt.jsm, but looking at it again, this would already work:
https://searchfox.org/comm-central/rev/67ff5d7aa78c03b51b1ccd06b8636f3ac28e5fbf/mail/extensions/openpgp/content/modules/mimeDecrypt.jsm#473

It was in fact a conscious choice of the Gloda authors not do index encrypted parts[1]. I don't know whether this choice is still valid in the day where (most) disks are encrypted anyway. To add another preference to the system, maybe indexing encrypted messages should be optional, or per account.

Patrick, maybe you can coordinate this with the OpenPGP team. If you/they decide to enable this function, you/they would need to use this patch. Feel free to assume authorship. For the p≡p Project we will potentially "patch" IndexMsg.jsm so it will index encrypted messages until a decision is made here.

If it's decided that decrypted parts will never be indexed, this bug should be closed as WONTFIX.

[1] https://searchfox.org/comm-central/rev/67ff5d7aa78c03b51b1ccd06b8636f3ac28e5fbf/mailnews/db/gloda/modules/MimeMessage.jsm#168

Attachment #9175635 - Attachment is obsolete: true
Flags: needinfo?(patrick)

Thanks for this. This is certainly the basis, but I think it needs a bit more surrounding work.

We discussed this already several times at OpenPGP Email Summits. The conclusion of the discussion is the following:
If we store decrypted data on the disk (which is what Gloda does), then we should only do this with the user's consent. That is, there should be a configuration option that allows the user to enable the functionality, and the default should be "off", unless you know that the data is stored on an encrypted disk.

As we don't know if the disk is encrypted in Thunderbird, we should foresee a preference value to handle this.

Flags: needinfo?(patrick)
Assignee: jorgk-bmo → patrick

How about using stop word lists to create a local search index which is protected with the TB master password so you don't have to store individual mails in a decrypted state but still have a search option across them that doesn't cost too much decryption performance?

This bug has been open for at least 16 years. I have been following it for only nine years.

We all know there is a simple solution. And that the solution accords with the general principles of confidentiality. (I'll write on that in a following comment. This comment is to add a new use case that is handled by one of the proposed solutions for this serious, longstanding, bug.)

This week I was asked to provide about a thousand messages from my archive. I could, of course, respond to this polite request by saying I would rather not. In due course a nice person would hand me an order from a Federal court, explaining the next step.* (Meanwhile, I would be looking for another job, but that is beside the point here.)

If Thunderbird followed the well established principle of cryptography, that confidentiality in transit and confidentiality in storage are two technical problems, and call for two technical solutions, then I'd simply copy the mbox, encrypt it (plenty of ways to do that, e.g. a file-zipper). Put the 873.8 megabytes on a storage device, mail it to the US Treasury counsel, and use Thunderbird to send the password in an encrypted message. The lawyers have tools for every kind of mail storage solutions. (But, as you know, can't decrypt messages encrypted by Thunderbird. And as you know, I am not about to give them my private key. (Nor, or course, have I been asked to.))

So, there we have another use case. It has nothing to do with the search use case, folks WANTING Thunderbird to--PLEASE--SEARCH MESSAGE BODIES.

But notice that it is something to think about, when thinking about the problems with implementing full search capabilities. There is one implementation that solves both use cases in a straightforward and--comparatively--extremely easy to implement way.

As you all know, I mean this: If the fix for this bug had been to work in the following way, the bug might have been fixed sixteen years ago:

"If we store decrypted data on the disk (which is what Gloda does), then we should only do this with the user's consent. That is, there should be a configuration option that allows the user to enable the functionality, and the default should be "off", unless you know that the data is stored on an encrypted disk."

 -- Patrick Brunschwig
     3 months ago.

[ * compliance vs. contempt ]

I know this is not the proper forum, but...

If someone know how I might prepare an mbox with a thousand messages, decrypted, or prepare one thousand files, each with a decrypted message, please post a link here.

I do have one procedure:
DO UNTIL done { Type(Command-P); Click(PDF); Click(Save as PDF); Type (Subject_text); IF duplicate(Subject) THEN Type(serial number); Click(Save); Type(Down-arrow) }
I figure that--once I get up to speed--I can do several a minute. 1000/(3*60). Maybe six hours, with a pee break or two, snack, and perhaps a tranquilizer.

I'm looking for something not quite so, uh, <don't get me started>...

Thanks for your tolerance for this out of order request.

Cordially, Joaquin

I think you want bug 1627962 which we want to do.

Thanks, Magnus. Sadly, I must use X.509. Not permitted to use PGP.

....... All praise to Zimmerman! .......

(Dear reader, do you use only post cards for your mail?
Or do you risk being considered a criminal or--worse--a terrorist, by hiding your message in an envelope before you post it?
If you do get a love letter in an envelope, and you want to keep it, do you carefully reseal the envelope?
Or do you leave the letter open, so anyone can read it?
Or, perhaps, do you leave the envelope unsealed but put your love letter in a safe place.
)

https://en.wikipedia.org/wiki/Pretty_Good_Privacy#Early_history

Reading the patch I assume the gloda already has support for indexing encrypted messages and we just need to a toggle to enable it when needed.

Do I assume correctly that when the setting is enabled, gloda starts to index new encrypted messages from that point only and can not search in messages received before? The option text may imply the user otherwise.

Also, does it mean gloda will index the contents of the encrypted messages in plain text in its word database as the message is received? I assume it does not decrypt each message on the fly just when a search is initiated. This would also need to be conveyed to the user as not encryption works only at transport and no longer secures the message in storage.

Duplicate of this bug: 1807679

This bug is not a duplicate of bug 180769.

This bug arises from the fact that Thunderbird does not follow this long established principle: data confidentiality in transit and data confidentiality in storage are distinct requirements and call for distinct implementations.

The result of this bug 188988, is this: if a message was encrypted in transit, Thunderbird can't search the body of that message.

bug 180769 arises from the fact that that the dialog for Menu>Edit>Find>Search Messages includes many choices of what to search for, but does not include "Encryption Status."
This has nothing to do with the fact that twenty years ago Thunderbird's choice was to implement data confidentiality in storage for encrypted messages by storing the message in its encrypted form.
The result of that bug, 1807679, is this: Thunderbird can't search for messages that were encrypted in transit.

bug 180769 may not be a bug In this sense: The choice, Menu>Edit>Find>Search Messages>Match>Subject>Customize..., allows the user to add other message headers to the choices in the list under Subject. For example, searching for Content-Type--contains--smime would do the job. (Or the corresponding text for an OpenPGP message.)
The Thunderbird experts will know.

I could check that in a minute. But I must go attend to something else. I'll post a general comment on this bug later.
Meanwhile: I do not have permission to change the Status of bug 180769.
Someone please do: thank you!

Apology for the boldface. I don't know the Markdown rules. (Nor, you see, do I know how to edit my post.)

No longer duplicate of this bug: 1807679

Thanks, Wayne!

I'm back from something else and just checked. Yes: Thunderbird will search for and find encrypted messages. (At least if S/MIME was used.)
1807679 asks for a Thunderbird feature to make that very easy.
My note on the 1807679 page tells how, with a small touch of quick and easy customization, to search for encrypted messages.

Now let's push forward on searching the Body encrypted messages.
Please.

(I am required to encrypt all my work messages. I have been waiting over fifteen years. Some have been waiting over twenty.
All encrypted messages. Thunderbird reports that I have 67,320.
[ Search Messages : Match all of the following : Date : is before : 1/18/2023 ]
)

You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: