Beginning on October 25th, 2016, Persona will no longer be an option for authentication on BMO. For more details see Persona Deprecated.
Last Comment Bug 188988 - encrypted emails are not searched
: encrypted emails are not searched
Status: NEW
Product: MailNews Core
Classification: Components
Component: Backend (show other bugs)
: Trunk
: All All
: -- major with 20 votes (vote)
: ---
Assigned To: Nobody; OK to take it and work on it
: 336628 644723 (view as bug list)
Depends on:
Blocks: glodafailtracker
  Show dependency treegraph
Reported: 2003-01-13 19:13 PST by Julien Pierre
Modified: 2016-02-17 07:54 PST (History)
36 users (show)
See Also:
Crash Signature:
QA Whiteboard:
Iteration: ---
Points: ---


Description Julien Pierre 2003-01-13 19:13:51 PST
When doing an "advanced" search of messages, ie. for example in the "body"
field, Mozilla does not search messages that have been encrypted.
This significantly reduces the usefulness of the mail program.

I would expect mozilla to be able to search messages when the user is logged in
to the necessary token containing the private key.

It might take a very long time to process however, in the case of a hardware
token, due to hardware access time. A partial solution would be for mozilla to
cache the symmetric keys for messages as long as the token hasn't been removed
or the user didn't log off from the token, so that one doesn't need to access
the token again. Ie. when you move back and forth between messages in mozilla,
it should only access the hw token the first time you view a given message, but
not again if you come back to it two seconds later.

Another solution to improve performance would involve storing the encrypted
messages differently. Perhaps they could be stored twice :
- once in their original format
- a second time decrypted and re-encrypted with a master symmetric key. This
would be for searching purposes only. All messages would be encrypted with the
same symmetric key; however you would still need access to the hardware token to
decrypt that key. But you could decrypt that key from any given encrypted
message stored. The benefit is that the process isn't limited to the software
security device, the messages aren't in the clear, and performance will be very
good due to only needing to access the hardware token once per search to get the
master symmetric key
Comment 1 Kai Engert (:kaie) 2003-01-16 07:22:03 PST
Just some thoughts, I'd say duplicating the required storage should be avoided.
Having the ability at all, to search through archived encrypted messages, would
be a good first step, even it were slow.
Comment 2 Tom Braun 2004-09-20 18:25:26 PDT
This is such a serious issue.  The bug has been filed long ago, but is there
anyone actually working on this? 

I have been trying to convince my employer to switch to Thunderbird and Enigmail
from Outlook, but it is really diffcult to do if you cannot search in your e-mail
messages.  This basically makes the otherwise wonderful mail client and encryption
almost useless in a serious setting.

I know this is not an enigmail issue, but the mail client knows that enigmail
needs to be called in order to display the message.  Therefore, it should be
similarly possible to know that enigmail needs to be called in order to search
the message, no?

There will certainly be performance issues, but something is better than nothing
at all.  Just get that feature in, and performance improvements can be worked
on in the future.  Please!

Mozilla (thunderbird, firefox, sunbird, etc.) has great stuff, but not being
able to search your messages really makes it useless in many situations.  :-(


Comment 3 Julien Pierre 2004-09-20 20:09:45 PDT

You are preaching to the choir here. You might want to try to locate some people
who actually work on mozilla mail and add them to the cc list of this bug. At
the very least, it should be assigned to a current bugzilla e-mail address.
Unfortunately, I don't know who the person would be for mail.
Comment 4 Bernhard Walle 2004-12-05 13:35:30 PST
This is not only a problem with Enigmal, but also with the integrated S/MIME
Comment 5 Wayne Mery (:wsmwk, NI for questions) 2007-09-06 13:51:43 PDT
do we know any cases currently where search fails for crypted messages?
Comment 6 Kai Engert (:kaie) 2007-09-14 09:24:01 PDT
(In reply to comment #5)
> do we know any cases currently where search fails for crypted messages?

Yes. I used this test case to avoid having to search through a lot of messages and waiting a long time:

- compose message yourself, in the body enter "test1234"
- send as encrypted message to yourself
- create folder "encrypted"
- move the message from yourself to folder encrypted
- in thunderbird, use edit/search/search for messages
- select "search in" folder encrypted
- select "message body" contains "test1234"
- hit search

actual results: nothing is found

Comment 7 tlauck 2008-07-27 17:53:48 PDT
In some things, the best is the enemy of the good.  In this case, I would be perfectly happy if all of my encrypted mails were stored in unencrypted form. They are actually in a folder that is protected by the Windows Encrypting file system in any event. My interest in encryption is in keeping ISPs and others who do not have access to my computer from reading my mail while it is in transit. Indeed, if mail were kept in decrypted form then email encryption could be almost completely transparent.

Others might want stronger encryption. When I first suggested this decades ago to the originators of PEM they said it was a bad idea. I disagree. Almost no one encrypts their email today and lack of transparency is one of the main reasons.

If Thunderbird is going to store encrypted email in decrypted format it will have to be a user interface option.
Comment 8 Julien Pierre 2008-07-28 15:24:51 PDT

That may be an acceptable option for you if you use POP and download the messages onto a file system you believe to be secure. Definitely it should not be the default.

And in the case of people using IMAP servers, we would never want to store decrypted copies of the messages onto the servers.
Comment 9 Vladislav Bolkhovitin 2011-03-25 10:59:52 PDT
Any update? This is critical problem if S/MIME is used. There's no use of an e-mail client if it can't find messages it's stored. It could force me to start looking on alternatives to very nice Thunderbird, which I like using very much.
Comment 10 tlauck 2011-03-25 11:24:54 PDT
I concur.  This problem makes the use of S/MIME useless for me, too.

I am appalled that this problem has been outstanding for so many years. It calls into question the viability of the entire Mozilla open source process.
Comment 11 David :Bienvenu 2011-03-25 12:01:32 PDT
gloda should be able to search encrypted e-mails, right, Andrew?
Comment 12 Andrew Sutherland [:asuth] 2011-03-25 12:28:41 PDT
Yes, since gloda stores the message body plaintext in SQLite, gloda would be the easiest solution to the searching problem.

If someone were interested in implementing this, they would need to teach gloda how to decrypt encrypted messages (is it a flag we tell the streaming layer?), as well as provide an explicit opt-in UI that causes gloda to enter a mode of operation where it will trigger the decryption process.  We would not want to start storing the encrypted messages in a decrypted form without explicit approval.

Vladislav, tlauck; would you be interested in trying to implement this?  The open source process is driven by people solving problems they are interested in fixing.  As you can probably tell, there basically no active contributors trying to improve the state of S/MIME and encrypted e-mail inside Thunderbird, which means we need new contributors who are interested in the state of S/MIME and encrypted e-mail in Thunderbird!  That could be you!

(The programming effort required in this case would not be complicated, it's more a question of learning about the better-documented internals of Thunderbird and the mozilla platform.)
Comment 13 Kai Engert (:kaie) 2011-03-25 12:35:09 PDT
Before implementing a feature that copies the contents of user's encrypted emails to a non-encrypted search database, a policy decision should be made.

Is the above approach acceptable or not?

Or should, for security reasons, searching in encrypted emails be restricted to linear walking through mails?

Maybe the behaviour of "copy encrypted messages to plain serach index"
should be based on a user preference.

Maybe the default of that preference should be in the more secure setting.

And I agree with the previous comment.
This is an open source project.
If you want features, we need contributions.
Contribution includes the process to find compromises acceptable to the project peers and helping with patches.
Comment 14 Ludovic Hirlimann [:Usul] 2011-03-27 09:00:11 PDT
*** Bug 644723 has been marked as a duplicate of this bug. ***
Comment 15 tlauck 2011-03-27 10:10:17 PDT
If decrypted emails are stored in plaintext form then there won't be a problem with searching, either in terms of performance or uncertainty as to what information may be leaked by searches. 

Storing the plaintext as well as cyphertext seems the easier way to deal with the search problem, given that there are already reasons why some people want to store the plaintext of decrypted mail in folders.  (One such reason concerns the problem of expired private keys. Requiring these to be stored is itself a security problem.)

If agreement can be reached on a suitable user interface (including defaults the user can set up) then it doesn't seem like a major problem to implement this capability. A message would reside in a folder either cipher text only (as at present) or cipher text plus plaintext (when the new capability has been applied to the message).  Messages in the former form could not be searchable, but messages in the new form would be. (By analogy the mail is in a sealed envelope or the mail has been opened and the document removed from the envelope but the envelope is retained to permit resealing, for evidence, etc.)

In the past there appeared to be some controversy as to whether or not this capability is a good idea.  If there can be consensus that this should be an option, then the next step would be to entertain suggestions as to what a suitable user interface might be.
Comment 16 Vladislav Bolkhovitin 2011-03-29 13:01:57 PDT
I don't think searching in encrypted emails stored locally should be restricted anyhow. I believe that purpose of S/MIME is only to protect messages during transfer and storage on intermediate mail servers. If they are in my local mail box, there's no point to protect them. At least, for me and people like me. If I wanted to have my mailbox encrypted, I'd rather use one of many products encrypting on the file system level, starting from encryptfs on Linux. Anyway, if you want your e-mails be encrypted, you'd definitely want to protect as well many other related things (address book, etc), where only file system level encryption can help you.

I believe problems of searching in S/MIME messages and storing or not them in decrypted form are quite orthogonal. If we start deciding to store decrypted messages, we would have to decide many related questions, like if to keep or not the encrypted form for future references, e.g. to check if messaged were transfered encrypted, with which keys, by which ciphers, ..., if not to store, then how and where to keep those reference data, etc.

If to store both encrypted and decrypted messages when encrypted only messages are not searchable, it would still keep the confusion that if I can with all the keys open see all my encrypted only messages by a mouse click, but a plain body search (not gloda) in them doesn't return anything. Such behavior is pretty disappointing and would distract people.

(Plus storing both encrypted and decrypted messages would be a big storage overuse.)

From other side the searching problem is simply about be able to find bodies of encrypted messages which users can read by a mouse click.

So, better to not confuse those 2 problems and solve the searching bug at first. Then later we can go ahead with the storing decrypted messages problem. I agree, better to store encrypted messages decrypted, but there's too many related problems to solve before making it possible.

I can look on implementing the search. But please somebody give me some step by step guide where to start looking to implement it, similarly as I wrote in Thunderbird and Mozilla are very big projects, so it would save a lot of time.
Comment 17 Andrew Sutherland [:asuth] 2011-03-29 13:37:06 PDT
Thank you for your interest, Vladislav!

The general overview for the global database mechanism is here:

The general documentation for libmime, the low-level layer that performs the actual parsing of messages is at the top of this file:

The libmime encryption overview header file is here:
with the encryption guts here:

The global database messaging process initiates streaming of the message here:

Our JavaScript wrapper around libmime that does this exposes its API here:
but the actual representation building happens here:

libmime is aware of the implementation of the JS mime emitter and has logic that sets specific flags to enable special behavior:

The unit test for the JS mime emitter lives here:

It knows how to make fake/gibberish S/MIME messages and uses this test library to do that:

You would run the test by building thunderbird, changing to the objdir/mailnews/db/gloda/test dir and running:
make SOLO_FILE=test_mime_emitter.js check-one

There are also other global database tests in that directory, you can run them all by doing the following from the same directory:
make xpcshell-tests

The general technical strategy I would propose as the first steps:
- Figure out how to cause the mime streaming process to cause decryption of messages, ideally exposing this as a flag that can be passed to mimemsg.js's streaming mechanism.
- Modify the indexing mechanism of the global database to pass that flag and attach it to a preference probably.

As Kai makes the excellent point, the defaults and such are something that do need some discussion and the appropriate venue for that is the tb-planning list:

I generally agree with your viewpoint that if you actually want your messages secured locally, you should be using an encrypted file-system for storage plus potentially encrypted swap.  Thunderbird makes zero guarantees about keeping decrypted message content safe once we decrypt it; it could easily end up in un-locked memory and swapped to disk, etc.

Please let me know if you have additional questions or need additional guidance outside of the actual S/MIME implementation.  I have basically zero knowledge about how that works, and Kai is the only person I am aware of with any familiarity with the implementation.  (And of course Enigmail/PGP is not directly supported in-tree at this time.)
Comment 18 Andrew Sutherland [:asuth] 2011-03-29 13:39:07 PDT
And the documentation for building Thunderbird should be here:

You may also want to drop into #maildev IRC channel or ask for feedback not specific to this bug in tb-planning.  Other documentation and info on those venues can be found here:
Comment 19 David.P 2011-08-29 02:32:31 PDT
Quote from Bug #260665:

"This limitation makes this otherwise wonderful mail-client and encryption module
BASICALLY UNUSABLE in a serious environment." (emphasis added)

That's exactly my feeling and experience. 

I will have to quit TB sooner or later for professional use if I won't be able to search my S/MIME and Enigmail encrypted mail. This feature (-> full text search in encrypted mails OR decrypted local mail storage) is so crucial that it single-handedly can make the difference between failure or success of my work (being a patent attorney).

Simply make it an option to STORE mail that has been TRANSFERRED (!) using encryption DECRYPTED locally.

Mail encryption of course is for TRANSFER (like a sealed envelope with paper mail) but not for local storage! Nobody is so stupid to re-seal and re-cut-open their paper letters every time they need to look up something, since this simply will KILL their productivity and thus their business! 

Rather and of course you keep your paper letters locked away in a safe cupboard (but with OPEN envelopes, or rather WITHOUT envelopes!) if they contain sensitive information.

I want to bang my head against the wall if I think about the ignorance at Mozilla that has been forcing people for almost a decade to keep (and re-seal every evening, like idiots) every single ******* envelope of every ******* letter that they store in their office cupboards.
Comment 20 David.P 2011-09-06 04:02:31 PDT
This Bug SUCKS me so much EVERY DAY.

Know what? Every time I need to read those ca. two lines of text in one of my encrypted e-mails that also contain like 15MB of attachments: my triple monitor multi CPU multi SSD monster PC stalls at 100% CPU load for like 10 seconds before it displays those stupid two lines of text that I need to read.

This not only SUCKS incredibly because it stupidly and unnecessarily disrupts one's workflow. Additionally, it also takes like 50 additional Watts of CPU power times 10 seconds which is exactly 0.1389 Watt Hours. 

This 20 times a day times 10 Million users makes 1.1575 megawatts of wasted electrical power (365/24/7) because of the STUPIDITY and IGNORANCE of that guy at Mozilla who supposedly has decided that users should not be allowed to store encrypted mail in decrypted state locally, nor detach or delete attachmens from their encrypted mail.

This Bug SUCKS me so much EVERY DAY.
Comment 21 Mark Banner (:standard8) 2011-09-06 05:48:06 PDT
David, while it's entirely reasonable to be frustrated by aspects of Thunderbird, Bugzilla is a venue specifically for driving the technical aspects of bugs forward.

If you feel the need to advocate or vent, <> is a more suitable place for that.

Please review <> before your next comment in Bugzilla so that you have a better understanding of what is and isn't appropriate here.  Thanks!
Comment 22 Wayne Mery (:wsmwk, NI for questions) 2011-09-06 06:16:34 PDT
*** Bug 336628 has been marked as a duplicate of this bug. ***
Comment 23 David.P 2011-09-06 06:23:11 PDT
Thanks. I however see no other way of making the point to PLEASE finally START WORKING on bugs like this WICH HAVE NOT BEING WORKED ON for a ******* decade!

Thank you all.
Comment 24 Joaquin Miller 2011-09-06 10:23:01 PDT
This is going to be long, so please bear with me, Dear Reader.

Vladislav, Andrew, and others who agree that S/MIME is for mail in transit, o n l y, please skip this post.

tlauk writes: In the past there appeared to be some controversy as to whether or not this capability is a good idea.

I would like to convince you that it is a good idea. 
That what we have here is not a bug, but a design flaw.

There is no blame, so it did not matter then, just as it does not matter now, nine years later, how this design flaw came to be in Thunderbird.

tlauck is right.  
In some things, the best is the enemy of the good.

In this case, the enemy is a misunderstanding of what is the best.

If you think about it, I believe you will agree that the question of confidentiality of data in transit is distinct from the question of confidentiality of data at rest.

The question of when and why and how to secure secure mail in transit is a different question from the question of of when and why and how to secure secure mail when in the recipient's control.

S/MIME is a solution for mail in transit.  It seems to do the job just fine. (The problems with keys and certificates and the whole (not required) PKI structure are not the fault of S/MIME.)

How to secure mail at rest?  

tlauk is right again.  
File system encryption is the solution for confidentiality of data at rest.

Forgive the choice of words, not offense intended, good hacks are excellent: 
One hack confidentiality of stored mail is to store the S/MIME attachment.

It is not a good hack, if it prevents searching messages.

Dear Reader, please take yourself back to the time that Julien Pierre posted this bug, six months before the release of Thunderbird 0.1, over eight years ago. Become an architect of Thunderbird. Thinking as a software architect, design an implementation of the confidentiality function for Thunderbird. 

I except you will say to your self: We'll use S/MIME, for all the obvious reasons.  That will provide confidentiality in transit.  Now, ask yourself: What about confidentiality at rest, on the user's machine?

Vladislav Bolkhovitin is right.  S/MIME is to protect messages in transit.

I feel the correct conclusion is this: The job of the mail reader is to create, send, receive, store, and search mail.  Security at rest is the job of the file system the mail reader using.

"Wrong," you say.  And you are correct: I ask, "Why?" and you reply, "Because the file systems our users are stuck with do not provide the confidentiality function."  I find I have to agree: certainly Windows and Mac OS did not provide confidentiality in a satisfactory way.  And TrueCrypt 1.0 was a year in the future. 

You may may feel that we need to look out for the less sophisticated user.  They will not understand, may not even think about the issues. If we store the message in the clear, we may be doing them a disservice. 

Perhaps that is how the decision was made back in 2003.  
But that was then.  This is now.

We have an adequate-for-many-users, well integrated encrypting file system on Windows.  We have pretty good full disk encryption on Mac OS (finally!). We have TrueCrypt. 

What we also have is that Thunderbird users can have either the message searching function or the confidentiality in transit function, whichever they prefer.  

But not both.

I have co-workers who want to use Thunderbird.  And did.  But finally gave up on Thunderbird.  It is  e x t r e m e l y  frustrating to not be able to search e-mail.  Makes it  h a r d  to do one's job.

Thanks, Dear Reader, for sticking with me. I must abjectly apologize for the fact that I am incompetent to be a contributor on this.

Thanks, Vladislav for volunteering.  Thanks, Andrew for coming right back to help. 


Q: Why save the encrypted form of the message?
A: (in the form of questions) Is that required for authentication, non-repudiation, and integrity? 

Is the signature calculated on the cleartext of the S/MIME attachment or on the encrypted text?

If so, then, until someone volunteers to provide the code to preserve these functions once only the cleartext is stored, the ciphertext will need to be stored, too.
Comment 25 Andrew Sutherland [:asuth] 2011-09-06 10:43:48 PDT
As a technical update about the amount of work required if someone wants to move things forward technically, bug 527927 (targeted at Thunderbird 8) has recently taught the JS Mime Emitter about encrypted messages.  This would likely reduce or eliminate any libmime hacking I discussed in comment #17.

The mechanism allows the JS mime emitter to see encrypted parts and their body contents and they are specially indicated as such.  This occurs for S/MIME encrypted messages and, when using enigmail, PGP-mime messages, but not PGP ASCII-armored messages which enigmail has to magically fix-up somehow using the message reader.  I believe (but could be wrong) that we only get the decrypted part if the relevant crypto subsystem has the crypto keys available and we won't cause prompting.  As such, someone wishing to enable pervasive indexing of decrypted messages would need to also address the need to make sure the keys are available to the indexing process and the fallout if the user decides not to enter the password, etc.

It is important to note that the patch causes the global database to intentionally avoid indexing the contents of such messages for consistency with our current behaviour of not indexing encrypted messages.
Comment 26 Vladislav Bolkhovitin 2011-09-06 13:53:10 PDT
I might be missing something, but it doesn't look like fixing 527927 will help anyhow fixing this bug.

We don't really want to search in encrypted e-mails. Instead, we want to have ALL SMIME/PGP/etc encrypted emails be stored locally decrypted in plain text and let lower FS level to handle encryption if necessary (see comments 20 and 24). Then we will get the search possibility automatically.

Storing the ciphertext should be optional, because for most cases storing only flags that authentication, integrity, etc. tests passed on the decryption phase + keys used (for reference) should be sufficient.

I'd love implementing it, but, unfortunately, it needs more concentration than I can afford at the moment :(
Comment 27 Julien Pierre 2011-09-06 14:10:54 PDT

I don't think you can rely on the local filesystem being properly encrypted. I would be very wary of any scheme that stored the e-mails in plain-text. Such a scheme would not be secure accross platforms and would not be a good solution, IMO.

The first time when building the index, you would have to actually decrypt all e-mails. However after that, you could store this index securely, without relying on platform specific capabilities. For S/MIME users, the index could be stored as a special message encrypted with the user's private key. For other users, it could be encrypted with a symmetric key such as the one that already protects passwords in Thunderbird/Firefox.

Note : I'm not involved with Thunderbird development in any way at this point. I just reported the bug against PSM in Seamonkey originally.
Comment 28 Vladislav Bolkhovitin 2011-09-06 14:37:05 PDT

SMIME/etc is a _transport_ level encryption and only it. Many users, including me, don't need local encryption (see coment 20).

Moreover, as soon as you indexed encrypted messages, this encryption gets useless, because the index will, basically, contain context of those messages. Or are you going to put the index under SMIME as well?

Hence, FS level encryption is the best way to go.

Let's not confuse fundamentally separate things and go the simplest way by making boys go right and girls - left.
Comment 29 Julien Pierre 2011-09-06 14:43:11 PDT

Yes, read comment 27, that's exactly what I suggested, put the index in an S/MIME message.

If you want something that decrypts things in plain text locally, I suspect you will have a hard time finding someone knowledgeable enough about security who would be willing to implement this, and an even harder time passing review. Unless perhaps the patch was able to detect the properties of the local FS and figure out it was encrypted and secure enough. However, that would be of limited usefulness, IMO, given that most users still don't use encrypted file systems.

Encrypting the index would be a more generic implementation that should be secure regardless of the file system.
Comment 30 tlauck 2011-09-06 15:06:37 PDT
Anyone who is competent in security understands the different threat models involved. Anyone who is competent in computer systems architecture understands the importance of locating functions in the correct place in the system. 

I've about given up on this issue.  I don't believe all the volunteers are incompetent. I think there are probably government agencies that are doing their best to delay the widespread use of encrypted email. One way of accomplishing this is to make common mail client software impractical to use.

In addition, one competent in security protocols would understand that retaining encryption keys for transmitted traffic is poor practice.  One wishes to provide as good an approximation to "perfect forward secrecy" as possible. Presently, it is necessary to retain encryption keys forever to decrypt old mail messages. This in itself is a known security risk.
Comment 31 Vladislav Bolkhovitin 2011-09-06 15:35:46 PDT
Once again, don't confuse 2 fundamentally separate things and let boys and girls know their rooms. There's a lot of FS level encryption tools on both Linux and Windows as well as on other platforms whose trustworthy is not less than of the S/MIME, so no S/MIME encrypted messages should be stored locally. KISS, you know.
Comment 32 tlauck 2011-09-12 12:03:09 PDT
Encrypting the index smacks of unneeded complexity. The complexity invades the software development process, confuses the users, and will degrade the responsiveness of the software.  In addition, if there are any encrypted messages it will be necessary for the user to provide a password to unlock them for searching to work properly. 

This is not only inconvenient, it is potentially insecure. In particular, old keys need to be retained long after their expiration lest old messages become unreadable. Even if the old messages have been thoroughly deleted from every computer system in which they were ever stored, there is a possibility that an encrypted form of these messages had been captured during their original transmission. If a computer containing a key for these messages should happen to become compromised the secrecy of these messages will be lost, even if they have been deleted.  (This is a variant on one of NSA's formerly classified threat scenarios, i.e. the present design of SMIME is far from providing "perfect forward secrecy".)

Whether incoming encrypted messages are automatically decrypted and whether they are stored in encrypted or decrypted form should be a user option. I would find it perfectly acceptable if the default were to decrypt when specifically requested and store in encrypted form. If the user has a secure file system or otherwise trusts his computer system then he can change these settings. If he doesn't then I would leave things as they presently are, i.e. leave the still encrypted messages unsearchable. The user always has the option of installing file system encryption and decrypting existing encrypted messages at any time.

This also has the virtue of being simple to implement.  :-)  

BTW, my email and all Thunderbird data associated with it is presently stored in encrypted form using the Windows Encrypting File System. At least after log out, none of this data will be available to someone who has physically captured my computer unless they can guess my very strong password. This works well to protect stored email and other client information.
Comment 33 Vladislav Bolkhovitin 2011-09-17 18:19:35 PDT
This is how I see this issue should be fixed.

TB would have 2 modes how to deal with encrypted and signed e-mails: "store decrypted" and legacy "stored encrypted", i.e. as it is at the moment.

Setting which mode to use would be available with the config editor. By default this settiong would not be set in any value, which means that TB seeing the first time an encrypted/signed message during any action (receiving, viewing, compacting, etc.) would ask user which mode to use offering by default new "store decrypted" mode.

In this mode TB would process encrypted/signed messages during any action (receiving, viewing, compacting, etc.) the same way. It would decrypt them and/or check signature, if the message is signed, then store it decrypted and set in its internal fields X-Mozilla-SMIME-Encrypt-Keys, X-Mozilla-SMIME-Sign-Keys and X-Mozilla-SMIME-Signature-Correct keys used to encrypt and/or sing this message as well as result of the signature verification (correct/incorrect) correspondingly. Then, if this message already saved, TB would delete the original.

This would make S/MIME messages full grade citizens and effectively make them fully searchable.
Comment 34 David.P 2011-09-22 03:39:40 PDT
First, thanks very much to Vladislav and Joaquin for their insightful contributions.

Anyway, this needs to be solved ASAP. I can't wait any more decades before I am allowed to take care of what I need to encrypt and what not.

I bl**dy need to full-text search my (growing) giant amount of business e-mail (and I need to be able to detach those thousands of useless 20MB attachments from my encrypted e-mails, see also comment 20). 


I suppose, what comes next is "well, then don't use it, sucker".


But thanks anyway for listening. 

PLEASE, someone who knows how do do this (I unfortunately don't) -- take some action and make a parser that crawls those local mbox files and decrypts them; and a patch that allows storage of new mails DEcrypted locally.

Or something :-(
Comment 35 Honza Bambas (:mayhemer) 2012-04-21 13:37:46 PDT
Can please somebody summary the status here in two/three lines?  I'm willing to work on this now.  My everyday work is ruined by this bug.  Thanks.
Comment 36 Andrew Sutherland [:asuth] 2012-04-21 14:45:08 PDT
(In reply to Honza Bambas (:mayhemer) from comment #35)
> Can please somebody summary the status here in two/three lines?  I'm willing
> to work on this now.  My everyday work is ruined by this bug.  Thanks.

Technical discussion on the bug has been primarily about teaching the global database to index encrypted messages.  Please see the posts by me for information on that; it should be fairly easy now.  If you are looking for the "advanced search" mechanism to search encrypted messages, I am unable to provide any input on the technical issues.

Kai Engert has also raised potential policy issues which have not been discussed (tb-planning would be the venue for that), but a patch to cause the global database to index encrypted messages only when a (hidden) preference is set should be minimal enough that no discussion is really required.  A unit test that verifies that we don't index encrypted e-mails by default and that we do when the preference is set would be required.  If you wanted to surface the preference in the UI, you would need UX signoff from Blake Winton (:bwinton).
Comment 37 Honza Bambas (:mayhemer) 2012-04-21 14:57:33 PDT
Thank you.

I think I can start with a hidden pref.  UI can be solved in a different bug.

I will create the test as well.
Comment 38 Joaquin Miller 2012-04-21 15:33:35 PDT
Honza: Praise you for volunteering!

Andrew: I don't know what the "advanced search" mechanism is. 

Menu Edit Find SearchMessages brings up a useful search window.  All the header fields of encrypted messages that mail readers leave in the clear (To:, From:, Date:, Subject:, ...) can be searched just fine.

<all caps> some of us want to search the body of the message </all caps>

(I write "want to."  It means: can't do their work if they can't search the body.)

I trust the proposed fix enables us to do that.


Kai: As to the policy question: I'm sure Mozilla procedures will take care of that.
I do hope everyone who participates in the policy discussions will agree to the base architectural principle: Security in transit and security in storage are separate items; architects need to keep them separate in their mind.

(There is little wrong with looking out for naive (best sense of the word) users by keeping encrypted what arrives encrypted; a reasonable hack (best sense of the word) for that may be to store the S/MIME attachment and not store the clear text body.
(If so: please fix Thunderbird so it can   a l s o  be used by  not-naive users who must encrypt in transit and must search in the body of their mail.  Those not-naive users will know if they need to encrypt in storage and how to accomplish that.)

Many thanks again to all.
Comment 39 Andrew Sutherland [:asuth] 2012-04-21 15:49:09 PDT
(In reply to Joaquin Miller from comment #38)
> Andrew: I don't know what the "advanced search" mechanism is. 
> Menu Edit Find SearchMessages brings up a useful search window.  All the
> header fields of encrypted messages that mail readers leave in the clear
> (To:, From:, Date:, Subject:, ...) can be searched just fine.

That's what I'm referring to as advanced search, although it's bad terminology since no part of the process has the word advanced in any captions, so I'll stop referring to it like that.  It's not hooked up to the global database mechanism in Thunderbird, although I think Eudora/Penelope has forked the dialog and there is an option to use the global search mechanism in their UI.

That mechanism can be made to work on encrypted messages too, I just can't speak as to why it doesn't work right now and so what would be required to make it work.  It's probably something small (which is not to say easy to find or entirely trivial to fix.)
Comment 40 Joaquin Miller 2012-04-21 16:07:34 PDT
Thanks for straightening me out, Andrew.

How shall I put this?

Darn.  A mail reader that won't let the user search her e-mail messages (the message part of the message: the body) ain't too good.
Comment 41 Magnus Melin 2012-04-22 23:08:28 PDT
xref bug 280588
Comment 42 Kai Eckert 2012-08-05 16:15:03 PDT
(In reply to Honza Bambas (:mayhemer) from comment #37)
> Thank you.
> I think I can start with a hidden pref.  UI can be solved in a different bug.
> I will create the test as well.

May I ask about your progress? I can't help, but would be willing to test it.
Comment 43 Honza Bambas (:mayhemer) 2012-08-06 04:17:08 PDT
(In reply to Kai Eckert from comment #42)
> May I ask about your progress?

None.  Releasing for now, I could get to it later (months).
Comment 44 Honza Bambas (:mayhemer) 2012-09-19 08:02:52 PDT
I don't intend to work on this bug any more, my priorities has unfortunately changed.
Comment 45 O. Furlong 2013-12-26 14:28:54 PST
I'm currently trying to organize a solution for bug 280588 which will solve this bug also. I'm not a programmer, but willing to organize a crowdfunding to finally remove this 10 year old trouble beast. Please input your comments in the other thread.
Comment 46 O. Furlong 2014-01-26 09:29:37 PST
We started a crowdfunding for a related issue, which will be a workaround for at least some users. Your help is appreciated:
Comment 47 Baturay Kaya 2014-05-07 14:10:19 PDT
Hello Everybody,

I have read this post and a couple related or similar bug posts and I thought maybe I can work on this bug. Before I start to work, I just wanted to ask if there is any progress?
Comment 48 Honza Bambas (:mayhemer) 2014-05-12 04:49:11 PDT
Definitely not from me.  Go ahead!
Comment 49 wayne 2014-11-24 06:50:21 PST
I'm not a programmer (although I plan to be one day!) but I wanted to simply add that this is, as everyone said, a kind if 'deal breaker' for most people to make the switch to a) Thunderbird and b) secure email with PGP.  I have about a year's worth of enrypted communications with my close friends, and one day I needed to search some keywords in those communications. I expected that I would be prompted for the PGP passphrase which would then allow the encrypted emails to be searched in addition to unencrypted.  Apparently this doesn't happen!  I could not find the content of the email I needed which caused me a good deal of concern.  If I can help in any way, let me know.
Comment 50 Tilo Freiwald 2015-03-20 10:39:23 PDT
Thanks to Janosch Rux, Patrick Brunschwig and others a filter rule to decrypt messages permanently was added to Enigmail. Maybe that solves this bug here?

Thank you so much!
Comment 51 David.P 2015-03-20 10:49:01 PDT
Whoa can this be true, after more than a decade of this Bug?

However, there is still the problem that S/MIME encrypted messages are still stored encrypted and can't be searched.
Comment 52 Sebastian Bitzer 2015-04-14 01:50:39 PDT
I would also like to point out again that the issue is the same for OpenPGP/MIME signed messages. In this case I understand it even less that Thunderbird can't search the body of the message. Yes, it's perhaps a bit more hidden than the body of a standard message, but at least its still available as clear text, no?
Comment 53 Ludovic Hirlimann [:Usul] 2015-09-25 05:46:59 PDT
Removing myslef on all the bugs I'm cced on. Please NI me if you need something on MailNews Core bugs from me.

Note You need to log in before you can comment on or make changes to this bug.