124486 - Enable Outlook Express importing when OE is not in default location or not installed (select files/folders)

Status: RESOLVED FIXED

(MailNews Core :: Import, enhancement)

Description

[Jeff Partridge]

The mail app can't find the Outlook Express address book when you try to import
it if the address book isn't in the default directory for that user. I keep my
address book and mail in a directory as far away from my C: drive as possible.
There's no place to tell Mozilla where to look for the address book and it never
found it. I had to open OE, export it to a csv file, and import it into Mozilla
from that.

Comment 1

[Jason M. Abels]

I believe this is the correct behaviour, as not many people would keep their
outlook express AB anywhere but the default application. WONTFIX seems likely
for this bug.

Reporter: can this be recreated on a modern build of mozilla?

Comment 2

[Todd Cohn]

Actually, I think this should be changed too.  There should be an option to
select a different location then the default.  I ran into this when I was trying
to import OE data from a seperate hard drive that was no longer bootable. 
Editing summary to make it mroe clear.

Comment 3

[Matthias Versen [:Matti]]

*** Bug 174962 has been marked as a duplicate of this bug. ***

Comment 4

[Daniel Wang]

*** Bug 153419 has been marked as a duplicate of this bug. ***

Comment 5

[Daniel Wang]

*** Bug 190469 has been marked as a duplicate of this bug. ***

Comment 6

[Aleksander Adamowski]

Seconding, apart from autodetecting (as it currently works) the location, there
should be a "browse..." button for importing Mail/Address books.

Some people are distrustful of such import facilities and want to run it on a
copy of original file(s), other people are moving to Mozilla after they had to
reformat their drive and restore mail from a backup - in those situations users
need to point to a file in a non-standard location.

Comment 7

[Jo Hermans]

*** Bug 228469 has been marked as a duplicate of this bug. ***

Comment 8

[OstGote!]

*** Bug 278212 has been marked as a duplicate of this bug. ***

Comment 9

[Albert Feller]

*** Bug 287173 has been marked as a duplicate of this bug. ***

Comment 10

[Eero Volotinen]

There must be way to import mails to thunderbird from user defined location.

Without it it makes impossible to switch to thunderbird. Hope this is fixed
really soon..

Comment 11

[Albert Feller]

*** Bug 281994 has been marked as a duplicate of this bug. ***

Comment 12

[Albert Feller]

*** Bug 288052 has been marked as a duplicate of this bug. ***

Comment 13

[Steve Laniel]

I'm kind of baffled why this is still an open bug. How difficult is it to stick
a file->open dialog somewhere in the sequence of import steps, and have the user
point to a PST file? This is *exactly* what I've needed for a client with many
thousands of email messages. Instead it seems that the easiest approach was to
create a dummy IMAP account, then upload the Outlook messages to that account. I
should have been able to do this all locally, with Thunderbird.

Comment 14

[Ben Slade]

Just a note, this problem is still active for me as of Thunderbird version 1.5
Beta 1 (20050908) on Mac OS X (10.4.2)

I was able to manually work around the problem by manually copying all the files
from my Mozilla profile (on Mac OS X, all the files under the
~/Library/Mozilla/Profiles/<myprofile>/*.slt/* ) to my Thunderbird profile(
(under ~/Library/Thunderbird/Profiles/default/*.slt/ ).  It's probably best to
make sure neither Mozilla or Thunderbird is running while you do this.

I haven't test this extensively yet, but it seems to have imported everything,
including addresses and security devices.

See http://www.mozilla.org/start/1.5/faq/profile.html#location to figure out
where your profiles are stored.

Comment 15

[Stebs]

Since Bug# 261295 importing mail from outlook requires outlook to be default email client
is a duplicate of this Bug (though that's not really correct imho):

A workaround would be to let Thunderbird Adressbook and Mail Import backup this Registry Value: [HKEY_LOCAL_MACHINE\SOFTWARE\Clients\Mail]@="Mozilla Thunderbird", set it to "Microsoft Outlook", do the import and then resetting the previous Value. That would prevent the annoying neccesarity having to manually set Outlook as default E-Mail Client.

Comment 16

[Magnus Melin [:mkmelin]]

*** Bug 319253 has been marked as a duplicate of this bug. ***

Comment 17

[Keith W Blackwell]

Some commenters have thus-far missed an important aspect here, which is illustrated by my situation.  I am trying to import an address book from an older system where Outlook Express was used, but the new system does not have Outlook Express, nor will I ever install it.  I don't even *know* the "default location", nor would any normal user in my situation.  I need to be able to tell Thunderbird where my old address book files are stored.

This is NOT an enhancement request.  Without this feature, I simply *cannot* import my old address book, period.  If someone has to go googling for information to figure some complicated process in order to work around the defect, then even if a workaround is found, that does not in any way diminish the seriousness of the defect or turn it into an enhancement request.  This is most assuredly a hole in the design that acts like a slap in the face to anyone who needs this functionality.  It is similar to the fact that I cannot import my old unix-style mbox-format mail folders simply because Thunderbird does not provide a way to do it, even though it should be trivial since Thunderbird uses the same format internally.  This is a design defect:  missing functionality that *should* be there, that prevents proper use of the application when it is *not* there.  Please change this from an "enhancement" to a "normal" defect.

This is still present as of 1.5.0.4.

Comment 18

[Andrea]

This should not be an enhancement in fact, because it seriously compromises the migration from other system.
In my case, I crashed my Office installation, and now 
I DO NOT WANT TO INSTALL OUTLOOK AGAIN 
just to import my email and contacts. 

I have several .pst files from my backups and I need to import from that files. 

The same bug applies to other mail systems, and expecially for the Contact List.

Comment 19

[timeless]

an enhancement is any time when you're adding code to handle something that was not handled. this wasn't handled, and thus it's an enhancement.

it doesn't matter. if you don't post a patch it will sit here for as long as it was going to sit whether or not it was marked as an enhancement.

so stop whining, and post a patch.

Comment 20

[Martin Andersen]

Sure, everyone know how to code, right? At least anybody worth a damn in your eyes...

Comment 22

[David Rankin]

(In reply to comment #0)
> The mail app can't find the Outlook Express address book when you try to import
> it if the address book isn't in the default directory for that user. I keep my
> address book and mail in a directory as far away from my C: drive as possible.
> There's no place to tell Mozilla where to look for the address book and it never
> found it. I had to open OE, export it to a csv file, and import it into Mozilla
> from that.

Even after 4+ years this is still a top bug. An independent OE mail import is a critical step for new users transitioning to Linux/t-bird. New users who have backed up their OE mailstore before installing Linux and thunderbird are frustrated to learn that thunderbird can't import their saved OE mail or contacts. I would vote for a fix/addition

Comment 23

[Paul]

I would add my vote to keep this issue alive -- or apparently, to make this issue alive, since it doesn't seem to be taken seriously by the development team.  Which I suppose is understandable, since they are presumably already using Thunderbird and not concerned with other peoples problems importing emails from other systems.

But if they are serious about encouraging adoption of Thunderbird, this is a higher priority issue than many other fancy new features, because people aren't going to switch to Thunderbird if they can't take their old mail with them.  If Thunderbird were a commercial product, a functional import of other mail formats would be the first thing developed.  

It's sad, because obviously the hard work of doing the import/conversion is already done -- it's only the relatively trivial work of adding a menu option & file dialog box to allow the existing conversion to do it's job.

Comment 24

[Magnus Melin [:mkmelin]]

I don't know the importer internals, but from reading other bugs I gather this isn't as simple as "adding a menu option & file dialog box to allow the existing conversion to do it's job". 

If I understand the problem correctly the importer goes through OE to get the data it needs, and for some reason that can't (?) be done when OE isn't installed at all, or in a non-standard location.

Comment 25

[Ken Young]

I don't understand why the development team is either ignoring this or technically challenged by it. If they simply modified the code to bring up a file explorer window and allow the user to select the location of 'any mail file' be it Outlook or any other brand, then it doesn't have to 'know where to look'. Implemented this way would also allow the import of multiple Outlook or other files to 'combine' multiple sources of contacts, etc. Other tools have taken this approach, which is where I got this idea to pass along - very little in life needs to be invented. I don't know how the code is written, but I can't imagine a modification that I describe above is years of work - the age of this string.

Comment 26

[Ken Young]

Apparently someone emailed me I think inferring I was being mean or inappropriate in my response above. Every developer is a nice person and no one has to take any action to fix anything - I agree. However, when a feature is lacking going back 5 years that some of us want to see 'fixed, improved, enhanced - pick your description that does not offend - and is not deleted or enhanced - then I don't get the purpose of all of us having this communication for 5 years. I use as much open source as I can - and not because its 'free' - it isn't - we all know the real cost is in lost/improved productivity, use, etc - not the cost of the software - although that can be substantial. Cry's for Improvement from us non-developers to those who can improve it is solely for the desire to get the 'best possible software out there' and provide praise to those who make it happen. If my use of english in my prior response to other frustrated people who want to use this software offended anyone - then you're way too sensitive. My response was to the others like myself that are a bit frustrated and shocked at the age of this string, especially, at least speaking from my experience, the 'fix/improvement' rate on bugs/features discussed on this site has been very good. I cast my 1 vote that we users give up on this item being addressed and move on. This is my sign off on this string. Best wishes to all!

Comment 28

[Scott A. Colcord]

I believe that it would be useful to note the current status of PST importing to Thunderbird.  Is it still the case that import requires installing Outlook and using MAPI?

If so, I think that it would be useful to note the reason(s) that direct import from .pst is not supported, as it is undeniably a adoption barrier for some users.  Two cited reasons for not doing direct import appear to be:

1)  A comment on Bug 153419 suggests that there is some kind of Microsoft-imposed legal barrier to reverse-engineering the PST file.  Is this a genuine concern?  If so, what is the basis for it?

2)  It's "too hard" to reverse-engineer the PST file.  I assume this means "There's no developer willing to put in the effort".  This is certainly a valid reason, but I'll observe that there are at least two community-driven efforts to enable access to the PST file, and there could be benefits to working together on it.

See:
  http://sourceforge.net/projects/libpff
  http://www.five-ten-sg.com/libpst

Comment 29

[Wayne Mery (:wsmwk)]

xref:
- http://blogs.msdn.com/interoperability/archive/2009/10/26/roadmap-for-outlook-personal-folders-pst-documentation.aspx Oct 2009
- http://msdn.microsoft.com/en-us/library/bb177050.aspx
- http://www.microsoft.com/interop/principles/default.mspx Feb 2008

Comment 30

[Nikolay Shopik]

(In reply to comment #29)
> xref:
> -
> http://blogs.msdn.com/interoperability/archive/2009/10/26/roadmap-for-outlook-personal-folders-pst-documentation.aspx
Outlook Express isn't stores any messages in PST if I remeber correctly, this only apply to Microsoft Outlook.

Comment 31

[Joshua Cranmer [:jcranmer]]

Pursuant to comment 29:
It's now released:
<http://msdn.microsoft.com/en-us/library/ff385210.aspx>.

Browsing the documentation:
* The actual database specification format is around 100 pages. I'd estimate the actual complexity of implementing the code to be around 10-30KLOC for read-only access (I've never implemented a database before, so my estimates here should be considered a Nonscientific Wild Ass Guess).

* The specification refers to three tiers of structures: pages (the NDB layer), tables (the LTP layer), and the actual objects (the Messaging layer). Without fully groking it, we would basically care about looking at the messaging layer.

* Storage of actual messages/attachments/everything else is not defined in that 203-page spec. There's another 84-page spec that contains (some? most?) message data attributes we would need to see ([MS-OXOMSG]), and there's another half-dozen specs of similarish size that would need to be at least checked for relevant data.

* Most of the actual property specs are actually listed under the Outlook Exchange protocol section. Presumably, then, the code here could be shared with code for bug 128284 (and other Exchange or Outlook requested features, e.g., those for Calendar code), if we do not use external libraries for an implementation. Possibly also bug 77811 as well, although I would have to reread the TNEF documentation more carefully to be able to ascertain.

If a purely in-house solution is developed (libpst is GPL, so we can't use it, unfortunately), I would personally be hesitant to accept a patch without a moderately comprehensive test suite, as this essentially boils down to reading a new database format.

In short, it's not impossible to be written from scratch, but I think Thunderbird and SeaMonkey would be better served bringing in an external (dedicated!) library for this rather than trying to create one from scratch for our own uses.

Comment 33

[Hiroyuki Ikezoe (:hiro)]

Attachment: Enable to import OE addressbook other than default location

This patch enables to import OE address book even if OE is uninstalled.

Comment 34

[Hiroyuki Ikezoe (:hiro)]

Attachment: Test for imporing addressbook

A test.

This test depends on the modification of import_helper.js in bug 700920.

Comment 35

[Mike Conley (:mconley) (:⚙️)]

Comment on attachment 578184 [details] [diff] [review]
Test for imporing addressbook

The code looks fine by me.  Assuming these tests properly pass, r=me.

Comment 36

[David :Bienvenu]

Comment on attachment 578184 [details] [diff] [review]
Test for imporing addressbook

Sorry for the delay in looking at this. The import_helper.js part of this patch has bit-rotted, and I'm not quite sure what the correct way to resolve the conflict is. Please let me know, thx!

Comment 37

[David :Bienvenu]

My guess is that it would be resolved like this:

  else if (aType == "CSV" || aType == "TAB" || aType == "Outlook Express")
  {
    this.mSupportedAttributes = supportedAttributes;
    this.mLdif = false;
  }
  else
    do_throw("Unexpected type passed to the AbImportHelper constructor");

Comment 38

[Hiroyuki Ikezoe (:hiro)]

(In reply to David :Bienvenu from comment #37)
> My guess is that it would be resolved like this:
> 
>   else if (aType == "CSV" || aType == "TAB" || aType == "Outlook Express")
>   {
>     this.mSupportedAttributes = supportedAttributes;
>     this.mLdif = false;
>   }
>   else
>     do_throw("Unexpected type passed to the AbImportHelper constructor");

Those lines are already removed by the fix of import_helper.js in bug 700920 I attached.

Comment 39

[David :Bienvenu]

Comment on attachment 578183 [details] [diff] [review]
Enable to import OE addressbook other than default location

can you attach a patch that applies cleanly against the trunk? Thx! Clearing review request until then.

Comment 40

[Hiroyuki Ikezoe (:hiro)]

Attachment: Adapt to the latest trunk

Comment 41

[David :Bienvenu]

Comment on attachment 597681 [details] [diff] [review]
Adapt to the latest trunk

thx for the patch; I can't easily test it, but I'm trusting that you have :-)

Comment 42

[Mark Banner (:standard8)]

Unfortunately the patch for the unit test in attachment 578184 [details] [diff] [review] is bitrotted, please can you update that and re-set checkin-needed when ready.

Comment 43

[Hiroyuki Ikezoe (:hiro)]

The test depends on bug 700920.

Comment 44

[Hiroyuki Ikezoe (:hiro)]

Attachment: De-rotted patch

Comment 45

[Hiroyuki Ikezoe (:hiro)]

Attachment: De-rotted test

Comment 46

[Hiroyuki Ikezoe (:hiro)]

Now it's the time to check-in.

Comment 47

[Hiroyuki Ikezoe (:hiro)]

Attachment: De-rotted patch

I did attach the patch before hg qrefresh.

Comment 48

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/comm-central/rev/a1b2e3523367
https://hg.mozilla.org/comm-central/rev/9f4fa3df291c

Comment 49

[Ryan VanderMeulen [:RyanVM]]

The test has issues on Win7 and Win64:

https://tbpl.mozilla.org/php/getParsedLog.php?id=12973417&tree=Thunderbird-Trunk

TEST-INFO | c:\talos-slave\test\build\xpcshell\tests\mailnews\import\test\unit\test_oe_addressbook.js | running test ...
TEST-UNEXPECTED-FAIL | c:\talos-slave\test\build\xpcshell\tests\mailnews\import\test\unit\test_oe_addressbook.js | test failed (with xpcshell return code: 0), see following log:
>>>>>>>

TEST-PASS | c:\talos-slave\test\build\xpcshell\head.js | [do_load_manifest : 666] true == true

TEST-INFO | (xpcshell/head.js) | test 1 pending

TEST-PASS | resources/import_helper.js | [GenericImportHelper : 28] [xpconnect wrapped nsIImportGeneric] != null

TEST-PASS | resources/import_helper.js | [null : 320] true == true

TEST-PASS | resources/import_helper.js | [null : 41] true == true

TEST-PASS | resources/import_helper.js | [null : 48] true == true

TEST-PASS | resources/import_helper.js | [null : 49] true == true

TEST-INFO | (xpcshell/head.js) | test 2 pending

TEST-PASS | resources/import_helper.js | [null : 87] true == true

TEST-PASS | resources/import_helper.js | [null : 88] true == true

TEST-PASS | resources/import_helper.js | [null : 258] true == true

TEST-PASS | resources/import_helper.js | [null : 233] [xpconnect wrapped nsIAbDirectory] != null

TEST-PASS | resources/import_helper.js | [null : 235] true == true

TEST-UNEXPECTED-FAIL | resources/import_helper.js | John Doe == cltbld - See following stack:
JS frame :: c:\talos-slave\test\build\xpcshell\head.js :: do_throw :: line 440
JS frame :: c:\talos-slave\test\build\xpcshell\head.js :: _do_check_eq :: line 534
JS frame :: c:\talos-slave\test\build\xpcshell\head.js :: do_check_eq :: line 555
JS frame :: resources/import_helper.js :: <TOP_LEVEL> :: line 282
JS frame :: resources/import_helper.js :: <TOP_LEVEL> :: line 241
JS frame :: resources/import_helper.js :: <TOP_LEVEL> :: line 95
JS frame :: resources/import_helper.js :: <TOP_LEVEL> :: line 51
JS frame :: c:/talos-slave/test/build/xpcshell/tests/mailnews/import/test/unit/test_oe_addressbook.js :: run_test :: line 12
JS frame :: c:\talos-slave\test\build\xpcshell\head.js :: _execute_test :: line 304
JS frame :: -e :: <TOP_LEVEL> :: line 1

TEST-INFO | (xpcshell/head.js) | exiting test

TEST-UNEXPECTED-FAIL | resources/import_helper.js | 2147500036 - See following stack:
JS frame :: c:\talos-slave\test\build\xpcshell\head.js :: do_throw :: line 440
JS frame :: resources/import_helper.js :: <TOP_LEVEL> :: line 247
JS frame :: resources/import_helper.js :: <TOP_LEVEL> :: line 95
JS frame :: resources/import_helper.js :: <TOP_LEVEL> :: line 51
JS frame :: c:/talos-slave/test/build/xpcshell/tests/mailnews/import/test/unit/test_oe_addressbook.js :: run_test :: line 12
JS frame :: c:\talos-slave\test\build\xpcshell\head.js :: _execute_test :: line 304
JS frame :: -e :: <TOP_LEVEL> :: line 1

TEST-INFO | (xpcshell/head.js) | exiting test

Comment 50

[Ryan VanderMeulen [:RyanVM]]

Disabled for now.
https://hg.mozilla.org/comm-central/rev/b8eee2f01dff

Comment 51

[Ryan VanderMeulen [:RyanVM]]

And backed out due to red caused by the disabling.
https://hg.mozilla.org/comm-central/rev/911e68201428

Comment 52

[Hiroyuki Ikezoe (:hiro)]

Thanks, I am investigating it.

Comment 53

[Hiroyuki Ikezoe (:hiro)]

On Windows 7 (and maybe Windows 64), WABOpen always opens the default address book even if file name is specified in WAB_PARAM. It seems like a windows bug...