Importing from Eudora and Outlook busted in TB 38 and trunk.

NEW
Unassigned

Status

MailNews Core
Import
--
critical
2 years ago
2 months ago

People

(Reporter: Jorg K (GMT+2), Unassigned)

Tracking

(Depends on: 1 bug, Blocks: 1 bug, {crash})

Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

(crash signature)

Attachments

(1 attachment, 7 obsolete attachments)

(Reporter)

Description

2 years ago
+++ This bug was initially created as a clone of Bug #1175055 +++

Importing from Eudora and Outlook is busted in TB 38 and trunk.

In bug 1175055 the options were disabled. The problem needs to be fixed and the options need to be re-enabled again.

Updated

2 years ago
Crash Signature: [@ nsXPCWrappedJS::CallMethod(unsigned short, XPTMethodDescriptor const*, nsXPTCMiniVariant*)] [@ nsXPCWrappedJS::AddRef()] [@ mozilla::dom::workers::GetCurrentThreadJSContext()] → [@ nsXPCWrappedJS::CallMethod(unsigned short, XPTMethodDescriptor const*, nsXPTCMiniVariant*)] [@ nsXPCWrappedJS::AddRef()] [@ mozilla::dom::workers::GetCurrentThreadJSContext()] [@ nsXPCWrappedJS::CallMethod] [@ nsXPCWrappedJS::AddRef] [@ moz…

Comment 1

2 years ago
Created attachment 8712815 [details] [diff] [review]
bug1176748_import.patch - main thread only

Totally untested (except that it compiles and tb starts up).

If it works, we could clean things up too, but ... who knows, maybe someone wants to fix it properly some day.
Attachment #8712815 - Flags: feedback?(mozilla)
(Reporter)

Comment 2

2 years ago
Created attachment 8712844 [details] [diff] [review]
Part 2: Undo stuff from bug 1175055 so that the import becomes available again.

BTW, this is based on the Eudora removal in bug 1243498. Patch won't apply unless you apply the patch from the other bug first.
(Reporter)

Comment 3

2 years ago
Comment on attachment 8712815 [details] [diff] [review]
bug1176748_import.patch - main thread only

I hate to give this an f- but it crashes ;-(

Hit MOZ_CRASH(nsMsgBrkMBoxStore not thread-safe) at c:/mozilla-source/comm-central/mailnews/local/src/nsMsgBrkMBoxStore.cpp:49
#01: NS_LogInit[xul +0x558720]
#02: NS_LogInit[xul +0x6b6ad0]
#03: NS_LogInit[xul +0x3d79e5]
#04: NS_LogInit[xul +0x560ba3]
#05: NSSBase64_EncodeItem[nss3 +0x290568]
#06: NSSBase64_EncodeItem[nss3 +0x27d90f]
#07: _get_flsindex[MSVCR120 +0x2c01d]
#08: _get_flsindex[MSVCR120 +0x2c001]
#09: BaseThreadInitThunk[kernel32 +0x4d0e9]
#10: RtlInitializeExceptionChain[ntdll +0x419bb]
#11: RtlInitializeExceptionChain[ntdll +0x4198e]

https://dxr.mozilla.org/comm-central/source/mailnews/local/src/nsMsgBrkMBoxStore.cpp#49

Here's what I did: I compiled on Win 7 and copied over the dist/bin folder to the Vista box that has Outlook installed. There I removed the TB profile for a fresh start. The first panel that comes up is the migration/import panel. I said OK to Outlook and it crashed.
Attachment #8712815 - Flags: feedback?(mozilla) → feedback-

Comment 4

2 years ago
Created attachment 8713327 [details] [diff] [review]
bug1176748_import.patch

Would you mind testing this too? 
I had forgot to actually get rid of the new thread creation. (There seems to be one more related to outlook ab import, haven't touched that)
Attachment #8712815 - Attachment is obsolete: true
Attachment #8713327 - Flags: feedback?(mozilla)
(Reporter)

Comment 5

2 years ago
Comment on attachment 8713327 [details] [diff] [review]
bug1176748_import.patch

I don't mind testing it as long as you don't mind another f- ;-)

Instead of crashing it now hangs.

The last debug is this:
[2076] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x8000FFFF: file c:/mozilla-source/comm-central/mailnews/base/src/nsMsgAccountManager.cpp, line 2316
[2076] WARNING: No valid default account found.: file c:/mozilla-source/comm-central/mailnews/base/src/nsMsgAccountManager.cpp, line 781
[2076] WARNING: Just using the first one (FIXME).: file c:/mozilla-source/comm-central/mailnews/base/src/nsMsgAccountManager.cpp, line 783

Yesterday the crash came right after these lines, now it hangs there. The Outlook inbox only has three messages, so that should be imported quickly. I finally killed it.

It had managed to create "Outlook Import" in "Local Folder" and in there an Inbox and Inbox.msf. But the Inbox has 0 bytes so as far as I can see, nothing got imported.
Attachment #8713327 - Flags: feedback?(mozilla) → feedback-
(Reporter)

Comment 6

2 years ago
Quoting from a private message from Joshua Cranmer:
The issue in question is that the importer code all runs off-main-thread, and a great deal of interfaces and implementations are designed to be main-thread-only--and when they're used off-main-thread, a fatal assert fires and the whole things comes crashing down. The "usual" trick for this is proxying the relevant interfaces.

The only reason that the import code tries to run off-main-thread appears to be so that the implementors can be lazy and do a giant synchronous run without bothering about returning control to the event loop. In practice, we'd have to proxy so much back to the main thread anyways; simply fixing the import code to process a single mail message at a time and then yielding back into the event loop (the best example code I can think of here is <https://dxr.mozilla.org/comm-central/source/mailnews/news/src/nsNNTPProtocol.cpp#2818-2883>) would achieve much the same effect with far less brittleness. It's probably possible to test this effect by ripping the proxies out and then figuring out where a more basic mail importer... but I don't think we actually have one of those :-(
No longer a topcrash
Keywords: topcrash-thunderbird
(Reporter)

Comment 8

3 months ago
Created attachment 8867566 [details] [diff] [review]
Undo stuff from bug 1175055 so that the import becomes available again

Refreshed.
Attachment #8712844 - Attachment is obsolete: true
(Reporter)

Comment 9

3 months ago
Since I have Outlook installed on my development machine now and I have a better understanding of the problem than in 2015, I thought I'd take another look at this. Sadly, on my x64 version I run into this error
===
Outlook:
Either there is no default mail client or the current mail client cannot
fulfill the messaging request. Please run Microsoft Outlook and set it as the
default mail client.
===
already seen in bug 1356881 comment #37.
(Reporter)

Comment 10

3 months ago
I've got a 32bit build now and I can trigger Outlook import (which crashes). I'll look into it further.

Comment 11

3 months ago
SysTools PST Converter
https://www.systoolsgroup.com/pst-converter.html
Convert MS Outlook PST File Items to Some of the Most Commonly Used File Formats
* Convert PST File Emails, Contacts, Calendars, Tasks, Notes etc
* Save Outlook Items in 4 Formats: EML / MSG / & vCard (.vcf)
* No MS Outlook Installation required for performing conversion
* Supports to convert corrupted and password protected file
* Preview: Mail, Calendar, Contacts, Task, Notes, Journal, etc.
* Quick Scan PST File and Generate Instant preview of all items
* No 2 GB PST Size Limitation: Convert both ANSI & UNICODE file types

Comment 12

3 months ago
java-libpst
https://github.com/rjohnsondev/java-libpst
A library to read PST files with java, without need for external libraries.

Comment 13

3 months ago
Outlook PST Viewer
https://www.voimakassoftware.com/pst-viewer.html
Voimakas Outlook PST Viewer provides a complete solution to view Outlook Personal Storage Table (PST) files. Voimakas Outlook PST Viewer recovers the PST file data. Its only view corrupted and non-corrupted PST file. PST viewer provides various facilities which are given below:
* View corrupted and non-corrupted PST file as calendars, contacts, journals, emails, notes and tasks etc.
* View PST files with attachment and message.
* We can view multiple PST files at a time.
* Show all folders in the original hierarchy.
* No limitation of PST file size.
* Fast scan and view PST files.
* View deleted items.
* Check and close multiple PST files.

Outlook PST Viewer Feature
* View PST files with attachment and message.
* Add multiple PST files.
* No size limit of PST file.
* Close single and multiple PST files.
* Support both 32 bit and 64 bit outlook version.
* Free version available.

Comment 14

3 months ago
Microsoft Outlook Developer Documentation
https://github.com/OfficeDev/outlook-dev-docs
(Reporter)

Comment 15

3 months ago
Created attachment 8868773 [details] [diff] [review]
1176748-enable-outlook.patch (v1).

This is what it takes to proxy nsOutlookMail::ImportMailbox() to the main thread.

Becky does the same:
https://dxr.mozilla.org/comm-central/rev/b7c66885ee8ba0ebeea9a2a241593bf9b28c3af7/mailnews/import/becky/src/nsBeckyMail.cpp#573

This is a work of a few hours mainly fighting with the compiler. About two years in the making :-(

Sadly, it still crashes, but somewhere else this time, not with "not thread-safe" any more.

To be continued.
(Reporter)

Comment 16

3 months ago
Created attachment 8868779 [details] [diff] [review]
WIP: 1176748-enable-outlook.patch (v2).

Fixed logic errors in previous version. No crash any more but now it hangs waiting for an event. Hmm.
Attachment #8713327 - Attachment is obsolete: true
Attachment #8867566 - Attachment is obsolete: true
Attachment #8868773 - Attachment is obsolete: true
(Reporter)

Comment 17

3 months ago
Created attachment 8868925 [details] [diff] [review]
WIP: 1176748-enable-outlook.patch (v3).

A little more elegant, but still the same idea, proxying nsOutlookMail::ImportMailbox() to the main thread as Becky does, see comment #15.

The debug only shows
== ProxyImportMailbox Deleted Items
and hence
  nsresult rv = NS_DispatchToMainThread(importMailbox, NS_DISPATCH_SYNC);
never returns.

Breaking in the debugger shows that the program is waiting for an event. Might it be waiting for a click on the progress panel that's displayed? Maybe the main tread is blocked on that panel.

Kent, any hints? You seemed to be interested in getting this fixed two years ago, see bug 1175055 comment #2.
Attachment #8868779 - Attachment is obsolete: true
Flags: needinfo?(rkent)
(Reporter)

Comment 18

3 months ago
Created attachment 8868927 [details] [diff] [review]
WIP: 1176748-enable-outlook.patch (v3b).

(Oops, removed remnants of wrong attempt.)
Attachment #8868925 - Attachment is obsolete: true

Comment 19

2 months ago
Sorry, I really cannot give any insights without spending a lot of time getting familiar with the code.
Flags: needinfo?(rkent)
You need to log in before you can comment on or make changes to this bug.