Closed Bug 1722938 Opened 1 year ago Closed 1 year ago

Importing an 8GB profile.zip hangs and fails with errors (Tools > Import > Import Everything > Thunderbird)

Categories

(Thunderbird :: Account Manager, defect)

Thunderbird 91
Desktop
Windows 10
defect

Tracking

(thunderbird_esr91 wontfix, thunderbird91+ wontfix, thunderbird92 wontfix)

RESOLVED FIXED
93 Branch
Tracking Status
thunderbird_esr91 --- wontfix
thunderbird91 + wontfix
thunderbird92 --- wontfix

People

(Reporter: tabguy1526, Assigned: rnons)

References

Details

User Story

Tentative STR (still a lot of guesswork here)

This is about the recently added feature to export and import full Thunderbird profile via tools menu.

- Somehow export your profile into a ZIP file (not sure if Tools > Export was used for that - why would that work if import fails due to size?)
- profile.zip file has 8GB file size
- Try to import 8GB profile.zip into a fresh install of Thunderbird (Tools > Import > Import Everything > Thunderbird)

Actual result
- hangs and fails with errors

Expected result
- import 8GB profile.zip correctly

Attachments

(4 files, 5 obsolete files)

Attached image usb hung (obsolete) —

import export does not work on beta 4 or latest 91 daily
on latest windows 10 Dell I7 ssd
import hangs two screens are open one asks for a tbird profile the other for a csv file
at times export on R91 beta 4 hangs on export to a 128gb USB drive also tried export o a local disk, external ssd and micro sd all fail ti import
tried with a net new windows and tbird install same thing

Attached image bad zip file (obsolete) —

Here's reporter's photo from attachment 9233715 [details] (at reduced size).
We can see clearly that the problem involves a USB flash disk... ;-)
(Sorry, I couldn't resist...)

(Please do not attach high resolution photos, and it's not clear how this photo contributes to understanding the bug...)

Component: Toolbars and Tabs → General
Summary: import expory hangs and does not import backup file → import export hangs and does not import backup file
Whiteboard: [support?]

Here's reporter's other SCREENSHOT (at reduced size), taken with DC (wasn't me)... :-)

old geek yyc, next time kindly google how to take/create a proper Screenshot image file on Windows 10. (Alt+)Print key, MSpaint.exe & Ctrl+V are your friends. Win+Shift+S is another one, select region and then click on the popup-preview to work with that further.

(In reply to old geek yyc from comment #0)

Created attachment 9233715 [details]
usb hung

import export does not work on beta 4 or latest 91 daily
on latest windows 10 Dell I7 ssd
import hangs two screens are open one asks for a tbird profile the other for a csv file
at times export on R91 beta 4 hangs on export to a 128gb USB drive also tried export o a local disk, external ssd and micro sd all fail ti import
tried with a net new windows and tbird install same thing

Old geek (reporter), thank you for your bug report. Unfortunately, it is not actionable. We don't even know if you have used Import from addon or builtin commands from TB menu. There are different workflows for inbuilt import, so we need to know exactly what you did in order to reproduce.
Please provide more detail and follow the prescribed structure:

  • Steps to reproduce (numbered list of every click)
  • Actual result (bulleted list of what actually happened)
  • Expected result (bulleted list of what should have happened)

In particular:

  • how do you define "hang"?
  • can you attach screenshots (image files, not DC photos, see my comment 3) of the two screens asking for TB profile and for csv file?
Flags: needinfo?(tabguy1526)
Attached file dual windows.odt (obsolete) —

dual popup from import

Flags: needinfo?(tabguy1526)

PROCESS
. downloaded latest daily version from softpedila.com
.install on a net new windows 10 install 64 bit core i5 500gb ssd 16 gb ram
. set to display menu bar
. installed usb stick into usb drive
. selected import
. asked to import all thunderbird
. selected export file from usb file
. nothing happened after 15 minutes
. terminated import
. used file explorer tool to make sure usb disk had no errors

HANG
.doing an import from a 128gb usb 3 stick
started thunderbird
selected import from tools menu
importing an 8gb file
selected zip file created on 32 bit version of tbird 91. beta 4
after 15 minutes no messages no activity from leds on usb stick or disk drive

what actually happened
. opened second popup for csv file
.. fter 15 minutes no messages no activity from leds on usb stick or disk drive
closed window using x

EXPECTED RESULT
. message sating import started using file name as import
. import status progress bar
. imported complete with a summary of accounts and messages imported

erro log from debug tools
downloaded and installed tbird from your website
used iobit uninstalled to delete other version

Thanks for reporting.

I just learned that nsIZipReader only supports zip file smaller than 2GB, see https://searchfox.org/mozilla-central/rev/9c91451cc2392d942a42493fc895f5aeeddde45d/modules/libjar/nsZipArchive.cpp#188. I don't have a good solution, some options:

  1. check the zip file size before passing to nsIZipReader, if larger than 2GB, ask the user to extract the zip then import the extracted folder.
  2. find another zip lib that supports larger zip file
  3. patch nsIZipReader, but according to bug 504864, seems the whole zip file is read into memory upfront.
  4. what we really need is just extracting the zip file to a tmp dir, I think it's possible to take some code from nsZipArchive to make our own zip extracting module.

Option 1 is technically the easiest but not very friendly to the user. I will take a look at option 4, what do you think?

Flags: needinfo?(mkmelin+mozilla)
Flags: needinfo?(geoff)
Flags: needinfo?(mkmelin+mozilla)
Flags: needinfo?(geoff)
Attachment #9233715 - Attachment is obsolete: true
Attachment #9233750 - Attachment is obsolete: true
Attachment #9233716 - Attachment is obsolete: true
Status: UNCONFIRMED → NEW
Component: General → Account Manager
Depends on: 271863
Ever confirmed: true
Whiteboard: [support?]

Comment on attachment 9233845 [details]
dual windows.odt

Trying to access this attachment...

Attachment #9233845 - Attachment mime type: image/jpeg → application/vnd.oasis.opendocument.text

Still a lot of guesswork here, but let's try to make this actionable.

I think Ping's comment 8 is crucial to understanding this - thank you Ping (:rnons), great find!!!

So this seems to be a size issue where we cannot handle zip files > 2GB, especially when it comes to re-importing the full profile. That would appear as a massive limitation which will hinder the correct use of the new feature for many. So far there is no information here about the export, so I am surprised how the export would have succeeded, assuming that it should have the same size limits.

I have extracted and cropped reporter's screenshots from the ODT files and will re-attach them as plain vanilla png image files below.
Thank you very much old geek yyc (reporter) for providing more information!

You have succeded to take useful screenshots! :-)) Next time just paste them into a graphics program like mspaint.exe, crop and black out private information as desired, save as .png or jpg and attach to the bug directly without embedding into ODT word processor file (you can even paste directly into the new attachment form, but make sure there's no private stuff in the image!). That way, your screenshots will become directly accessible here and we can focus on understanding the bug rather than extracting information. Maybe this screenshot how-to has some more tricks...

(In reply to Ping Chen (:rnons) from comment #9)

Found https://searchfox.org/comm-central/source/mozilla/devtools/client/shared/vendor/jszip.js, will see if it works.

What are the prospects for accomplishing a solidly tested implementation by say Tuesday, for shipment in esr91?

If not possible then perhaps we should disable the feature initially on 91.0, because option #1 is not feasible (strings).

(In reply to Wayne Mery (:wsmwk) from comment #12)
^^

User Story: (updated)
Flags: needinfo?(remotenonsense)
Summary: import export hangs and does not import backup file → Importing an 8GB profile.zip hangs and fails with errors (Tools > Import > Import Everything > Thunderbird)

Reporter's attachment 9233845 [details] extracted.

So the Import TB Profile Wizard dialog window becomes unresponsive.

Attachment #9233845 - Attachment is obsolete: true

Here's reporter's screenshot from comment 7, attachment 9233847 [details], extracted and cropped.

Error console shows NS_ERROR_FILE_TOO_BIG from ThunderbirdProfileMigrator.jsm.

Comment on attachment 9233847 [details]
error log.odt from latest trail used tbird from your website

Still learning to obsolete on the go ;-)

Attachment #9233847 - Attachment is obsolete: true

Comment on attachment 9233767 [details]
SCREENSHOT 1: Reporter's photo from attachment 9233716 [details] - "bad zip file" (8.31 GB)

.

Attachment #9233767 - Attachment description: SCREENSHOT 1: Reporter's photo from attachment 9233716 - "bad zip file" → SCREENSHOT 1: Reporter's photo from attachment 9233716 - "bad zip file" (8.31 GB)

(In reply to Wayne Mery (:wsmwk) from comment #12)

(In reply to Ping Chen (:rnons) from comment #9)

Found https://searchfox.org/comm-central/source/mozilla/devtools/client/shared/vendor/jszip.js, will see if it works.

What are the prospects for accomplishing a solidly tested implementation by say Tuesday, for shipment in esr91?

I should be able to get jszip working today, but jszip also reads the whole zip file into the memory, there won't be 2GB limit, but will depend on user's memory and profile size. It's still easy to get out-of-memory for really large profile. So we still need to change a few strings in the import dialog, and support importing from a plain profile dir.

Flags: needinfo?(remotenonsense)

My own backup on a 16gb Macbook Pro is 14gb, so I surely would have run out of memory using jszip. I'm sure there are people with bigger profiles.

THHERE IS A VERY CRITCAL NEED FOR THIS TOOL
IT WILL BE KEY FOR MIGRATING TO WINDOWS 11 AND MANY PCS
THAT NOW EXIST CANNOT RUN WINDOWS 11 AND REPLACEMENT SYSTEMS FOR MANY PRODUCTOIN PLATFORMS WILL BE NEEDED
I TESTED 6 SYSTEMS ONLY 1 PASSED THE WINDOWS 11 TEST

Any attachments I submit are l
Libre Office word processing documents and done using the screen print function

Showing the usb drive picture was indented to show no disk reads;
large files cannot be emailed and use of external device is
forced, the use of a USB drive for extra large files is common for Yumi and Rufus (both public domain)
why not look at how these programs handle large files like 4 to 6 gb iso files

There is a user import export tool that seems to work on Tbird 78
"Thunderbird Import Export Tools NG" can anything be learned from that tool?

Anybody who submits a bug should attach the error log

Can PEAZIP or some other public domain file compression too be used loading an 8gb file or more could cause issues

As an alternative how about better easy to follow instructors to copy the profile directory from one pc tp another
and guidelines such things as do I need the same window's ids
does the copied profile have to be copied to the same place ?

Becasue nsIZipReader only supports zip file smaller than 2GB, we have to support importing from a profile dir. Because we can't get nsIFilePicker to support folder and zip file at the same time, ask user to select prefs.js instead.

This patch also adds a failed page to the migration dialog, so that user know importing has stopped.

Assignee: nobody → remotenonsense
Status: NEW → ASSIGNED

Please post documentation as to how to export a profile and
the exact steps to import a profile
thanks

Target Milestone: --- → 92 Branch

TOOLS USED
am using the tool provided via the import export function under the standard supplied tools menu
AM NOT USING AN EXTENSION AS THEY DO NOT WORK WITH CURRENT TBIRD VERSIONS

TESTED AUG 5 21 USING LATEST TBIRD DAILY
NO CHANGES IN RESULTS NOTHING WORKS WITH A LARGE EXPORT FILE

A fix is still in the progress, you can check the "Phabricator Revisions" section on this page, once the fix is ready, the status will become published.

PROFILE IMPORT SOURCE
i have used serval devices to import the profile zip file created on
a usb 2 and 3 flash drive 128gb
a 128gb sd card
an external 2.5 usb ssd drive
copied the export file to the same partition tbird is installed on

All input devices produce the same results
import on the source computer goes into a wait state all most immediately
THIS BUG IS NOT MEDIA DEPANDANT AND YES I USED THE TBIRD IMPORT EXPORT FEATURE UNDER THE TOOLS MENU

(In reply to old geek yyc from comment #22)

Please post documentation as to how to export a profile and
the exact steps to import a profile
thanks

This should help for manual profile export/import:
https://support.mozilla.org/en-US/kb/profiles-where-thunderbird-stores-user-data

Attachment #9234194 - Attachment description: Bug 1722938 - Support selecting prefs.js in ThunderbirdProfileMigrator.jsm. r=darktrojan → Bug 1722938 - Support importing from profile folder in ThunderbirdProfileMigrator.jsm. r=darktrojan,mkmelin
See Also: → 1726224
Target Milestone: 92 Branch → 93 Branch

Pushed by mkmelin@iki.fi:
https://hg.mozilla.org/comm-central/rev/206ee4d91f33
Support importing from profile folder in ThunderbirdProfileMigrator.jsm. r=darktrojan,mkmelin

Status: ASSIGNED → RESOLVED
Closed: 1 year ago
Resolution: --- → FIXED
Duplicate of this bug: 1723472
You need to log in before you can comment on or make changes to this bug.