452465 - Migration wizard gets confused with non-matching directory-rel and directory prefs

Status: RESOLVED FIXED

(SeaMonkey :: Startup & Profiles, defect)

Description

[Martin Mokrejs]

User-Agent:       Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.16) Gecko/20080722 SeaMonkey/1.1.11
Build Identifier: Build Identifier: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.2pre) Gecko/2008082711 SeaMonkey/2.0a1pre

In mailer in seamonkey 1.1.11 I had "Local Folders" with some emails. The migration wizard did not include them in the conversion process.

Reproducible: Always

Steps to Reproduce:
1.
2.
3.



Build platform
target
i686-pc-linux-gnu

Build tools
Compiler        Version         Compiler flags
gcc     gcc version 4.3.1 (Gentoo 4.3.1-r1 p1.1)        -Wall -W -Wno-unused
-Wpointer-arith -Wcast-align -W -Wno-long-long -pedantic -fno-strict-aliasing
-pthread -pipe
c++     gcc version 4.3.1 (Gentoo 4.3.1-r1 p1.1)        -fno-rtti
-fno-exceptions -Wall -Wconversion -Wpointer-arith -Woverloaded-virtual -Wsynth
-Wno-ctor-dtor-privacy -Wno-non-virtual-dtor -Wcast-align -Wno-long-long
-pedantic -fno-strict-aliasing -fshort-wchar -pthread -pipe

Configure arguments
--disable-optimize '--enable-debug=-g3\ -O0\ -ggdb' --enable-debug-modules=all
--enable-debugger-info-modules --enable-detect-webshell-leaks --enable-svg
--enable-svg-renderer-libart --enable-image-decoders=all --with-qtdir=/usr/qt/3
--enable-application=suite --disable-freetype2 --enable-jprof
--enable-default-toolkit=cairo-gtk2 --enable-xft --disable-gssapi

Comment 1

[Frank Wein [:mcsmurf]]

Can you open with SeaMonkey 1.1.11 the URL about:config and enter "Local Folders" in the filter field at the top? There should appear three or four different preferences, possible all starting with mail.server.server1 (or another serverX entry). If yes, does the path under mail.server.server1.directory match your local path on the file system to Local Folders?
BTW: Do you still use CVS (we use Mercurial now) or did you manually change your user-agent? Just wondering because the current rv is rv:1.9.1a2pre.

Comment 2

[Martin Mokrejs]

It was a "CVS TRUNK", sorry didn't know that time about mercurial. Does it explain the situation?

Inserting "Local" already gives nothing, not even "Local Folders". I searched for "folder" instead. I gave me some hits, just guessing what might be of importance:

mail.accountmanager.localfoldersserver: server2
mail.identity.default.draft_folder: mailbox://nobody@Local%20Folders/Drafts
mail.identity.default.fcc_folder: mailbox://nobody@Local%20Folders/Sent
mail.identity.default.stationery_folder: mailbox://nobody@Local%20Folders/Templates
mail.identity.id1.draft_folder: mailbox://[e-mail]/Drafts
mail.identity.id1.fcc_folder: mailbox://[e-mail]/Sent
mail.server.server1.spamActionTargetFolder: mailbox://nobody@Local%20Folders/SPAM
mail.server.server2.spamActionTargetFolder: mailbox://nobody@Local%20Folders/Junk
mail.ui.folderpane.version: 3

shell$ find .mozilla -name \*Folders
.mozilla/default/9o1z0pti.slt/Mail/Local Folders
.mozilla/seamonkey/5jz5qf46.default/Mail/Local Folders
shell$ find .mozilla -name SPAM*
.mozilla/default/9o1z0pti.slt/Mail/Local Folders/SPAM.msf
.mozilla/default/9o1z0pti.slt/Mail/Local Folders/SPAM
shell$

Comment 3

[Frank Wein [:mcsmurf]]

Ok, were there other pref subentries under mail.server.server2.*? The Local Folders entry was displayed just fine in the account pane, right?

As far as I know the migration wizard works by looking at all the mail.server.* entries and copying over the mail.server.serverX.directory folders to the new profile (for local mail folders).

Comment 4

[Martin Mokrejs]

I cannot select all the entries from about:config at once, so am pasting just each name and value separately. :(( So, just some name/value pairs, but nothing nested more deeply.

mail.server.server2.directory: /home/mmokrejs/.mozilla/Default User/prhfzrw7.slt/Mail/Local Folders
mail.server.server2.directory-rel: [ProfD]Mail/Local Folders
mail.server.server2.hostname: Local Folders
mail.server.server2.name: Local Folders
mail.server.server2.spamActionTargetAccount: mailbox://nobody@Local%20Folders
mail.server.server2.spamActionTargetFolder: mailbox://nobody@Local%20Folders/Junk
mail.server.server2.userName: nobody

Yes, the Folders pane shows the folder correctly with its subfolders. Right-clicking
on the folder name a selecting properties gives me the full path:
/home/mmokrejs/.mozilla/default/9o1z0pti.slt/Mail/Local
and account name "Local Folders".

Comment 5

[Frank Wein [:mcsmurf]]

Oh: mail.server.server2.directory: /home/mmokrejs/.mozilla/Default User/prhfzrw7.slt/Mail/Local Folders
vs.
/home/mmokrejs/.mozilla/default/9o1z0pti.slt/Mail/Local

Did you have two SeaMonkey 1.x profiles (default vs. Default User)? And did it ask you what profile you want to import?

Comment 6

[Martin Mokrejs]

I don't think it asked me to choose between Default and default. Yes, I remember myself and others commenting on the mess with "default" and "Default" in bugzilla. ;-)

I think the second directory got created for me once when I tried some devel/TRUNK version, like a year old of seamonkey? Or it could be a remnant from using seamonkey on "Digital Unix/OSF1" some years ago? Anyway, there is no "Default/" anymore! At least the conversion should ensure a directory specified in the configuration does exist. Probably that is why it did allow me to choose?
$ ls -la ~/.mozilla/default/
total 2492
drwx------ 4 mmokrejs mmokrejs    4096 Sep  2 21:06 .
drwx------ 7 mmokrejs users       4096 Aug 28 15:43 ..
drwx------ 7 mmokrejs mmokrejs    4096 Sep  2 21:15 9o1z0pti.slt
drwx------ 2 mmokrejs mmokrejs    4096 Nov  8  2007 Cache
-rw------- 1 mmokrejs mmokrejs 2529454 Sep  2 21:13 XUL.mfasl
$ ls -la ~/.mozilla
total 56
drwx------  7 mmokrejs users     4096 Aug 28 15:43 .
drwx--x--x 91 mmokrejs users    12288 Sep  2 18:13 ..
-rw-------  1 mmokrejs mmokrejs  1046 Sep  2 21:06 appreg
drwx------  4 mmokrejs mmokrejs  4096 Sep  2 21:06 default
drwx------  4 mmokrejs mmokrejs  4096 Jun 11 10:24 extensions
drwx------  2 mmokrejs mmokrejs  4096 Apr 28 14:49 firefox
-rw-------  1 mmokrejs mmokrejs  4898 Mar  3  2008 mozver.dat
-rw-------  1 mmokrejs mmokrejs  5221 Sep  2 21:15 pluginreg.dat
drwx------  2 mmokrejs mmokrejs  4096 Mar  2  2008 plugins
drwx------  4 mmokrejs mmokrejs  4096 Aug 28 15:43 seamonkey
$

Comment 7

[Frank Wein [:mcsmurf]]

Note that it said "Default User" and "default" :).
Was there any pref under about:config which included the path /home/mmokrejs/.mozilla/default/9o1z0pti.slt/Mail/Local (so the real path to your local folders)?

Comment 8

[Frank Wein [:mcsmurf]]

Oh and just enter parts of the path because of the escaping. And you said in Comment 2 the path was ".mozilla/default/9o1z0pti.slt/Mail/Local Folders", so I guess you forgot " Folders" in the last part of Comment 4?

Comment 9

[Frank Wein [:mcsmurf]]

Just one more thing: If you have time, can you try to install Thunderbird and see if you get the same import problems as with SeaMonkey trunk? Thunderbird should be able to import SeaMonkey mail settings and folders.

Comment 10

[Frank Wein [:mcsmurf]]

Martin: One last question: Did you move/rename your old SeaMonkey profile once? That might explain the problem (server2.directory vs. server2.directory-rel).

Comment 11

[Martin Mokrejs]

(In reply to comment #8)
> Oh and just enter parts of the path because of the escaping. And you said in
> Comment 2 the path was ".mozilla/default/9o1z0pti.slt/Mail/Local Folders", so
> I guess you forgot " Folders" in the last part of Comment 4?

$ ls -la /home/mmokrejs/.mozilla/default/9o1z0pti.slt/Mail
total 44
drwx------ 10 mmokrejs mmokrejs 4096 Nov  8  2007 .
drwx------  7 mmokrejs mmokrejs 4096 Sep  9 20:23 ..
drwx------  8 mmokrejs mmokrejs 4096 Sep  9 19:54 Local Folders
-rw-------  1 mmokrejs mmokrejs 1142 Dec  6  2006 Local Folders.msf
drwx------  2 mmokrejs mmokrejs 4096 Sep 20  2004 a
drwx------  2 mmokrejs mmokrejs 4096 Mar 27  2007 mail.natur.cuni.cz
drwx------  2 mmokrejs mmokrejs 4096 Feb 17  2004 pf-i400.natur.cuni-1.cz
drwx------  2 mmokrejs mmokrejs 4096 Feb  5  2004 pf-i400.natur.cuni.cz
drwx------  4 mmokrejs mmokrejs 4096 Oct 25  2003 primus.gsf.de
drwx------ 23 mmokrejs mmokrejs 4096 Sep  9 19:54 ribosome.natur.cuni-1.cz
drwx------  9 mmokrejs mmokrejs 4096 Sep  9 19:54 ribosome.natur.cuni.cz
$

But to answer you question completely, I have to rebuild again seamonkey and try the TRUNK version and see if it did not miss the string after the space (I have deleted the old CVS-based tree).

Comment 12

[Martin Mokrejs]

(In reply to comment #10)
> Martin: One last question: Did you move/rename your old SeaMonkey profile once?
> That might explain the problem (server2.directory vs. server2.directory-rel).

I think yes, I think some years ago moved data from a seamonkey running on Alpha
over to my Linux laptop when I bought it. Was around 2002, though.

Comment 13

[Robert Kaiser]

So this looks to me like we might have an underlying issue there (rel vs. non-rel priority conflict between parts of the app), and it's probably a good idea to get that fixed, but probably not blocking alpha 1 for this.
Maybe worth a relnote, marking as such, if you have a good text basing on when this really happens, this would be good.

Confirming and adjusting subject due to the finding of migrator dealing differently with directory-rel and directory prefs than the actual mailnews code.

Also, profile migrator is "Startup & Profiles" component.

Comment 14

[Frank Wein [:mcsmurf]]

Martin: I think you can migrate your profile by starting up SeaMonkey 1.1.x, right clicking on the server2.directory pref and clicking on Reset. SeaMonkey 1.1.x should recreate that pref then from the directory-rel one and your Local Folders should migrate (if you want to try agaun).

Comment 15

[Frank Wein [:mcsmurf]]

Oh, I forgot: You need to launch MailNews once then in SeaMonkey 1.1.x and open your Local Folders once so that the pref gets re-created.

Comment 16

[Martin Mokrejs]

I haven't built the debug build of current hg TRUNK yet but let me mention that now when I have upgraded to 

Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.16) Gecko/20080924 SeaMonkey/1.1.11

I got

Removing /home/mmokrejs/.mozilla/seamonkey/5jz5qf46.default/compreg.dat leftover from older seamonkey
Removing /home/mmokrejs/.mozilla/seamonkey/5jz5qf46.default/XUL.mfasl leftover from older seamonkey

on startup. But it recognizes my mail folder from the ~/.mozilla/default/ path so am a bit puzzled why it cares about the newly migrated, broken profile at all. ;-)

Comment 17

[Frank Wein [:mcsmurf]]

You got those two messages in the console?

Comment 18

[Martin Mokrejs]

Into the parent xterm window.

Comment 19

[Martin Mokrejs]

(In reply to comment #14 and #15)
> Martin: I think you can migrate your profile by starting up SeaMonkey 1.1.x,
> right clicking on the server2.directory pref and clicking on Reset. SeaMonkey
> 1.1.x should recreate that pref then from the directory-rel one and your Local
> Folders should migrate (if you want to try again).

So I tried with "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.18pre) Gecko/20080924 SeaMonkey/1.1.13pre". I had before:

mail.server.server1.directory: /home/mmokrejs/.mozilla/default/9o1z0pti.slt/Mail/ribosome.natur.cuni.cz
mail.server.server1.directory-rel: [ProfD]Mail/ribosome.natur.cuni.cz

mail.server.server2.directory: /home/mmokrejs/.mozilla/Default User/prhfzrw7.slt/Mail/Local Folders
mail.server.server2.directory-rel: [ProfD]Mail/Local Folders

mail.server.server3.directory: /home/mmokrejs/.mozilla/default/9o1z0pti.slt/Mail/ribosome.natur.cuni-1.cz
mail.server.server3.directory-rel: [ProfD]Mail/ribosome.natur.cuni-1.cz

Please note there is only ~/.mozilla/default/ directory and probably due to
some magic entries point to "Default\ User/" subdirectory. Sometimes I see in the "about:config" output "Local Folders" while sometimes "Local%20Folders", btw.

> Oh, I forgot: You need to launch MailNews once then in SeaMonkey 1.1.x and
> open your Local Folders once so that the pref gets re-created.

I clicked around in Local Folders in my mailer, selected some emails etc.
but the mail.server.server2.directory did not get re-created, nor the other
ones I have deleted as well. I shutdown seamonkey and started again for several times, with subsequent startup of MailNews but it did not re-create them.
Summary of the commands I issued and diff of former and later content of .mozilla/default/9o1z0pti.slt/prefs.js will follow.

Did you mean to use Seamonkey2.0pre to do the migration after the "Reset"?

Comment 20

[Martin Mokrejs]

Attachment: log of the session with seamonkey-1.1.13pre.en-US.linux-i686-Sep24

Comment 21

[Martin Mokrejs]

$ rm -rf ~/.mozilla/seamonkey/
$ cd ../seamonkey-2.0a1pre.en-US.linux-i686-Sep24/
$ ./seamonkey &

"Import Settings and Data"
  Import All items from: Seamonkey 1.x, Netscape 6/7 or Mozilla 1.x

"Home Page Selection"
  Please select the home page you wish to use: Import your home page from Seamonkey 1.x, Netscape 6/7 or Mozilla 1.x


It started import and in a second or two it said: "Import complete" and opened browser with the projects homepage, not my homepage (I have no homepage set, the value is empty).

Clicking on MailNews icon I get two password prompting windows for my two POP3 accounts. The upper window is unlabeled but having Cancel and OK buttons. The one below is labeled "Enter your password" and contains the remote server name/address as expected and behind these two was the mailer window.

I pressed cancel in both cases not to download my new emails. Let me say add that even when I later on quit seamoneky and started it again it again raised one of these two windows unlabeled. The Error Console messages below might explain why.

Anyway, back to the moment when I ran MailNews for the first time after the Import Wizard. The left frame containing overview of my folders showed those my 2 accounts but with only Inbox and Trash as suspected and no other folders at all. The "Local Folders" contains Outbox, Trash, Junk.

In summary, current TRUNK behaviour is even worse than few weeks before! Please provide capability to log results of the Migration Wizard and wait for the user to acknowledge the results before continuing execution.


about:config gave me:

mail.server.server1.directory;/home/mmokrejs/.mozilla/seamonkey/8m2rrzph.default/Mail/ribosome.natur.cuni.cz
mail.server.server1.directory-rel;[ProfD]Mail/ribosome.natur.cuni.cz
  
mail.server.server2.directory;/home/mmokrejs/.mozilla/seamonkey/8m2rrzph.default/Mail/Local Folders
mail.server.server2.directory-rel;[ProfD]Mail/Local Folders
  
mail.server.server3.directory;/home/mmokrejs/.mozilla/seamonkey/8m2rrzph.default/Mail/ribosome.natur.cuni-1.cz
mail.server.server3.directory-rel;[ProfD]Mail/ribosome.natur.cuni-1.cz
  
So, seamonkey2.0a1pre re-created the .directory as you have probably proposed in comments #14 and #15.
  
  

In Error Console I see:

No chrome package registered for chrome://wallet/locale/walletTasksOverlay.dtd

Error: undefined entity
Source File: chrome://wallet/content/walletTasksOverlay.xul
Line: 78, Column: 5
Source Code:
    <menu id="menu_passwordManager"


Error: redeclaration of const kNonStringDataLength
Source File: chrome://global/content/nsDragAndDrop.js
Line: 4


Warning: Key event not available on some keyboard layouts: key="f" modifiers="control,alt"
Source File: chrome://navigator/content/navigator.xul
Line: 0

Comment 22

[Frank Wein [:mcsmurf]]

Oh excuse me. I've read the source again and discovered it does not actually recreate the .directory entries :/. So you have to re-create the .directory entries via about:config for migration to work and correct the path in the server2 entry to where the Local Folders actually are.
So conclusion of this bug is probably that there is no bug after all. Guess you had the wrong path in that pref due to a profile move, but because you also had the .directory-rel entry, it was no problem for MailNews, only for our new migration code. Maybe we can/should write some code to check if .directory and .directory-rel point to the same location, but I think it's rather an exception that they do not.

Comment 23

[Martin Mokrejs]

For curiosity, I tried thunderbird-2.0.0.17pre.en-US.linux-i686.tar.gz as of Sep 23. I did not bother to zap the broken ~/.mozilla/seamonkey/ directory missing my email folders. I though it will probably create ~/.mozilla/thunderbird and start the Migration Wizard but it created ~/.thunderbird/.

The "Import Wizard" said:

"Import Preferences, Account Settings, Addressbook and other data from":
  and I selected "Nestcape 6, 7 or Mozilla 1.x"

In a second or two it "finished" import and has asked me for passwords on 10.8.0.1 POP3 servers, both password prompting windows were labeled. I folders overview left frame I see my 2 pop3 accounts and "Local Folders" with only Inbox and Trash in each.

After quitting the application I browsed the direcotry structure created:

$ ls -la ~/.thunderbird/wjwts2v7.default/
total 3548
drwx------ 5 mmokrejs mmokrejs    4096 Sep 24 22:26 .
drwx------ 3 mmokrejs mmokrejs    4096 Sep 24 22:20 ..
-rw------- 1 mmokrejs mmokrejs       0 Sep 24 22:22 .parentlock
-rw------- 1 mmokrejs mmokrejs     216 Sep 24 22:22 .signature
-rw------- 1 mmokrejs mmokrejs    4878 Sep 24 22:22 75716314.s
drwx------ 5 mmokrejs mmokrejs    4096 Sep 24 22:22 Mail
drwx------ 2 mmokrejs mmokrejs    4096 Sep 24 22:20 US
-rw------- 1 mmokrejs mmokrejs 1132355 Sep 24 22:20 XPC.mfasl
-rw------- 1 mmokrejs mmokrejs 1170725 Sep 24 22:22 XUL.mfasl
-rw------- 1 mmokrejs mmokrejs  671185 Sep 24 22:22 abook.mab
-rw------- 1 mmokrejs mmokrejs   65536 Sep 24 22:26 cert8.db
-rw------- 1 mmokrejs mmokrejs     181 Sep 24 22:20 compatibility.ini
-rw------- 1 mmokrejs mmokrejs  169359 Sep 24 22:22 compreg.dat
drwx------ 2 mmokrejs mmokrejs    4096 Sep 24 22:20 extensions
-rw------- 1 mmokrejs mmokrejs     176 Sep 24 22:22 extensions.cache
-rw------- 1 mmokrejs mmokrejs     268 Sep 24 22:22 extensions.ini
-rw------- 1 mmokrejs mmokrejs    2557 Sep 24 22:22 extensions.rdf
-rw------- 1 mmokrejs mmokrejs    1438 Sep 24 22:22 history.mab
-rw------- 1 mmokrejs mmokrejs   16384 Sep 24 22:26 key3.db
-rw------- 1 mmokrejs mmokrejs    4888 Sep 24 22:22 localstore.rdf
-rw------- 1 mmokrejs mmokrejs   13950 Sep 24 22:22 mimeTypes.rdf
-rw------- 1 mmokrejs mmokrejs    2281 Sep 24 22:26 panacea.dat
-rw------- 1 mmokrejs mmokrejs      58 Sep 24 22:22 persdict.dat
-rwx------ 1 mmokrejs mmokrejs   10596 Sep 24 22:26 prefs.js
-rw------- 1 mmokrejs mmokrejs   16384 Sep 24 22:22 secmod.db
-rw------- 1 mmokrejs mmokrejs  141216 Sep 24 22:22 training.dat
-rw------- 1 mmokrejs mmokrejs      10 Sep 24 22:26 virtualFolders.dat
-rw------- 1 mmokrejs mmokrejs  109397 Sep 24 22:22 xpti.dat
$ ls -la ~/.thunderbird/wjwts2v7.default/Mail/
total 20
drwx------ 5 mmokrejs mmokrejs 4096 Sep 24 22:22 .
drwx------ 5 mmokrejs mmokrejs 4096 Sep 24 22:26 ..
drwx------ 2 mmokrejs mmokrejs 4096 Sep 24 22:22 Local Folders
drwx------ 2 mmokrejs mmokrejs 4096 Sep 24 22:22 ribosome.natur.cuni-1.cz
drwx------ 2 mmokrejs mmokrejs 4096 Sep 24 22:22 ribosome.natur.cuni.cz
$ ls -la ~/.thunderbird/wjwts2v7.default/Mail/Local\ Folders/
total 16
drwx------ 2 mmokrejs mmokrejs 4096 Sep 24 22:22 .
drwx------ 5 mmokrejs mmokrejs 4096 Sep 24 22:22 ..
-rw------- 1 mmokrejs mmokrejs    0 Sep 24 22:22 Trash
-rw------- 1 mmokrejs mmokrejs 1221 Sep 24 22:26 Trash.msf
-rw------- 1 mmokrejs mmokrejs    0 Sep 24 22:22 Unsent Messages
-rw------- 1 mmokrejs mmokrejs 1222 Sep 24 22:26 Unsent Messages.msf
$ ls -la ~/.thunderbird/wjwts2v7.default/Mail/ribosome.natur.cuni.cz/
total 16
drwx------ 2 mmokrejs mmokrejs 4096 Sep 24 22:22 .
drwx------ 5 mmokrejs mmokrejs 4096 Sep 24 22:22 ..
-rw------- 1 mmokrejs mmokrejs    0 Sep 24 22:22 Inbox
-rw------- 1 mmokrejs mmokrejs 1549 Sep 24 22:26 Inbox.msf
-rw------- 1 mmokrejs mmokrejs    0 Sep 24 22:22 Trash
-rw------- 1 mmokrejs mmokrejs 1465 Sep 24 22:26 Trash.msf
$

abook.mab seems to contain my addresses.
history.mab is "empty" like ~/.mozilla/default/9o1z0pti.slt/history.dat seems to be.
training.dat contains some data probably extracted from my emails.

$ ls -la ~/.mozilla/default/9o1z0pti.slt/training.dat
-rw------- 1 mmokrejs mmokrejs 141216 Sep  9 10:40 /home/mmokrejs/.mozilla/default/9o1z0pti.slt/training.dat
$ ls -la ~/.thunderbird/wjwts2v7.default/training.dat
-rw------- 1 mmokrejs mmokrejs 141216 Sep 24 22:22 /home/mmokrejs/.thunderbird/wjwts2v7.default/training.dat
$

Comment 24

[Martin Mokrejs]

(In reply to comment #22)
> Oh excuse me. I've read the source again and discovered it does not actually
> recreate the .directory entries :/. So you have to re-create the .directory
> entries via about:config for migration to work and correct the path in the
> server2 entry to where the Local Folders actually are.
> So conclusion of this bug is probably that there is no bug after all. Guess 
> you had the wrong path in that pref due to a profile move, but because you 
> also had the .directory-rel entry, it was no problem for MailNews, only for 
> our new migration code. Maybe we can/should write some code to check if 
> .directory and .directory-rel point to the same location, but I think it's 
> rather an exception that they do not.

Please do provide the check and rename/move functionality. Concurrently I would
propose to remove the magic code converting "Default User" to "default" when
evaluating the directory path.

If somebody would agree on renaming "Local Folders" to something without the space things would be clearer, "find . -name blah | xargs cmd" would not break on the space and I guess many other programs as well.

Comment 25

[Frank Wein [:mcsmurf]]

There is no magic code that converts the path, it's just: When .directory-rel is present, ignore .directory (in MailNews).

Comment 26

[Robert Kaiser]

(In reply to comment #25)
> There is no magic code that converts the path, it's just: When .directory-rel
> is present, ignore .directory (in MailNews).

...and IIRC migration only uses .directory - and there's where we run into the problem.

Comment 27

[Rich Gray (:rbgray)]

I'm seeing this migrating from 
Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:1.8.1.21) Gecko/20090303 SeaMonkey/1.1.15 to
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1b3pre) Gecko/20090223 SeaMonkey/2.0a3.

Changing platform to All/All.

In my case, the problem manifested itself as the contents of one of my many pop accounts having the contents of another.  The contents of the former account were nowhere to be seen (dataloss?) and the later account was empty.  

What I _think_ caused the problem is that several years ago, my profile took a some sort of hit, so I created a 'new' profile and manually moved my mail accounts to it.  At some later time, deciding I hadn't lost anything, I deleted 'default' (or 'default user'?) and moved 'new' back to 'default'.  (I don't remember the steps I took to do all this.) 

(In reply to comment #25)
> There is no magic code that converts the path, it's just: When .directory-rel
> is present, ignore .directory (in MailNews).

Why can't the migration code do the same?  (Or modify SM 1.1.6 so that it corrects such problems before migration if that is easier?) Why even have .directory?

Comment 28

[Martin Mokrejs]

While testing current comm-central build my local folders got converted. At least in the imported preferences I see .directory point to a full path, .directory-rel points to [ProfD]/Mail/$blah, where blah is also trailing string in .directory value. So, it seems I either fixed the paths myself in the past in my nowadays seamonkey-1.1.17 profile or the Import wizard fixed that for me. Either way, the Local Folders were imported into the new profile, so the original bug reported here is gone. Well, I can check the about:config in seamonkey-1.1.17 if somebody wants me to do so. Just have to quit the currently running:

Build identifier: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.3pre) Gecko/20090801 SeaMonkey/2.0b2pre

Comment 29

[Martin Mokrejs]

(In reply to comment #28)

So, it seems I either fixed the paths myself in the past
> in my nowadays seamonkey-1.1.17 profile or the Import wizard fixed that for me.

I fixed that manually in the past (so I haven't actually verified with current import whether a difference between .directory and .directory-rel contents screws the import process still or not):

https://bugzilla.mozilla.org/show_bug.cgi?id=452468#c12

Comment 30

[Robert Kaiser]

This comes up in testing again and again, we'll surely stumble over it when we do a release with it. I still don't understand why we can't resolve relative directories in the migration code - after all, the migration code knows the path of the old profile so that it can copy files over from there, and it reads those prefs, so shouldn't it just be able to resolve those relative paths on that basis?

Comment 31

[Robert Kaiser]

As a note, this is at least related to, if not a dupe of, bug 382870.

Comment 32

[Mark Banner (:standard8)]

(In reply to comment #30)
> This comes up in testing again and again, we'll surely stumble over it when we
> do a release with it. I still don't understand why we can't resolve relative
> directories in the migration code - after all, the migration code knows the
> path of the old profile so that it can copy files over from there, and it reads
> those prefs, so shouldn't it just be able to resolve those relative paths on
> that basis?

I've looked at this sort of bug in the past. I'm a bit rusty, but remembering some of it.

http://mxr.mozilla.org/comm-central/source/suite/profile/migration/src/nsNetscapeProfileMigratorBase.cpp#1016

Is where the code starts in CopyMailFolderPrefs. The prefs are literally just all the mail.server.server<n>.* prefs not in server order. iirc I came to the conclusion that to fix this bug you'd have to:

- Iterate through the list of servers, building up a picture of servers and if they are relative or not.
- Go through the list again adjusting the folder prefs (or not) and copying the folders (or not).

There may be a simpler way, but that's the place to start.

Unfortunately I didn't have time to fix and test it then, and I don't now.

Comment 33

[Ian Neal]

Do we know how many people this is likely to affect? How many will have directory-rel preferences?

Comment 34

[Robert Kaiser]

(In reply to comment #33)
> Do we know how many people this is likely to affect? How many will have
> directory-rel preferences?

Everyone has directory-rel preferences, the question is in how many cases they differ from the absolute paths. MailNews _always_ uses the directory-rel and only falls back to the absolute one if -rel isn't set. Migration only uses the absolute one right now though, so it's a bug in any case.
I'm frequently reading of the problem that "my mail folders disappeared when migrating" which we could always track back to this issue.

Comment 36

[Frank Wein [:mcsmurf]]

Attachment: Patch

Not sure if this is the best approach for this, basically I copied some code from libpref and converted to the frozen linkage string API.

Comment 37

[Frank Wein [:mcsmurf]]

Attachment: Patch 2

Modified patch with your suggestions from IRC.

Comment 38

[Frank Wein [:mcsmurf]]

> +  nsresult GetRelativeFilePref(const nsACString& aPrefValue, void * * _retval);

Forgot to remove that line.

Comment 39

[[e-mail]]

Comment on attachment 403870 [details] [diff] [review]
Patch 2

>+    const char *keyBegin, *strEnd;
>+    prefValue.BeginReading(&keyBegin, &strEnd);
Not used.

>+    nsCAutoString relPath(Substring(prefValue, 7));
Don't bother with this, just pass the Substring to SetRelativeDescriptor

>+    *aReturnFile = theFile;
>+    NS_ADDREF(static_cast<nsILocalFile*>(*aReturnFile));
theFile.forget(*aReturnFile);

>+    rv = aPrefBranch->GetComplexValue(aPrefName,
>+                                      NS_GET_IID(nsILocalFile),
>+                                      reinterpret_cast<void**>(aReturnFile));
Eww, this is ugly. Two possibilities come to mind:
1. Don't bother with non-relative prefs here. Make the caller do the work.
nsCOMPtr<nsILocalFile> sourceMailFolder;
nsresult rv = GetRelativeFile(serverBranch, "directory-rel",
                              getter_AddRefs(sourceMailFolder));
if (NS_FAILED(rv))
  rv = serverBranch->GetComplexValue("directory", NS_GET_IID(nsILocalFile),
                                     getter_AddRefs(sourceMailFolder));
2. Declare theFile outside the if, then use
} else {
  rv = aPrefBranch->GetComplexValue(aPrefName,
                                    NS_GET_IID(nsILocalFile),
                                    getter_AddRefs(theFile));
}
theFile.forget(*aReturnFile);
return rv;

>+    if (NS_FAILED(rv))
>+      return rv;
>+  }
>+
>+  return NS_OK;
Could just return rv here. (Probably unnecessary for possibility 1 though.)

>+      nsCString directoryPref;
Not used.

>+  for (PRInt32 i = count-1; i > 0; --i) {
I'd prefer (PRInt32 i = count; i-- > 0; ) [or --i >= 0]

>+  nsresult GetRelativeFilePref(const nsACString& aPrefValue, void * * _retval);
Not used.

Comment 40

[Frank Wein [:mcsmurf]]

Attachment: Patch 3 [Checkin: Comment 42]

Fixed all issues mentioned.

Comment 41

[[e-mail]]

Comment on attachment 404673 [details] [diff] [review]
Patch 3
[Checkin: Comment 42]

>+
>+    *aReturnFile = theFile;
>+    NS_ADDREF(static_cast<nsILocalFile*>(*aReturnFile));
Don't need this, you're forgetting it anyway. (And it's not a void** any more.) r=me with that fixed.

Comment 42

[Frank Wein [:mcsmurf]]

http://hg.mozilla.org/comm-central/rev/d78cb3baccd9

Comment 43

[[e-mail]]

(In reply to comment #41)
>(From update of attachment 404673 [details] [diff] [review])
>>+
>>+    *aReturnFile = theFile;
>>+    NS_ADDREF(static_cast<nsILocalFile*>(*aReturnFile));
>Don't need this, you're forgetting it anyway. (And it's not a void** any more.)
>r=me with that fixed.
You only removed one of the two lines. Fortunately .forget is safer than .swap, so you won't crash or leak anything, and may even get optimised out!

Comment 44

[Frank Wein [:mcsmurf]]

Attachment: Patch [Checkin: Comment 45]

Already got review for this, just forgot to remove this line as stated in the review comment. Should be a safe patch, minimal to no risk.

Comment 45

[Frank Wein [:mcsmurf]]

Comment on attachment 405108 [details] [diff] [review]
Patch
[Checkin: Comment 45]

http://hg.mozilla.org/comm-central/rev/626e14c3c742