44764 - Profile migration fails when the profile to migrate from has a Japanese name

Status: VERIFIED FIXED

(Core Graveyard :: Profile: Migration, defect, P1)

Description

[Katsuhiko Momoi]

** Observed with 7/6/2000 Win32 build **

Bug 37398 is about Profile Manager not being to display 
and proceed with profile migration when the original 
profile to migrate from has a Japanese name -- most probably any
non-Latin 1 names. racham actually has fixed the problem --
we looked at the proposed fix and confirmed that the display
part now works. 
draggg said in Bug 37398 that when the display and the
migration process failure, i.e. migration simply does not
proceed at all, are fixed, we can seriously look at the
migration failure issue. 

Well, now we can and the news is not good. I confirmed that
the migration dialog comes up and the process seems to complete.
It creates a new profile with exactly the same name as the 4.x profile
it is migrating but the contents are not migrated at all.

In particular, 

1. When I start up Messenger, I am asked to create a mail account
   since there is NONE. My earlier profile contained at least
   one mail account. Looking in the IMAP and Mail folders,
   I see no IMAP server folders created. There needs to be at least
   one such server folder created with the inbox summary file in it.
2. bookmarks.html is not migrated and we get instead a default one.
3. prefs.js is the default one not the migrated one with all
   the info from the earlier profile.

..., etc.


This problem does not occur if the profile to migrate has an ASCII name.

I am nominating this bug for nsbeta2 because profile migration for
Japanese users should succeed when the profile names are in Japanese.
A lot of Japanese users have profiles in their own language in 4.x.
Migration of such profiles is important. This is listed in
PR2 criteria for international.

We had a blocking Bug 37398 which prevented us to see this 
problem. Now that that bug is almost done with, we now know
that migration will fail.

Comment 1

[Katsuhiko Momoi]

QA contact to teruko as this is classified as Browse bug but
I will share QA'ing this one because it has a major impact on
mail account migration as well.

Comment 2

[dbragg]

I need just a little clarification.  When you say "the contents are not migrated
at all" are you saying there are no files in the new profile directory?  Or are
you saying that the files are there but none of the new prefs have been set?

Comment 3

[Katsuhiko Momoi]

The files such bookmarks.html and prefs.js are there but the content
of these files do not reflect waht was in the original. I have 
a feeling that we are just using the fallback default for
these, i.e. whatever the default settings for these when
a profile is created totally new. Here are items I see in a 
newly created profile if the original profile name was in Japanese:

Files:

bookmarks.html
cookies.txt
history.day
localestore.rdf
panels.rdf
prefs.js
search.rdf

Folder: (all empty)
Cache
ImapMail
Mail
News

Comment 4

[Katsuhiko Momoi]

When the original profile name is in English,
I see a lot more files than the above created, e.g.

cert7.db
many Address Book related files ending in .na2
launch.aim
liprefs.js
etc.

Comment 5

[Jay Patel [:jay]]

Putting on [nsbeta2+] radar for beta2 fix. 

Comment 6

[dbragg]

Actually, I think this is a bug in profile.cpp.  Specifically line 1367.  
Appending a PRUnichar to the newProfileDir (which is a fileSpec) is probably 
mangling the name.  Reassigning to racham.  (We should discuss this Bhuvan.)

Comment 7

[Katsuhiko Momoi]

Bhuvan, Don, do we have a progress on this bug?

Comment 8

[dbragg]

Bhuvan spoke with Tao.  He's working on the profile manager side of the fix.
After that's done we'll be able to determine if there is a profile migration
side of the bug.  Through inspection of the code I don't think there is but I'll
only be able to tell for sure after the profile manager work is done.

Comment 9

[racham]

I spoke to Tao about this problem. We need to use a debug build on Japanese OS 
machine to track the problem further. Tao has requested his group members for 
such a machine. We are going work on this tomorrow to come up with required 
solution. Changing ETA to 7/18.

Adding Tao to the cc list.

Comment 10

[racham]

With timely help from Tao and Naoki, I have a fix that migrates a japanese 
profile properly (reflected via prefs, bookmarks and mail settings). I also 
migrated a english profile with these changes. That also went throught fine. All 
these activities on windows machine. 

As this fix involves the code that other platforms also execute, we need some 
testing to be done on other platforms too. Planning on doing extensive testing 
with incorporating this fix. Mac migration is totally broken and Linux is 
already doing the right thing. When I said that we need to test on other 
platforms, I would like us to make sure that english builds are not affected 
with these changes in anyway (theoritically they should not). I will take teruko 
and momoi's help for testing.

I have couple of more things to check. When I am ready, I would like to have 
this fix reviewed by Frank, Naoki and Tao and make sure that we are not 
overlooking anything. 

Comment 11

[racham]

When I said mac migration is broken in  my previous update...I was referring to 
the migration of japanese profiles on Mac (bug 45183), not english profiles.

English profiles are migrated successfully on all platforms as of today.

Comment 12

[racham]

I have been doing some testing to see if the migrated information has been 
consumed properly. During the testing I did last time, as an acocunt that 
refers my 4x imap account showed in the folderPane, I thought things are going 
fine. But that is not right, I opened the Users50 folder to find a directory 
with some weird chars and that folder contained ImapMail folder and 
corresponding subfolders. So, that means the ImapMail folder inside the current 
profile directory (a migrated directory) has been ignored. 

When the imapFolder code tried to get the ImapMail folder from the profile 
directory, the profile name in the path got messed up and the file system ended 
up thinking that we are talking about a new folder and creates  it (hence that 
new weird directory). nsFilePath, a mechanism used in deriving the imapFolder 
name might have messed up things here...I am going to try some things to see if 
we can get rid of this problem.

Then I need to check if we need to the same thing with LocalFolders and News 
code. If I have to modify the code in all these locations, we may be adding some 
risk here....!

Adding Scott Putterman, David Bienvenu and Scott MacGregor to the cc list.

Comment 13

[Katsuhiko Momoi]

You know that IMAP mailbox folder names when they contain
8-bit data are actually represented using modified-UTF7 encoding.
So, something like the following is not unusual under Mozilla.

&MMYwuTDI-.msf

I don't think we are doing the same thing for Local folders,
however.

Comment 14

[racham]

I have been debugging the japanese migration task to findout additional efforts 
we need to put forward to make this happen (besides the patch I have).

* After the prefmigrator completes the data migration (i.e., copyig 
4x to 6.0), browser comes up. Clicking on messenger icon kicks off 
MessengerMigrator service. MessengerMigrator starts it's task of setting 
ImapFolder prefs (probably other prefs too). I found that it builds FilePath 
prefs in UCS2 as it operates and then at the end it writes them to prefs file 
(written into the prefs file in UTF-8 format). Then the folderpane starts 
querying for imapfolder and prefs returns the utf-8 info it has...and we  
STATIC_CAST that value to nsFilePath in AddDirectorySeparator routine 
 and assign it to nsAutoString after doing a conversion (nsImapFolder.cpp:175). 
That operation messes up the foldername and we are creating a new folder with 
that name (thinking it is new) and we proceed further (for ascii characters we 
don't find this weirdness as STATIC_CAST does nothing bad in that case). Any 
caller since the messenger is triggered are going through GetPath to get the 
folder value and if we have japanese migration, that catch here is, we get 
different values depending on whether the caller is MessengerMigrator or other 
service. I think we should be doing CopyUnicharPref while reading the prefs and 
use nsIFile to get and set the data. I noticed newsFolder doing the similar  
casting and conversion in it's outine. LocalFolders seems to getting 
NativePathString which is good. Better solution in all cases is to use nsIFile. 
If I have overlooked something here, please do add commets.

* Currently, News folder is not visisble on the folderpane inspite of having the 
data in the profile directory...this could be another path manipulation or 
pref-migrator not dropping all required files. 

* The patch for profilemanager (will post as an attachment) is full of 
NS_Converts back and forth. The major profilemanager functions GetProfileDir and 
SetProfileDir have been modified with these conversions. That calls for 
extensive testing on profilemanager (mozilla and commercial), say even when we 
come with other fixes.

Though we see bookmarks and pref settings (of 4x) being honored, not having the 
mail piece makes the whole migration process very weak.

So, the real solution lies in using nsIFile (which can't be done at this point 
as it will change function declarations and disturb all callers and so on....). 

Given the number of things that we need to do and the risk we may end up adding 
with the non-nsIFile solution, I think we should not pursue this bug for nsbeta2 
anymore and do it in nsbeta3.

Modifying the status from nsbeta2+ to nsbeta- and adding the keyword nsbeta3.  
changing "fix in hand" to "partial fix in hand". 

Adding Steve Elmer to the cc list.

Comment 15

[racham]

Attachment: profilemanager patch

Comment 16

[racham]

*** Bug 45183 has been marked as a duplicate of this bug. ***

Comment 17

[selmer (gone)]

OK, given all the potential badness I don't think we should pursue the full
solution for this now in beta3 either.  However, we need to do _something_.
What I'd recommmend is making sure we can get all the contents from the old
profile and then migrate into a name that everyone can handle.  We should prompt
the user for a new ASCII-ONLY name for the new Seamonkey profile.  If that's not
feasible, then using "default" is better than not being able to migrate at all...

Comment 18

[lchiang]

+ per mail/migration triage for items selmer wrote above.

Comment 19

[Teruko Kobayashi]

What is the status of this bug?

Comment 20

[racham]

nsIFile support has been implemented in the profilemanager. We shall be able to
migrate information across. I will look into the japanese builds and see what's
going on..? There might stillbe some mail issues (MessengerMigrator and Imap
Access issues) which need to be ironed out. I will work on a debug build
(Naoki's machine) and see how far are we going here..

Adding Seth to the cc list.

Comment 21

[selmer (gone)]

Adding actual nsbeta3+ marker now.  Bhuvan, what's your estimate on how long
it'll take to fix this?

Comment 22

[racham]

Yesterday night, I worked on japanese build. There are still some issues in
getting the directory name in the correct format (everything is stored in the
UTF8 format and we need things in UCS2 when we operate). I am going to work
further on this today. I will be able to give estimates at the end of the day...

Comment 23

[Phil Peterson]

PDT agrees P1

Comment 24

[racham]

While I transfer the data to the new registry (4x->6.0), I get a localFile 
version of the source directory (4.x directory) and get a persistentDescriptor 
string out of it. So, in order to assign it back to nsString, I am converting 
the sting back to UCS2 from UTF8 (NS_ConvertUTF8toUCS2) and the conversion is 
failing at this point. I will debug this furher. I may need to talk to tao or 
Frank to findout what's going on here..Will update the bug with any further 
findings.

Comment 25

[Peter Trudelle]

PDT agrees this is needed on branch, marking nsbeta3++

Comment 26

[racham]

Trying to transfer the data without any loss (with hope of not using too many 
conversion routines in t the middle), I ran into the situation where 
GetPersistentDescriptor is not defined (for nsILocalFile) to return a Unicode 
String. So, that calls for introduction of conversion routine and also not to do 
the conversion to the system charset earlier than required..Will continue to 
work on this with all the routes I can take. But after solving this there is 
another part to this that is the mailnews side story. So, the ETA for the total 
solution is unknown at this point. Updating the status whiteboard accordingly.

Comment 27

[racham]

Some breakthrough here (thanks to naoki). GetPersistentDescriptor for Unicode is
not implemented but the required functions to achieve the end result to such an
effect are available. I used GetUnicodePath() and InitWithUnicodePath() in place
of GetPersistentDescriptorUnicodeString() and
SetPersistentDescriptorUnicodeString(). I am able to migrate the files across
now. I see my bookmarks and prefs getting reflected properly.
I haven't checked the equivalency of other files.

Coming to the Mail story, it did display my migrated account. HOwver when I
started using the account, it created a new folder (name in some unreadable
format and that one has ImapMial, Mail and News folders) in the Users50 folder
and stored all .msf files of my Imap account in it's ImapMail folder instead of
storing them under folder Users50/<japanese named folder>/ImapMail. This is not
encouraging.

I will post diffs that brought us to this stage.

Comment 28

[racham]

Attachment: updated profilemanager patch

Comment 29

[racham]

The patch posted (ID=15634) is not the complete solution. It only fixes the part
of the bringing the data across i.e., from 4.x folder to 6.0 folder.

I will be working with Seth to findout the issues involved with mail.

Comment 30

[racham]

I think we have the complete fix.

Seth has modified the small amount of code in pref-migrator and that avoided the 
problems of creating directories with weird names to hold mail folders. 
Migration code today does a utf-8 conversion on some special prefs that come 
4.x. Some of those prefs include path names. So, they were coming in system 
charset and this utf-8 conversion function tried to convert those path names. 
As a result, we have those odd-looking prefs. The code that accesses these prefs 
has been changed to use GetFileXPref and we don't need to convert those prefs to  
UTF8 format anymore and use the prefs that are coming in the native system 
format as they come in. So, all the those special prefs with a possibility 
of having file name or path in native charset have been removed from prefs 
listed in migrator code (all prefs -
http://lxr.mozilla.org/seamonkey/source/profile/pref-migrator/src/nsPrefMigratio
n.cpp#2070). Seth says all we should be OK with this change. I will post the 
diff related to that in my next update.

So, we still have to test these patches on mac and linux JS systems. I will 
build optimized dlls/libraries and pass then over to QA engineers in i18n team. 

Comment 31

[racham]

Attachment: pref-migrator diffs done by seth

Comment 32

[racham]

We need Naoki Hotta, Frank Tang and Alec Flett's help in reviewing and 
super-reviewing these changes.

Attachment ID 15634 : ProfileManager fix to get and set the profile directory 
paths in Persistent Unicode format.

Attachment ID 15785 : ProfileMigrator fix to leave alone some of the prefs which 
contain native path in the value of the pref. 

Comment 33

[(not reading, please use [e-mail] instead)]

a note about why that change to nsPrefMigration.cpp is safe:

we are just removing the "path" prefs from the list of prefs that we need to
convert into UTF8.

For english users, the "path" prefs that were already in ASCII which means we
never converted them anyways.

Comment 34

[racham]

btw, seth mentioned to me that he will be adding a comment to the fix he 
proposed asking to not to add any prefs that can possibly contain native 
file/directory paths.

Comment 35

[Alec Flett]

for the first patch, you can actually declare nsLocalFile on the stack, as in 
nsLocalFile locProfileDir;
locProfileDir.InitWithUnicodePath((profileItem->profileLocation).GetUnicode())

etc

I'm curious how this will work on the mac, because I don't know how
profileItem->profileLocation is used. if it's used as a path, then this wont
work on mac.

but sr=alecf on both patches

Comment 36

[Alec Flett]

oops, I mis-read the patch

Comment 37

[Alec Flett]

this looks fine. I should say I'm surprised it ever worked on mac, but still
sr=alecf

Comment 38

[nhottanscp]

r=nhotta

Comment 39

[(not reading, please use [e-mail] instead)]

racham:  feel free to put a comment in nsPrefMigration.cpp (about why not to put
path prefs in the list of prefs to convert) for me.

Comment 40

[racham]

Totally unrelated to this bug, I ran into bug 50625 (Can't compile
nsPrefMigration.cpp optimized) while building mac optimized
build. Recompiling with proposed fix for that bug (Adding a pragma optimization
level define). So, optimized builds on the way....

Comment 41

[lchiang]

(I thought this was fixed last night.  Has the bug just not been updated yet?)

This morning's verification builds are already done and we were hoping that to 
be the final builds.

Comment 42

[(not reading, please use [e-mail] instead)]

looking at the tree, this fix has not been checked in yet.

Comment 43

[racham]

No. It is not not checked in yet. I don't want to check this in until we clear
all tests on japanese and english builds. It is yet to pass all verfications on
both english and japanese machines for all the profile manager functionalities.
Teruko will be doing japanese testing. I am testing on english OS platforms.

Comment 44

[racham]

Tests on all platforms (en : Mac, Win, Linux && ja : Win, Linux) are passed OK
except for japanese mac machine. We have some profile information migration
problems there. Need to debug on Mac Japanese machine to find out the problem.
Naoki & Frank, please let me know if anyone has Mac Ja OS machine. thanks. Now
this is a rtm problem.

Adding rtm keyword to this.

Comment 45

[nhottanscp]

I don't have Japanese MacOS installed on my machine. But we may be able to
reproduce the problem using MacRoman charset. That can be tested on US system.
Please try some European characters (e.g. option+tilda+'a').

Comment 46

[scottputterman]

Marking [rtm need info]. When you have the reviews and final patches please 
remove the need info.

Comment 47

[racham]

sure. Currently working with profiles with European chars to debug on mac. So,
there will be one more patch and then we will be ready for the next step.

Comment 48

[racham]

Naoki helped me figured the weirdness with eupropean profile migration problems
on mac. I can migrate profiles with european characters successfully now. I will
pass on the optimized library with these changes bits to teruko soon and we
shall get the status on japanese also. Thanks Naoki.

Comment 49

[Frank Tang]

nhotta- can you review the patch ?

Comment 50

[racham]

Sat with teruko to verify the fix on Japanese mac cmahine. Japanese mac is still
giving problems and away from the progress made on european profiles. Profiles
with european chars worked find mac OS english machine. Mac JS OS has much more
basic problems likenot even displaying Japanese name properly. Now, we seriously
need to debug on Mac as substitute mechanisms aren't helping us as we thought.
Currently installing Japanese OS on Naoki's machine. Will start debugging on
that soon.

Comment 51

[racham]

Yes...! We are almost there..! I was able migrate profile also successfully using 
a debiug build on japanese mac machine. Thanks to Naoki for settign up this 
machine and helping me deciphering i18n stuff.

4.x mac has all the information already in unicode and it needed no further 
conversion. So, that fact provided  break-through. We were able to list all 4.x 
japanese profiles. A japanese 4.x profile created using english communicator 
was migrted successfully. We ran into some problems while trying to migrate 
a japanese profile created using a japanese communicator  (bug filed on Seth : 
55449).  Adding dependency on 55449.

Will post what would be a complete patch for this bug. Naoki, the additional 
changes since the last update are the only i18n related. Please review them 
one more time. Thanks.

Comment 52

[racham]

Attachment: latest profilemanager patch

Comment 53

[racham]

If the fix for 55449 is going to be complicated, we may want to release note
that. i18n and Marketing should probabaly give a call on that.

Aslo, there is another related bug 55444 to migration (again a bug that may be
hit by a small number of people...but need to be fixed oneday) which probably is
also another release note candidate, if the fix is any complicated.

Comment 54

[nhottanscp]

As we talked, the mac-roman hack is not needed, otherwise looks fine, r=nhotta.

Some comments on the related bug (bug 55449 and bug 55444).
* If both are not fixed that means migration from Japanese Mac 4.x always fails.
* If only 55444 is fixed then migration does not fails but local mail and news
are not migrated.
* If 55449 is fixed then the user can migrate Japanese Mac 4.x profile.

Comment 55

[racham]

yeah..I remember and I thought I took them out. I will take out that hack and
for records post the diffs again in the bug. Thanks.

Comment 56

[racham]

Attachment: profmgr diffs (no mac-roman hack)

Comment 57

[Frank Tang]

sr=ftang the "09/28/00 17:12" and "10/06/00 13:14" patch looks good.

Comment 58

[Phil Peterson]

rtm++

Comment 59

[Teruko Kobayashi]

Bhuvan's fix does not work completely in Win32 build.
I tested his patch in 2000-09-30 Win32 build. 

I found out this worked fine depending on which 4.x profile directory was created. 

1. In Naoki's NT machine, 
   You created 4.x profile under C:\Program files\Netscape\users directory. 
   Netscape 6 is located in D:\Program files\Netscape\Netscape 6  directory. 

  This works. 

2. My one NT 4.0J machine, 
    I created 4.x profile under D:\475\Users directory. 
    Netscape 6 is located in D:\Program files\Netscape\Netscape 6  directory. 

   This does not work. 
  
3. My 2nd NT 4.0J machine, 
    I created 4.x profile under D:\   directory. 
    Netscape 6 is located in D:\Program files\Netscape\Netscape 6  directory. 

   This works fine.  I tried this in the first NTJ.  This works fine in the machine, also. 

4. My Win95J machine, 
    You created 4.x profile under C:\Program files\Netscape\users directory. 
    Netscape 6 is located in C:\Program files\Netscape\Netscape 6  directory. 

   This does not work. 

Comment 60

[racham]

Teruko,

I will come and take a look at your cube and check to see why are you runnning
into these differences. May be you are running into the other migration problems
we have. Anyway, I will take a look.

bhuvan

Comment 61

[Teruko Kobayashi]

Bhuvan and I checked his dll with 10-06 Win32 build.  When we created new Japanese profie in 4.75, it migrated correctly.
The Japanese profile name was not migrated was some special case.  It was 4.x bug.  It does not create profile 
Japanese characters whose tail byte is 5c correctly.  I will find the bug.  We need this for release note.
I will investigate this issue. 
If we use the different Japanese name, the migration works fine.  

I will test Bhuvan's fix on Mac build next Monday. (He has debug build.)  

Comment 62

[racham]

As mentioned by Teruko, all cases on all windows platforms are working fine
except for profiles with that special 4.x japanese character she mentioned. I am
holding off my checkins until monday on Teruko's request as she wants to test
these changes on optimized build built on japanese mac machine.

Comment 63

[Jaime Rodriguez, Jr.]

Racham - Please attached your patches to this bug, and obtain the propoer
reviews prior to Maybe, you can get the reviews, while Teruko does the
testing??? This could save us some time.

Comment 64

[racham]

Reviews are already done and also teruko tested this morning on the mac.
Japanese profiles created using english communicator 4.x are migrated properly.
Japanese profiles created using japanese communicator 4.x will run into bug 55449.

I am going to check this in today. Just updating trees on all platforms to pick
up weekend changes.

Comment 65

[Teruko Kobayashi]

I found the bug report about 5c problem in 4.x.  366745 was resolved as Wontfix.
I tested Bhuvan's fix in Win32 again.  His fix works fine except this 5c profile name.

Comment 66

[racham]

Fix checked in. Landed both on branch and trunk.

We should be able to migrate japanese profiles now (expect for one case I 
mentioned in my previous update, which should be taken care by fix to bug 
55449). 

This bug demanded attention & help from many people. Thanks everyone.

Comment 67

[racham]

forgot to change the resolution status. marking fixed.

Comment 68

[Teruko Kobayashi]

Added relnoteRTM in keywords.

Comment 69

[Teruko Kobayashi]

Verified as fixed in 2000-10-11-09 MN6 build.