424641 - Write profile migrator to make old Thunderbird profiles available under "Mozilla", "~/mozilla", or "Application Support"

Status: RESOLVED WONTFIX

(Thunderbird :: OS Integration, defect, P5)

Description

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Bug 247973 explains the simple change in nsMailApp.cpp to store Thunderbird profiles within the same hierarchy as Firefox. At the moment all the profiles are created one hierarchy above. Following changes have to be done:

Windows:  "%AppData%\Thunderbird" => "%AppData%\Mozilla\Thunderbird"
Linux:    "~/thunderbird" => "~/mozilla/thunderbird"
Mac OS X: "~/Library/Thunderbird" => "~/Library/Application Support/Thunderbird"

Because already existing users would lose their profiles when doing this change, there has to be implemented a profile migrator which moves the current profiles to the new location, create a symbolic link or whatever to make it available. It also has to be taken into account that a switch back to Thunderbird 2 should still offer this migrated profiles.

In bug 247973 comment 62 Mark Banner has given following possible idea:

> 5. NO PROFILE MOVE but reference a RELATIVE location for existing profiles.
> 
> This means that:
> - New Profiles are created in the correct place.
> - Existing Profiles aren't moved. No file copying/moving. Nothing to mess up.
> Nothing to miss out. Just referenced from where they currently exist.

Marking this bug as blocker of bug 247973. If we will get such a profile migrator in the future we could do the switch to use the new profile folder.

Comment 1

[Mark Banner (:standard8)]

I'm taking this bug. I think I've got the best idea as to what needs to be done, but I expect this bug in its current form will be won't fixed, as we're not going to *move* profiles.

I'm going to target a solution/conclusion (whatever that may be) to TB 3.0a1, but I'm not going to get to it round to looking at it in any more detail until the 7th April at least.

Comment 2

[Benjamin Kraus]

Although the title includes the word "move", the description offers the suggestion of creating a relative link from the new location back to the old location, which means no profile (files, folders, etc) actually needs to be moved.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Indeed. Sorry for the bad wording. Adjusting summary accordingly.

Comment 4

[Joshua Cranmer [:jcranmer]]

Just a nit: my Linux builds store stuff in ~/.mozilla, ~/.thunderbird, etc.

I think it may be nice to also have new profiles set up in ~/.mozilla/thunderbird instead of ~/.thunderbird/ (even if old profiles stay in ~/.thunderbird).

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to comment #4)
> Just a nit: my Linux builds store stuff in ~/.mozilla, ~/.thunderbird, etc.

Yes, that's the current behavior.
 
> I think it may be nice to also have new profiles set up in
> ~/.mozilla/thunderbird instead of ~/.thunderbird/ (even if old profiles stay in
> ~/.thunderbird).

With the fix on bug 247973 new profiles will be created at this location. And with the profile migrator old profiles should be accessible from there (respecting downgrade capabilities). 

Comment 6

[Mark Banner (:standard8)]

Attachment: The fix

This patch does what I want it to (I'm going to use Linux directory references here for ease of reference):

* If there is no profiles.ini in ~/.mozilla/thunderbird, then look in ~/.thunderbird, if there is no profiles.ini there, treat as if first start-up and create a new profile in ~/.mozilla/thunderbird
* If there is no profiles.ini in ~/.mozilla/thunderbird, and there is one in ~/.thunderbird, then fill out ~/.mozilla/thunderbird/profiles.ini with profile locations (and defaults/flags), but profile locations will now be absolute.
* Remove the various ifdefs, relating to the old ~/.thunderbird location (hence everything will work correctly).

If there is a profiles.ini in ~/.mozilla/thunderbird then the profile service/start up code will pick it up automatically and this code won't be invoked.

I still have to determine the changes to the tinderbox set-ups to get this to work correctly.

I have tested this patch on Mac OS X and Linux, could someone test on Windows please?

Comment 7

[Mark Banner (:standard8)]

This isn't going to make Alpha 1. Probably Alpha 2. I don't think there's much point in requesting wanted/blocking, I'll just drive it through.

Comment 8

[Mark Banner (:standard8)]

Attachment: The fix v2

Better patch -> returns PR_TRUE at the right time (and removes redundant PR_TRUE from for loop).

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Mark, I have started a TryServer build with your patch 2. So we could also run a test on Windows in about an hour. I'll post the link when it's finished.

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Sorry, but the TryServer cannot build Thunderbird at the moment. We still need someone who can build Thunderbird on Windows. I'll comment in bug 247973.

Comment 11

[Robert Kaiser]

This might need updating of tinderbox configs when it lands, possibly even an update to tinderbox code to make sure it finds the profile after creation.

Comment 12

[Mark Banner (:standard8)]

Comment on attachment 318390 [details] [diff] [review]
The fix v2

I'm happy this should work on all platforms. I'll request toolkit review once I know Thunderbird is happy with this.

Note: gets rid of lots of TB-specific stuff from toolkit & core.

Comment 13

[Mark Banner (:standard8)]

Attachment: Patch for Tinderboxes

Tinderbox script patch, again I'll wait for the main one to be reviewed before requesting review.

Comment 14

[Mark Banner (:standard8)]

Attachment: Tinderbox config changes

Here's the additional changes required to the tinderbox configs. Mac actually doesn't need the change, VendorName is already defined as Mozilla, it just gets away with it because the current build-seamonkey-util.pl effectively ignores VendorName for Thunderbird.

Comment 15

[Mark Banner (:standard8)]

I just wanted to expand on what it does slightly:

(In reply to comment #6)
> * If there is no profiles.ini in ~/.mozilla/thunderbird, then look in
> ~/.thunderbird, if there is no profiles.ini there, treat as if first start-up
> and create a new profile in ~/.mozilla/thunderbird
> * If there is no profiles.ini in ~/.mozilla/thunderbird, and there is one in
> ~/.thunderbird, then fill out ~/.mozilla/thunderbird/profiles.ini with profile
> locations (and defaults/flags), but profile locations will now be absolute.

So if you have TB 2 and move to TB trunk (3.x) and don't add/delete any profiles you won't notice a thing, you can use the profiles in either TB 2 and TB 3.

- If you get TB trunk and create new profiles, they won't show up in TB 2.
- If you revert to TB 2 and create new profiles, they won't show up in TB 3.

In both these cases you can manually add the absolute directory to the profiles.ini.

Comment 16

[Ronald Killmer]

For the Windows platform, will this take into account the differences between Vista and XP User App Data paths.  Vista has some differences for Mozilla folder, example:
C:\Users\xxx\AppData\Roaming\Mozilla\Extensions\{3550f703-e582-4d05-9a08-453d09bdfdc6}
There is no "Crash Reports" folder related to Mozilla/Firefox, but Tb has one.

Another oddity of Vista is it also uses the following path:
C:\Users\xxx\AppData\Local\Mozilla\Firefox\

I do not know enough about any Windows NT family of OS to understand any of the in's and out's of %AppData% path variables and whether they complcate the patch in terms of the absolute addressing of profiles.

Comment 17

[Onno Ekker [:nONoNonO UTC+1]]

I don't understand the need to make the path absolute.

When profiles.ini get stored in Mozilla/Thunderbird (on Windows), you can reach the old profiles by going to ../../Thunderbird/, so you can change the TB2 relative path Profiles/26ij3ng4.default to the new TB3 relative path ../../Thunderbird/Profiles/26ij3ng4.default, specifying the same directory.

Making this absolute makes it probably harder to implement and harder for users to move their profiles to the right folder manually.

Comment 18

[Mark Banner (:standard8)]

(In reply to comment #17)
> Making this absolute makes it probably harder to implement and harder for users
> to move their profiles to the right folder manually.

Its certainly not intended that users need to move their profiles.

I also don't see how this makes it harder for users to move their profiles - given that they will have other paths within the profile to change, it isn't exactly hard to work out what you have to change to fix this if you really want to move your profile.

Comment 19

[Mark Banner (:standard8)]

(In reply to comment #17)
> I don't understand the need to make the path absolute.

I have just taken another look at this. For some reason, the toolkit profile service (that we use) forces the relative/absolute behaviour as:

- If profile directory is with the application data directory, then save the location as relative.
- If the profile directory is outside of the application data directory, then save the location as absolute.

http://bonsai.mozilla.org/cvsblame.cgi?file=/mozilla/toolkit/profile/src/nsToolkitProfileService.cpp&rev=1.23&mark=788-798#787

I expect this was done to ease someone moving the whole app data dir to a different place, though I don't know for sure because we haven't got the history there. Unfortunately that limits what we can do here.

Comment 20

[Onno Ekker [:nONoNonO UTC+1]]

(In reply to comment #19)
> > I don't understand the need to make the path absolute.
> 
> I have just taken another look at this.

Like you said in comment #18, it's not that much of a problem is the path gets absolute. I just thought it would make the patch more complex to make the path absolute, but if it's forced to absolute (at least outside profile directory), I would leave it that way...

Comment 21

[Mark Banner (:standard8)]

Comment on attachment 318390 [details] [diff] [review]
The fix v2

This patch will in fact break roaming profiles (on Windows), due to using absolute paths, therefore cancelling reviews. We can't make them relative due to how toolkit currently works.

I'll take a fresh look at this and possible solutions after alpha 2 is out.

Comment 22

[Magnus Melin [:mkmelin]]

Let's keep this on the wanted‑thunderbird3+ list since this would be great to get fixed... (I don't know how realistic it is though.)

Comment 23

[Mark Banner (:standard8)]

I'm not working on this at the moment reverting to default assignee.

Comment 24

[Kent James (:rkent)]

Something that I have never understood about TB's profile usage on Windows is the use of the "..\(User)\Application Data" folder instead of the "..\(User)\Local Settings\Application Data" folder for storing of local email mbox files. I haven't looked at this in a few years, but as I recall the "Local Settings" version was used by Microsoft applications for storing emails, because that directory was not copied for roaming users, and the large size of email files can be too much to copy for roaming users. The entire profile is copied to a local version whenever you login, so if you have several gigabytes of email files, then login was too slow to be practical if the "Local Settings" folder was not used.

If you are going to go to all of the effort of changing the profile location, you might want to consider this issue as well.

Comment 25

[Ronald Killmer]

The case that Kent makes is only true for WinN Pre-Vista.  In Vista there are two sub-folders of a users App Data, Local and Roaming and it is the Roaming folder that holds the Tunderbird/Profiles/xxxx.default/Mail/Mbox files. The Local is where the Tb Update and Profiles/*.mfl and Cache folders are stored. Windows 7 may alter this yet again.

Comment 26

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #17)
> I don't understand the need to make the path absolute.

Any of relative and absolute can be set, but "relative path" is far easy/simple(addition of "../../" only) as you say.

1. set Vender=any-string-you-want in application.ini
> profiles.ini on MS Win-XP:
> C:\Documents and Settings\wada\Application Data\any-string-you-want\Thunderbird\profiles.ini
2. Define a profile by Tb, and change path to next
> Name=xxx
> IsRelative=1
> ../../Thunderbird/Profiles/eu9kojjh.xxx
3. After restart of Tb using the profile, profiles.ini was changed to absolute path format.
> Name=xxx
> IsRelative=0
> Path=C:\Documents and Settings\wada\Application Data\Thunderbird\Profiles\eu9kojjh.xxx

Comment 27

[WADA:World Anti-bad-Duping Agency]

If direct use of "profile directory used by Tb3/2/1.5" is required, migration of profiles.ini only is sufficient.
(directory can be shared by Tb4 and Tb3/2/1.5)
(1) Append old Tb's profiles.ini entries to new Tb's profiles.ini
(2) If duplicate name happens, or rename is required, change Name= value.
(3) Change Path
    Path change due to application.ini,Vendor=Moziila, action is common.
    Tb3 : IsRelative=1, Path=Profiles/xxx.yyy(Win), xxx.yyy(Linux,Mac)
    => Win    Path=../../Thunderbird/Profiles/xxx.yyy
    => Linux  Path=../../thunderbird/xxx.yyy
    => Mac    Path=../../Thunderbird/xxx.yyy
    Tb3 : IsRelative=0, Path=...
    => Path=... (no need to change)
After restart of Tb, IsRelative=/Path= was changed to absolute by Tb on Win.

Above is same as next.
 - Start profile manager of new Tb
 - Create profile(s) with existent profile directory(absolute is used)

Comment 28

[Mark Banner (:standard8)]

This has outstanding issues that are difficult to solve, additionally it doesn't actually adversly affect most people, and is just a profile location issue that the vast majority of users won't care about. Therefore not tracking.

Comment 29

[Techlive Zheng]

(In reply to Mark Banner (:standard8) from comment #28)
> This has outstanding issues that are difficult to solve, additionally it
> doesn't actually adversly affect most people, and is just a profile location
> issue that the vast majority of users won't care about. Therefore not
> tracking.

Under *nix, firefox defaultly located its profile under ~/.mozilla/firefox, this is the request to move the profile location of thunderbird from ~/.thunderbird to ~/.mozilla/thunderbird.

Comment 30

[Tobias B. Besemer [:BesTo] (QA)]

(In reply to Mark Banner (:standard8) (afk until 20th July) from comment #19)
> (In reply to comment #17)
> > I don't understand the need to make the path absolute.
> 
> I have just taken another look at this. For some reason, the toolkit profile
> service (that we use) forces the relative/absolute behaviour as:
> 
> - If profile directory is with the application data directory, then save the
> location as relative.
> - If the profile directory is outside of the application data directory,
> then save the location as absolute.
> 
> http://bonsai.mozilla.org/cvsblame.cgi?file=/mozilla/toolkit/profile/src/
> nsToolkitProfileService.cpp&rev=1.23&mark=788-798#787
> 
> I expect this was done to ease someone moving the whole app data dir to a
> different place, though I don't know for sure because we haven't got the
> history there. Unfortunately that limits what we can do here.

(In reply to Mark Banner (:standard8) (afk until 20th July) from comment #21)
> Comment on attachment 318390 [details] [diff] [review]
> The fix v2
> 
> This patch will in fact break roaming profiles (on Windows), due to using
> absolute paths, therefore cancelling reviews. We can't make them relative
> due to how toolkit currently works.
> 
> I'll take a fresh look at this and possible solutions after alpha 2 is out.

That's all old stuff. Any news to this?

Comment 31

[Mark Banner (:standard8)]

(In reply to Tobias B. Besemer [:BesTo] from comment #30)
> That's all old stuff. Any news to this?

AFAIK nothing has changed in the profile management code that would affect this.

The relative/absolute paths in the profile management are the issue with moving existing profiles, and hence it needs thinking about and doing in a way that wouldn't break existing users (which the patches suggested would do).

Comment 32

[Tobias B. Besemer [:BesTo] (QA)]

(In reply to Mark Banner (:standard8) from comment #31)
> The relative/absolute paths in the profile management are the issue with
> moving existing profiles, and hence it needs thinking about and doing in a
> way that wouldn't break existing users (which the patches suggested would
> do).

Are the paths relative or absolute?
What's about a solution that just save new created profiles in the new path?
Or where's exactly the problem?

Comment 33

[Tobias B. Besemer [:BesTo] (QA)]

(In reply to Mark Banner (:standard8) from comment #19)
> (In reply to comment #17)
> > I don't understand the need to make the path absolute.
> 
> I have just taken another look at this. For some reason, the toolkit profile
> service (that we use) forces the relative/absolute behaviour as:
> 
> - If profile directory is with the application data directory, then save the
> location as relative.
> - If the profile directory is outside of the application data directory,
> then save the location as absolute.
> 
> http://bonsai.mozilla.org/cvsblame.cgi?file=/mozilla/toolkit/profile/src/
> nsToolkitProfileService.cpp&rev=1.23&mark=788-798#787
> 
> I expect this was done to ease someone moving the whole app data dir to a
> different place, though I don't know for sure because we haven't got the
> history there. Unfortunately that limits what we can do here.

(In reply to Mark Banner (:standard8) from comment #21)
> Comment on attachment 318390 [details] [diff] [review]
> The fix v2
> 
> This patch will in fact break roaming profiles (on Windows), due to using
> absolute paths, therefore cancelling reviews. We can't make them relative
> due to how toolkit currently works.
> 
> I'll take a fresh look at this and possible solutions after alpha 2 is out.

Or is, or should we fill a new bug for the toolkit?
Making paths relative if the app is installed on the system, too.
(Guess the relatives are for USB Sticks ...)
Or a routine to move the directories on startup of the app?

This bug is from 2008 and now over 7 years old ... :-/

Comment 34

[Wayne Mery (:wsmwk)]

Given the closure of bug 247973, let's put this bug to sleep.

N.B. comment 19 and comment 21.