7067 - All profile contents should use cross-platform formats

Status: RESOLVED INCOMPLETE

(Core Graveyard :: Profile: BackEnd, enhancement, P4)

Description

[Chris McAfee]

Win32 and Linux should be able to share prefs, profiles.
E.g. a dual-booted system would use the same profile location
for either OS.

[[e-mail] says:]
This should probably be filed as at least one enhancement request bug.  It's
not simply a matter of profile manager or even prefs behavior, everything
that lives in the profile would need to be in a cross-platform format.  Each
item in the profile directory and its subdirectories represents some set of
Seamonkey features that needs to be examined to see if they support this
configuration.  Specific bugs that detail how it fails is the best way to
figure out what needs to be done and determine who/when/if it will get done.

I think it's reasonable to claim that all files in a profile should be in
cross-platform formats (and probably I18Nified too.)  What I don't know is
whether we're already there or very far away since it depends on doing some
investigation for multiple modules.

Comment 1

[selmer (gone)]

I'll accept this bug, but: 1) it's a low-priority enhancement request, 2) it
doesn't contain the information about what's actually "broken," 3) other bugs
are likely to be required once it's understood which parts of a profile don't
already work properly.

I've changed the summary to reflect the underlying problem rather than the
symptom.  Please add module owners to the CC list if you think their area may
violate the cross-platform rule.

Comment 2

[Phil Peterson]

I emailed this bug report to mailnewsstaff for comment. Offhand, I can't think
of a reason that mail/news profile data wouldn't be XP.

Comment 3

[davidmc]

Added myself to the cc list.  I also don't know of any problems, but in
the past I did lots and lots of thinking about what constitutes a cross
platform format, so I can help analyze any such problem completely.

There are only two main problems I think one should watch for: 1) line
endings, and 2) integers larger than one byte in size written in binary.

The first is solved if all variations of CR, LF, and CRLF are taken as
equivalents on all platform readers.

The second is no problem if all content is composed of one-byte chars,
or two byte chars with a specific byte ordering like in utf8.  When this
is not true, you have a problem.

When integers are written in binary, the address ordering of bytes is
either little endian or big endian.  The following figure illustrates:

   hex         little endian (e.g. Win)    big endian (e.g. Mac)
+------------+--------------------------+--------------------------+
| 0x4321     |  0x21  0x43              |  0x43  0x21              |
+------------+--------------------------+--------------------------+
| 0x87654321 |  0x21  0x43  0x65  0x87  |  0x87  0x65  0x43  0x21  |
+------------+--------------------------+--------------------------+

One can actually check platform endianess dynamically at runtime with
the following very fast code snippet:

  unsigned short twoBytes = 0x4321;
  BOOL isBigEndian = ( ((char*) &twoBytes) != 0x21 );
  // Note this works correctly even when sizeof(unsigned short) > 2,
  // because little endian puts 0x21 in first byte no matter how big.

A multibyte int written on one platform will read with inverted bytes
on a platform with the other endianess.  There are three main ways to
handle this: 1) convert ints to ascii strings with defined byte order,
2) write in one endianess in all platforms as a canonical standard,
or 3) annotate each int or group of ints written with its endianess.

If profiles contain mostly text, then we are most likely to get into
trouble with i18n content that contains any two-byte integers, either
containing a character or else a run length field for associated text.

For example, naked Unicode can be expressed by a size field plus some
sequence of two byte characters, and all the characters as well as the
size field are endian sensitive.

Comment 4

[davidmc]

D'oh, bug fix for code snippet (remember to deref the pointer):
  BOOL isBigEndian = ( *((char*) &twoBytes) != 0x21 );

Comment 5

[Matthew Tuck [:CodeMachine]]

Anyone going to do an audit of this?

Comment 6

[selmer (gone)]

We need to get to this sooner than M15, changing to M12.

Comment 7

[Matthew Tuck [:CodeMachine]]

This should really be totally XP if possible, some people might want to carry
their profile around on a floppy or zip.

Comment 8

[daver]

moving to M13

Comment 9

[Simon Fraser [no longer active]]

There are two ways to deal with the line termination issues that sharing a text
file between platforms entails:
1. Use a standard delimiter on all platforms (e.g. LF)
2. Use the platform-native delimiter, and ensure that the file reading code can
   deal with other types of line termination.

Because users may want to manually edit files in their profile with a native
text editor that is not smart about line breaks (e.g. SimpleText on the Mac),
then solution 2. is preferred.

Comment 10

[Benjamin Smedberg]

I don't mind making the profiles XP, but please read the thread "new
location for Users50 directory on Windows" thread on <A
HREF="news://news.mozilla.org/netscape.public.mozilla.seamonkey">mozilla.seamonk
ey</A>. It is very important to
many sysadmins on shared computers (including myself) that the personal
profile information be stored in the OS-Proper location, which means for
WinNT in /WinNT/Profiles/{User}/Application Data, and on unix in the $HOME
directory.

Comment 11

[Chris McAfee]

I believe Unix uses $(HOME)/.mozilla/<profile> as the
profile directory, mentioned thread only applies to Win32.

Comment 12

[selmer (gone)]

Bug 6464 contains the profile directory location discussion.  This bug is for
the contents of profiles only.

Comment 13

[leger]

Moving all libPref component bugs to new Preferences: Backend component.  
libPref component will be deleted.

Comment 14

[Paul MacQuiddy]

spam: moving qa contact on some bugs from paulmac to [e-mail]

Comment 15

[sairuh (rarely reading bugmail)]

->gbush for profile mgr.

Comment 16

[Matthew Tuck [:CodeMachine]]

Err, did this get done for M14?

Comment 17

[moshev]

How come this is still M14? M14 is already out!

Comment 18

[Peter "jag" Annema]

And so is M16.

Comment 19

[Mike]

M16 has been out for a while now, these bugs target milestones need to be 
updated.

Comment 20

[Oliver Klee]

Milestone 0.8 has been released. We should either resolve this bug or update its
milestone.

Comment 21

[timeless]

temporary reassign, please add info and pass around.

Comment 22

[Conrad Carlen (not reading bugmail)]

This is big. I'm very interested in it and would like to pursue it. There's
several issues here:
(1) Most of the files which we store in a profile dir are text. I agree with
Simon that we should write the text files in platform native line endings so
people can edit them and make sure whatever reads them can deal with CR, LF, and
CR/LF line endings.
(2) Prefs (especially mail) contain file prefs which are written in a
platform-native way. We need to instead use XP relative file specs in the
profile registry and pref files. That is discussed in bug 12911 and I posted
something on n.p.m.xpcom about that.
(3) There's an easy way to deal with endianess - the method used by the Mac
header Endian.h. It has macros such as EndianU32_BtoN which converts a big
endian Uint32 (the macros come in all sizes) to the native endianess. This means
that the files are always written in the same endianess using the _BtoN version
of the macro. Depending on what the std endianess of the file and what native
endianess is, the macros become no-ops. Very clean technique. I say we copy it.
Depending on what the std endianess of the files is, the platform on which the
macros are no-ops wins performance wise. If we voted, I'm sure the little-endian
hordes would win :-(

Issue #2 I would like to deal with first because it has benefits on its own as
well as contributing to the solution of this bug. The three points I gave are a
lot of work. May be better to divide and conquer and make this bug dependent on
those. 

Comment 23

[Conrad Carlen (not reading bugmail)]

Whoops, typo - the sentence:

> This means that the files are always written in the same endianess using the
> _BtoN version of the macro.

should say _NtoB (if the decided format was big-endian)

Comment 24

[Mike Kaply [:mkaply]]

I would love to see this, but I'll throw one more thing in the mix just to make 
it more painful...

8.3 filenames :)

Comment 25

[Arthur]

Cc'ing myself. Are files already enough XP to run mozilla with the same profile
under Linux, Solaris and Win NT? I think I could set the necessary links without
to much pain.

Comment 26

[Xavier Poinsard]

I did some tests with dual-boot W2000/Linux and the main problem is the path to
the mail storage folders.
I remember there is the same problem with the PSM.

Comment 27

[Conrad Carlen (not reading bugmail)]

Yep. That's bug 12911.

Comment 28

[timeless]

Why do people comment w/o reading the bug?  Prefs.js has file refs which aren't 
XP friendly (marking blocker).  For reference, you don't need to comment to add 
yourself as a CC.

Offhand, the mail stuff isn't XP because the mail folder locations isn't XP 
safe

Comment 29

[Conrad Carlen (not reading bugmail)]

*** Bug 93875 has been marked as a duplicate of this bug. ***

Comment 30

[Grace Bush]

*** This bug has been marked as a duplicate of 137978 ***

Comment 31

[Grace Bush]

sorry marked wrong bug

Comment 32

[Aleksander Adamowski]

I'd like to nominate bug 137006 (the follow-up to bug 12911) for inclusion in
dependencies here.

Comment 33

[Malcolm Ferguson]

Perhaps more of the files in the profiles should be XML.  As I understand it,
XML parsers (at least the ones I've used) are supposed to be able to handle
little vs big endian.  Using UTF-8 or UCS-2/4 with byte order marks (BOMs) makes
this trivial.  Or am I barking up the wrong tree here?

Comment 34

[Bayard]

Having just bought a USB keychain with the aim of sharing a single profile
between many machines [really just the bookmarks.html, but why not the rest] I'm
looking forward to this functionality appearing in Mozilla. There currently
doesn't seem to be a way to select the profile to use, you have to create a new one.

Comment 35

[Travis Chase]

*** Bug 187332 has been marked as a duplicate of this bug. ***

Comment 36

[Marcelo de G. Malheiros]

I´ve been trying to share profiles between Linux and Windows XP (both running
Firefox 1.0 r.v.1.7.5 Gecko20041118), but with partial success.

I can force both systems to use the same profile, by automatically by creating
specific profiles that can be shared through a network drive woith SAMBA, which
mount the user homes in /home and H:. I used the following commands:

  firefox -createprofile "default /home/<user>/.mozilla/firefox/profile"
  firefox -createprofile "default H:\.mozilla\firefox\profile"

This makes possible to ensure all system users (both on Linux and on Windows) to
have their profiles automatically created when they login (if the profile
already exists, nothing is changed). Then I can force the opening of the created
profiles with the command (same on both plataforms):

  firefox -P default

However the catch is that some files inside the profile are not the same,
because of specific component names, paths and plataform-dependent libraries:

In "compreg.dat":

 [COMPONENTS]
-rel:libjar50.so,1100780760000
+rel:jar50.dll,1100783820000
-rel:nsSetDefaultBrowser.js,1100780760000
+rel:nsSetDefaultBrowser.js,1100783820000
-rel:nsDictionary.js,1100780760000
+rel:nsDictionary.js,1100783820000
-rel:nsCloseAllWindows.js,1100780760000
+rel:nsCloseAllWindows.js,1100783820000
...

[CLASSIDS]
-{6a8c0dd4-1dd2-11b2-9a8f-f82f9df25b07},,application/x-mozilla-static,,nsGfxGTKModule
+{6a8c0dd4-1dd2-11b2-9a8f-f82f9df25b07},,application/x-mozilla-static,,nsGfxModule
-{8b5314ba-db01-11d2-96ce-0060b0fb9956},,application/x-mozilla-static,,nsWidgetGtk2Module
+{8b5314ba-db01-11d2-96ce-0060b0fb9956},,application/x-mozilla-static,,nsWidgetModule
...

Also, in "xpti.dat":

[Header,2]
 0,Version,2,0
-1,AppDir,/usr/local/firefox1.0
+1,AppDir,C:\ARQUIV~1\MOZILL~1

 [Directories,2]
-0,/usr/local/firefox1.0/components
-1,/usr/local/firefox1.0/plugins
+0,C:\ARQUIV~1\MOZILL~1\components
+1,C:\ARQUIV~1\MOZILL~1\plugins

-[Files,2]
-0,browser.xpt,0,267762,1100780760000
-1,qfaservices.xpt,0,144,1100780760000
+[Files,1]
+0,browser.xpt,0,264450,1100783820000

Finally, the "XUL" files are even not the same, as there is "XUL.mfasl" on
Windows and "XUL.mfl" on Linux.

Maybe a possibility is to put these different files in platform-specific
directories, or append a platform-specific suffix, so they can coexist without
interference...

I guess this "feature" is important for the widespread use of
Mozilla/Firefox/Thunderbird.

Comment 37

[Nigel McFarlane]

Regarding comment 36. I think there's an agenda to reduce the number
of profile file-formats now (not that it helps with portability
specifically). Some thoughts on your problem files:

compreg.dat:

the [COMPONENTS] part is quite hard to fix.

the [CLASSID] part could be addressed by making module names
port-independant. That would require changes to nsIModule so
that a port-specific "instance" attribute could be added. That
way port-specific module instances would still be uniquely
identified in the code base, even if only one instance of any
module appears in any product bundle.

xpti.dat:

Windows can handle forward /'s as well as back \'s. It would be
interesting to hack an install and test if "C:\USR\LOCAL\FIREFOX1.0"
(the chosen windows install directory) could be reduced to
"/usr/local/firefox1.0" (with default volume C:). Might require
XPCOM changes to support.

The [Files,2] part is different because you installed Talkback
on Windows only (ie your two installs differ).

xul.mfl:

The XUL cache part is of debateable value for a local app install that
has a network-based profile. Unless the network is blindingly fast
(Fast Gigabit), the profile'd cache is probably a performance bottleneck.
For this kind of install, the best XUL cache file format is "no file at all".

- N.

Comment 38

[timeless]

erm. fastload files are based on platform specific bits. you don't want the
fastload file from linux on windows, you'd lose functionality. you don't want
the windows version on linux, you'd get strange errors.

as for magically mapping / on windows, i'll cry fowl. use relative paths.

Comment 39

[Benjamin Smedberg]

compreg.dat is also a machine-specific file, and should not be shared between
multiple machines of the same OS, much less between machines of different OSes.

Comment 40

[Elmar Ludwig]

*** Bug 310510 has been marked as a duplicate of this bug. ***

Comment 41

[Ryan]

(In reply to comment #40)
> *** Bug 310510 has been marked as a duplicate of this bug. ***

Coming from a university research environment with multi *nix platforms.  This
is   a high priority for us.  We run Sun and Linux with NIS.   This bug rears
its head everytime a user jumps from a sun to linux workstation to do their
research.

Comment 42

[Willy_ Foo_Foo]

One of the beauty of Mozilla is its platform independent
MAC_OS/WINDOWS/*NIX same profile will work everywhere(just copy the profile folder if you dual boot or have many devices, please keep it that way)


But implementation of web apps seems to be breaking this
So i would request to have an options to disable this because i prefer cross platform profile usability over a web app, i don't want to sync profiles(using internet) to test stuff under same profile on various platforms


My advise would be to install all files in the profile directory called Web Apps/appname
so it is cross platform(can be launched from inside firefox if profile is shared cross platform[similar to addons])& easily uninstalled like extensions

Comment 43

[Bill Walker [:bwalker] [@wfwalker]]

(In reply to Sillius Soddus from comment #42)
> One of the beauty of Mozilla is its platform independent
> MAC_OS/WINDOWS/*NIX same profile will work everywhere(just copy the profile
> folder if you dual boot or have many devices, please keep it that way)
> 
> 
> But implementation of web apps seems to be breaking this
> So i would request to have an options to disable this because i prefer cross
> platform profile usability over a web app, i don't want to sync
> profiles(using internet) to test stuff under same profile on various
> platforms
> 
> 
> My advise would be to install all files in the profile directory called Web
> Apps/appname
> so it is cross platform(can be launched from inside firefox if profile is
> shared cross platform[similar to addons])& easily uninstalled like extensions




We _are_ committed to providing an awesome Apps experience for every Web Runtime for every App we list in our Marketplace. As Tim says, our implementation involves a modest amount of platform-specific information for each native installation.

Rather than managing Apps by copying profiles around, we want users to take advantage of our Apps in the Cloud (AITC) service[1]. With AITC, the user natively installs their Apps on all their devices. Making the actual profiles cross-platform is not a project goal right now, since we don't expect users to manage their Apps that way.

[1] https://wiki.mozilla.org/Services/AppsInTheCloud

Comment 44

[Benjamin Smedberg]

This bug is filed in a bugzilla component related to pre-Firefox code which no longer exists. I believe it is no longer relevant and I am therefore closing it INCOMPLETE.

If you believe that this bug is still valid and needs to be fixed, please reopen it and move it to the Toolkit:Startup and Profile System product/component.