The default bug view has changed. See this FAQ.

[FEATURE]Prefs backend code must be changed to use UTF8

VERIFIED INVALID

Status

()

Core
Preferences: Backend
P3
normal
VERIFIED INVALID
18 years ago
17 years ago

People

(Reporter: mcmullen, Assigned: neeti)

Tracking

Trunk
All
Mac System 8.5
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

18 years ago
I'm adding this task as a bug to help me track it.
(Reporter)

Updated

18 years ago
Component: Pref UI → libPref
(Reporter)

Comment 1

18 years ago
Setting the right component.
(Reporter)

Updated

18 years ago
Hardware: Macintosh → All
(Reporter)

Comment 2

18 years ago
Platform=all
(Reporter)

Updated

18 years ago
Status: NEW → ASSIGNED
Target Milestone: M5
(Reporter)

Comment 3

18 years ago
Changing plumbing bugs to M5 milestone, and accepting them.

Updated

18 years ago
Target Milestone: M5 → M6

Comment 4

18 years ago
Changed target milestone to M6.
(Reporter)

Comment 5

18 years ago
Changing to m7
(Reporter)

Updated

18 years ago
Target Milestone: M7 → M8
(Reporter)

Comment 6

18 years ago
This has been scheduled for M8

Updated

18 years ago
Blocks: 7228

Comment 7

18 years ago
While the majority of prefs are probably limited to ASCII values, some can
contain non-ASCII strings.  Examples:

  - File or directory names (encoded in the localized OS script/codepage)
  - Font names may (encoded in the localized OS script/codepage)
  - Real names (e.g., mail/vcard identities) can have non-ASCII LDAP v3
    prefs are stored in UTF-8

The 4.x mish-mash is due to historical evolution.  This worked sorta OK as long
as these files were used only on the local system.

We want to improve 5.0 and beyond.  This is a chance for us to change the
file format.  After 5.0, we will want to preserve forward compatibility.

If we ever want prefs to be served remotely (a la something like Location
Independence), the 4.x model breaks.  It is a known problem that LI does
not interoperate between platforms.

Any code reading the prefs should convert from UTF-8 to the appropriate
encoding (e.g., UCS-2 for internal processing or the OS script/codepage
to access the filesystem).  Hopefully the pref API will provide some
easy mechanism for this.

Since the 4.x data is not tagged, the migration tool should probably guess
that the old prefs are encoded in the current OS script/codepage.  (I18N
can provide APIs to determine this.)  And then convert to UTF-8.  The
tool will need to special case the LDAP prefs (already in UTF-8) and not
convert those entries.  Do we want to provide a UI for the end-user to
change the default (guessed) conversion?

The other issue is how to let users edit non-ASCII prefs.  Most text
editors today on all platforms, do not support UTF-8 (NT editors can
handle UCS-2.).  Two options:

   1. Provide something similar to the Java native2ascii tool to convert
      from UTF-8 to another editable encoding.
   2. Rely upon Ender to edit plain text UTF-8 files.  Theoretically this
      should work since Ender support both plain text and UTF-8.   But
      we'd need to do some QA.  Also, it may be a chicken and the egg
      problem, if you need to hack to prefs to bring up Communicator.
      But maybe that's an edge case that can be handled by creating a
      new profile.

Comment 8

18 years ago
M9, giving to don for re-assigning.

Updated

18 years ago
Assignee: don → dp
Target Milestone: M9

Comment 9

18 years ago
dp, can your team eventually handle this issue?

Updated

18 years ago
Assignee: dp → neeti
(Assignee)

Updated

18 years ago
Status: NEW → ASSIGNED
Target Milestone: M11
(Assignee)

Updated

18 years ago
Summary: Prefs backend code must be changed to use UTF8 → [Feature]Prefs backend code must be changed to use UTF8
(Assignee)

Updated

18 years ago
Summary: [Feature]Prefs backend code must be changed to use UTF8 → [FEATURE]Prefs backend code must be changed to use UTF8
(Assignee)

Updated

18 years ago
Blocks: 11408

Comment 10

18 years ago
Adding lyecies@netscape.com to cc list.
(Assignee)

Updated

18 years ago
Assignee: neeti → bobj
Status: ASSIGNED → NEW
(Assignee)

Comment 11

18 years ago
Bob,
I have discussed this bug with Erik Van der Poel and Frank Tang to figure out
the changes required for the prefs backend code(libpref) to use UTF8. Currently
libpref uses char* in its apis for e.g.
PREF_SetCharPref(const char *pref_name, const char *value)
PREF_GetCharPref(const char *pref_name, char * return_buffer, int * length).

Both Frank and Erik agree that we should not be adding or changing apis in
libpref, but always treat the char* in the above apis as UTF8 strings. The code
calling these apis should make sure that the char* is a UTF8 string, if the data
is non-ascii. So, once the old prefs files are converted to UTF8 using the
migration tool, libpref could handle utf8.

I am assigning you the bug so that you could reassign it to the appropriate
persons to write the migration tool utility.

Neeti

The r

Updated

18 years ago
Assignee: bobj → ftang

Comment 12

18 years ago
Bob is on sabbatical and Erik is on vacation.
re-assigning to ftang for now..

Updated

18 years ago
Status: NEW → ASSIGNED

Comment 13

18 years ago
The issue should be solve in the code which CALL
PREF_SetCharPref(const char *pref_name, const char *value)
PREF_GetCharPref(const char *pref_name, char * return_buffer, int * length).

but not themself. For these two function. Make sure we clearly spec out the
char* mean UTF8 .

Neeti, could  you do the spec out part ?
(Assignee)

Comment 14

18 years ago
Frank, I can spec out spec out the char* mean UTF8 . But, I think this will only
be effective, once we have the migration tool ready.

Comment 15

18 years ago
I think we need to change the API - in
http://lxr.mozilla.org/mozilla/source/modules/libpref/public/nsIPref.idl
100   void SetCharPref(in string pref, in string value);
should be
100   void SetCharPref(in string pref, in wstring value);

and
126   void SetDefaultCharPref(in string pref, in string value);
should be
126   void SetDefaultCharPref(in string pref, in wstring value);

and
137   string CopyCharPref(in string pref);
should be
137   wstring CopyCharPref(in string pref);

If we do that, then we also need to change the following callers-

http://lxr.mozilla.org/mozilla/ident?i=CopyCharPref
http://lxr.mozilla.org/mozilla/ident?i=SetDefaultCharPref
http://lxr.mozilla.org/mozilla/ident?i=SetCharPref

a safer plan is first add the new wstring version (in C++ it will be
PRUnichar* or nsString) and then graduage change the callers. And finally remove
the string version.

Reassign this back to neeti for adding the wstring version.
Assignee: ftang → neeti
Status: ASSIGNED → NEW
(Assignee)

Updated

18 years ago
Status: NEW → ASSIGNED
Depends on: 14349
(Assignee)

Comment 16

18 years ago
Opened a new bug for the migration tool utility - 14349. Bug 5130 if
implementing the three new apis which Frank mentioned above.
(Assignee)

Updated

18 years ago
Target Milestone: M11 → M15
(Assignee)

Comment 17

18 years ago
Moving target milestone to M15
spam: added self to cc list as this might affect my realm.
(Assignee)

Comment 19

17 years ago
I think we should not be adding or changing apis in libpref, but always treat
the char* in the above apis as UTF8 strings. The code calling these apis should
make sure that the char* is a UTF8 string, if the data is non-ascii. So, once
the old prefs files are converted to UTF8 using the migration tool, libpref
could handle utf8.
(Assignee)

Comment 20

17 years ago
The libpref api need not know anything about anything about the various
encodings. It just stores the string it was passed in, and returns the same.

The callers of the api should make sure that char* means UTF-8

Comment 21

17 years ago
Alec has already started adding APIs to libpref to allow callers to get/set
Unicode (PRUnichar) strings. I think we need to continue going down this path,
because it is too painful for the rest of Mozilla to do the UTF-8 <-> Unicode
conversion all over the place, when it could easily be done in one place
(libpref).

Comment 22

17 years ago
Alec, see my previous comment in this bug.

Comment 23

17 years ago
Eric, How would that be ?

Any code using libpref should already be doing that. I see two cases:

1. Code storing strings into pref are already using utf 8 in which case there is
no change

2. Code storing strings into pref are using ascii or whatever and need to be
converted to utf8. The api's you are adding arent going to solve the problem as
now instead of changing these code from converting to utf8, you would be
changing this code to convert to unicode. Is that so much simpler to do the
later ?

I dont understand the reasoning behind this api addition to libpref.

Comment 24

17 years ago
The existing APIs are defined in terms of XPIDL "string". These are, and should
remain, UTF-8. I am asking for the addition of APIs defined in terms of XPIDL
"wstring", which are of course PRUnichar (Unicode). And as I said, Alec has
already started adding such APIs. I'm asking for that work to be completed.
There is a separate bug report for that completion work. I don't have the bug
number, off-hand.

Comment 25

17 years ago
Ok I talked with eric about this. I think we need to talk about this together. I
will try to huddle with alecf, neeti, eric and selmer to close on this.

Comment 26

17 years ago
I agree with erik that the API needs to be extended.. there are many places
where you need to retrieve a unicode string from prefs, and it makes no sense to
make all these call sites do the conversion over and over, in possibly different
or incorrect ways.
... please take a look at
http://lxr.mozilla.org/seamonkey/source/modules/libpref/src/nsPref.cpp#708

The way I did it there was approved by the i18n team some time back... if there
are any revisions or improvements, we only have to do it in one place, not
fifty.

Comment 27

17 years ago
I see. So are we planning on continuing the unicode and char * apis. Sounds like
should actively deprecate the char * api.

Comment 28

17 years ago
I think it would be good to deprecate the char* (string) APIs. However, it might
be a good idea to see how much code will be affected by such a change.

Comment 29

17 years ago
no! that's not right either. Sometimes you just want a char* pref... for
instance in mail we use a char* pref to determine the server type like "imap" or
"pop3" - it would be rediculous to convert those from UTF8 to PRUnichar* and
back to UTF8 if I just want to compare it with a char string constant like
"imap"

Comment 30

17 years ago
OK, I don't feel too strongly about this, so leave the char* APIs in, but add
PRUnichar* APIs correspondingly.

Comment 31

17 years ago
ok, I've checked in the fix to add get/set of default Unichar prefs, so I'd say
mark this INVALID or WONTFIX.
(Assignee)

Comment 32

17 years ago
Marking this invalid, since the existing APIs are defined in terms of XPIDL 
"string". These are UTF-8. Alec has checked in the APIs defined in terms of 
XPIDL "wstring", which are PRUnichar (Unicode).
Status: ASSIGNED → RESOLVED
Last Resolved: 17 years ago
Resolution: --- → INVALID

Comment 33

17 years ago
Moving all libPref component bugs to new Preferences: Backend component.  
libPref component will be deleted.
Component: libPref → Preferences: Backend

Comment 34

17 years ago
sorry for the spam, changing QA contact.
QA Contact: paulmac → sairuh
verif.
Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.