328981 - History DB page titles are displayed corrupt in url bar autocomplete, endian problem turns Latin characters into Chinese

Status: RESOLVED FIXED

(Firefox :: Bookmarks & History, defect)

Description

[Mark Mentovai]

Strings stored in history.dat are stored as UCS2.  This includes page titles.  history.dat is supposed to have a native endianness, set when the file is created or reset.  When using a history.dat file whose native endianness does not match the host architecture's endianness, existing page titles in the database are not being swapped (or are double-swapped?) for display in the url bar autocomplete.  This results in corrupt page titles being displayed in the history autocomplete dropdown.

Although bytes aren't swapped when reading from history.dat, they are swapped as needed when writing to it.  This means that if you have a history.dat file set as other-endian, even new pages you visit will wind up with bad page titles being displayed.  Other reads from history seem unaffected: the Go menu and history sidebar do not have corrupt text.

This will affect people who had run Firefox under Rosetta on x86 and switch to a native version, people who migrate from a ppc to x86 and copy their profile (or have their profile copied by a migration assistant), and people who wish to flip Rosetta on and off to use plugins that haven't yet been ported to x86.

This was fixed for Core:History in bug 108134.  The byte-swapping code is still in the toolkit history implementation, we need to find out why the swaps don't happen (or happen twice?) during reads from history for url bar autocomplete display.

Comment 1

[Steve England [:stevee]]

Is this problem going to be nullified with the Places landing I wonder?

Comment 2

[Mark Mentovai]

Yeah, that's why I set the version to 1.5.0.x.

Comment 3

[Mark Mentovai]

Nominating because of the effect this has on the x86 transition.

Comment 4

[Mark Mentovai]

nsAutoCompleteController (toolkit) isn't getting its strings from nsGlobalHistory, here:

http://bonsai.mozilla.org/cvsblame.cgi?file=mozilla/toolkit/components/history/src/nsGlobalHistory.cpp&rev=1.58.2.2.2.2&mark=992-1004#976

Instead, it's getting them from nsAutoCompleteMdbResult, here:

http://bonsai.mozilla.org/cvsblame.cgi?file=mozilla/toolkit/components/autocomplete/src/nsAutoCompleteMdbResult.cpp&rev=1.9&mark=249-250#228

nsAutoCompleteMdbResult doesn't do any byte-swapping.

Comment 5

[Mark Mentovai]

Attachment: Byte-swap in nsAutoCompleteMdbResult

This patch was prepared against the 1.8.0 branch, because Places nullifies it on the trunk and 1.8 branch.  Even so, I intend to bake it there - that code shouldn't fall behind until it's permanently disabled.  This applies to the trunk with fuzz 7 (UCS2 APIs were deprecated in favor of UTF16).

Comment 6

[Mark Mentovai]

Attachment: v2, avoid an AND in SwapBytes

Updated with feedback from 328982

Comment 7

[Brian Ryner (not reading)]

i thought we weren't going to change interfaces for 1.5.0.x. releases...?

Comment 8

[Mark Mentovai]

Attachment: v3, with extended interface

Well, I doubt anyone else is using anything as specific as nsIAutoCompleteMdbResult, but OK.  Here, the change is forked out into an extended nsIAutoCompleteMdbResult2 interface.

Comment 9

[Mark Mentovai]

Comment on attachment 213851 [details] [diff] [review]
v3, with extended interface

Checked in on the trunk, but with Places on, it's not part of the build there.

Comment 10

[Brian Ryner (not reading)]

Oops, my comment about new/delete vs malloc/free from bug 328982 applies here as well.

Comment 11

[Mark Mentovai]

Comment 10 addressed on trunk.

Comment 12

[Tim Riley [:timr]]

Comment on attachment 213851 [details] [diff] [review]
v3, with extended interface

a=timr for drivers.  We were waiting for bryner's review.  Got that.  Land it!

Comment 13

[Mark Mentovai]

Landed on 1_8 and 1_8_0 branches.

Comment 14

[Jay Patel [:jay]]

Changing OS to MacOS X.  This is MacOS X only, right?  From Mark's comments, someone with an x86 Mac should verify this.  Cc'ing Marcia in case she can do it.

Comment 15

[Mark Mentovai]

This is not Mac OS X only, it affects any profile shared beteween different-endian systems.  Practically speaking, though, it's OS X-only, because that's the only platform for which official builds are produced for big-endian processors.

Comment 16

[Dave Liebreich [:davel]]

verified on 180 branch using universal binary built on maya - Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:1.8.0.2) Gecko/20060310 Firefox/1.5.0.2

Comment 17

[Phil Ringnalda (:philor)]

*** Bug 330249 has been marked as a duplicate of this bug. ***

Comment 18

[Quiroga]

How to install this attach "v3, with extended interface"?? I have I Imac Intel. 

I need to copile??  If I click on the link just open a "text" code.

Comment 19

[Mark Mentovai]

Firefox 1.5.0.2, release planned soon, will ship with this fix.  In the mean time, you can download a nightly universal build (http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/experimental/universal-binaries/latest-Mozilla1.8.0/).  For help compiling Mozilla from source yourself, see http://developer.mozilla.org/en/docs/Mac_OS_X_Build_Prerequisites .