Last Comment Bug 454708 - storage-Legacy can throw when calling ConvertToUnicode
: storage-Legacy can throw when calling ConvertToUnicode
Status: VERIFIED FIXED
Please read comment 48 before adding ...
: regression, verified1.9.0.3, verified1.9.0.4
Product: Toolkit
Classification: Components
Component: Password Manager (show other bugs)
: 1.9.0 Branch
: All All
: -- major with 2 votes (vote)
: mozilla1.9.1b1
Assigned To: Justin Dolske [:Dolske]
:
Mentors:
: 454993 456983 457066 457212 457259 (view as bug list)
Depends on: 457358
Blocks: 451155 456835
  Show dependency treegraph
 
Reported: 2008-09-10 17:03 PDT by Ed
Modified: 2010-10-06 06:01 PDT (History)
45 users (show)
dveditz: blocking1.9.0.3+
stephen.donner: in‑testsuite+
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---


Attachments
a sample of corrupted signons3.txt (223 bytes, text/plain)
2008-09-17 20:34 PDT, Atsushi Sakai
no flags Details
key3.db for the sample signons3.txt (16.00 KB, application/octet-stream)
2008-09-17 20:37 PDT, Atsushi Sakai
no flags Details
Patch v.1 (4.35 KB, patch)
2008-09-19 13:26 PDT, Justin Dolske [:Dolske]
no flags Details | Diff | Splinter Review
Patch v.2 (3.36 KB, patch)
2008-09-23 16:50 PDT, Justin Dolske [:Dolske]
gavin.sharp: review+
mbeltzner: approval1.9.0.3+
Details | Diff | Splinter Review
Patch v.2 (for branch) (3.97 KB, patch)
2008-09-24 10:41 PDT, Justin Dolske [:Dolske]
no flags Details | Diff | Splinter Review
A simple testcase - saving as the default latin-1 will fail. UTF-8 will not (464 bytes, text/plain)
2008-09-24 10:50 PDT, Jesper Hansen
no flags Details

Description Ed 2008-09-10 17:03:20 PDT
User-Agent:       Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.4; en-US; rv:1.9.0.1) Gecko/2008070206 Firefox/3.0.1
Build Identifier: Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.4; en-US; rv:1.9.0.1) Gecko/2008070206 Firefox/3.0.1

Ever since upgrading, the new Firefox is not remembering passwords or asking if I want to. My preferences are set to REMEMBER PASSWORDS FOR SITES and there are no exceptions in the EXCEPTIONS box.

Reproducible: Always

Steps to Reproduce:
1.I type user name and password for any site, paypal, wachovia, etc. 
2. It takes me to the logged in site without asking if I want the password remembered.
3.
Actual Results:  
It happens at every site that requires a password.
Comment 1 Chris C 2008-09-12 00:29:15 PDT
Build identifier : Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.2) Gecko/2008090514 Firefox/3.0.2

I'm also having this issue since upgrading to the newest version. signons3.txt is still there, but it seems no passwords are actually read from this file.
The passwords manager display stays blank, although the file it's supposed to read from...isn't blank.
Comment 2 Ria Klaassen (not reading all bugmail) 2008-09-12 00:36:39 PDT
(In reply to comment #1)
> Build identifier : Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.2)
> Gecko/2008090514 Firefox/3.0.2
> 
> I'm also having this issue since upgrading to the newest version. signons3.txt
> is still there, but it seems no passwords are actually read from this file.
> The passwords manager display stays blank, although the file it's supposed to
> read from...isn't blank.
In that case I'd say: make a screenshot of your passwords while it's still possible.
Comment 3 Justin Dolske [:Dolske] 2008-09-15 21:53:42 PDT
Please attach some debug output per https://wiki.mozilla.org/Firefox:Password_Manager_Debugging.
Comment 4 Chris C 2008-09-16 14:24:01 PDT
Debugging shows the following :

Login Manager: Initialization of storage component failed: [Exception... "Component returned failure code: 0x8000ffff (NS_ERROR_UNEXPECTED) [nsIScriptableUnicodeConverter.ConvertToUnicode]"  nsresult: "0x8000ffff (NS_ERROR_UNEXPECTED)"  location: "JS frame :: file:///C:/Program%20Files/Mozilla%20Firefox%203/components/storage-Legacy.js :: anonymous :: line 853"  data: no]

That line 853 reads :
line.value = this._utfConverter.ConvertToUnicode(line.value);
Comment 5 Justin Dolske [:Dolske] 2008-09-17 15:07:42 PDT
Bah. This is probably a regression from bug 451155... The bug fixed buy 451155 could result in invalid UTF8 being written to the file, so now that we try to convert it from UTF8, the bogus data will cause it to fail. :(

We can't recover the login (due to lossy UTF16->ascii conversion), but we should at least try/catch the UTF8 conversion and delete the login.

I'd be curious to see which line in your profile's signon3.txt is causing the problem. Could you email me this file? (The passwords are encrypted, but you can edit out encrypted-looking lines if you'd like.)
Comment 6 Atsushi Sakai 2008-09-17 20:34:47 PDT
Created attachment 339174 [details]
a sample of corrupted signons3.txt

(In reply to comment #5)
> I'd be curious to see which line in your profile's signon3.txt is causing the
> problem. 

The sample is here.
I made it on https://うっかり.jp/members/
Comment 7 Atsushi Sakai 2008-09-17 20:37:52 PDT
Created attachment 339175 [details]
key3.db for the sample signons3.txt

key3.db is here.
Not to worry because real password is not included.
Comment 8 Justin Dolske [:Dolske] 2008-09-19 13:26:12 PDT
Created attachment 339502 [details] [diff] [review]
Patch v.1

The core fix here is to wrap the charset conversion in a try/catch. The tricky bit is deciding what to do with a corrupted entry...

One option is to do nothing, and keep it around. It won't get used anywhere, but it's a safe and minimal change.

The other option is to flag corrupted entries, and drop them when loading the file. This makes for tidy housekeeping, but adds complexity, risk, and more to test. The particular risk here is a "drop this entry" code path adds the possibility of having a bug that results in more dataloss (eg, forgetting to reset the "drop this entry" flag, and dropping all following entries).

I've implemented the second option here (untested!), but I'm sort of leaning towards just doing the first option instead.

Gavin, thoughts?
Comment 9 :Gavin Sharp [email: gavin@gavinsharp.com] 2008-09-19 13:47:23 PDT
option 1 sounds best - cost of ignoring them is pretty low I think.
Comment 10 Paul O'Shannessy [:zpao] (not reading much bugmail, email directly) 2008-09-19 14:08:16 PDT
If we keep it around (option 1), will that effect importing with the switch to mozStorage? Or will we just import a corrupted entry?
Comment 11 Justin Dolske [:Dolske] 2008-09-23 16:50:39 PDT
Created attachment 340035 [details] [diff] [review]
Patch v.2

Updated to implement option 1, with tests.

(In reply to comment #10)
> If we keep it around (option 1), will that effect importing with the switch to
> mozStorage? Or will we just import a corrupted entry?

No. It's a valid string once it's in memory, the problem only happens when we try converting the on-disk value from UTF8. The mozStorage code is unaffected by this.
Comment 12 Justin Dolske [:Dolske] 2008-09-23 17:03:45 PDT
Updating summary.

Gavin noted on IRC that this bug can't be the same problem that the original reporter (Ed) was having. The patch that caused this regression didn't land until Firefox 3.0.2. This bug was filed against FF3.0.1. So, although the symptoms are the same, they must be separate problems. *sigh*

Since this bug has already mutated, I've filed bug 456661 to track the original problem from comment 0.
Comment 13 Mike Beltzner [:beltzner, not reading bugmail] 2008-09-24 06:54:13 PDT
Comment on attachment 340035 [details] [diff] [review]
Patch v.2

a=beltzner, drop it like it's hot
Comment 14 Mike Beltzner [:beltzner, not reading bugmail] 2008-09-24 07:43:06 PDT
Uhm, I suppose we should have this on mozilla-central before crash landing it on the cvs trunk.
Comment 15 Justin Dolske [:Dolske] 2008-09-24 09:14:12 PDT
Pushed changeset 5ba271f60a30.

(Branch landing coming up shortly)
Comment 16 Ivan Pavlov 2008-09-24 09:58:27 PDT
sorry for breaking in, i'm complete novice here and haven't touch the Firefox code, but I had same problem, landed here and saw the proposed options to fix.
I think that the options to drop out entries would cause a lot of problems to international users. 
What I mean - I have a lot of entries in signons3.txt like:
http://www.navet.government.bg (phpMyAdmin работи на localhost)

so dropping out these entries will disable almost all my phpMyAdmin accounts. This would be fully discouraging for me to migrate to 3.0.2

So may be you could consider also following options:
- as Firefox works fine with such entries in 3.0.1 and not in 3.0.2 maybe there is "easy" way to make a migration tool (best automatic on update, or at least an external tool)
- to remove only the invalid characters if they are not inside the URL. Maybe this would not break the whole entry and it will be still usable. This would at least reduce the damages.
Comment 17 Justin Dolske [:Dolske] 2008-09-24 10:24:41 PDT
(In reply to comment #16)
> sorry for breaking in, i'm complete novice here and haven't touch the Firefox
> code, but I had same problem, landed here and saw the proposed options to fix.
> I think that the options to drop out entries would cause a lot of problems to
> international users.

Nope.

Any existing logins that worked in 3.0.1 will continue to work, and any logins you could save with 3.0.1 will continue to be savable. We're not breaking any functionality.

The logins we were talking about "dropping" in this bug were already corrupted, and wouldn't be working. Bug 451155 fixed this problem -- you can now save logins on IDN sites and have them work when you revisit the site. Before that fix, the logins were corrupted when saved. This bug is about what we do when reading in those already-corrupted logins.

> - to remove only the invalid characters if they are not inside the URL. Maybe
> this would not break the whole entry and it will be still usable.

That would break the logins even more, and they still wouldn't work.
Comment 18 Justin Dolske [:Dolske] 2008-09-24 10:41:35 PDT
Created attachment 340164 [details] [diff] [review]
Patch v.2 (for branch)

Checked in on FF3.0 branch:

Checking in toolkit/components/passwordmgr/src/storage-Legacy.js;
  new revision: 1.29; previous revision: 1.28
Checking in toolkit/components/passwordmgr/test/unit/test_storage_legacy_3.js;
  new revision: 1.8; previous revision: 1.7
Checking in toolkit/components/passwordmgr/test/unit/data/signons-454708.txt;
  initial revision: 1.1
Comment 19 Jesper Hansen 2008-09-24 10:50:00 PDT
Created attachment 340168 [details]
A simple testcase - saving as the default latin-1 will fail. UTF-8 will not

Attached is a signons3.txt file as provided by Firefox 3.0.2
Now the original signons3.txt from a upgraded profile from 2.0 => 3.0.0 => 3.0.1 => 3.0.2 made the file latin-1. All new profiles create it as utf-8. 
If saving the file as latin-1 then firefox 3.0.2 will fail.
If saving the file as utf-8 then firefox 3.0.2 will not fail.

Use it for testing. 

Requested by Tomcat.
Comment 20 Justin Dolske [:Dolske] 2008-09-24 11:27:08 PDT
Landed on GECKO190_20080827_RELBRANCH (for 3.0.3).

Checking in toolkit/components/passwordmgr/src/storage-Legacy.js;
  new revision: 1.28.2.1; previous revision: 1.28
Checking in toolkit/components/passwordmgr/test/unit/test_storage_legacy_3.js;
  new revision: 1.7.2.1; previous revision: 1.7
Checking in toolkit/components/passwordmgr/test/unit/data/signons-454708.txt;
  new revision: 1.1.2.1; previous revision: 1.1
Comment 21 Alex 2008-09-24 13:35:42 PDT
I've got into the same problem after update to FF 3.0.2. It is great to see the status as "RESOLVED FIXED", however in what way it can be applied? I really need my saved passwords back!
Thanks.
Comment 23 Henrik Skupin (:whimboo) 2008-09-24 16:00:46 PDT
*** Bug 454993 has been marked as a duplicate of this bug. ***
Comment 24 Mike Beltzner [:beltzner, not reading bugmail] 2008-09-24 18:30:49 PDT
Comment on attachment 340035 [details] [diff] [review]
Patch v.2

(fixing flags after a mass flag reset)
Comment 25 Ivan Pavlov 2008-09-25 01:00:30 PDT
(In reply to comment #19)
> Now the original signons3.txt from a upgraded profile from 2.0 => 3.0.0 =>
> 3.0.1 => 3.0.2 made the file latin-1. All new profiles create it as utf-8. 
> If saving the file as latin-1 then firefox 3.0.2 will fail.
> If saving the file as utf-8 then firefox 3.0.2 will not fail.

I used my own signons3.txt from version 3.0.1 - opened it as latin-1 (ISO-8859-1) and saved it back as utf-8. After installation of 3.0.2 the passwords were on place, even entries with unreadable characters are working.
Comment 26 dsr 2008-09-25 03:53:37 PDT
P L E A S E  tell all people which are not software developers and who have this problems with passwords too :

1. open notepad.exe
2. look for a file signons3.txt in the folder "documents and settings" 
   subfolder= nameof user\userdata\Mozilla\Firefox\Profiles
   (Dokumente und Einstellungen\name anwender\Anwendungsdaten\Mozilla\Firefox
    \Profiles)
3. open the file signons3.txt in notepad
4. save the file as ! : leave the name and choose the option save as UTF-8
   ANSI is the standard 
5. restart firefox and afterwards you should have your passwords back :)

   für alle verzweifelten.
Comment 27 Gyorgy Kapolnas 2008-09-25 06:28:06 PDT
Tank you, you just saved my life :)
Comment 28 dsr 2008-09-25 06:54:16 PDT
... thought it would be helpful ...
kind regards
Comment 29 Benjamin Smedberg [:bsmedberg] 2008-09-25 07:26:42 PDT
*** Bug 456983 has been marked as a duplicate of this bug. ***
Comment 30 David Tenser [:djst] 2008-09-25 08:10:52 PDT
*** Bug 454993 has been marked as a duplicate of this bug. ***
Comment 31 Tracy Walker [:tracy] 2008-09-25 09:14:36 PDT
Verified this with the test case in bug 451155 created in 3.0.1. Uninstalled then installed 3.0.3 build1 on Win XP.  The Password Manager is not blank like it was 3.0.1->3.0.2.  

Marking verified.  note: the password entry created in 3.0.1 is invalid and isn't corrected by upgrade.  It has to be recreated by the user to make a valid entry in PWM. I made a relnote comment in 451155 about it.
Comment 32 Joel Maher ( :jmaher) 2008-09-25 10:49:54 PDT
verified in linux in a similar fashion to Tracy from comment #31:
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.3) Gecko/2008092416 Firefox/3.0.3
Comment 33 Matthias Versen [:Matti] 2008-09-25 13:03:38 PDT
*** Bug 457066 has been marked as a duplicate of this bug. ***
Comment 34 Atwill Europe 2008-09-26 03:19:27 PDT
You can also mark Bug 457178 as a duplicate of this bug...
Comment 35 Jim Jeffery not reading bug-mail 1/2/11 2008-09-26 05:46:30 PDT
*** Bug 457212 has been marked as a duplicate of this bug. ***
Comment 36 Carsten Book [:Tomcat] - PTO-back Sept 4th 2008-09-26 10:10:13 PDT
*** Bug 457259 has been marked as a duplicate of this bug. ***
Comment 37 Tuomo Tanskanen 2008-09-27 03:41:49 PDT
Autoupdated to 3.0.3, passwords still not saved and the list is empty, just like 3.0.2!
Comment 38 Florian Wolf 2008-09-27 04:25:37 PDT
Made a downgrade from 3.0.2 to 3.0.1
Then patched to 3.0.2 again and applied 3.0.3

Passwords still won't work.

OS: WinVista x64
Comment 39 Mardeg 2008-09-27 04:36:18 PDT
(In reply to comment #38)
> Passwords still won't work.
..and for anyone else visiting this bug to add comments, there are reports on irc that in 3.0.3 just adding a new password (by going to any new site with a form for logging in and entering a bogus one and choosing to save it when prompted) then restarting firefox fixes any carry-over problems.

Could you please try that and mention whether doing that works?
Comment 40 Tuomo Tanskanen 2008-09-27 05:02:08 PDT
Comment #39: It does indeed work and fixes the issue. However, the regular user does not appreciate the non-automatic solution and may not be able to fix it, frustrates and leaves Mozilla for good.
Comment 41 Florian Wolf 2008-09-27 05:03:35 PDT
Didn't try.
I fixed it with downgrading to 3.0.1 again and applying 3.0.3 directly without the mentioned UTF-8 fixes above. Thx anyways :)
Comment 42 Aureliano 2008-09-27 09:56:49 PDT
It worked for me.
Comment 43 aglamer 2008-09-27 12:09:07 PDT
Fixed only after applying method from comment #39. Thanks to Mardeg and shame to developers. Nobody should know tricks in order to fix the problem.
Comment 44 Alex 2008-09-28 11:03:25 PDT
#43 +1
Comment 45 VikeDragon 2008-09-28 19:36:37 PDT
Fire Fox 3.0.3 is not remembering passwords too ... No accessing to password...
Comment 46 Echi 2008-09-28 23:51:01 PDT
The Patch didn't work for me, too. I fixed it with re-installing FireFox.
Comment 47 PREMARTIN 2008-09-29 00:15:24 PDT
the patch works on my computer, old password are back and it propose to save new ones.
thanks
Comment 48 :Gavin Sharp [email: gavin@gavinsharp.com] 2008-09-29 01:03:39 PDT
The problem people are seeing in 3.0.3 is most likely bug 457358. Comment 39 is a known workaround for that bug. We're going to be landing the fix for that bug in Firefox 3.0.4. Please refrain from adding comments here or there unless there is new information available.
Comment 49 beckfield1 2008-09-29 18:41:58 PDT
The workaround in comment 39 did not work for me.  I am running FF 3.0.3 on Kubuntu Linux 8.04.  After following the instructions in comment 39, I also tried deleting passwords from the password manager and re-establishing them.  This did not work either.

I'm not sure what relevant information I can provide.  Please ask and I'll do my best.
Comment 50 Justin Dolske [:Dolske] 2008-09-29 18:45:12 PDT
If you're still seeing problems after adding or removing a login, please file a new bug and attach debugging output per https://wiki.mozilla.org/Firefox:Password_Manager_Debugging
Comment 51 md 2008-09-30 01:17:14 PDT
Hi, the Passwort Manager still does not work for me (XPSP2), since the Update to 3.0.3 I am asked for my master password when I go to a password protected Website, but the Passwort is not filled in. Here is the debug output:

PwMgr Storage: Failed to decrypt string: MDoEEPgAAAAAAAAAAAAAAAAAAAEwFAYIKoZIhvcNAwcECCeWezDqxdTnBBCh+2tmsj2ZH4jatEAMdNpH (NS_ERROR_UNEXPECTED)
.. (repeated) ..
PwMgr Storage: Failed to decrypt string: MEIEEPgAAAAAAAAAAAAAAAAAAAEwFAYIKoZIhvcNAwcECB1OY9s/BVzXBBjEOs6YRhvakDV4WeaPXSfl/DHSPnl1BJ8= (NS_ERROR_UNEXPECTED)
.. (repeated) ..
Login Manager: form[0]: found 0 matching logins.
Login Manager: (form ignored -- no password fields.)

Any Idea? When I reinstall 3.0.1 it works fine.
Comment 52 md 2008-09-30 01:37:18 PDT
OK, Comment #39 also works for me.
Comment 53 Angus Ireland 2008-10-02 12:38:56 PDT
Same as Comment #51 for me.
For a short while my passwords worked fine and now...
PwMgr Storage: Failed to decrypt string: VeryLongStringWhichI'veTruncated (NS_ERROR_UNEXPECTED)

Theres ~10 passwords that work but apart from them... all errors like above
Comment 54 Dan 2008-10-08 16:54:16 PDT
Have tried all solutions listed here.  My firefox 3.0.3 installation simply stopped letting me use passwords.  Rebooting, reinstalling firefox, saving as UTF8.  None of these things worked.

This just happened today, but I recently upgraded to 3.0.3 so maybe it's been going on for a couple days

Dan
Comment 55 Dan 2008-10-08 17:49:21 PDT
I have downgraded to 3.0.1, but passwords still inaccessible.  It is now allowing me to add passwords, but the old ones aren't showing up.  It appears they may be lost forever, even though they are in the signons3.txt in encrypted form.

-Dan
Comment 56 Justin Dolske [:Dolske] 2008-10-08 18:02:34 PDT
Dan: Please file a new bug, and attach debugging as described in https://wiki.mozilla.org/Firefox:Password_Manager_Debugging.
Comment 57 Justin Dolske [:Dolske] 2008-10-08 18:04:55 PDT
(removing fixed1904 keyword, I think it was accidentally added when things were shuffled around for 3.0.3/3.0.4. This bug is already correctly marked verified1.9.0.3).
Comment 58 Samuel Sidler (old account; do not CC) 2008-10-09 07:38:00 PDT
fixed1.9.0.4 was added because this patch was fixed in two places: the relbranch for the firedrilled Firefox 3.0.3 and the main cvs trunk for Firefox 3.0.4. This needs to be reverified for the next release.
Comment 59 Al Billings [:abillings] 2008-10-23 16:40:57 PDT
Verified for 1.9.0.4 with Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.0.4pre) Gecko/2008102304 GranParadiso/3.0.4pre.
Comment 60 Jesper Hansen 2008-10-23 17:45:21 PDT
Verified fixed Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.4pre) Gecko/2008102304 GranParadiso/3.0.4pre

latin-1 passwords (Realm) are being listed in the password manager now without any other "user-fixes".

One strange thing though. When I save a new password, it convert the text in the file to UTF-8. However the file is still being opened as latin-1 by gedit making the UTF-8 chars in the file being displayed as latin-1 by gedit. Line should read "begrænset", but reads it "begrænset".
It is being reed as the correct "begrænset" however.
See the testcase.

Can be ignored?
Comment 61 Alexander Hörnlein 2009-11-12 06:39:52 PST
Happened me on update to 3.5.5 (windows 7)- first the update couldn't be applied, retried to update and update finished but passwords were gone - had to change encoding of signon3.txt to utf-8 -> passwords are back

Note You need to log in before you can comment on or make changes to this bug.