storage-Legacy can throw when calling ConvertToUnicode

VERIFIED FIXED in mozilla1.9.1b1

Status

()

Toolkit
Password Manager
--
major
VERIFIED FIXED
9 years ago
7 years ago

People

(Reporter: Ed, Assigned: Dolske)

Tracking

({regression, verified1.9.0.3, verified1.9.0.4})

1.9.0 Branch
mozilla1.9.1b1
regression, verified1.9.0.3, verified1.9.0.4
Points:
---
Dependency tree / graph
Bug Flags:
blocking1.9.0.3 +
in-testsuite +

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: Please read comment 48 before adding a new one)

Attachments

(3 attachments, 3 obsolete attachments)

(Reporter)

Description

9 years ago
User-Agent:       Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.4; en-US; rv:1.9.0.1) Gecko/2008070206 Firefox/3.0.1
Build Identifier: Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.4; en-US; rv:1.9.0.1) Gecko/2008070206 Firefox/3.0.1

Ever since upgrading, the new Firefox is not remembering passwords or asking if I want to. My preferences are set to REMEMBER PASSWORDS FOR SITES and there are no exceptions in the EXCEPTIONS box.

Reproducible: Always

Steps to Reproduce:
1.I type user name and password for any site, paypal, wachovia, etc. 
2. It takes me to the logged in site without asking if I want the password remembered.
3.
Actual Results:  
It happens at every site that requires a password.
(Reporter)

Updated

9 years ago
Version: unspecified → 3.0 Branch
Component: Preferences → Password Manager
Product: Firefox → Toolkit
QA Contact: preferences → password.manager
Version: 3.0 Branch → 1.9.0 Branch

Comment 1

9 years ago
Build identifier : Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.2) Gecko/2008090514 Firefox/3.0.2

I'm also having this issue since upgrading to the newest version. signons3.txt is still there, but it seems no passwords are actually read from this file.
The passwords manager display stays blank, although the file it's supposed to read from...isn't blank.
(In reply to comment #1)
> Build identifier : Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.2)
> Gecko/2008090514 Firefox/3.0.2
> 
> I'm also having this issue since upgrading to the newest version. signons3.txt
> is still there, but it seems no passwords are actually read from this file.
> The passwords manager display stays blank, although the file it's supposed to
> read from...isn't blank.
In that case I'd say: make a screenshot of your passwords while it's still possible.
(Assignee)

Comment 3

9 years ago
Please attach some debug output per https://wiki.mozilla.org/Firefox:Password_Manager_Debugging.

Comment 4

9 years ago
Debugging shows the following :

Login Manager: Initialization of storage component failed: [Exception... "Component returned failure code: 0x8000ffff (NS_ERROR_UNEXPECTED) [nsIScriptableUnicodeConverter.ConvertToUnicode]"  nsresult: "0x8000ffff (NS_ERROR_UNEXPECTED)"  location: "JS frame :: file:///C:/Program%20Files/Mozilla%20Firefox%203/components/storage-Legacy.js :: anonymous :: line 853"  data: no]

That line 853 reads :
line.value = this._utfConverter.ConvertToUnicode(line.value);
(Assignee)

Comment 5

9 years ago
Bah. This is probably a regression from bug 451155... The bug fixed buy 451155 could result in invalid UTF8 being written to the file, so now that we try to convert it from UTF8, the bogus data will cause it to fail. :(

We can't recover the login (due to lossy UTF16->ascii conversion), but we should at least try/catch the UTF8 conversion and delete the login.

I'd be curious to see which line in your profile's signon3.txt is causing the problem. Could you email me this file? (The passwords are encrypted, but you can edit out encrypted-looking lines if you'd like.)
Assignee: nobody → dolske
Blocks: 451155
OS: Mac OS X → All
Hardware: Macintosh → All
Target Milestone: --- → mozilla1.9.1b1
(Assignee)

Updated

9 years ago
Status: UNCONFIRMED → NEW
Ever confirmed: true

Comment 6

9 years ago
Created attachment 339174 [details]
a sample of corrupted signons3.txt

(In reply to comment #5)
> I'd be curious to see which line in your profile's signon3.txt is causing the
> problem. 

The sample is here.
I made it on https://うっかり.jp/members/

Comment 7

9 years ago
Created attachment 339175 [details]
key3.db for the sample signons3.txt

key3.db is here.
Not to worry because real password is not included.
(Assignee)

Comment 8

9 years ago
Created attachment 339502 [details] [diff] [review]
Patch v.1

The core fix here is to wrap the charset conversion in a try/catch. The tricky bit is deciding what to do with a corrupted entry...

One option is to do nothing, and keep it around. It won't get used anywhere, but it's a safe and minimal change.

The other option is to flag corrupted entries, and drop them when loading the file. This makes for tidy housekeeping, but adds complexity, risk, and more to test. The particular risk here is a "drop this entry" code path adds the possibility of having a bug that results in more dataloss (eg, forgetting to reset the "drop this entry" flag, and dropping all following entries).

I've implemented the second option here (untested!), but I'm sort of leaning towards just doing the first option instead.

Gavin, thoughts?
option 1 sounds best - cost of ignoring them is pretty low I think.
If we keep it around (option 1), will that effect importing with the switch to mozStorage? Or will we just import a corrupted entry?
(Assignee)

Comment 11

9 years ago
Created attachment 340035 [details] [diff] [review]
Patch v.2

Updated to implement option 1, with tests.

(In reply to comment #10)
> If we keep it around (option 1), will that effect importing with the switch to
> mozStorage? Or will we just import a corrupted entry?

No. It's a valid string once it's in memory, the problem only happens when we try converting the on-disk value from UTF8. The mozStorage code is unaffected by this.
Attachment #339174 - Attachment is obsolete: true
Attachment #339175 - Attachment is obsolete: true
Attachment #339502 - Attachment is obsolete: true
Attachment #340035 - Flags: review?(gavin.sharp)
(Assignee)

Comment 12

9 years ago
Updating summary.

Gavin noted on IRC that this bug can't be the same problem that the original reporter (Ed) was having. The patch that caused this regression didn't land until Firefox 3.0.2. This bug was filed against FF3.0.1. So, although the symptoms are the same, they must be separate problems. *sigh*

Since this bug has already mutated, I've filed bug 456661 to track the original problem from comment 0.
Summary: Firefox 3.0.1 not remembering passwords → storage-Legacy can throw when calling ConvertToUnicode
Attachment #340035 - Flags: review?(gavin.sharp) → review+
Flags: blocking1.9.0.3+
Keywords: regression
Comment on attachment 340035 [details] [diff] [review]
Patch v.2

a=beltzner, drop it like it's hot
Attachment #340035 - Flags: approval1.9.0.3+
Uhm, I suppose we should have this on mozilla-central before crash landing it on the cvs trunk.
(Assignee)

Comment 15

9 years ago
Pushed changeset 5ba271f60a30.

(Branch landing coming up shortly)
Status: NEW → RESOLVED
Last Resolved: 9 years ago
Resolution: --- → FIXED

Comment 16

9 years ago
sorry for breaking in, i'm complete novice here and haven't touch the Firefox code, but I had same problem, landed here and saw the proposed options to fix.
I think that the options to drop out entries would cause a lot of problems to international users. 
What I mean - I have a lot of entries in signons3.txt like:
http://www.navet.government.bg (phpMyAdmin работи на localhost)

so dropping out these entries will disable almost all my phpMyAdmin accounts. This would be fully discouraging for me to migrate to 3.0.2

So may be you could consider also following options:
- as Firefox works fine with such entries in 3.0.1 and not in 3.0.2 maybe there is "easy" way to make a migration tool (best automatic on update, or at least an external tool)
- to remove only the invalid characters if they are not inside the URL. Maybe this would not break the whole entry and it will be still usable. This would at least reduce the damages.

Updated

9 years ago
Blocks: 454993
(Assignee)

Comment 17

9 years ago
(In reply to comment #16)
> sorry for breaking in, i'm complete novice here and haven't touch the Firefox
> code, but I had same problem, landed here and saw the proposed options to fix.
> I think that the options to drop out entries would cause a lot of problems to
> international users.

Nope.

Any existing logins that worked in 3.0.1 will continue to work, and any logins you could save with 3.0.1 will continue to be savable. We're not breaking any functionality.

The logins we were talking about "dropping" in this bug were already corrupted, and wouldn't be working. Bug 451155 fixed this problem -- you can now save logins on IDN sites and have them work when you revisit the site. Before that fix, the logins were corrupted when saved. This bug is about what we do when reading in those already-corrupted logins.

> - to remove only the invalid characters if they are not inside the URL. Maybe
> this would not break the whole entry and it will be still usable.

That would break the logins even more, and they still wouldn't work.
(Assignee)

Comment 18

9 years ago
Created attachment 340164 [details] [diff] [review]
Patch v.2 (for branch)

Checked in on FF3.0 branch:

Checking in toolkit/components/passwordmgr/src/storage-Legacy.js;
  new revision: 1.29; previous revision: 1.28
Checking in toolkit/components/passwordmgr/test/unit/test_storage_legacy_3.js;
  new revision: 1.8; previous revision: 1.7
Checking in toolkit/components/passwordmgr/test/unit/data/signons-454708.txt;
  initial revision: 1.1
(Assignee)

Updated

9 years ago
Keywords: fixed1.9.0.3

Comment 19

9 years ago
Created attachment 340168 [details]
A simple testcase - saving as the default latin-1 will fail. UTF-8 will not

Attached is a signons3.txt file as provided by Firefox 3.0.2
Now the original signons3.txt from a upgraded profile from 2.0 => 3.0.0 => 3.0.1 => 3.0.2 made the file latin-1. All new profiles create it as utf-8. 
If saving the file as latin-1 then firefox 3.0.2 will fail.
If saving the file as utf-8 then firefox 3.0.2 will not fail.

Use it for testing. 

Requested by Tomcat.
(Assignee)

Comment 20

9 years ago
Landed on GECKO190_20080827_RELBRANCH (for 3.0.3).

Checking in toolkit/components/passwordmgr/src/storage-Legacy.js;
  new revision: 1.28.2.1; previous revision: 1.28
Checking in toolkit/components/passwordmgr/test/unit/test_storage_legacy_3.js;
  new revision: 1.7.2.1; previous revision: 1.7
Checking in toolkit/components/passwordmgr/test/unit/data/signons-454708.txt;
  new revision: 1.1.2.1; previous revision: 1.1
Flags: in-testsuite+
Blocks: 456835

Comment 21

9 years ago
I've got into the same problem after update to FF 3.0.2. It is great to see the status as "RESOLVED FIXED", however in what way it can be applied? I really need my saved passwords back!
Thanks.

Comment 22

9 years ago
http://support.mozilla.com/en-US/kb/Cannot+use+or+save+passwords+after+upgrading+Firefox
Flags: blocking1.9.0.4+ → blocking1.9.0.3+
Keywords: fixed1.9.0.3
No longer blocks: 454993
Duplicate of this bug: 454993
Comment on attachment 340035 [details] [diff] [review]
Patch v.2

(fixing flags after a mass flag reset)
Attachment #340035 - Flags: approval1.9.0.4+ → approval1.9.0.3+

Comment 25

9 years ago
(In reply to comment #19)
> Now the original signons3.txt from a upgraded profile from 2.0 => 3.0.0 =>
> 3.0.1 => 3.0.2 made the file latin-1. All new profiles create it as utf-8. 
> If saving the file as latin-1 then firefox 3.0.2 will fail.
> If saving the file as utf-8 then firefox 3.0.2 will not fail.

I used my own signons3.txt from version 3.0.1 - opened it as latin-1 (ISO-8859-1) and saved it back as utf-8. After installation of 3.0.2 the passwords were on place, even entries with unreadable characters are working.

Comment 26

9 years ago
P L E A S E  tell all people which are not software developers and who have this problems with passwords too :

1. open notepad.exe
2. look for a file signons3.txt in the folder "documents and settings" 
   subfolder= nameof user\userdata\Mozilla\Firefox\Profiles
   (Dokumente und Einstellungen\name anwender\Anwendungsdaten\Mozilla\Firefox
    \Profiles)
3. open the file signons3.txt in notepad
4. save the file as ! : leave the name and choose the option save as UTF-8
   ANSI is the standard 
5. restart firefox and afterwards you should have your passwords back :)

   für alle verzweifelten.

Comment 27

9 years ago
Tank you, you just saved my life :)

Comment 28

9 years ago
... thought it would be helpful ...
kind regards
Duplicate of this bug: 456983

Updated

9 years ago
Blocks: 454993

Updated

9 years ago
No longer blocks: 454993
Duplicate of this bug: 454993
Verified this with the test case in bug 451155 created in 3.0.1. Uninstalled then installed 3.0.3 build1 on Win XP.  The Password Manager is not blank like it was 3.0.1->3.0.2.  

Marking verified.  note: the password entry created in 3.0.1 is invalid and isn't corrected by upgrade.  It has to be recreated by the user to make a valid entry in PWM. I made a relnote comment in 451155 about it.
Status: RESOLVED → VERIFIED
Keywords: fixed1.9.0.3 → verified1.9.0.3
verified in linux in a similar fashion to Tracy from comment #31:
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.3) Gecko/2008092416 Firefox/3.0.3
Duplicate of this bug: 457066

Comment 34

9 years ago
You can also mark Bug 457178 as a duplicate of this bug...
Duplicate of this bug: 457212

Updated

9 years ago
Duplicate of this bug: 457259

Comment 37

9 years ago
Autoupdated to 3.0.3, passwords still not saved and the list is empty, just like 3.0.2!

Comment 38

9 years ago
Made a downgrade from 3.0.2 to 3.0.1
Then patched to 3.0.2 again and applied 3.0.3

Passwords still won't work.

OS: WinVista x64

Comment 39

9 years ago
(In reply to comment #38)
> Passwords still won't work.
..and for anyone else visiting this bug to add comments, there are reports on irc that in 3.0.3 just adding a new password (by going to any new site with a form for logging in and entering a bogus one and choosing to save it when prompted) then restarting firefox fixes any carry-over problems.

Could you please try that and mention whether doing that works?

Comment 40

9 years ago
Comment #39: It does indeed work and fixes the issue. However, the regular user does not appreciate the non-automatic solution and may not be able to fix it, frustrates and leaves Mozilla for good.

Comment 41

9 years ago
Didn't try.
I fixed it with downgrading to 3.0.1 again and applying 3.0.3 directly without the mentioned UTF-8 fixes above. Thx anyways :)

Comment 42

9 years ago
It worked for me.

Comment 43

9 years ago
Fixed only after applying method from comment #39. Thanks to Mardeg and shame to developers. Nobody should know tricks in order to fix the problem.
(Assignee)

Updated

9 years ago
Depends on: 457358

Comment 44

9 years ago
#43 +1

Comment 45

9 years ago
Fire Fox 3.0.3 is not remembering passwords too ... No accessing to password...

Comment 46

9 years ago
The Patch didn't work for me, too. I fixed it with re-installing FireFox.

Comment 47

9 years ago
the patch works on my computer, old password are back and it propose to save new ones.
thanks
The problem people are seeing in 3.0.3 is most likely bug 457358. Comment 39 is a known workaround for that bug. We're going to be landing the fix for that bug in Firefox 3.0.4. Please refrain from adding comments here or there unless there is new information available.
Whiteboard: Please read comment 48 before adding a new one

Comment 49

9 years ago
The workaround in comment 39 did not work for me.  I am running FF 3.0.3 on Kubuntu Linux 8.04.  After following the instructions in comment 39, I also tried deleting passwords from the password manager and re-establishing them.  This did not work either.

I'm not sure what relevant information I can provide.  Please ask and I'll do my best.
(Assignee)

Comment 50

9 years ago
If you're still seeing problems after adding or removing a login, please file a new bug and attach debugging output per https://wiki.mozilla.org/Firefox:Password_Manager_Debugging

Comment 51

9 years ago
Hi, the Passwort Manager still does not work for me (XPSP2), since the Update to 3.0.3 I am asked for my master password when I go to a password protected Website, but the Passwort is not filled in. Here is the debug output:

PwMgr Storage: Failed to decrypt string: MDoEEPgAAAAAAAAAAAAAAAAAAAEwFAYIKoZIhvcNAwcECCeWezDqxdTnBBCh+2tmsj2ZH4jatEAMdNpH (NS_ERROR_UNEXPECTED)
.. (repeated) ..
PwMgr Storage: Failed to decrypt string: MEIEEPgAAAAAAAAAAAAAAAAAAAEwFAYIKoZIhvcNAwcECB1OY9s/BVzXBBjEOs6YRhvakDV4WeaPXSfl/DHSPnl1BJ8= (NS_ERROR_UNEXPECTED)
.. (repeated) ..
Login Manager: form[0]: found 0 matching logins.
Login Manager: (form ignored -- no password fields.)

Any Idea? When I reinstall 3.0.1 it works fine.

Comment 52

9 years ago
OK, Comment #39 also works for me.

Comment 53

9 years ago
Same as Comment #51 for me.
For a short while my passwords worked fine and now...
PwMgr Storage: Failed to decrypt string: VeryLongStringWhichI'veTruncated (NS_ERROR_UNEXPECTED)

Theres ~10 passwords that work but apart from them... all errors like above

Comment 54

9 years ago
Have tried all solutions listed here.  My firefox 3.0.3 installation simply stopped letting me use passwords.  Rebooting, reinstalling firefox, saving as UTF8.  None of these things worked.

This just happened today, but I recently upgraded to 3.0.3 so maybe it's been going on for a couple days

Dan

Comment 55

9 years ago
I have downgraded to 3.0.1, but passwords still inaccessible.  It is now allowing me to add passwords, but the old ones aren't showing up.  It appears they may be lost forever, even though they are in the signons3.txt in encrypted form.

-Dan
(Assignee)

Comment 56

9 years ago
Dan: Please file a new bug, and attach debugging as described in https://wiki.mozilla.org/Firefox:Password_Manager_Debugging.
(Assignee)

Comment 57

9 years ago
(removing fixed1904 keyword, I think it was accidentally added when things were shuffled around for 3.0.3/3.0.4. This bug is already correctly marked verified1.9.0.3).
Keywords: fixed1.9.0.4
fixed1.9.0.4 was added because this patch was fixed in two places: the relbranch for the firedrilled Firefox 3.0.3 and the main cvs trunk for Firefox 3.0.4. This needs to be reverified for the next release.
Keywords: fixed1.9.0.4
Verified for 1.9.0.4 with Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.0.4pre) Gecko/2008102304 GranParadiso/3.0.4pre.
Keywords: fixed1.9.0.4 → verified1.9.0.4

Comment 60

9 years ago
Verified fixed Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.4pre) Gecko/2008102304 GranParadiso/3.0.4pre

latin-1 passwords (Realm) are being listed in the password manager now without any other "user-fixes".

One strange thing though. When I save a new password, it convert the text in the file to UTF-8. However the file is still being opened as latin-1 by gedit making the UTF-8 chars in the file being displayed as latin-1 by gedit. Line should read "begrænset", but reads it "begrænset".
It is being reed as the correct "begrænset" however.
See the testcase.

Can be ignored?

Comment 61

8 years ago
Happened me on update to 3.5.5 (windows 7)- first the update couldn't be applied, retried to update and update finished but passwords were gone - had to change encoding of signon3.txt to utf-8 -> passwords are back

Updated

7 years ago
You need to log in before you can comment on or make changes to this bug.