Closed Bug 454708 Opened 16 years ago Closed 16 years ago

storage-Legacy can throw when calling ConvertToUnicode

Categories

(Toolkit :: Password Manager, defect)

1.9.0 Branch
defect
Not set
major

Tracking

()

VERIFIED FIXED
mozilla1.9.1b1

People

(Reporter: haddit, Assigned: Dolske)

References

Details

(Keywords: regression, verified1.9.0.3, verified1.9.0.4, Whiteboard: Please read comment 48 before adding a new one)

Attachments

(3 files, 3 obsolete files)

User-Agent:       Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.4; en-US; rv:1.9.0.1) Gecko/2008070206 Firefox/3.0.1
Build Identifier: Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.4; en-US; rv:1.9.0.1) Gecko/2008070206 Firefox/3.0.1

Ever since upgrading, the new Firefox is not remembering passwords or asking if I want to. My preferences are set to REMEMBER PASSWORDS FOR SITES and there are no exceptions in the EXCEPTIONS box.

Reproducible: Always

Steps to Reproduce:
1.I type user name and password for any site, paypal, wachovia, etc. 
2. It takes me to the logged in site without asking if I want the password remembered.
3.
Actual Results:  
It happens at every site that requires a password.
Version: unspecified → 3.0 Branch
Component: Preferences → Password Manager
Product: Firefox → Toolkit
QA Contact: preferences → password.manager
Version: 3.0 Branch → 1.9.0 Branch
Build identifier : Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.2) Gecko/2008090514 Firefox/3.0.2

I'm also having this issue since upgrading to the newest version. signons3.txt is still there, but it seems no passwords are actually read from this file.
The passwords manager display stays blank, although the file it's supposed to read from...isn't blank.
(In reply to comment #1)
> Build identifier : Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.2)
> Gecko/2008090514 Firefox/3.0.2
> 
> I'm also having this issue since upgrading to the newest version. signons3.txt
> is still there, but it seems no passwords are actually read from this file.
> The passwords manager display stays blank, although the file it's supposed to
> read from...isn't blank.
In that case I'd say: make a screenshot of your passwords while it's still possible.
Please attach some debug output per https://wiki.mozilla.org/Firefox:Password_Manager_Debugging.
Debugging shows the following :

Login Manager: Initialization of storage component failed: [Exception... "Component returned failure code: 0x8000ffff (NS_ERROR_UNEXPECTED) [nsIScriptableUnicodeConverter.ConvertToUnicode]"  nsresult: "0x8000ffff (NS_ERROR_UNEXPECTED)"  location: "JS frame :: file:///C:/Program%20Files/Mozilla%20Firefox%203/components/storage-Legacy.js :: anonymous :: line 853"  data: no]

That line 853 reads :
line.value = this._utfConverter.ConvertToUnicode(line.value);
Bah. This is probably a regression from bug 451155... The bug fixed buy 451155 could result in invalid UTF8 being written to the file, so now that we try to convert it from UTF8, the bogus data will cause it to fail. :(

We can't recover the login (due to lossy UTF16->ascii conversion), but we should at least try/catch the UTF8 conversion and delete the login.

I'd be curious to see which line in your profile's signon3.txt is causing the problem. Could you email me this file? (The passwords are encrypted, but you can edit out encrypted-looking lines if you'd like.)
Assignee: nobody → dolske
Blocks: 451155
OS: Mac OS X → All
Hardware: Macintosh → All
Target Milestone: --- → mozilla1.9.1b1
Status: UNCONFIRMED → NEW
Ever confirmed: true
Attached file a sample of corrupted signons3.txt (obsolete) —
(In reply to comment #5)
> I'd be curious to see which line in your profile's signon3.txt is causing the
> problem. 

The sample is here.
I made it on https://うっかり.jp/members/
Attached file key3.db for the sample signons3.txt (obsolete) —
key3.db is here.
Not to worry because real password is not included.
Attached patch Patch v.1 (obsolete) — Splinter Review
The core fix here is to wrap the charset conversion in a try/catch. The tricky bit is deciding what to do with a corrupted entry...

One option is to do nothing, and keep it around. It won't get used anywhere, but it's a safe and minimal change.

The other option is to flag corrupted entries, and drop them when loading the file. This makes for tidy housekeeping, but adds complexity, risk, and more to test. The particular risk here is a "drop this entry" code path adds the possibility of having a bug that results in more dataloss (eg, forgetting to reset the "drop this entry" flag, and dropping all following entries).

I've implemented the second option here (untested!), but I'm sort of leaning towards just doing the first option instead.

Gavin, thoughts?
option 1 sounds best - cost of ignoring them is pretty low I think.
If we keep it around (option 1), will that effect importing with the switch to mozStorage? Or will we just import a corrupted entry?
Attached patch Patch v.2Splinter Review
Updated to implement option 1, with tests.

(In reply to comment #10)
> If we keep it around (option 1), will that effect importing with the switch to
> mozStorage? Or will we just import a corrupted entry?

No. It's a valid string once it's in memory, the problem only happens when we try converting the on-disk value from UTF8. The mozStorage code is unaffected by this.
Attachment #339174 - Attachment is obsolete: true
Attachment #339175 - Attachment is obsolete: true
Attachment #339502 - Attachment is obsolete: true
Attachment #340035 - Flags: review?(gavin.sharp)
Updating summary.

Gavin noted on IRC that this bug can't be the same problem that the original reporter (Ed) was having. The patch that caused this regression didn't land until Firefox 3.0.2. This bug was filed against FF3.0.1. So, although the symptoms are the same, they must be separate problems. *sigh*

Since this bug has already mutated, I've filed bug 456661 to track the original problem from comment 0.
Summary: Firefox 3.0.1 not remembering passwords → storage-Legacy can throw when calling ConvertToUnicode
Attachment #340035 - Flags: review?(gavin.sharp) → review+
Flags: blocking1.9.0.3+
Keywords: regression
Comment on attachment 340035 [details] [diff] [review]
Patch v.2

a=beltzner, drop it like it's hot
Attachment #340035 - Flags: approval1.9.0.3+
Uhm, I suppose we should have this on mozilla-central before crash landing it on the cvs trunk.
Pushed changeset 5ba271f60a30.

(Branch landing coming up shortly)
Status: NEW → RESOLVED
Closed: 16 years ago
Resolution: --- → FIXED
sorry for breaking in, i'm complete novice here and haven't touch the Firefox code, but I had same problem, landed here and saw the proposed options to fix.
I think that the options to drop out entries would cause a lot of problems to international users. 
What I mean - I have a lot of entries in signons3.txt like:
http://www.navet.government.bg (phpMyAdmin работи на localhost)

so dropping out these entries will disable almost all my phpMyAdmin accounts. This would be fully discouraging for me to migrate to 3.0.2

So may be you could consider also following options:
- as Firefox works fine with such entries in 3.0.1 and not in 3.0.2 maybe there is "easy" way to make a migration tool (best automatic on update, or at least an external tool)
- to remove only the invalid characters if they are not inside the URL. Maybe this would not break the whole entry and it will be still usable. This would at least reduce the damages.
Blocks: 454993
(In reply to comment #16)
> sorry for breaking in, i'm complete novice here and haven't touch the Firefox
> code, but I had same problem, landed here and saw the proposed options to fix.
> I think that the options to drop out entries would cause a lot of problems to
> international users.

Nope.

Any existing logins that worked in 3.0.1 will continue to work, and any logins you could save with 3.0.1 will continue to be savable. We're not breaking any functionality.

The logins we were talking about "dropping" in this bug were already corrupted, and wouldn't be working. Bug 451155 fixed this problem -- you can now save logins on IDN sites and have them work when you revisit the site. Before that fix, the logins were corrupted when saved. This bug is about what we do when reading in those already-corrupted logins.

> - to remove only the invalid characters if they are not inside the URL. Maybe
> this would not break the whole entry and it will be still usable.

That would break the logins even more, and they still wouldn't work.
Checked in on FF3.0 branch:

Checking in toolkit/components/passwordmgr/src/storage-Legacy.js;
  new revision: 1.29; previous revision: 1.28
Checking in toolkit/components/passwordmgr/test/unit/test_storage_legacy_3.js;
  new revision: 1.8; previous revision: 1.7
Checking in toolkit/components/passwordmgr/test/unit/data/signons-454708.txt;
  initial revision: 1.1
Keywords: fixed1.9.0.3
Attached is a signons3.txt file as provided by Firefox 3.0.2
Now the original signons3.txt from a upgraded profile from 2.0 => 3.0.0 => 3.0.1 => 3.0.2 made the file latin-1. All new profiles create it as utf-8. 
If saving the file as latin-1 then firefox 3.0.2 will fail.
If saving the file as utf-8 then firefox 3.0.2 will not fail.

Use it for testing. 

Requested by Tomcat.
Landed on GECKO190_20080827_RELBRANCH (for 3.0.3).

Checking in toolkit/components/passwordmgr/src/storage-Legacy.js;
  new revision: 1.28.2.1; previous revision: 1.28
Checking in toolkit/components/passwordmgr/test/unit/test_storage_legacy_3.js;
  new revision: 1.7.2.1; previous revision: 1.7
Checking in toolkit/components/passwordmgr/test/unit/data/signons-454708.txt;
  new revision: 1.1.2.1; previous revision: 1.1
I've got into the same problem after update to FF 3.0.2. It is great to see the status as "RESOLVED FIXED", however in what way it can be applied? I really need my saved passwords back!
Thanks.
Flags: blocking1.9.0.4+ → blocking1.9.0.3+
Keywords: fixed1.9.0.3
No longer blocks: 454993
Comment on attachment 340035 [details] [diff] [review]
Patch v.2

(fixing flags after a mass flag reset)
Attachment #340035 - Flags: approval1.9.0.4+ → approval1.9.0.3+
(In reply to comment #19)
> Now the original signons3.txt from a upgraded profile from 2.0 => 3.0.0 =>
> 3.0.1 => 3.0.2 made the file latin-1. All new profiles create it as utf-8. 
> If saving the file as latin-1 then firefox 3.0.2 will fail.
> If saving the file as utf-8 then firefox 3.0.2 will not fail.

I used my own signons3.txt from version 3.0.1 - opened it as latin-1 (ISO-8859-1) and saved it back as utf-8. After installation of 3.0.2 the passwords were on place, even entries with unreadable characters are working.
P L E A S E  tell all people which are not software developers and who have this problems with passwords too :

1. open notepad.exe
2. look for a file signons3.txt in the folder "documents and settings" 
   subfolder= nameof user\userdata\Mozilla\Firefox\Profiles
   (Dokumente und Einstellungen\name anwender\Anwendungsdaten\Mozilla\Firefox
    \Profiles)
3. open the file signons3.txt in notepad
4. save the file as ! : leave the name and choose the option save as UTF-8
   ANSI is the standard 
5. restart firefox and afterwards you should have your passwords back :)

   für alle verzweifelten.
Tank you, you just saved my life :)
... thought it would be helpful ...
kind regards
Blocks: 454993
No longer blocks: 454993
Verified this with the test case in bug 451155 created in 3.0.1. Uninstalled then installed 3.0.3 build1 on Win XP.  The Password Manager is not blank like it was 3.0.1->3.0.2.  

Marking verified.  note: the password entry created in 3.0.1 is invalid and isn't corrected by upgrade.  It has to be recreated by the user to make a valid entry in PWM. I made a relnote comment in 451155 about it.
Status: RESOLVED → VERIFIED
verified in linux in a similar fashion to Tracy from comment #31:
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.3) Gecko/2008092416 Firefox/3.0.3
You can also mark Bug 457178 as a duplicate of this bug...
Autoupdated to 3.0.3, passwords still not saved and the list is empty, just like 3.0.2!
Made a downgrade from 3.0.2 to 3.0.1
Then patched to 3.0.2 again and applied 3.0.3

Passwords still won't work.

OS: WinVista x64
(In reply to comment #38)
> Passwords still won't work.
..and for anyone else visiting this bug to add comments, there are reports on irc that in 3.0.3 just adding a new password (by going to any new site with a form for logging in and entering a bogus one and choosing to save it when prompted) then restarting firefox fixes any carry-over problems.

Could you please try that and mention whether doing that works?
Comment #39: It does indeed work and fixes the issue. However, the regular user does not appreciate the non-automatic solution and may not be able to fix it, frustrates and leaves Mozilla for good.
Didn't try.
I fixed it with downgrading to 3.0.1 again and applying 3.0.3 directly without the mentioned UTF-8 fixes above. Thx anyways :)
It worked for me.
Fixed only after applying method from comment #39. Thanks to Mardeg and shame to developers. Nobody should know tricks in order to fix the problem.
Depends on: 457358
#43 +1
Fire Fox 3.0.3 is not remembering passwords too ... No accessing to password...
The Patch didn't work for me, too. I fixed it with re-installing FireFox.
the patch works on my computer, old password are back and it propose to save new ones.
thanks
The problem people are seeing in 3.0.3 is most likely bug 457358. Comment 39 is a known workaround for that bug. We're going to be landing the fix for that bug in Firefox 3.0.4. Please refrain from adding comments here or there unless there is new information available.
Whiteboard: Please read comment 48 before adding a new one
The workaround in comment 39 did not work for me.  I am running FF 3.0.3 on Kubuntu Linux 8.04.  After following the instructions in comment 39, I also tried deleting passwords from the password manager and re-establishing them.  This did not work either.

I'm not sure what relevant information I can provide.  Please ask and I'll do my best.
If you're still seeing problems after adding or removing a login, please file a new bug and attach debugging output per https://wiki.mozilla.org/Firefox:Password_Manager_Debugging
Hi, the Passwort Manager still does not work for me (XPSP2), since the Update to 3.0.3 I am asked for my master password when I go to a password protected Website, but the Passwort is not filled in. Here is the debug output:

PwMgr Storage: Failed to decrypt string: MDoEEPgAAAAAAAAAAAAAAAAAAAEwFAYIKoZIhvcNAwcECCeWezDqxdTnBBCh+2tmsj2ZH4jatEAMdNpH (NS_ERROR_UNEXPECTED)
.. (repeated) ..
PwMgr Storage: Failed to decrypt string: MEIEEPgAAAAAAAAAAAAAAAAAAAEwFAYIKoZIhvcNAwcECB1OY9s/BVzXBBjEOs6YRhvakDV4WeaPXSfl/DHSPnl1BJ8= (NS_ERROR_UNEXPECTED)
.. (repeated) ..
Login Manager: form[0]: found 0 matching logins.
Login Manager: (form ignored -- no password fields.)

Any Idea? When I reinstall 3.0.1 it works fine.
OK, Comment #39 also works for me.
Same as Comment #51 for me.
For a short while my passwords worked fine and now...
PwMgr Storage: Failed to decrypt string: VeryLongStringWhichI'veTruncated (NS_ERROR_UNEXPECTED)

Theres ~10 passwords that work but apart from them... all errors like above
Have tried all solutions listed here.  My firefox 3.0.3 installation simply stopped letting me use passwords.  Rebooting, reinstalling firefox, saving as UTF8.  None of these things worked.

This just happened today, but I recently upgraded to 3.0.3 so maybe it's been going on for a couple days

Dan
I have downgraded to 3.0.1, but passwords still inaccessible.  It is now allowing me to add passwords, but the old ones aren't showing up.  It appears they may be lost forever, even though they are in the signons3.txt in encrypted form.

-Dan
Dan: Please file a new bug, and attach debugging as described in https://wiki.mozilla.org/Firefox:Password_Manager_Debugging.
(removing fixed1904 keyword, I think it was accidentally added when things were shuffled around for 3.0.3/3.0.4. This bug is already correctly marked verified1.9.0.3).
Keywords: fixed1.9.0.4
fixed1.9.0.4 was added because this patch was fixed in two places: the relbranch for the firedrilled Firefox 3.0.3 and the main cvs trunk for Firefox 3.0.4. This needs to be reverified for the next release.
Keywords: fixed1.9.0.4
Verified for 1.9.0.4 with Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.0.4pre) Gecko/2008102304 GranParadiso/3.0.4pre.
Verified fixed Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.4pre) Gecko/2008102304 GranParadiso/3.0.4pre

latin-1 passwords (Realm) are being listed in the password manager now without any other "user-fixes".

One strange thing though. When I save a new password, it convert the text in the file to UTF-8. However the file is still being opened as latin-1 by gedit making the UTF-8 chars in the file being displayed as latin-1 by gedit. Line should read "begrænset", but reads it "begrænset".
It is being reed as the correct "begrænset" however.
See the testcase.

Can be ignored?
Happened me on update to 3.5.5 (windows 7)- first the update couldn't be applied, retried to update and update finished but passwords were gone - had to change encoding of signon3.txt to utf-8 -> passwords are back
You need to log in before you can comment on or make changes to this bug.