Bugzilla

Comment 2

•

7 months ago

Hi Wayne,

Sorry for how frustrating this is. Can you tell me more about what you mean by a "compressed" backup of your bookmarks? Have you zipped up the HTML bookmark file? Or do you mean something else entirely?

Flags: needinfo?(wayne)

Assignee

Comment 3

•

7 months ago

(they could mean jsonlz4 backups Places generates daily)

Reporter

Comment 4

•

7 months ago

Yes, by compressed Bookmark backups, I mean what are apparently called the jsonLz4 backups that Firefox normally makes. I have also tried Importing the uncompressed Bookmark backup files (HTML) and have an issue doing them also! The whole thing has become a nightmare as your regular Support has not resolved the issue, it is taking too long, I need the Bookmarks for research and there is no phone number to call for Support. Sometimes, for problems like I am having, a person needs to talk to someone who is an "expert" with Firefox!

Firefox needs to simply be able to restore both the regular and the HTML Bookmark backups. But preferably in the first place not have any Bookmark glitches so I don't have to try to restore my Bookmarks in the first place. (For some reason many Bookmarks disappeared in a folder.)

Hopefully, my issues with not being able to simply restore my Bookmarks will be resolved soon as it is taking way too long.

Thank you.

Comment 5

•

7 months ago

As I don't have a link to your Support post, I can't see what has been tried. Can you provide a link to that, please?

Can I presume you've tried to import the compressed bookmarks file via the Library window (via the App Menu > Bookmarks > Manage Bookmarks), and then via the "Restore > Choose File" option from the "Import and Backup" menu at the top of that window?

See attached screenshot.

Comment 6

•

7 months ago

Attached image Restore > Choose File option — Details

Reporter

Comment 7

•

7 months ago

Here is the link to the regular Firefox Support website: https://support.mozilla.org/en-US/questions/1441292?utm_campaign=questions-reply&utm_source=notification&utm_medium=email#answer-1640085

Yes, I have tried many times to use the Bookmarks Manager Library to import both the compressed and HTML Bookmarks backups. Only once did importing a HTML backup worked. I also tried the built in default Import option shown after a new profile is made after I tried unsuccessfully to Import Bookmark backups as I recall but that didn't work. I got all my Bookmarks, I presume including the Toolbar Bookmarks. I then proceeded to move each Toolbar Bookmark to the Toolbar and then when filled many more to the folder ad the right end of the Toolbar. Who knows how many, may 50 so it took awhile. I happened to close the only open Tab though so Firefox closed and I did NOT have a backup of the newly set up Firefox! As a result, I could not Restore it and no Restore function has worked since. This is frustrating!

Flags: needinfo?(wayne)

Comment 8

•

7 months ago

Okay, thank you wayne. I've read through the support forum. It looks like you've tried a number of things.

It's clear from those posts and the ones here that you're frustrated, and that you're highly motivated to get this addressed soon. The people on support.mozilla.org have been trying various things with you mainly because they don't have access to your bookmarks backups, so they (like me) are making educated guesses as to what is wrong and offering diagnostic steps to give us more information about what might be happening.

A faster route, if you're willing, is to send me the bookmarks backups. You are obviously under no obligation to send me the bookmarks backup, but if you do, chances are I'll be able to get you an answer more quickly about what's happening, and whether or not we can retrieve your bookmarks.

If that works for you, let me know, and I can email you some secure file transfer options.

Flags: needinfo?(wayne)

Reporter

Comment 9

•

7 months ago

Yes, you are 100% correct that I am frustrated. Firefox Support needs to be a lot better. Sometimes there are strange things that require more expertise or investigation. In the meantime I need to have access to all of my Bookmarks!

I can send you some Bookmarks so please let me know how I can send them. I received another email after this one which includes the following link: https://bugzilla.mozilla.org/userprefs.cgi?tab=email. The link opened a webpage but I did not see anything about security. Is there something to set correctly so I can receive secure files to you?

Flags: needinfo?(wayne)

Comment 10

•

7 months ago

Hi wayne, I have sent you email to the address associated with your Bugzilla account with secure upload instructions.

Reporter

Comment 11

•

7 months ago

I could not view the attachments in your second message or any instructions and when I clicked on the link which was included in the first email today which is: https://forms.gle/E1ScxjpMCJDRzpVRA, I get a webpage that says: "You Need Permission, This form can only be viewed by users in the owner's organization. Try contacting the owner of the form if you think this is a mistake. Learn More." So this won't work.

As for using PairDrop, the issue is being online at the same time. I do not know when you are available. I can be available this afternoon. It would be very helpful to have a phone conversation as this back and forth electronic communication is delaying the resolution.

Comment 12

•

7 months ago

Hi Wayne.

The attachments are my OpenGPG signature for the message, which you can ignore. I've sent another email with more instructions.

Reporter

Comment 13

•

7 months ago

After clicking on the link in the first email, I was able to upload my latest good Bookmarks compressed backup without a problem and no need for instructions. I also need to restore my other items, such as password, cookies, extensions, and etc. Hopefully, this can be resolved very soon.

Comment 14

•

7 months ago

Okay, thank you Wayne. I have received your bookmarks jsonlz4 file. I'm examining it.

Comment 15

•

7 months ago

(Hey Wayne, this next comment is for the other engineers working in this space.)

The problem appears to be here: https://searchfox.org/mozilla-central/rev/ff6e63804e6f7b548a26338708663093ce16be11/toolkit/components/places/Bookmarks.sys.mjs#586-588

Wayne has a large number of bookmarks in some of his folders. When we call into getBookmarkDetailMap, we run this SQL query: https://searchfox.org/mozilla-central/rev/ff6e63804e6f7b548a26338708663093ce16be11/toolkit/components/places/Bookmarks.sys.mjs#3334-3356

and that one throws an exception from SQLite: "too many SQL variables". When we enter that routine, we're passing just under 80000 GUIDs to getBookmarkDetailMap, and I think that's causing SQLite to explode when it tries to bind that many GUIDs to the SQL query.

Summary: Cannot import compressed or HTML Bookmarks → Cannot import compressed or HTML Bookmarks for very large sets of bookmarks

Updated

•

7 months ago

Component: Migration → Places

Product: Firefox → Toolkit

Comment 16

•

7 months ago

Hey mak,

Any suggestions on how you'd like to proceed? I was able to import the bookmarks by just returning the insertInfos here, and skipping all of the rest. Presumably, we'll want to break up getting the itemDetailMaps into reasonable sized chunks so that SQLite doesn't fall over. How did you want to proceed?

And can you think of any steps for me to walk Wayne through to get him up and running that don't require waiting for us to ship a fix for this?

Flags: needinfo?(mak)

Assignee

Comment 17

•

7 months ago

(In reply to Mike Conley (:mconley) (:⚙️) from comment #16)

Hey mak,

Any suggestions on how you'd like to proceed?

We must fix the bug. While long term we should remove getBookmarkDetailMap, and implement array binding, the short term fix is to use lazy.PlacesUtils.chunkArray(aGuids, db.variableLimit).

I'm a bit surprised we go over the limit for bookmarks, as it's 32766, though I know very few users have more bookmarks, and that's usually caused by add-ons creating bookmarks or tags infinitely. It sounds like this is one of those cases with 80k bookmarks, unless there's many duplicate guids for some reason? How many entries are in the moz_bookmarks table at the end of the import?

Since wayne created a new profile to import these bookmarks, you could import them locally into a new profile (with a patch) and then pass over places.sqlite to them.
Then we should try to push a patch to 124, at least, so in the worst case they could use Beta to import.

Flags: needinfo?(mak) → needinfo?(mconley)

Comment 18

•

7 months ago

Thanks, Marco. Let's see if we can get Wayne up and running and then we'll figure out the right fix for the long-term.

Hey Wayne, we think we have a plan. The plan goes like this:

We revert your Firefox Refresh via the "Old Firefox Data" that should (hopefully) still be on your Desktop. You still won't have the bookmarks that you wanted from the backup, but you should get your history, logins, addons and preferences back.
I'll need you to send me your places.sqlite database from your profile folder using the same technique that you used to send me the bookmarks backup. Keep in mind that this will send me your navigation history as well.
I'll use a specially prepared local build of Firefox to import from your bookmarks backup into the places.sqlite database that you send me.
I'll send you back the places.sqlite database, which you can put inside of your profile directory

And that should get you into a workable state.

The only catch is step 2, where you'll need to send me your places.sqlite database, which includes your browsing history (but no passwords, payment methods, etc). If you're not comfortable doing that, you might have to wait until we can ship a fix to let you do the import yourself.

If you're ready to proceed with that, then for Step 1, please follow the instructions here to move your "Old Firefox Data" into the profile that you refreshed into: https://support.mozilla.org/en-US/kb/recovering-important-data-from-an-old-profile#w_copying-files-between-profile-folders

Flags: needinfo?(mconley) → needinfo?(wayne)

Assignee

Updated

•

7 months ago

Assignee: nobody → mak

Severity: -- → S2

Status: UNCONFIRMED → NEW

Ever confirmed: true

Priority: -- → P1

Assignee

Comment 19

•

7 months ago

Attached file Bug 1884308 - Chunk bookmark GUIDs when retrieving details to avoid going over Sqlite variables limit. r=mconley — Details

Reporter

Comment 20

•

7 months ago

Hopefully, this will be resolved very soon! The one only time I was miraculously able to import the Bookmarks backup as a HTML file, I found a folder that could be deleted that had many duplicated Bookmarks in it and deleted it. (It was a folder titled "From Internet Explorer" and if I'm not mistaken, it was for some reason in my Bookmarks Toolbar Bookmarks. If this was deleted, it would make the Bookmark file size smaller.

Flags: needinfo?(wayne)

Reporter

Comment 21

•

7 months ago

For some reason, the screen was refreshed before I finished my previous response and I cannot edit it and include a closing parentheses. I wanted to mention as reported before that the reason I was not able to continue to use the imported Bookmarks as just mentioned above is that I managed to close the only open Tab so Firefox closed and I hadn't backed up the Bookmarks.

Thank you for working on this.

Comment 22

•

7 months ago

Alright Wayne, I deleted the "From Internet Explorer" folder and the size did indeed go down. Can you email me with the Google Account that you uploaded your JSON file with? That way I can securely share the new backup file with you without the "From Internet Explorer" folder.

Flags: needinfo?(wayne)

Reporter

Comment 23

•

7 months ago

I need to mention in response to your message #18 do have some functionality as a result of somehow using a previous Bookmarks backup but I have no idea how I got it. (I was waylaid as a result of working with a Microsoft Support Engineer concerning another issue so now I can respond again.) I will need the passwords, cookies, and Toolbar Bookmarks restored.

I found the link you sent earlier and clicked on it and here is the URL for it: https://docs.google.com/forms/d/e/1FAIpQLSchypJWYLfV8Z4wFDnxHvStbN4k452r0-eBQrZQTCHaFcXorQ/viewform Is this what you are wanting but the only option on the webpage is to Upload a file. Will there be an option to download a file if you upload a file there? I am not familiar with using this. How do I get the file you upload? Just copy it? Perhaps you can explain if not easily done.

I uploaded the places.sqlite from my Old Firefox Data files on my Desktop.

Thanks

Flags: needinfo?(wayne)

Comment 24

•

7 months ago

Hi Wayne,

I've gone ahead and shared a new bookmarks JSON file for you that has the "From Internet Explorer" folder removed. It's significantly smaller, and should import correctly in your copy of Firefox. Let me know when you receive it and if it works properly.

Updated

•

7 months ago

Flags: needinfo?(wayne)

Pulsebot

Comment 25

•

7 months ago

Pushed by mak77@bonardo.net: https://hg.mozilla.org/integration/autoland/rev/3f338b935edb Chunk bookmark GUIDs when retrieving details to avoid going over Sqlite variables limit. r=mconley,places-reviewers,Standard8

Assignee

Updated

•

7 months ago

status-firefox124: --- → affected

status-firefox125: --- → affected

status-firefox-esr115: --- → unaffected

Keywords: regression

Regressed by: 1829579

Assignee

Comment 26

•

7 months ago

Attached file Bug 1884308 - Chunk bookmark GUIDs when retrieving details to avoid going over Sqlite variables limit. r=mconley — Details

Original Revision: https://phabricator.services.mozilla.com/D204660

Phabricator Automation

Updated

•

7 months ago

Attachment #9391474 - Flags: approval-mozilla-release?

Phabricator Automation

Comment 27

•

7 months ago

Uplift Approval Request

Explanation of risk level: We're reusing a code helper that is commonly used in similar cases and it's thus well tested. Most of the changes are necessary just for the unit test.
String changes made/needed: none
Code covered by automated testing: yes
Steps to reproduce for manual QE testing: I think the automated test is sufficient, otherwise QA should generate a profile with 33000 bookmarks and try to backup and restore it.
Risk associated with taking this patch: low
User impact if declined: Users with more than 32766 bookmarks would be unable to restore a bookmarks backup.
Is Android affected?: no
Needs manual QE test: no
Fix verified in Nightly: yes

Assignee

Comment 28

•

7 months ago

This affects a small subset of users (< 0,1%, we don't have more precise numbers) but it's a serious dataloss bug for them.

Flags: qe-verify+

Reporter

Comment 29

•

7 months ago

Mike, How can I get the Bookmarks file? I can upload files to you but I see no way to download a file. I asked about this earlier but details about obtaining the file wasn't explained. Also, what about the places.sqlite file I uploaded?

Hopefully, my issues can be resolved soon.

Flags: needinfo?(wayne)

Comment 30

•

7 months ago

Hi Wayne,

I shared the file with your GMail account, which should have received an email about it. Just in case, I've also just sent an email to the address we've already been using to communicate with the link.

Flags: needinfo?(wayne)

Reporter

Comment 31

•

7 months ago

Mike,

I can see the text but what is the best way to get the Bookmarks? There is a Download icon on the right. I presume it would be what I should use. I want to do things right so I don't have more problems so I am hesitant at doing some things!

Also, nothing was mentioned about the places.sqlite file yet I uploaded. Again, what about my passwords, cookies, and extensions?

Hopefully, large Bookmark files can be made to work correctly with Firefox.

Thanks

Flags: needinfo?(wayne)

Comment 32

•

7 months ago

Yes, download the file using the icon in the top right. Let's just ensure that you can import the bookmarks from that file, and then we'll work on recovering your old profile data and making it your default profile data again.

Flags: needinfo?(wayne)

Reporter

Comment 33

•

7 months ago

Okay, the Bookmarks backup file is now in my Profiles folder in Firefox. Nothing more has still has been mentioned about how I can get my passwords, cookies, and extensions back yet. Without them, I won't really be able to use Firefox as I need to.

Flags: needinfo?(wayne)

Comment 34

•

7 months ago

Alright. The next step is to restore your profile from before the refresh. This is the profile stored in the "Old Firefox Data" folder on your desktop. The instructions for doing this are linked to at the end of comment 18.

Flags: needinfo?(wayne)

Natalia Csoregi [:nataliaCs]

Reporter

Comment 35

•

7 months ago

The instructions you mentioned I did not notice address importing a Bookmarks backup. I went ahead and imported the Bookmarks file you sent into my current Profile using the Bookmarks Manager and I got what appears all of my Bookmark toolbar items which is nice but NO Bookmarks (most of them) were Imported that appear in the Bookmarks Sidebar---anywhere!

I need to get all of my Bookmarks back soon! How can I? I need more exact instructions. It should not be so complicated to restore or Import Bookmarks.

Flags: needinfo?(wayne)

Comment 36

•

7 months ago

bugherder

https://hg.mozilla.org/mozilla-central/rev/3f338b935edb

Status: NEW → RESOLVED

Closed: 7 months ago

status-firefox125: affected → fixed

Resolution: --- → FIXED

Target Milestone: --- → 125 Branch

Ryan VanderMeulen [:RyanVM]

Updated

•

7 months ago

status-firefox123: --- → wontfix

Reporter

Comment 37

•

7 months ago

When will Mozilla Support fix the Bookmarks issue I have reported nine days ago?

Flags: needinfo?(mak)

Reporter

Comment 38

•

7 months ago

The first box below this window was mistakenly selected so the case was closed but then I tried to Reopen it but was unsuccessful for some reason. I will click on the Reopen button again and hopefully it will Reopen the case.

I just discovered that my request to delete all my Bookmarks from Internet Explorer in my Bookmarks Toolbar was not done! They are still there. Instead, it appears that all my Bookmarks were deleted from Internet Explorer in one of the other main Bookmarks folders. This is apparently why most of my Bookmarks were NOT imported. See Comment #20 and #22.

I first reported this issue 10 days ago and Mozilla Support still has not resolved it!

Status: RESOLVED → REOPENED

Resolution: FIXED → ---

Assignee

Comment 39

•

7 months ago

•

Edited

Please don't touch the bug flags, as we use them to track the work.

The bug will be fixed in Firefox 125 Beta (available this week), and a few weeks later in Firefox Release 124.1.

Status: REOPENED → RESOLVED

Closed: 7 months ago → 7 months ago

Flags: needinfo?(mak)

Resolution: --- → FIXED

Comment 40

•

7 months ago

Hi Wayne,

We've fixed the bug that was preventing import of the large bookmarks backup file. That fix will ride out to the release channel of Firefox soon. In the meantime, we've closed the bug as RESOLVED because there's no remaining engineering work that we've identified from the bug report.

That's not to say that your particular issue is resolved - it's clearly not. But we believe we've fixed the underlying Firefox bug.

I first reported this issue 10 days ago and Mozilla Support still has not resolved it!

As a reminder, most if not all of the people on the Mozilla Support boards are volunteers. They volunteer their time and expertise to help users with difficulties that they're having with Firefox.

Similarly, I'm going to volunteer my time to help you with the remainder of your problem. However, I'm going to do so over email. I will respond to your comment 38 in a follow-up email, and we can continue our discussion there outside of Bugzilla (where we track our engineering work).

Reporter

Comment 41

•

7 months ago

I understand the bug was fixed that prevented importing the large Bookmark file which is good but how much longer will it take to get my current problem resolved being most of my Bookmarks cannot be imported? You can of course contact me by email since no other person has resolved the issue. Again, the very last issue occurred as a result of the wrong "From Internet Explorer" folder being deleted (not by me). I wrote basically that it can be deleted from the Bookmarks Toolbar Bookmarks since somehow a duplicate copy was in there. Instead, apparently it was deleted from the main Bookmarks folder. This is surely why all what I call my main Bookmarks were missing, upon importing the Bookmarks file I was sent.

I really need my all my Bookmarks soon! As you know, volunteers, so far, have not been able to resolve my Bookmarks issue and the case was opened 11 days ago. I need to do research and not having most of my Bookmarks and or not being able to save them is a VERY important issue. Your support for Firefox needs to be able to address issues promptly since if people loose very important functionality, they will tend to move to another browser. Bookmarks are a more important feature than some people may think they are. I even donated to Mozilla!