1300378 - Sync bookmarks large list: payload is too large JSON

Status: RESOLVED INCOMPLETE

(Firefox :: Sync, defect)

Description

[Wellington Torrejais da Silva]

Attachment: Captura de Tela (299).png

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:51.0) Gecko/20100101 Firefox/51.0
Build ID: 20160903030202

Steps to reproduce:

Sync bookmarks large list: payload is too large JSON

When POST send multimedia unresponsive and browser hang


Actual results:

Problems with synchronization because Sync try retry infinity.
And always fail. 


Expected results:

- Sync work.
- Split bookmarks to sync in small parts
- Increase JSON limit
- Use separated tread and process to sync data
- Sync POST and errors isolated from video at browser....

Comment 1

[Mark Hammond [:markh] [:mhammond]]

Thanks for the report, and apologies for the problems you are having. It appears your "Unfiled/Other Bookmarks" folder is huge, probably due to the number of bookmarks in that folder. I opened bug 1300451 to increase this size, but would it be possible for you to tell me how many bookmarks you have in this folder?

Comment 2

[Wellington Torrejais da Silva]

(In reply to Mark Hammond [:markh] from comment #1)
> Thanks for the report, and apologies for the problems you are having. It
> appears your "Unfiled/Other Bookmarks" folder is huge, probably due to the
> number of bookmarks in that folder. I opened bug 1300451 to increase this
> size, but would it be possible for you to tell me how many bookmarks you
> have in this folder?

Thanks. I have just 40.000 bookmarks...
In another groups/folders more 30.000.

Comment 3

[Mark Hammond [:markh] [:mhammond]]

Thanks for the response - that's alot of bookmarks! There's not much we can do in the short term, but bug 1300451 should offer some relief later. It's certainly possible that creating some subfolders and organizing your bookmarks so the number of items in a single folder is around ~15000 might also cause this problem to go away for you.

Comment 4

[Richard Newman [:rnewman]]

My recommendation for how to work around this:

* Turn off bookmark syncing by unchecking it in Preferences. This will avoid lots of syncs happening while you're making changes, and reduce the chance of other problems.

* Move bookmarks into folders so that you have no more than about 10,000 bookmarks per folder. The ideal is much less than that, of course! You might organize by date range, or by topic. Hierarchy is fine.

* Run the Places Maintenance add-on to make sure that there's nothing else structurally wrong with your bookmarks database (which will also make things faster). You might also choose to use an add-on like Bookmark Deduplicator to make sure you don't have duplicates.

* When you're on a reliable, fast network, turn on bookmark syncing again, and wait for the sync to finish. If there are still problems at this point, please let us see an error log from about:sync-log.

Comment 5

[Wellington Torrejais da Silva]

(In reply to Richard Newman [:rnewman] from comment #4)
> My recommendation for how to work around this:
> 
> * Turn off bookmark syncing by unchecking it in Preferences. This will avoid
> lots of syncs happening while you're making changes, and reduce the chance
> of other problems.
> 
> * Move bookmarks into folders so that you have no more than about 10,000
> bookmarks per folder. The ideal is much less than that, of course! You might
> organize by date range, or by topic. Hierarchy is fine.
> 
> * Run the Places Maintenance add-on to make sure that there's nothing else
> structurally wrong with your bookmarks database (which will also make things
> faster). You might also choose to use an add-on like Bookmark Deduplicator
> to make sure you don't have duplicates.
> 
> * When you're on a reliable, fast network, turn on bookmark syncing again,
> and wait for the sync to finish. If there are still problems at this point,
> please let us see an error log from about:sync-log.

I don't have duplicates.
I've tried many times but the Library Manager don't work more.
Any operation cause errors. For example:

-----
A script on this page may be busy, or it may have stopped responding. You can stop the script now, open the script in the debugger, or let the script continue.

Script: chrome://browser/content/places/treeView.js:824
---

I'm trying to organize them for years.
:-(

Comment 6

[Mark Hammond [:markh] [:mhammond]]

(In reply to Wellington Torrejais da Silva from comment #5)
> I've tried many times but the Library Manager don't work more.
> Any operation cause errors. For example:
> 
> -----
> A script on this page may be busy, or it may have stopped responding. You
> can stop the script now, open the script in the debugger, or let the script
> continue.
> 
> Script: chrome://browser/content/places/treeView.js:824
> ---
> 
> I'm trying to organize them for years.
> :-(

Mak, do you have any advice how these bookmarks could be organized given the volume?

Comment 7

[Wellington Torrejais da Silva]

Attachment: Library-not-working.zip

About these errors?
Please help me to solve these problem...

Comment 8

[Marco Bonardo [:mak]]

(In reply to Mark Hammond [:markh] from comment #6)
> Mak, do you have any advice how these bookmarks could be organized given the
> volume?

For now it should be done in chunks, something like 1000 at a time should do.
In the past we made micro optimizations for large bookmarks moves, but with the change to async bookmarking API we may have lost some of that. Unfortunately it's nonsense to micro optimize again until we are done with deprecating and removing usage of the old API, cause we'd do the work twice. I hope to be able to set the bookmarking API technical debt payment as a goal for the next quarter, the fact Sync is moving to those should help with that.

The script error is not critical, often it's enough to ask it to continue to complete the work, so that could also be tried.

In case you're completely unable to move bookmarks even in small chunks, let me know and I can take a look at what's the worst offenter.
I suspect bug 1107364 would help a lot also this case.

Comment 9

[Mark Hammond [:markh] [:mhammond]]

Closing this - there's not much we can do in this bug - everything we can do has been split off into other bugs.