849294 - Firefox Sync server appears to be down

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Server: Sync, defect, P1)

Description

[Andrew Miller]

User Agent: Mozilla/5.0 (Windows NT 6.1; rv:19.0) Gecko/20100101 Firefox/19.0
Build ID: 20130215130331

Steps to reproduce:

Opened the Tools menu and selected "Sync Now".


Actual results:

A popup appeared on the bottom of the page saying "Sync encountered an error while syncing: Firefox Sync server maintenance is underway, syncing will resume automatically. Sync will automatically retry this action." It has been displaying this message for the last 24 hours and I believe there may be a server down. cf. http://support.mozilla.org/en-US/questions/924979


Expected results:

It should sync my account using Firefox Sync.

Comment 1

[Toby Elliott [:telliott]]

We are having a couple of issues. Can you attach your log as shown in https://philikon.wordpress.com/2011/06/13/how-to-file-a-good-sync-bug/

Thanks!

Comment 2

[Andrew Miller]

Attachment: Sync bug error log for this issue

Comment 3

[Markus Popp]

I see Firefox Sync server issues quite frequently, actually like every 3rd or 4th time I need to sync. This happens way too frequently and makes Sync unusable.

Getting no error messages at all (no warning that sync didn't work) and it made me lose all bookmarks on mobile, at least until Sync works again so I can finally do this sync.

This is an unacceptable situation. At the very least, a workaround should be implemented, see bug #837229 so there is always a way to get bookmarks back until these Sync problems are solved at the very root (which may well require redoing Sync from scratch, as broken as it appears to be).

Comment 4

[Markus Popp]

Any update on this? Is anybody even looking into this already?

Comment 5

[Sheeri Cabral [:sheeri]]

Having talked with Markus, sync is relied on very heavily to use both Firefox on Android and Firefox on Desktop. Sync is critical to both Desktop and Android, because it makes them work together seamlessly. When it doesn't work seamlessly, like this weekend, it devalues the Firefox brand on both Desktop and Mobile.

Comment 6

[Justin Lazaro [:jlaz] (use needinfo)]

Hello Markus,

That particular db has been experiencing some load issues.  We've migrated users from that host over the weekend, which means that you will be reassigned to a healthier db.  Are you continuing to experience any other issues?  Are you seeing bookmarks disappear from your devices?

Comment 7

[Markus Popp]

Yesterday evening I finally managed to sync successfully, so now I've got everything back. To be honest, I'm not very eager to do further experiments and risk losing them again (until I have to sync due to some bookmark updates).

It seems like bookmarks are updated in the database immediately when the sync process begins, and if there is a problem or even failure, the database is left with what's been synced until that point, leaving an inconsistent state. So what's missing from my point of view is that the sync process is done in a transactional way, so changes in the database are only committed when the entire sync process has ended successfully. In the case of failure, the user would at least still have the previous state (so bookmarks unchanged) rather than potentially losing bookmarks.

I could also imagine that some checksum could be useful. To make the sync process work something like this:

* create checksum of user's bookmarks
* send checksum to server
* start sync in a transactional way
* when finished, compare server checksum to client checksum (make sure bookmarks are exactly what they should be)
* if check is positive, commit transactional sync
* if check is negative, rollback transaction, display error message to user.

Here comes another problem into play: currently the user has no way to reliably verify if a sync was successful. All I get to see is the rotating sync icon and only by how long the sync procedure takes I can estimate whether or not it was probably successful. In many cases there is no error message displayed at all, even if sync fails. So I don't even know if it is safe to sync the changes back to Firefox Mobile because the sync process may seem to have been successful, even if it wasn't.

Also helpful could be a web interface where users can see the bookmarks as they are currently stored on the server. Opera Link has something like this.

Comment 8

[Toby Elliott [:telliott]]

Thanks for the thoughts. Sync treats the client as the source of truth for an item, not the server, which is effectively a transmission channel. The only way the client should be losing data because of an inconsistent sync state is if the client is manually purging and trying to restore from bookmarks after a failed sync. Sync isn't designed to be a backup service, and if a sync has only partially succeeded, syncing the changes back to your mobile device will successfully update those bookmarks that were synced, which seems better than not syncing any of them.

Bookmarks are updated immediately because they can be tracked individually, and, if there's a failed write, we let the client know so that it can try again with the bookmarks that failed (or, if there's a full db problem, we let them know that the whole thing failed). Checksums are nice in theory, and would be a good idea if we were running, say, a daily backup, but having to checksum 2,000+ records every five minutes would require a whole lot of resources. (Interestingly, this would be pretty easy to add for someone running their own server - the code provides hooks to make this straightforward. But doing it for multiple million people would melt down quickly.)

A web interface is something that we've wanted for a long time, but there's a major hurdle - the records are encrypted client-side, so we don't see anything other than random blobs of data. In order to make use of the web interface, you'd need to manually enter your sync key, which is not something that the vast majority of users know about at all. That interface conundrum is the main reason that we haven't managed to make forward progress on this.

I'd encourage you to get involved with the PiCL project, which is exploring some of these very questions: https://wiki.mozilla.org/Identity/AttachedServices

Comment 9

[Markus Popp]

Part of the problem is that the sync from desktop to the server didn't show any error, so left me thinking that it was successful, and when I did the sync on Mobile, the bookmarks were gone (sometimes entirely, sometimes partially as I tried again and again).

And once they were gone, there was no way to get them back as the Sync server was having issues and the damage on Mobile was done already.

In the end it took me 3 days until I could do a successful full sync again, so 3 days without access to my bookmarks in Firefox Mobile. Which from a user experience perspective is extremely bad, and definitely needs to be fixed.

Comment 10

[Toby Elliott [:telliott]]

While I agree that it's bad that you didn't have access to new bookmarks on your phone for 3 days, that should not have impacted any existing bookmarks on your phone. The phone doesn't try to make its state match the current online state, it just asks for any new updates that have been submitted since the last time it checked.

If there was deletion of bookmarks that weren't also deleted on the desktop, then it sounds like there was a deeper problem than the server delay.

Comment 11

[Markus Popp]

https://bugzilla.mozilla.org/show_bug.cgi?id=837229#c12

Comment 12

[Toby Elliott [:telliott]]

Underlying issues should be resolved on this one, and the client issues are being handled elsewhere.

Comment 13

[kjr06e]

Firefox sync servers are down again. I have been trying to sync my bookmarks and tabs since yesterday 04/30/2020. I am just getting a loading icon it wont take to me to the screen to sign into my Firefox account.

Comment 14

[Rachel Tublitz [:rachel]]

kjr06e, to confirm here: it sounds like you are not logged in to your Firefox account? Can you try:

1. Logging in, and trying again?
2. If that doesn't work, assuming you are on a desktop machine can you attach your sync logs (you can find them by entering 'about:sync-log' in the url bar) so we can take a look?

Comment 15

[kjr06e]

file:///C:/Users/tawan/AppData/Roaming/Mozilla/Firefox/Profiles/6bfyje8o.default-1588287685425/weave/logs/error-sync-1588327772318.txt

file:///C:/Users/tawan/AppData/Roaming/Mozilla/Firefox/Profiles/6bfyje8o.default-1588287685425/weave/logs/error-sync-1588338442824.txt

The sync log errors I am getting I cannot log in or sync my Firefox account

Comment 16

[Rachel Tublitz [:rachel]]

kjr06e, can you upload the files instead of pasting the paths?

Those are definitely the right files though!

There's an 'Attach New File' button at the top of this page that you can use to add attachments.