Closed Bug 1541496 Opened 7 months ago Closed 26 days ago

Extension sync storage not synced

Categories

(WebExtensions :: Storage, defect, P1)

66 Branch
Desktop
Windows
defect

Tracking

(firefox69 verified, firefox70 verified, firefox71 verified)

VERIFIED FIXED
mozilla70
Tracking Status
firefox69 --- verified
firefox70 --- verified
firefox71 --- verified

People

(Reporter: binami, Assigned: rpl)

References

Details

Attachments

(1 file)

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0

Steps to reproduce:

I have several Extentions that use Firefox sync. Like uBlock Origin or LiveMarks. I setup a Firefox Sync account on multiple computers. Then I let sync run multiple times on all computers and repeated this in the following days and weeks but I always get the same results.

Actual results:

Sync works for bookmarks and passwords, but NOT for addon sync storage. So everything that is on storage-sync.sqlite is NOT synced across my computers. I even inspected this sqlite file with DB Browser and while each individual Firefox installation indeed contains its local addon settings in this storage file, they are all different for each Firefox installation. Sync does NOT WORK! I do not receive any error messages. "about:sync-log" does not contain any recent logs that correspond to this issue. Everything seems to be working but in reality is it not. It is driving me crazy because there is no way to even start tackling this issue with you not able to build a browser that displays basic error messages.

Expected results:

Extention sync storage should be synced of course, but at least your Firefox developers should make it possible to debug. What is the issue, why is this not synced? What did I or what did you wrong? This is very important information that every software should be able to display to the user. Thank you very much!

Summary: Extention sync storage not synced → Extension sync storage not synced
Component: Untriaged → Sync
OS: Unspecified → Windows
Hardware: Unspecified → Desktop

Hi binami, I'm sorry you've been running into this, and your frustration is totally understandable. Extension storage uses a different backend than the rest of Sync, so I'm moving this bug to that component for visibility.

Component: Sync → Storage
Product: Firefox → WebExtensions

I'm curious if our QA can reproduce this.

Flags: needinfo?(kraj)

I can confirm that the settings modifications for an add-on are not synced between computers when using Firefox Sync with Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0. I used ublock Origin and LiveMarks for testing with following results:

uBlock Origin :

  • under settings I checked all checkboxes - NOT synced at all
  • under filter list - Build-in - checked all checkboxes - NOT synced at all
  • under filter list - Ads - checked all checkboxes - NOT synced at all

LiveMarks :

  • I addes some new livemarks - they were correctly synced
  • More Options - changed refresh interval - setting was NOT synced

Normal bookmarks and passwords were correctly synced.

Please let me know if I correctly understood and tested this bug based on scenario from above.

Flags: needinfo?(kraj) → needinfo?(binami)
Status: UNCONFIRMED → NEW
Ever confirmed: true

I took a brief look to this, in particular looking into uBlock storage.sync API usage and why the data which have been stored locally is not getting synced on the cloud storage, and it seems to be because of a "wrong assumption" from ExtensionStorageSync, which is currently going to sync the extension data only if the extension page that uses the storage.sync API methods is still open.

For uBlock the page that is using the storage.sync API methods seems to be its "dashboard.html" extension page (the one opened when pressing the preferences button from about:addons).

On the contrary if the page that uses the storage.sync API methods is the extension background page, then ExtensionStorageSync will always detect the extension as one that needs to be synced (because in Firefox that extension page is currently only destroyed when the extension is shutting down).

In my opinion we should fix the issue by making sure that ExtensionStorageSync does sync existing extension data even if the extension page that used the API has been closed.


Some additional details from the investigation I did locally:

Looking into the "criteria" that ExtensionStorageSync.syncAll is currently using to decide if there is anything to sync,
it seems that it is trying to detect if any existing extension context is currently using the storage.sync API, e.g. see:

ExtensionStorageSync.registerInUse is being called by ExtensionStorageSync.getCollection and ExtensionStorageSync.addOnChangedListener

If the sync happens while the uBlock's "dashboard.html" extension page is closed, in the sync logs there will be the following log:

1555604499651	Sync.Engine.Extension-Storage	DEBUG	Syncing extension settings for []

and so for ExtensionStorageSync there is nothing to sync, because there is no extension pages that called one of the storage.sync API methods still around.

On the contrary, if the user press the Firefox "Sync now" button while uBlock's "dashboard.html" extension page is still open then, based on the sync logs available from "about:sync-log", the uBlock sync data is getting synced, e.g. the following log line appear in the new sync logs:

1555604715169	Sync.Engine.Extension-Storage	DEBUG	Syncing extension settings for ["uBlock0@raymondhill.net"]
1555604715172	Sync.Engine.Extension-Storage	INFO	Need to create keys and/or salts for ["uBlock0@raymondhill.net"]
1555604716777	Sync.Engine.Extension-Storage	INFO	Detected a new UUID ({e9810deb-1614-4355-be6a-94b7f942753d}, was {3b55a047-c9dd-4ed9-95b3-c67f7690e734}). Resetting sync status for everything.
1555604716786	Sync.Engine.Extension-Storage	INFO	Need to create keys and/or salts for ["uBlock0@raymondhill.net"]
1555604717062	Sync.Engine.Extension-Storage	INFO	Need to create keys and/or salts for ["uBlock0@raymondhill.net"]
Flags: needinfo?(binami)
Priority: -- → P1

Thanks for pointing out the existence of this logfile. Indeed, if you enable the preference "services.sync.log.appender.file.logOnSuccess", there are more log files present and they show you what is synced and what not. This is very helpful and allowed me to further investigate this issue.

So what I found out is that it is quite difficult to actually get the things synced you would like to have synced. There are some not very intuitive choices which also changed from version to version. If you want "addon preferences" to be synced, there are multiple choices. You could either assume this is done when selecting "addon" or "preferences". In the past, the "preferences" checkbox was responsible for this. I clearly remeber that this was the case for NoScript and some earlier sync version. Nowadays, what I found out, the "addon" checkbox is the correct one. Ok, this solves one issue. Now I can indeed sync my addon settings. BUT... and here is the catch. Now I am forced to have exactly the same addons installed on every Firefox installation, i.e. also addon installation and enable state are synced. This is clearly not what I want. I clearly prefer the old behavior where addon settings are synced once the "preferences" checkbox is selected. I can give you one very easy to understand example why someone does not want to have the same addons everywhere. Take "HTTPS Everywhere" for example. This addon makes sense on a mobile device such a laptop but is useless on you stationary PC with a protected internet connection.

What you could also do is to exclude the addon enable state from sync. I this this would be a sufficient and easy to implement solution if you don't want to change the rules again. Also describe the individual checkboxes of sync more precisely. I always hate when you have to go through trail & error.

And please, make "services.sync.log.appender.file.logOnSuccess" the default! Maybe limit it to the past few days only. But it is always an Odyssey to find out where relevant information and log files are written to.

Assignee: nobody → lgreco
Status: NEW → ASSIGNED
Pushed by luca.greco@alcacoop.it:
https://hg.mozilla.org/integration/autoland/rev/cf81db9c8887
Ensure storage.sync does sync the data for previously synced extensions without an active extension context. r=glasserc

(In reply to Arthur Iakab [arthur_iakab] from comment #8)

Luca can you please take a look?

Sure, I'm on it (I reproduced the test failure locally and applied the needed changes on the attached patch, I'll re-land the patch once the reviewer had a chance to look to the last additions).

Flags: needinfo?(lgreco)
Pushed by luca.greco@alcacoop.it:
https://hg.mozilla.org/integration/autoland/rev/0f50c2174889
Ensure storage.sync does sync the data for previously synced extensions without an active extension context. r=glasserc
Status: ASSIGNED → RESOLVED
Closed: 3 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla70
Regressions: 1566839

I have checked the fix using the latest Nightly (71.0a1/20190919184237), Beta (70.0b8/20190919164641) and Release (69.0.1/20190917135527) on Windows 10 Pro 64-bit and MacOS High Sierra 10.13.6.

uBlock Origin, LiveMarks, Tree Style Tabs have been used for testing, with the following results:

ublock Origin:

  • selected all checkboxes under the Settings tab – NOT synced at all
  • in the Filter lists tab, selected all filters from Built-in, except “uBlock filters – Privacy” and unchecked all other filters – NOT synced at all
  • set a shortcut for “Enter element picker mode” in the Shortcuts tab – NOT synced

LiveMarks:

  • added several livemarks – synced correctly
  • modified the Refresh Interval – NOT synced
  • modified the Extension Icon to the one in the middle – NOT synced

Tree Style Tabs:

  • changed Theme to Plain Dark – NOT synced
  • selected “Close Descendants” from Context Menu – synced

Considering the above, the fix appears to have failed and will reopen the issue.

Status: RESOLVED → REOPENED
Resolution: FIXED → ---

(In reply to Alex Cornestean from comment #13)

I have checked the fix using the latest Nightly (71.0a1/20190919184237), Beta (70.0b8/20190919164641) and Release (69.0.1/20190917135527) on Windows 10 Pro 64-bit and MacOS High Sierra 10.13.6.

uBlock Origin, LiveMarks, Tree Style Tabs have been used for testing, with the following results:

ublock Origin:

  • selected all checkboxes under the Settings tab – NOT synced at all
  • in the Filter lists tab, selected all filters from Built-in, except “uBlock filters – Privacy” and unchecked all other filters – NOT synced at all
  • set a shortcut for “Enter element picker mode” in the Shortcuts tab – NOT synced

LiveMarks:

  • added several livemarks – synced correctly
  • modified the Refresh Interval – NOT synced
  • modified the Extension Icon to the one in the middle – NOT synced

Tree Style Tabs:

  • changed Theme to Plain Dark – NOT synced
  • selected “Close Descendants” from Context Menu – synced

Considering the above, the fix appears to have failed and will reopen the issue.

Hi Alex,
I looked into the extension sources to see how the storage.sync API is being used and double-check if the behavior described were actually the expected ones:

For uBlock Origin syncing to work, as I also mentioned briefly in comment 4, there is an initial setup needed (the one described in the following uBlock wiki page: https://github.com/gorhill/uBlock/wiki/Cloud-storage) because uBlock only uses the storage.sync API from the "Cloud storage setup page", and so the user needs to setup the uBlock settings sync once and then the settings should be now synced even without explicitly opening that page before clicking Firefox's "Sync Now" button.

LiveMarks seems to only be storing the feeds into storage.sync, and so the behavior described in comment 13 seems to be the expected one.

Tree Style Tabs only stores part of its settings in storage.sync, and style is a local-only one (See https://github.com/piroor/treestyletab/commit/272a87359d34c6f9644c6e7e5cce2e8af46088da), and so the behavior described in comment 13 should be the expected one.

Flags: needinfo?(alexandru.cornestean)

Just to be sure sync storage usage in uBlock Origin is not misunderstood:

the extension page that uses the storage.sync API methods is still open

When importing/exporting from sync storage, the sync storage is always accessed from the background page, not the extension page. The extension merely send a message to the background page to perform the read/write from/to sync storage.

the settings should be now synced even without explicitly opening that page before clicking Firefox's "Sync Now" button

uBlock Origin import/export automatically the sync storage content, the user has to expressly import/export them. This is emphasized in the documentation, the user must click import or export buttons.

Hello,

Retested the issue based on the new provided information on the latest Nightly (71.0a1/20190925215653), Beta (70.0b9/20190923154733) and Release (69.0.1/20190917135527) under Windows 10 Pro 64-bit and MacOS High Sierra 10.13.6.

Retested with uBlock Origin, Tree Style Tabs and HTTPS Everywhere, with the following results, confirming the issue is resolved:

uBlock Origin:

I managed to sync the extensions settings as mentioned in the uBlock documentation. Did this by using 2 machines on which I logged in with the same account. I have enabled the uBlock cloud storage support on both. After changing settings on machine 1 (M1), I pressed the “Export to cloud storage button” in the uBlock dashboard and afterwards the “Sync Now” button of Firefox. Switched to the second machine (M2) and did the same, but reversed – first “Sync Now” and afterwards “Import from cloud storage”. In addition, I managed to export/import from M2 to M1 with success.

Tree Style Tabs:

Attempted once more to sync settings based on the new information provided regarding what settings actually use storage.sync and it was successful.

HTTPS Everywhere:

Unchecked the “Auto-update rulesets” from the General Settings tab and checked “Enable mixed content rulesets” from Advanced Settings. Settings are properly synced across machines. Attempted to reverse sync as well, with success.

Flags: needinfo?(alexandru.cornestean)
Status: REOPENED → RESOLVED
Closed: 3 months ago26 days ago
Resolution: --- → FIXED
Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.