1484162 - [Remote Settings] The Sync Functionality no longer works if a Review is approved where a previously created Record is deleted

Status: VERIFIED FIXED

(Cloud Services :: Server: Remote Settings, defect)

Description

[Rares Doghi, Desktop QA]

[Affected versions]: 
Nightly 63.0a1

[Affected platforms]:
Platforms: Windows 10 x 64, Windows 10 x 86, Mac OS X 10.12

[Steps to reproduce]:
1. Reach About:Debugging and load the Temporary addon.
2. Reach The About:remotesettings-pi and click Sync - Data is displayed.
3. Open a new tab Reach https://settings-writer.stage.mozaws.net/v1/admin/ 
4. Add 2 New Records and Request and Approve the Review.
5. Reach The About:remotesettings-pi and click Sync - Data with the 2 extra records is displayed.
6. Reach https://settings-writer.stage.mozaws.net/v1/admin/ and Delete the 2 Extra files that were previously created And Request a Review. 
7. Add a New Record and request another Review.
8. Approve the Request with a different account.
9. Reach The About:remotesettings-pi and click the Reset and then Sync.

Expected Results:
The files should be displayed without issues.

Actual Results:
The Data is no longer displayed when clicking Sync, Reset or Refresh.
The following errors are displayed in the Browser Console:

DEPRECATION WARNING: Search service falling back to synchronous initialization. This is generally the consequence of an add-on using a deprecated search service API. You may find more details about this deprecation at: https://developer.mozilla.org/en-US/docs/XPCOM_Interface_Reference/nsIBrowserSearchService#async_warning jar:file:///C:/Users/rares.doghi/Desktop/Firefoxes/ff62_0a1_9_Jun/firefox/omni.ja!/components/nsSearchService.js 2604 SRCH_SVC__ensureInitialized jar:file:///C:/Users/rares.doghi/Desktop/Firefoxes/ff62_0a1_9_Jun/firefox/omni.ja!/components/nsSearchService.js 4016 get currentEngine resource://activity-stream/lib/TopSitesFeed.jsm 55 getShortURLForCurrentSearch resource://activity-stream/lib/TopSitesFeed.jsm 79 init resource://activity-stream/lib/TopSitesFeed.jsm 645 onAction resource://activity-stream/lib/Store.jsm 51 _middleware/</< resource://activity-stream/lib/Store.jsm 29 Store/this[method] resource://activity-stream/lib/Store.jsm 140 init

NetworkError when attempting to fetch resource.
IndexedDB UnknownErr: ActorsParent.cpp:13788
Error: Invalid content/signature

Comment 1

[Mathieu Leplatre [:leplatrem]]

Thanks for reporting! 

If you can reproduce then it could probably be an error on the server side. We'll investigate!

Comment 2

[Mathieu Leplatre [:leplatrem]]

Rares, do you know if it's something that used to work (for example at the time of Bug 1473239) and no longer works?

Comment 3

[Rares Doghi, Desktop QA]

Hi Mathieu, I am unsure if it used to work that way since there's been a few changes made to Admin as well, I can't really say. It's possible I wasn't trying this exact scenario, the steps are not that user friendly.

Comment 4

[Mathieu Leplatre [:leplatrem]]

Cristian, 

Could you please make sure we reproduce this? 

I'm trying to work on it but can't reproduce locally. Maybe it comes from the STAGE architecture that goes through a CDN etc.  And I don't want to touch your collection.

- Could you please tell me if the "invalid signature" error also appears if we don't create a new record (step 7)?
- Could you please tell me if the signature error persists in time (e.g. leave the server in the same state and retry later)?

In the case you can reproduce, could we please setup a meeting and doing over Vidyo? That maybe more efficient.

Thanks a lot :)

Comment 5

[Cristian Baica [:cbaica], Release Desktop QA]

Hi Mathieu,

As we have discussed, I was able to reproduce this on windows, as well as ubuntu.(In reply to Mathieu Leplatre from comment #4)

> - Could you please tell me if the "invalid signature" error also appears if
> we don't create a new record (step 7)?

Indeed, the issue occurs even without step 7. So after deleting the 2 added records and having the delete change approved, then going directly to step 9.

> - Could you please tell me if the signature error persists in time (e.g.
> leave the server in the same state and retry later)?

I have left the server in the same state for 1h 20m. Clicking the sync button again successfully synced the data on the server.

Comment 6

[Mathieu Leplatre [:leplatrem]]

I can confirm that this comes from a CDN cache problem. I could see a TooManyInvalidation error on the server.

There is a probability that this problem occurred because each step of the test described above was executed very quickly and generated many invalidation requests to the CDN in a very short time.

This means that the same problem could occur in PROD if many users review/approve collections in a short period of time. 

We believe that the easiest way to solve this would be to get rid of cache invalidation and cache busting instead (see Bug 1497159).

Comment 7

[Mathieu Leplatre [:leplatrem]]

Cristian could you try again? Can we mark it as resolved?
Thanks!

Comment 8

[Cristian Baica [:cbaica], Release Desktop QA]

Tested on the latest beta available (Fx64b3).
No error is received when performing the sync action, after the 2 created records are deleted or after deleting them AND uploading another 2 records.
Closing the ticket as verified fixed.