1481348 - Intermittent JavaScript Error: "Polling for changes failed: Server error 404 Not Found: "JSON.parse: unexpected character at line 1 column 1 of the JSON data

Status: RESOLVED FIXED

(Firefox :: Remote Settings Client, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: aciure [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=192309708&repo=mozilla-central

https://queue.taskcluster.net/v1/task/I6eqEZi6TWW6LSF0RLQ5Xw/runs/0/artifacts/public/logs/live_backing.log

8:26:55     INFO - TEST-PASS | browser/components/payments/test/browser/browser_total.js | Check unmodified total currency amount - "$2.00 USD" == "$2.00 USD" - 
18:26:55     INFO - Buffered messages logged at 18:26:55
18:26:55     INFO - Leaving test bound test_modifier_with_no_method_selected
18:26:55     INFO - Entering test bound test_modifier_with_no_method_selected
18:26:55     INFO - adding a basic-card
18:26:55     INFO - Console message: [JavaScript Error: "Polling for changes failed: Server error 404 Not Found: "JSON.parse: unexpected character at line 1 column 1 of the JSON data"." {file: "resource://services-settings/remote-settings.js" line: 717}]
18:26:55     INFO - remoteSettingsFunction/remoteSettings.pollChanges@resource://services-settings/remote-settings.js:717:13
18:26:55     INFO - async*notify@jar:file:///Z:/task_1533576206/build/application/firefox/omni.ja!/components/RemoteSettingsComponents.js:24:5
18:26:55     INFO - TM_notify/<@jar:file:///Z:/task_1533576206/build/application/firefox/omni.ja!/components/nsUpdateTimerManager.js:197:11
18:26:55     INFO - TM_notify@jar:file:///Z:/task_1533576206/build/application/firefox/omni.ja!/components/nsUpdateTimerManager.js:244:7
18:26:55     INFO - 
18:26:55     INFO - Buffered messages finished
18:26:55     INFO - TEST-UNEXPECTED-FAIL | browser/components/payments/test/browser/browser_total.js | [JavaScript Error: "Polling for changes failed: Server error 404 Not Found: "JSON.parse: unexpected character at line 1 column 1 of the JSON data"." {file: "resource://services-settings/remote-settings.js" line: 717}]
18:26:55     INFO - remoteSettingsFunction/remoteSettings.pollChanges@resource://services-settings/remote-settings.js:717:13
18:26:55     INFO - async*notify@jar:file:///Z:/task_1533576206/build/application/firefox/omni.ja!/components/RemoteSettingsComponents.js:24:5
18:26:55     INFO - TM_notify/<@jar:file:///Z:/task_1533576206/build/application/firefox/omni.ja!/components/nsUpdateTimerManager.js:197:11
18:26:55     INFO - TM_notify@jar:file:///Z:/task_1533576206/build/application/firefox/omni.ja!/components/nsUpdateTimerManager.js:244:7
18:26:55     INFO -  - 
18:26:55     INFO - Stack trace:
18:26:55     INFO - chrome://mochitests/content/browser/browser/components/payments/test/browser/head.js:onConsoleMessage:331
18:26:55     INFO - chrome://mochikit/content/tests/SimpleTest/specialpowersAPI.js:observe/<:385
18:26:56     INFO - waitForMessageFromWidget: paymentDialogReady
18:26:56     INFO - GECKO(6148) | ++DOCSHELL 000001E2F6443000 == 6 [pid = 6148] [id = {03178357-fb44-4043-9727-ffdc65613ccf}]
18:26:56     INFO - GECKO(6148) | ++DOMWINDOW == 14 (000001E2F6506400) [pid = 6148] [serial = 146] [outer = 0000000000000000]
18:26:56     INFO - GECKO(6148) | ++DOMWINDOW == 15 (000001E2E819E400) [pid = 6148] [serial = 147] [outer = 000001E2F6506400]
18:26:56     INFO - GECKO(6148) | --DOMWINDOW == 3 (0000025EBE523600) [pid = 7244] [serial = 116] [outer = 0000000000000000] [url = https://example.com/browser/browser/components/payments/test/browser/blank_page.html]
18:26:56     INFO - GECKO(6148) | [Parent 6148, Main Thread] WARNING: NS_ENSURE_TRUE(root) failed: file z:/build/build/src/layout/base/nsDocumentViewer.cpp, line 3544
18:26:56     INFO - GECKO(6148) | ++DOCSHELL 000001BFA566B800 == 2 [pid = 6536] [id = {93d44052-10eb-45b7-b3d2-e397f6ba2c02}]
18:26:56     INFO - GECKO(6148) | ++DOMWINDOW == 13 (000001BFA5627200) [pid = 6536] [serial = 181] [outer = 0000000000000000]
18:26:56     INFO - GECKO(6148) | ++DOMWINDOW == 14 (000001BFA7432C00) [pid = 6536] [serial = 182] [outer = 000001BFA5627200]
18:26:56     INFO - GECKO(6148) | ++DOMWINDOW == 15 (000001BFA7C0C000) [pid = 6536] [serial = 183] [outer = 000001BFA5627200]
18:26:56     INFO - GECKO(6148) | ++DOCSHELL 000001BFA7C72000 == 3 [pid = 6536] [id = {3b323304-a971-4f8a-84e9-47dae5a39039}]
18:26:56     INFO - GECKO(6148) | ++DOMWINDOW == 16 (000001BFA7C4B400) [pid = 6536] [serial = 184] [outer = 0000000000000000]
18:26:56     INFO - GECKO(6148) | ++DOMWINDOW == 17 (000001BFA8A1A000) [pid = 6536] [serial = 185] [outer = 000001BFA7C4B400]
18:26:57     INFO - Got paymentDialogReady from widget

Comment 1

[Intermittent Failures Robot]

3 failures in 2561 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-central: 3

Platform breakdown:
* windows10-64-ccov: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1481348&startday=2018-08-05&endday=2018-08-11&tree=all

Comment 2

[Intermittent Failures Robot]

3 failures in 2579 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-central: 3

Platform breakdown:
* windows10-64-ccov: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1481348&startday=2018-08-06&endday=2018-08-12&tree=all

Comment 4

[Mathieu Leplatre [:leplatrem]]

Attachment: Bug 1481348 - Prevent intermittent failures with fake RemoteSettings server r?glasserc

Comment 5

[Mathieu Leplatre [:leplatrem]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=6d4bae5015e03c4118a5f9f6a94b30299f847eae

Comment 6

[Ethan Glasser-Camp (:glasserc)]

Comment on attachment 9001719 [details]
Bug 1481348 - Prevent intermittent failures with fake RemoteSettings server r?glasserc

Ethan Glasser-Camp (:glasserc) has approved the revision.

Comment 7

[Mathieu Leplatre [:leplatrem]]

Thanks Ethan for your review.

I'm aware that the current patch is not entirely elegant. I'm going to be off for the next 2 weeks. If this issue becomes too annoying, we can land it as it is, and iterate later.

> Can you confirm that we don't expect any users to self-host RemoteSettings? If so, I'm OK with it. If not, I'm a little nervous about the test, because it could mask server misconfiguration.

Indeed, users that self-host remote settings could have some server configuration error masked.

Or for example, when using STAGE, changing the polling endpoint preference to a bad value would silently fail.

Comment 8

[Intermittent Failures Robot]

6 failures in 2672 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 6

Platform breakdown:
* linux64: 5
* linux32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1481348&startday=2018-08-13&endday=2018-08-19&tree=all

Comment 9

[Intermittent Failures Robot]

2 failures in 2568 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-central: 2

Platform breakdown:
* windows10-64-ccov: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1481348&startday=2018-08-20&endday=2018-08-26&tree=all

Comment 10

[Eliza Balazs [:ebalazs_]]

leplatrem: Hi, this is perma failing on the last merge: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=95b4b8e25577093b1e2358ce08cdd8c3ee191cc2&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=usercancel&filter-resultStatus=runnable&filter-searchStr=windows10-64-ccov%20debug%20Mochitests%20with%20e10s%20test-windows10-64-ccov%2Fdebug-mochitest-browser-chrome-e10s-1%20M-e10s(bc1)&selectedJob=196587280

Comment 12

[Intermittent Failures Robot]

22 failures in 538 pushes (0.041 failures/push) were associated with this bug yesterday.

Repository breakdown:
* mozilla-central: 22

Platform breakdown:
* windows10-64-ccov: 22

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1481348&startday=2018-08-30&endday=2018-08-30&tree=all

Comment 13

[Eliza Balazs [:ebalazs_]]

:marco, Hi, I see that Mathieu Leplatre its on PTO, could you find time to have a look at this? 

Its a perma failing windows10-64-ccov debug: 

https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=runnable&filter-searchStr=windows10-64-ccov%20debug%20Mochitests%20with%20e10s%20test-windows10-64-ccov%2Fdebug-mochitest-browser-chrome-e10s-1%20M-e10s(bc1)&selectedJob=196827877

Thanks!

Comment 14

[Intermittent Failures Robot]

46 failures in 2545 pushes (0.018 failures/push) were associated with this bug in the last 7 days.

This is the #36 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. ** 

Repository breakdown:
* mozilla-central: 46

Platform breakdown:
* windows10-64-ccov: 46

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1481348&startday=2018-08-27&endday=2018-09-02&tree=all

Comment 15

[Mathieu Leplatre [:leplatrem]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=7ecc78b06f14036173e2cd298a79debb7bf704c5

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/1cb35a652277
Prevent intermittent failures with fake RemoteSettings server r=glasserc

Comment 17

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/1cb35a652277

Comment 18

[Intermittent Failures Robot]

22 failures in 515 pushes (0.043 failures/push) were associated with this bug yesterday.

Repository breakdown:
* mozilla-central: 22

Platform breakdown:
* windows10-64-ccov: 22

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1481348&startday=2018-09-04&endday=2018-09-04&tree=all

Comment 19

[Intermittent Failures Robot]

34 failures in 2667 pushes (0.013 failures/push) were associated with this bug in the last 7 days.

This is the #48 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. ** 

Repository breakdown:
* mozilla-central: 34

Platform breakdown:
* windows10-64-ccov: 34

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1481348&startday=2018-09-03&endday=2018-09-09&tree=all

Comment 20

[Mathieu Leplatre [:leplatrem]]

The bot keeps pinging here saying «Resolving this bug is a high priority» while the graph shows a drop :)

Eliza, could you please confirm that this is resolved?
Thanks!

Comment 21

[Eliza Balazs [:ebalazs_]]

Yes, this is resolved, no failures in the last 7 days (5th-12th Sept).

The failures from Comment 19 are from the 4th of Sept, and this was fixed on the 5th of Sept.