1574569 - Local Storage Next Generation breaks travis-ci.org site on my computer / NS_ERROR_FAILURE: tracer.js:98

Status: RESOLVED FIXED

(Core :: Storage: localStorage & sessionStorage, defect, P1)

Description

[ailin.nemui]

Attachment: Screenshot from 2019-08-14 11-01-10.png

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0

Steps to reproduce:

Go to this site:

https://travis-ci.org/irssi/irssi/builds/571386478?utm_source=github_status&utm_medium=notification

Actual results:

Blank (grey) page, error NS_ERROR_FAILURE: tracer.js:98 visible in console

I asked other users with Nightly, but they could not reproduce this issue. I am not sure where the difference is. My system is openSUSE x86-64, Nightly auto-updates from mozilla

Expected results:

Travis-ci website should render and show Results of tests

Setting dom.storage.next_gen to false works around the problem.

Comment 1

[ailin.nemui]

I used mozregression to find this pushlog:

https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=5e8cc2d02203362b91e84197585de9a82eeeff64&tochange=456bdb1cc727ce71bb5dc70533e43e12191e0071

Comment 2

[Andrew Sutherland [:asuth] (he/him)]

Can you open https://firefox-storage-test.glitch.me/ and let us know if any of the other subsystems besides LocalStorage report problems?

Comment 3

[ailin.nemui]

all is totally working

Comment 4

[ailin.nemui]

Attachment: Screenshot from 2019-08-16 22-33-16.png

screenshot of " Good: Totally Working. (fullyOperational)" in fresh profile (white), daily profile (black), broken travis page (grey)

Comment 5

[Andrew Sutherland [:asuth] (he/him)]

LocalStorage NextGen stores data on a per-site/origin basis. This suggests the origin is experiencing troubles. Ideally we would like to move the data for the site out of your profile directory so 1) Firefox will re-create the necessary database and hopefully work and 2) so you could perhaps send us the corrupt files to diagnose what's going on, if 1 fixes it.

I suspect you're already well aware of how to find the profile directory, but if not:

• Open "about:profiles"
• Locate the line "This is the profile in use and it cannot be deleted." That's the profile the browser is using.
• The directory of interest is the "Root Directory". Either click the "Open Directory" button or copy and paste the location into your file browser/command line of choice.

From your profile directory, the directory we're interested in should be storage/default/https+++travis-ci.org. Given that LSNG isn't working and you're on linux, you should probably be able to move the directory somewhere outside of the storage/ subtree without trouble. (Moving the directory within the subtree will make QuotaManager mad in other ways.)

This may cause the site to start working immediately when you refresh the page, or you may need to restart Firefox. But I think this should all be safe to do while Firefox is open and just refreshing should work.

In the event moving the directory to the side works, it would be great if you could zip up the directory and email it to [e-mail] and [e-mail]. At least for me, all the database contains is a single entry with key pusherTransportTLS and value {"timestamp":1565988192644,"transport":"ws","latency":259}, but if you use Travis a lot there may be more stuff in there.

Comment 6

[ailin.nemui]

I am having this problem also on an empty/fresh profile.

The storage/default folder is empty.

please advise

Comment 7

[Andrew Sutherland [:asuth] (he/him)]

So, the console error line number seems wrong for the "tracer.js" I'm being served. I wonder if there's some kind of A/B testing going on.

If I open up the devtools and click on the "Debugger" tab, then use "ctrl-p" to bring up "tracer.js", and then I right-click on the tab and do "copy source URI", I get:
https://cdn.travis-ci.org/assets/travis/initializers/tracer.js

Is that what you get too?

Also, can you expand the stack trace of the error you're seeing in the devtools console and screenshot that too? Maybe there's something else interesting in that backtrace.

Comment 8

[ailin.nemui]

copy source uri -> https://cdn.travis-ci.org/assets/travis/initializers/tracer.js ... same as you

is there any way I can have firefox print more detailed info about the error?

Comment 9

[ailin.nemui]

Attachment: Screenshot from 2019-08-16 23-56-49.png

screenshot with error expanded

Comment 10

[ailin.nemui]

nb if I try to click the link to the tracer.js, I don't get any javascript source at all. but the debugger does show javascript; the line which it marks as leading to the error is:

return Tracer;

Comment 11

[Andrew Sutherland [:asuth] (he/him)]

Thanks for the screenshot and all your help in digging into this error! Based on the expanded call-stack, the line of source that's breaking is return window.localStorage['apiTrace'] === 'true'; which is a straightforward read.

I think the last thing we might want to check is the "Browser Console". It's accessible from the "Web Developer" menu of the main menu, or on linux by hitting control-shift-j. It might potentially be displaying some interesting QuotaManager errors. (It's very confusing that your storage/default folder is empty, too. I wonder if it's the wrong profile or you accidentally looking in the ".cache" directory? As the firefox-storage-test shouldn't be able to pass if there isn't stuff in that directory...

Comment 12

[ailin.nemui]

The folder is not empty any more; now it contains a https+++firefox-storage-test.glitch.me/ directory

I don't think the Browser Console shows anything useful. I'm attaching a screen shot, but it has shown this content since the browser launch

Comment 13

[ailin.nemui]

Attachment: browser console screenshot

Comment 14

[Andrew Sutherland [:asuth] (he/him)]

So, this is all a bit of a puzzler at this point, especially given that the problem reproduces with a clean profile for you but not for others (ex: me) and the errors are a generic failure error code, not a generic security error code. I think the last thing we should check without adding more instrumentation to LSNG is whether 1) disabling "enhanced tracking protection" from the little shield in the location bar doorhanger helps and whether 2) disabling tracking protection from about:preferences helps. You'd want to refresh the page after each attempt. Doing this in a new profile is fine.

Also, please note if you're doing anything special to set up the new profile. Also, also, if you could check if it's possible that you have globally installed addons that show up in about:addons that might be coming from a system-wide install that could somehow be doing something? (And/or if your Firefox says it's centrally managed, which I think might be indicative of that?)

Comment 15

[ailin.nemui]

I really appreciate the effort you're putting in trying to solve this issue with me. It's also very puzzling to me!

I created a new profile called empty using "firefox -ProfileManager"

I launch the profile with "firefox -P empty --no-remote"

-> Enhanced tracking protection is OFF: no change, still broken

-> Privacy & Security -> Custom -> Uncheck Cookies/Tracking content/Cryptominers/Fingerprinters: no change

about:addons -> Shows me recommendations of things to add to firefox, I can't see any addons that would be installed.

where would firefox say that it's being managed?

Comment 16

[Andrew Sutherland [:asuth] (he/him)]

A little bar shows up at the top of about:preferences, I believe. Also about:policies would show something. I wasn't really expecting it; it's something more likely on Windows with anti-virus software that likes to meddle, but I also don't know what various distributions do. (For example, when using gnome-shell, it likes to install a webextension to let you control gnome-shell's extensions. But I'm unsure if things have changed in relation to that or not.)

I'm going to needinfo Jan for his thoughts. I think we may need to add more logging for exceptional cases.

Comment 17

[Jan Varga [:janv]]

I think running a debug build could reveal where the code fails.

Do you know how to download a debug build from [1] ?

[1] https://treeherder.mozilla.org/#/jobs?repo=mozilla-central

Comment 18

[ailin.nemui]

[Child 24912, Main Thread] WARNING: 'gPendingSyncMessage', file /builds/worker/workspace/build/src/dom/localstorage/LSObject.cpp, line 1132
[Child 24912, Main Thread] WARNING: 'NS_FAILED(rv)', file /builds/worker/workspace/build/src/dom/localstorage/LSObject.cpp, line 824
[Child 24912, Main Thread] WARNING: 'NS_FAILED(rv)', file /builds/worker/workspace/build/src/dom/localstorage/LSObject.cpp, line 876
[Child 24912, Main Thread] WARNING: 'NS_FAILED(rv)', file /builds/worker/workspace/build/src/dom/localstorage/LSObject.cpp, line 576
[Child 24912, ImageIO] WARNING: Appending an extra chunk for SourceBuffer: file /builds/worker/workspace/build/src/image/SourceBuffer.cpp, line 137
[Child 24912, ImageIO] WARNING: Appending an extra chunk for SourceBuffer: file /builds/worker/workspace/build/src/image/SourceBuffer.cpp, line 137
[Child 24912, ImageIO] WARNING: Appending an extra chunk for SourceBuffer: file /builds/worker/workspace/build/src/image/SourceBuffer.cpp, line 137
JavaScript error: https://cdn.travis-ci.org/assets/travis-a8a26a0a760d89d75f2f8f6752a6cf69.js, line 1086: NS_ERROR_FAILURE:

give me more detailed instructions if there is something I can try

Comment 19

[Jan Varga [:janv]]

(In reply to ailin.nemui from comment #18)

[Child 24912, Main Thread] WARNING: 'gPendingSyncMessage', file /builds/worker/workspace/build/src/dom/localstorage/LSObject.cpp, line 1132
[Child 24912, Main Thread] WARNING: 'NS_FAILED(rv)', file /builds/worker/workspace/build/src/dom/localstorage/LSObject.cpp, line 824
[Child 24912, Main Thread] WARNING: 'NS_FAILED(rv)', file /builds/worker/workspace/build/src/dom/localstorage/LSObject.cpp, line 876
[Child 24912, Main Thread] WARNING: 'NS_FAILED(rv)', file /builds/worker/workspace/build/src/dom/localstorage/LSObject.cpp, line 576

Ok, so I think I know what's going on. I guess you have accessibility enabled, so parent process synchronously calls into content processes sometimes which can collide with LSNG. We made some changes in the meantime and the gPendingSyncMessage check shouldn't be needed in theory. I propose to add a new pref for this, so you will be able to change the pref and see if LS works for you. Once we are confident everything works as expected we can remove all the gPendingSyncMessage stuff along with the pref.

Comment 20

[ailin.nemui]

I can confirm, force disabling a11y also works around this issue. that is a strange interrelation...

Comment 21

[Jan Varga [:janv]]

Attachment: Bug 1574569 - Don't abort LocalStorage requests when a sync message from parent is detected; r=asuth

We keep the original behavior on windows for now since accessibility on windows
uses sync COM calls (not sync IPC messages).

Comment 22

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/0303ad759e46
Don't abort LocalStorage requests when a sync message from parent is detected; r=asuth

Comment 23

[Andrei Ciure[:aciure]]

https://hg.mozilla.org/mozilla-central/rev/0303ad759e46

Comment 24

[Cristian Fogel [:cfogel]]

Unfortunately the issue does not appear to reproduce on Ubuntu(16.04) - 70.0a1 (Build ID 20190816215314).
@ailin mind checking if the issue is fixed with the current builds?

Thank you in advance!

Comment 25

[ailin.nemui]

works for me with dom.storage.abort_on_sync_parent_to_child_messages false and patch 0303ad759e46

to reproduce, set abort to true and use accessibility services (on openSUSE in my case, not sure if it would matter)

Comment 26

[Jan Varga [:janv]]

The pref is set to false by default, see https://searchfox.org/mozilla-central/rev/17756e2a5c180d980a4b08d99f8cc0c97290ae8d/modules/libpref/init/StaticPrefList.yaml#2208
So this should be fixed for everyone.