1332627 - ABORT: file resource://gre/modules/PlacesUtils.jsm line 1910 for OSX network accounts

Status: RESOLVED INCOMPLETE

(Toolkit :: Places, defect)

Description

[HiFrank]

Attachment: OSX_Firefox_Places.db-issues_Crash-log.txt

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:52.0) Gecko/20100101 Firefox/52.0
Build ID: 20170120004016

Steps to reproduce:

- Client machine OS X 10.11.6
- Firefox 45-52 for Apple OSX
- OSX-account is a OSX-linked "LocalNetworkAccount", i.e.: home folder located on OS X Server (Server.app 5.2)
- Launch Firefox in stock configuration


Actual results:

Firefox will crash after some time.
Restart of Firefox fails; requires killing process and launch from scratch.
Intermittendly back-button is deactivated, Bookmarks can't be saved.


Expected results:

Firefox should be stable and usable as it would be on the same machine using an account local to the client mac, instead of using a OSX-network-account on that same mac.

Comment 1

[HiFrank]

Hello,
please excuse my non-technical comment, i hope it is still of use as it is meant to put the *assumed/perceived* issue into operational perspective: it *appears* to me that the current FF (as the 40+ releases obviously) may not be compatible with mass deployments on OSX as part of roaming profiles in a institutional environment, as the frequent crashes stop users using FF. 
Which is 100% my individual scope of experience, no doubt.
Thank you for taking care.

Comment 2

[Marco Bonardo [:mak] (Away Apr 25 - May 5)]

Please try setting the "storage.nfs_filesystem" preference to true in about:config
Sqlite is known to have issues with some network file systems, since they lie about locking, thus it's really easy to corrupt any database.

Comment 3

[HiFrank]

Hello,
thanks a lot four your input. I need a day to find a test machine but will report back then!
Best

Comment 4

[HiFrank]

(In reply to Marco Bonardo [::mak] from comment #2)
> Please try setting the "storage.nfs_filesystem" preference to true in
> about:config
> Sqlite is known to have issues with some network file systems, since they
> lie about locking, thus it's really easy to corrupt any database.

Hello,

sorry for the delay, but it was a bit dfficult to get hold of logs from the field.
I had applied the "storage.nfs_filesystem" preference for FF 52.0.1 (64-bit) as you proposed, but obviously the problem has stayed...

I attached one of the logfiles, and according to it still the problem seem to be writepermissions, if i understand that right?

We didn't face this issue until FF35, and this still holds, as a testdeployment of a control group of Mac-clients verified.

Maybe technological changes after 35 may hint at possible causes for this problem?

Best

Comment 5

[HiFrank]

Sorry,

did not find an option to attach a file, therefore here inline:



===============================================
AbortMessage: ###!!! ABORT: file resource://gre/modules/PlacesUtils.jsm, line 2075

AdapterDeviceID: 0x6741

AdapterVendorID: 0x1002

Add-ons: %7B972ce4c6-7e08-4474-a285-3208198ce6fd%7D:52.0.1,aushelper%40mozilla.org:2.0,e10srollout%40mozilla.org:1.9,firefox%40getpocket.com:1.0.5,webcompat%40mozilla.org:1.0

AddonsShouldHaveBlockedE10s: 0

AsyncPluginInit: 0

AsyncPluginShutdownStates: {WidevineCdm:{11f08ac00:"S=CloseActive"},clearkey:{10ec2e400:"S=CloseActive"},gmpopenh264:{11d8c8400:"S=CloseActive"},-:{-:"012345=Async shutdown complete"}}

AsyncShutdownTimeout: {"phase":"places-will-close-connection","conditions":[{"name":"PlacesUtils read-only connection closing as part of Places shutdown","state":"1. Service has initiated shutdown","filename":"resource://gre/modules/PlacesUtils.jsm","lineNumber":2075,"stack":["resource://gre/modules/PlacesUtils.jsm:setupDbForShutdown/promiseClosed<:2075","resource://gre/modules/Promise.jsm -> resource://gre/modules/Promise-backend.js:Promise:386","resource://gre/modules/PlacesUtils.jsm:setupDbForShutdown:2070","resource://gre/modules/PlacesUtils.jsm:null:2113"]}]}

BuildID: 20170316213829

ContentSandboxCapable: 1

ContentSandboxLevel: 1

CrashTime: 1493024859

DOMIPCEnabled: 1

E10SCohort: test

EMCheckCompatibility: true

FramePoisonBase: 7ffffffff0dea000

FramePoisonSize: 4096

InstallTime: 1493022728

MozCrashReason: MOZ_CRASH()

Notes: AdapterVendorID: 0x1002, AdapterDeviceID: 0x6741FP(D00-L1000-W00000000-T0100) GL Layers? GL Context? GL Context+ GL Layers+ xpcom_runtime_abort(###!!! ABORT: file resource://gre/modules/PlacesUtils.jsm, line 2075)

ProductID: {ec8030f7-c20a-464f-9b0e-13a3a9e97384}

ProductName: Firefox

ReleaseChannel: release

SafeMode: 0

SecondsSinceLastCrash: 2036

ShutdownProgress: profile-before-change

StartupCrash: 0

StartupTime: 1493024458

TelemetryEnvironment: {"build":{"applicationId":"{ec8030f7-c20a-464f-9b0e-13a3a9e97384}","applicationName":"Firefox","architecture":"x86-64","buildId":"20170316213829","version":"52.0.1","vendor":"Mozilla","platformVersion":"52.0.1","xpcomAbi":"x86_64-gcc3","hotfixVersion":null,"architecturesInBinary":"i386-x86_64"},"partner":{"distributionId":null,"distributionVersion":null,"partnerId":null,"distributor":null,"distributorChannel":null,"partnerNames":[]},"system":{"memoryMB":20480,"virtualMaxMB":null,"cpu":{"count":4,"cores":4,"vendor":"GenuineIntel","family":6,"model":42,"stepping":7,"l2cacheKB":256,"l3cacheKB":6144,"speedMHz":2500,"extensions":["hasMMX","hasSSE","hasSSE2","hasSSE3","hasSSSE3","hasSSE4_1","hasSSE4_2","hasAVX"]},"os":{"name":"Darwin","version":"15.6.0","locale":"de-DE@currency=EUR"},"hdd":{"profile":{"model":null,"revision":null},"binary":{"model":null,"revision":null},"system":{"model":null,"revision":null}},"gfx":{"D2DEnabled":null,"DWriteEnabled":null,"ContentBackend":"Skia","adapters":[{"description":null,"vendorID":"0x1002","deviceID":"0x6741","subsysID":null,"RAM":null,"driver":null,"driverVersion":null,"driverDate":null,"GPUActive":true}],"monitors":[{"screenWidth":1920,"screenHeight":1080,"scale":1}],"features":{"compositor":"opengl"}}},"settings":{"blocklistEnabled":true,"e10sEnabled":true,"e10sCohort":"test","telemetryEnabled":false,"locale":"en-US","update":{"channel":"release","enabled":false,"autoDownload":false},"userPrefs":{"browser.cache.disk.capacity":358400,"browser.newtabpage.enhanced":true,"pdfjs.disabled":true},"addonCompatibilityCheckEnabled":true,"isDefaultBrowser":true,"defaultSearchEngine":"google","defaultSearchEngineData":{"name":"Google","loadPath":"jar:[app]/omni.ja!browser/google.xml","origin":"default","submissionURL":"https://www.google.com/search?q=&ie=utf-8&oe=utf-8&client=firefox-b"}},"profile":{"creationDate":17280},"addons":{"activeAddons":{"[e-mail]":{"blocklisted":false,"description":"Sets value(s) in the update url based on custom checks.","name":"Application Update Service Helper","userDisabled":false,"appDisabled":false,"version":"2.0","scope":1,"type":"extension","foreignInstall":false,"hasBinaryComponents":false,"installDay":17242,"updateDay":17242,"isSystem":true},"[e-mail]":{"blocklisted":false,"description":"Staged rollout of Firefox multi-process feature.","name":"Multi-process staged rollout","userDisabled":false,"appDisabled":false,"version":"1.9","scope":1,"type":"extension","foreignInstall":false,"hasBinaryComponents":false,"installDay":17242,"updateDay":17242,"isSystem":true},"[e-mail]":{"blocklisted":false,"description":"When you find something you want to view later, put it in Pocket.","name":"Pocket","userDisabled":false,"appDisabled":false,"version":"1.0.5","scope":1,"type":"extension","foreignInstall":false,"hasBinaryComponents":false,"installDay":17242,"updateDay":17242,"isSystem":true},"[e-mail]":{"blocklisted":false,"description":"Urgent post-release fixes for web compatibility.","name":"Web Compat","userDisabled":false,"appDisabled":false,"version":"1.0","scope":1,"type":"extension","foreignInstall":false,"hasBinaryComponents":false,"installDay":17242,"updateDay":17242,"isSystem":true}},"theme":{"id":"{972ce4c6-7e08-4474-a285-3208198ce6fd}","blocklisted":false,"description":"The default theme.","name":"Default","userDisabled":false,"appDisabled":false,"version":"52.0.1","scope":4,"foreignInstall":false,"hasBinaryComponents":false,"installDay":17242,"updateDay":17242},"activePlugins":[{"name":"Shockwave Flash","version":"25.0.0.163","description":"Shockwave Flash 25.0 r0","blocklisted":false,"disabled":false,"clicktoplay":false,"mimeTypes":["application/x-shockwave-flash","application/futuresplash"],"updateDay":17280}],"activeGMPlugins":{"gmp-gmpopenh264":{"version":"1.6","userDisabled":false,"applyBackgroundUpdates":1},"gmp-widevinecdm":{"version":"1.4.8.903","userDisabled":false,"applyBackgroundUpdates":1}},"activeExperiment":{},"persona":null}}

Theme: classic/1.0

Throttleable: 1

URL: https://www.mendeley.com/forgot/?routeTo=https%3A%2F%2Fwww.mendeley.com%2Fnewsfeed&email=mustermann%40drfz.de&emailLocked=

UptimeTS: 401.31234439

Vendor: Mozilla

Version: 52.0.1

useragent_locale: en-US


This report also contains technical information about the state of the application when it crashed.
===============================================

Comment 6

[Marco Bonardo [:mak] (Away Apr 25 - May 5)]

(In reply to HiFrank from comment #5)
> AsyncShutdownTimeout:
> {"phase":"places-will-close-connection","conditions":[{"name":"PlacesUtils
> read-only connection closing as part of Places shutdown","state":"1. Service
> has initiated
> shutdown","filename":"resource://gre/modules/PlacesUtils.jsm","lineNumber":
> 2075,"stack":["resource://gre/modules/PlacesUtils.jsm:setupDbForShutdown/
> promiseClosed<:2075","resource://gre/modules/Promise.jsm ->
> resource://gre/modules/Promise-backend.js:Promise:386","resource://gre/
> modules/PlacesUtils.jsm:setupDbForShutdown:2070","resource://gre/modules/
> PlacesUtils.jsm:null:2113"]}]}

There doesn't seem to be a relation with "storage.nfs_filesystem", that you likely still want to be set.

This seems to be due to the fact for some reason we are unable to close the database connection in a meaningful amount of time.
You could try increasing toolkit.asyncshutdown.crash_timeout from the current value (60000ms, or 1 minute) and see if it's just matter of the network taking a long time for this. Try with 3 or 5 minutes and see if that helps.

I'm assuming you don't have add-ons installed that may make use of the history/bookmarks database connection.

If even increasing the crash_timeout to 5 minutes doesn't help, it's possible that we just fail to close connections on this filesystem, and then we may need to introduce a workaround (maybe just don't wait closing if storage.nfs_filesystem is set or such).

Comment 7

[HiFrank]

(In reply to Marco Bonardo [::mak] from comment #6)
> (In reply to HiFrank from comment #5)
> > AsyncShutdownTimeout:
> > {"phase":"places-will-close-connection","conditions":[{"name":"PlacesUtils
> > read-only connection closing as part of Places shutdown","state":"1. Service
> > has initiated
> > shutdown","filename":"resource://gre/modules/PlacesUtils.jsm","lineNumber":
> > 2075,"stack":["resource://gre/modules/PlacesUtils.jsm:setupDbForShutdown/
> > promiseClosed<:2075","resource://gre/modules/Promise.jsm ->
> > resource://gre/modules/Promise-backend.js:Promise:386","resource://gre/
> > modules/PlacesUtils.jsm:setupDbForShutdown:2070","resource://gre/modules/
> > PlacesUtils.jsm:null:2113"]}]}
> 
> There doesn't seem to be a relation with "storage.nfs_filesystem", that you
> likely still want to be set.
> 
> This seems to be due to the fact for some reason we are unable to close the
> database connection in a meaningful amount of time.
> You could try increasing toolkit.asyncshutdown.crash_timeout from the
> current value (60000ms, or 1 minute) and see if it's just matter of the
> network taking a long time for this. Try with 3 or 5 minutes and see if that
> helps.
> 
> I'm assuming you don't have add-ons installed that may make use of the
> history/bookmarks database connection.
> 
> If even increasing the crash_timeout to 5 minutes doesn't help, it's
> possible that we just fail to close connections on this filesystem, and then
> we may need to introduce a workaround (maybe just don't wait closing if
> storage.nfs_filesystem is set or such).

Hello Marco,

thank you for your proposal.

As i have experience with other applications having a hard time with OSX network homes as well (MS OneNote for instance), i still fancy the issue may be rooted in a OSX sandbox side effect? It's the "read-only connection closing" part that strikes me here...

I'll first check your measure immediately. It may take a bit until i receive the first feedback...

Best

Comment 8

[Marco Bonardo [:mak] (Away Apr 25 - May 5)]

(In reply to HiFrank from comment #7)
> As i have experience with other applications having a hard time with OSX
> network homes as well (MS OneNote for instance), i still fancy the issue may
> be rooted in a OSX sandbox side effect? It's the "read-only connection
> closing" part that strikes me here...

That just means that a connection to the Sqlite database was took more than 1 minute to close, and thus the process was forcibly crashed to avoid hanging around. The read-only is just a note about the specific type of connection, nothing to do with permissions.

Comment 9

[HiFrank]

(In reply to Marco Bonardo [::mak] from comment #8)
> (In reply to HiFrank from comment #7)
> > As i have experience with other applications having a hard time with OSX
> > network homes as well (MS OneNote for instance), i still fancy the issue may
> > be rooted in a OSX sandbox side effect? It's the "read-only connection
> > closing" part that strikes me here...
> 
> That just means that a connection to the Sqlite database was took more than
> 1 minute to close, and thus the process was forcibly crashed to avoid
> hanging around. The read-only is just a note about the specific type of
> connection, nothing to do with permissions.

Hello Marco,
thank you for your explanation. I'll take up the track you proposed and report back a.s.a.p.

Best

Comment 10

[HiFrank]

Attachment: OSX_Firefox_Places.db-issues_Crash-log_1704281801.txt

Crash log with custom settings:

FF 52.0.1
toolkit.asyncshutdown.crash_timeout =: 300000ms
storage.nfs_filesystem =: true

Comment 11

[HiFrank]

(In reply to Marco Bonardo [::mak] from comment #8)
> (In reply to HiFrank from comment #7)
> > As i have experience with other applications having a hard time with OSX
> > network homes as well (MS OneNote for instance), i still fancy the issue may
> > be rooted in a OSX sandbox side effect? It's the "read-only connection
> > closing" part that strikes me here...
> 
> That just means that a connection to the Sqlite database was took more than
> 1 minute to close, and thus the process was forcibly crashed to avoid
> hanging around. The read-only is just a note about the specific type of
> connection, nothing to do with permissions.

Hello Marco,

it looks like the situation has not improved. 

This time i used a different machine and different account, but same configuration, modified solely with what you proposed (toolkit.asyncshutdown.crash_timeout =: 300000ms). I attached a log (OSX_Firefox_Places.db-issues_Crash-log_1704281801.txt).
From the user perspective: adding Bookmarks works, history works as well. Shutting down the application also works - basically. It takes a substantial amount of time to finish that, the application may report to be not responding in the meantime, but eventually will succeed. The actual failure cases are 
the regular crashes when 

- using the application, or 
- leaving the application running passively in the background, with not user interaction keeping it busy.

The core differentiator that triggers this behaviour is the user having it's home directory AFP-based on an OSX-Server-network-homes-share. No user account local to the very same client machine will experience any issues...

It's a fixed line gigabit network, latency is normal, ping times are around 0,366ms, server has over 60% of headroom in all metrices (memory, cpu, bandwidth)...

May there be s.th. other i could check out?

Best regards

Comment 12

[Marco Bonardo [:mak] (Away Apr 25 - May 5)]

I don't have many ideas. All the crashes you shown are shutdown crashes. So I'm not sure I understand comment 11 about shutdown being ok, but crashes during other tasks. These are all shutdown crashes.

Looks like the Sqlite.jsm call to yield conn.close(); just never returns in your environment.
There may be a logic error in the Sqlite.jsm shutdown code, but it needs to be debugged in the actual environment causing the issue to be figured out. We always had problems with nfs and afp, because that filesystem is bogus and lies to Sqlite about locking. You can see the many dependencies of bug 719952.

Comment 13

[Marco Bonardo [:mak] (Away Apr 25 - May 5)]

If you can code js, you could try making a debug build of firefox, and debugging (even just through dump() calls) what happens in Sqlite.jsm::close() method, and where we stop.

Comment 14

[HiFrank]

(In reply to Marco Bonardo [::mak] from comment #13)
> If you can code js, you could try making a debug build of firefox, and
> debugging (even just through dump() calls) what happens in
> Sqlite.jsm::close() method, and where we stop.

Hello Marco, 

sorry for being late. 
Phew... o.k. ;-)
That makes sense for sure to diagnose the issue in situ. It may take a bit of time as i'm in the midst of a migration project and also have no developer installation (XCode) right at hand immediately, but i'll go there if it helps.

Thank you very much for your help so far!

Best

Comment 15

[Marco Bonardo [:mak] (Away Apr 25 - May 5)]

Unfortunately we don't have a way to reproduce the problem :(

Comment 16

[HiFrank]

Hello Marco,

thank your for having a look!

Unfortunately i could not make the case for investing a bigger time slot into this issue down here as i hoped in my last post, as my "environment" prefers to question the entire OSX deployment rather to ask for options how to make it work :-/

In this context i investigated OSX 10.12.x and 10.13.x and their respective APPLE OSX-Server releases in test deployments, whether improvements have been made with regard how the assumed problematic behaviour of SQLite & network storage is handled at the APPLE end, and unfortunately the same issues basically persist in all nee releases (affecting all APPLE MacOS "Internetaccounts" applications such as Mail, Calendar, Contacts, but also MS Office 2016 and OneNote and Firefox).

To be precise:
- OSX "Internet account"-applications behave better now, as they still tend to crash at times, but can be revitalized upon a relogin of the network user account. At the same time, they seem to tend to hang earlier/more often...
- Outlook 2016 now fails differently (can't store passwords in keychain), as still does OneNote (still no login possible)
- the Firefox situation improved sort of with OSX 10.13.x, as now Bookmarks can be set, and won't get lost, when Firefox then crashes at some time, which still happens. In earlier times, often no Bookmarks could be set at some point in time, and often Booksmarks got lost, when a new Places DB had to be created after a crash. So Firefox now simply needs to be restarted, but it's not good enough for critical sessions yet...

My comclusion so far is that APPLE seems not to be too heavily leaned towards having customers deploy network-home directories, and as i don't think i will get away any longer with promising "my" users improvements somewhere in the future with the next MacOS release, i'll start considering radically different approaches...

If this verbal "update" should be backed by any kind of liog files, i'd be happy to gather them.

Best so far