1827756 - [Fission Android] Timeout [but not actually: Font face deadlock] in dom/security/test/csp/test_bug1452037.html

Status: RESOLVED FIXED

(GeckoView Graveyard :: Sandboxing, defect, P1)

Description

[[:owlish] 🦉 PST]

Debug build.
Push with failure
Log

Comment 1

[Nika Layzell [:nika] (ni? for response)]

In the logcat log, I see this immediately after the test_start for the test:

04-12 21:59:49.329 11661 11676 I GeckoDump: ⰲ겿{"action":"test_start","time":1681336789329,"thread":null,"pid":null,"source":"mochitest","test":"/tests/dom/security/test/csp/test_bug1452037.html","js_source":"TestRunner.js"}ⰲ겿
04-12 21:59:49.331 11661 11676 I Gecko   : [Child 11661, Main Thread] WARNING: Potential deadlock detected:
04-12 21:59:49.331 11661 11676 I Gecko   : Cyclical dependency starts at
04-12 21:59:49.331 11661 11676 I Gecko   : Mutex : gfxFontEntry lock
04-12 21:59:49.331 11661 11676 I Gecko   : Next dependency:
04-12 21:59:49.331 11661 11676 I Gecko   : Mutex : SharedFTFace::mLock (currently acquired)
04-12 21:59:49.331 11661 11676 I Gecko   : Cycle completed at
04-12 21:59:49.331 11661 11676 I Gecko   : Mutex : gfxFontEntry lock
04-12 21:59:49.331 11661 11676 I Gecko   : Deadlock may happen for some other execution
04-12 21:59:49.331 11661 11676 I Gecko   : 
04-12 21:59:49.331 11661 11676 I Gecko   : : file /builds/worker/checkouts/gecko/xpcom/threads/BlockingResourceBase.cpp:247
04-12 21:59:49.331 11661 11676 I Gecko   : [Child 11661, Main Thread] WARNING: Potential deadlock detected:
04-12 21:59:49.331 11661 11676 I Gecko   : Cyclical dependency starts at
04-12 21:59:49.331 11661 11676 I Gecko   : Mutex : gfxFontEntry lock
04-12 21:59:49.331 11661 11676 I Gecko   : Next dependency:
04-12 21:59:49.331 11661 11676 I Gecko   : Mutex : SharedFTFace::mLock (currently acquired)
04-12 21:59:49.331 11661 11676 I Gecko   : Cycle completed at
04-12 21:59:49.331 11661 11676 I Gecko   : Mutex : gfxFontEntry lock
04-12 21:59:49.331 11661 11676 I Gecko   : Deadlock may happen for some other execution
04-12 21:59:49.331 11661 11676 I Gecko   : 
04-12 21:59:49.331 11661 11676 I Gecko   : : file /builds/worker/checkouts/gecko/xpcom/threads/BlockingResourceBase.cpp:247
04-12 21:59:49.331 11661 11676 I Gecko   : [Child 11661, Main Thread] WARNING: Potential deadlock detected:
04-12 21:59:49.331 11661 11676 I Gecko   : Cyclical dependency starts at
04-12 21:59:49.331 11661 11676 I Gecko   : Mutex : gfxFontEntry lock
04-12 21:59:49.331 11661 11676 I Gecko   : Next dependency:
04-12 21:59:49.331 11661 11676 I Gecko   : Mutex : SharedFTFace::mLock (currently acquired)
04-12 21:59:49.331 11661 11676 I Gecko   : Cycle completed at
04-12 21:59:49.331 11661 11676 I Gecko   : Mutex : gfxFontEntry lock
04-12 21:59:49.331 11661 11676 I Gecko   : Deadlock may happen for some other execution
04-12 21:59:49.331 11661 11676 I Gecko   : 
04-12 21:59:49.331 11661 11676 I Gecko   : : file /builds/worker/checkouts/gecko/xpcom/threads/BlockingResourceBase.cpp:247
04-12 21:59:49.331 11661 11676 I Gecko   : [Child 11661, Main Thread] WARNING: Potential deadlock detected:
04-12 21:59:49.331 11661 11676 I Gecko   : Cyclical dependency starts at
04-12 21:59:49.331 11661 11676 I Gecko   : Mutex : gfxFontEntry lock
04-12 21:59:49.331 11661 11676 I Gecko   : Next dependency:
04-12 21:59:49.331 11661 11676 I Gecko   : Mutex : SharedFTFace::mLock (currently acquired)
04-12 21:59:49.331 11661 11676 I Gecko   : Cycle completed at
04-12 21:59:49.331 11661 11676 I Gecko   : Mutex : gfxFontEntry lock
04-12 21:59:49.331 11661 11676 I Gecko   : Deadlock may happen for some other execution
04-12 21:59:49.331 11661 11676 I Gecko   : 
04-12 21:59:49.331 11661 11676 I Gecko   : : file /builds/worker/checkouts/gecko/xpcom/threads/BlockingResourceBase.cpp:247

It looks like there's a deadlock issue on android between SharedFTFace::mLock and gfxFontEntry lock, which is leading to timeouts in some geckoview android tests (at least I assume that potential deadlock is likely repsonsible). Leaving a ni? to :emilio to look into it.

Comment 7

[Emilio Cobos Álvarez (:emilio)]

Last change in those locks was introduced in bug 1763414... Jonathan, do you have cycles to take a look?

Comment 8

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1763414

Comment 10

[Chris Peterson [:cpeterson]]

Jonathan, this bug blocks Android Fission because this font deadlock causes multiple tests to fail in Android Fission mode. As such, the Firefox Android team is eager for this test to be fixed. If you have questions about running Android tests in Fission mode, reach out to :owlish in the #fission channel on Slack or Matrix.

Comment 11

[Olivia Hall [:olivia]]

Not entirely sure if bug 1804027 is related, but similar font deadlock logging occurs before a test timeout, so linking it as a reference.

Comment 12

[Jonathan Kew [:jfkthame]]

I've been looking into the potential deadlock warnings here, and I don't think this is actually related to the test timeouts. Note that the warnings are of potential deadlock; they do not tell us that a deadlock is in fact happening.

Looking at tests such as test_bug1452037.html, I don't see anything that seems likely to trigger a font-related deadlock, unless there were a deadlock that's liable to happen on pretty much any invocation. But that doesn't happen; the timeouts seem reliably associated with particular tests, while the tests in question are not doing anything "interesting" with fonts.

Moreover, checking logs from other test runs, it appears that these font-related warnings always appear, with or without Fission, and they appear on all runs, not just the ones that end up timing out.

The warning is genuine, inasmuch as that there's a potential locking cycle, and we should fix that; but it won't resolve the test timeout here.

Comment 13

[Jonathan Kew [:jfkthame]]

According to this try run, the patch in bug 1835110 will resolve the font-related "potential deadlock" being reported here; but test_bug1452037 will still time out just the same.

Comment 14

[Jonathan Kew [:jfkthame]]

Moving this back to GV:Sandboxing for re-examination, instead of Layout:Text & Fonts, as the font-related warnings were not actually relevant to the timeout happening in this test.

Comment 15

[Chris Peterson [:cpeterson]]

(In reply to Jonathan Kew [:jfkthame] from comment #13)

According to this try run, the patch in bug 1835110 will resolve the font-related "potential deadlock" being reported here; but test_bug1452037 will still time out just the same.

Jonathan, thanks for investigating!

Irene, can you please retest this mochitest with Fission? Jonathan fixed the font deadlock warnings with bug 1835110, but it won't fix this test (since these deadlock warnings are logged with or without Fission). So we need a new theory for why this test is timing out.

When originally categorizing the Fission mochitest failures, you suspected this bug is the cause of at least four other mochitest failures. So fixing this bug is a high priority for unblocking other Fission tests.

Comment 16

[[:owlish] 🦉 PST]

The test now passes for me locally. I'll make a try push to see what is going on in automation

Comment 17

[[:owlish] 🦉 PST]

Ok, this particular test passes in automation as well, and I see some of the other tests pass now also. I will continue retesting to see what else got fixed, including the tests that are blocked on this. I will also enable the tests that are now passing.

Comment 18

[[:owlish] 🦉 PST]

Apologies, I am reopening this bug to make a patch to enable this test.

Comment 19

[[:owlish] 🦉 PST]

Attachment: Bug 1827756 - Enable dom/security/test/csp/test_bug1452037 test for Android Fission builds

Comment 20

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/0b3723566862
Enable dom/security/test/csp/test_bug1452037 test for Android Fission builds r=nika

Comment 21

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/0b3723566862