1723597 - Citrix? Crash in [@ __GI___umask]

Status: RESOLVED WORKSFORME

(Core :: Security: Process Sandboxing, defect, P3)

Description

[Chris Peterson [:cpeterson]]

100% of these crash reports have Fission enabled.

67 crash reports from 14 different installs of Nightly 91 and 92.

Crash report: https://crash-stats.mozilla.org/report/index/2c77ada0-375d-42af-b5ed-fd8f60210802

Reason: SIGSYS

Top 2 frames of crashing thread:

0 libc.so.6 __GI___umask 
1 libAppProtection.so.1.6.7.10 libAppProtection.so.1.6.7.10@0x19bf2 

Comment 1

[Chris Peterson [:cpeterson]]

Jed, could your fd leak fix for sandbox file broker bug 1719391 have caused this crash regression?

The earliest crashing buildid is 20210711212832 and bug 1719391 is the only change that looks related to Linux or sandboxing in the pushlog for the two days before buildid 20210711212832:

https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=329106017f97def520114f314233b6baa3334a30&tochange=877be9f91d38f4166c9a7363cd176f387456916b

Comment 2

[Jed Davis [:jld] ⟨⏰|UTC-7⟩ ⟦he/him⟧]

I am deeply suspicious of this libAppProtection.so thing, for one because it's using umask at all, and for another because it's unknown to our crash reporter (which does know about the various other system libraries).

According to people on StackOverflow this library is due to Citrix; I assume they're globally preloading it. (See also the many many problems we have with library injection on Windows.) But also, note that people are reporting problems with release Firefox, so this isn't just the Nightly-only intentional crash; it's apparently breaking things even when we just return the error ENOSYS.

This means that it wouldn't be enough to add it to kLibsThatWillCrash. Ideally we'd just block the library from loading, but that's difficult; LD_AUDIT should be early enough in the loading process, but the last time I tried to use it, it caused a crash on a version of Ubuntu we still had to support at the time, and I don't know if the bugs have been fixed.

Also, notice that the StackOverflow question was reported months before bug 1719391 landed, so the timing seems to be a coincidence. I wonder if there's something about this situation that also interferes with sending crash reports.

I'll need to investigate this some more.

Comment 3

[Chris Peterson [:cpeterson]]

Removing "Fission" from the bug summary since it sounds like this is an issue caused by Citrix or some other external software, not Fission.

Comment 4

[Julien Cristau [:jcristau]]

AFAIK we don't crash on sandbox violations outside of nightly.

Comment 5

[Jed Davis [:jld] ⟨⏰|UTC-7⟩ ⟦he/him⟧]

(In reply to Julien Cristau [:jcristau] from comment #4)

AFAIK we don't crash on sandbox violations outside of nightly.

That is true, but in this case there are user reports (see the StackOverflow link in comment #2) that release is affected, possibly because of something else the injected library is doing; I'll investigate further and report back.

Comment 6

[Julien Cristau [:jcristau]]

Hi :gcp, what's the next step here, are we likely to do something about it in the 93 time frame?

Comment 7

[Gian-Carlo Pascutto [:gcp]]

I'm moving this back to P2. Jed looked at it and concluded there's no good fix. Perhaps juggling with LD_AUDIT might work but we're not even sure.

We can contact the vendor here and point out that their usage of umask here is almost 100% certain to be broken.

Comment 8

[Julien Cristau [:jcristau]]

Thanks.

Comment 9

[Gian-Carlo Pascutto [:gcp]]

I'm dropping this - we have no good idea how to work around this and crash volume is low.

Comment 10

[Jed Davis [:jld] ⟨⏰|UTC-7⟩ ⟦he/him⟧]

To answer comment #1: nothing we've done is related (other than turning sandboxing on in the first place), and this is basically an “external software affecting Firefox” type of issue (although, as mentioned, we might be able to work around it on our end to some extent if we needed to).

Comment 11

[BugBot [:suhaib / :marco/ :calixte]]

Closing because no crashes reported for 12 weeks.