1386826 - Reftest and crashtest doesn't run locally on linux, after sandbox tightening

Status: RESOLVED FIXED

(Core :: Security: Process Sandboxing, defect, P2)

Description

[Daniel Holbert [:dholbert]]

STR:
 In a mozilla-central build (debug in my case), run:
  ./mach reftest
    ....or on a single directory, e.g.:
  ./mach reftest layout/reftests/layers/

EXPECTED RESULTS:
Reftests run.  Reftest logging like REFTEST TEST-LOAD should start appearing in terminal.

ACTUAL RESULTS:
After the reftest canvas window appears, I see this logged to my terminal:
==========
Chrome file doesn't exist: /scratch/work/builds/mozilla-inbound/obj/_tests/reftest/reftest/chrome/reftest/content/reftest-content.js
[Child 27730] WARNING: NS_ENSURE_SUCCESS_VOID(rv) failed with result 0x80520015: file ../../../mozilla/dom/base/nsFrameMessageManager.cpp, line 1618
JavaScript error: file:///scratch/work/builds/mozilla-inbound/obj/dist/bin/browser/components/nsBrowserGlue.js, line 993: TypeError: win is null
[Parent 27603] WARNING: attempt to modify an immutable nsStandardURL: file /scratch/work/builds/mozilla-inbound/mozilla/netwerk/base/nsStandardURL.cpp, line 1807
[Parent 27603] WARNING: Failed to retarget HTML data delivery to the parser thread.: file /scratch/work/builds/mozilla-inbound/mozilla/parser/html/nsHtml5StreamParser.cpp, line 977
==========

And then it just sits there doing nothing.  No reftests are loaded, and no REFTEST TEST-LOAD logging happens.


"hg bisect" tells me this was a regression from bug 1308400.


I initially reported this on bug 1386502 (which seems like this same issue except on Windows).  I'm spinning this off as its own bug, since it looks like this regressed (for me) due to a linux-specific change, so I'm guessing there will be platform-specific fixes that are required as well.

Comment 1

[Daniel Holbert [:dholbert]]

(In reply to Daniel Holbert [:dholbert] (PTO August 3-7) from comment #0)
> "hg bisect" tells me this was a regression from bug 1308400.

Specifically, the first-bad build is from this changeset:
 https://hg.mozilla.org/integration/mozilla-inbound/rev/167f91f87172

Comment 2

[Haik Aftandilian [:haik]]

In my case, this works when my object dir is within the repo, but when its configured to be outside the repo, the test hangs.

Comment 3

[Haik Aftandilian [:haik]]

(In reply to Haik Aftandilian [:haik] from comment #2)
> In my case, this works when my object dir is within the repo, but when its
> configured to be outside the repo, the test hangs.

I don't know if it's related to the problem, but I checked with dholbert and he also has the object dir outside the repo (despite the above path "/scratch/work/builds/mozilla-inbound/obj" suggesting otherwise.)

Comment 4

[Daniel Holbert [:dholbert]]

Yes, my objdir is indeed outside of my repo -- I have:
  source: /scratch/work/builds/mozilla-inbound/mozilla
  obj:    /scratch/work/builds/mozilla-inbound/obj

So it's likely that my problem is caused by the whitelist not including objdir-outside-the-repo, as haik suggests.

Comment 5

[Daniel Holbert [:dholbert]]

(I verified that reftests run correctly if I remove my "MOZ_OBJDIR=@TOPSRCDIR@/../obj" line from .mozconfig and rebuild. So indeed, my results match haik's.)

(Apparently this doesn't help on Windows, though, per Bug 1386502 comment 10.)

Comment 6

[Jed Davis [:jld] ⟨⏰|UTC-7⟩ ⟦he/him⟧]

I also have my objdir outside the source dir, and I reproduced this with MOZ_SANDBOX_LOGGING=t:

Sandbox: SandboxBroker: denied op=stat rflags=400000 perms=0 path=/home/jld/src/obj.gecko-dev/obj-x86_64-pc-linux-gnu/_tests/reftest/reftest/chrome/reftest/content/reftest-content.js for pid=69071
Sandbox: Failed errno -13 op 2 flags 0400000 path /home/jld/src/obj.gecko-dev/obj-x86_64-pc-linux-gnu/_tests/reftest/reftest/chrome/reftest/content/reftest-content.js
Sandbox: SandboxBroker: denied op=access rflags=0 perms=0 path=/home/jld/src/obj.gecko-dev/obj-x86_64-pc-linux-gnu/_tests/reftest/reftest/chrome/reftest/content/reftest-content.js for pid=69071
Sandbox: Failed errno -13 op 1 flags 00 path /home/jld/src/obj.gecko-dev/obj-x86_64-pc-linux-gnu/_tests/reftest/reftest/chrome/reftest/content/reftest-content.js
Chrome file doesn't exist: /home/jld/src/obj.gecko-dev/obj-x86_64-pc-linux-gnu/_tests/reftest/reftest/chrome/reftest/content/reftest-content.js

I'd suggest just allowing access to all directories in the chrome:// registry, but it looks like it might be nontrivial to extract that information.

Comment 7

[Haik Aftandilian [:haik]]

(In reply to Jed Davis [:jld] (⏰UTC-6) from comment #6)
> I'd suggest just allowing access to all directories in the chrome://
> registry, but it looks like it might be nontrivial to extract that
> information.

As an alternative, what about using NS_GRE_DIR/../..? (Only on development builds.)

Comment 8

[Gian-Carlo Pascutto [:gcp]]

(In reply to Haik Aftandilian [:haik] from comment #7)
> As an alternative, what about using NS_GRE_DIR/../..? (Only on development
> builds.)

If we're going to do things like that then I think I prefer lowering the sandbox to level 2 for these testsuites.

I think the fundamental problem here is that the tests need MOZ_DEVELOPER_OBJ_DIR which is currently not used on Linux, meaning that this is also a simple fix, but also meaning we won't be able to get rid of those env vars just yet.

Note that this isn't needed for try itself, which is why my test on removing them didn't notice it. (And there was no OBJ_DIR to remove on Linux in the first place.

Comment 9

[Gian-Carlo Pascutto [:gcp]]

After some digging, the problem isn't MOZ_DEVELOPER_OBJ_DIR (because Linux doesn't actually rely on those). We need to duplicate:
https://hg.mozilla.org/integration/autoland/rev/b57aac875b65

But pointing to the objdir, instead of the repodir. It's possible Linux actually only strictly needs the objdir now that it has symlink handling, but macOS likely needs both (until symlinks are replaced by hardlinks).

Comment 10

[Gian-Carlo Pascutto [:gcp]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=1e49d38cd3dbf04c647de982dec6f0a895efff93

Comment 11

[Gian-Carlo Pascutto [:gcp]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=42d91f771793073ddd55582b52b33c9a2329beb1

Comment 12

[Gian-Carlo Pascutto [:gcp]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=9a91f1cfe66b99e02c9d00867e0e5a1b5878c0ee

Comment 13

[Gian-Carlo Pascutto [:gcp]]

Attachment: Bug 1386826 - Whitelist the objdir, so _tests gets covered.

Review commit: https://reviewboard.mozilla.org/r/169670/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/169670/

Comment 14

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8898301 [details]
Bug 1386826 - Whitelist the objdir, so _tests gets covered.

https://reviewboard.mozilla.org/r/169670/#review174890

thanks

Comment 15

[Gian-Carlo Pascutto [:gcp]]

(In reply to Gian-Carlo Pascutto [:gcp] from comment #9)
> It's possible Linux actually only strictly needs the objdir now that it has symlink handling,

From the try pushes (which included a variation that removed srcdir handling) this seem not to be true.

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/40a865254b32
Whitelist the objdir, so _tests gets covered. r=jmaher

Comment 17

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/40a865254b32