1896221 - stop setting MOZHARNESS_TEST_PATHS as in some cases it ends up with a very long string and causes xsession to segfault

Status: RESOLVED FIXED

(Testing :: General, task)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

in researching the cause for backing out bug 1895092, I found that we set MOZHARNESS_TEST_PATHS to a very large (11K+ characters) string. This ends up causing xsession to segfault. Setting it to <8K characters allows things to progress normally.

when xsession fails, we lose out on a window manager, dbus, and pulseaudio- basically a very limited window manager if not any at all. It appears that the memory for a command includes ALL env vars and the whole command line, so moving data from env vars -> command line is not ideal. Although right now we define the values in the env var and then send them on the command line (mozharness adds the manifests to the command line), so changing this to only have one copy would work.

This doesn't scale long term, and different OS environments could put pressures, I would rather solve this a more definitive way.

To solve this, we can use manifests-by-task.json.gz (which is an existing artifact from the decision task), this artifact is basically:
{'task_label': [manifest_path, ...], ...}

Here we would pass into the harness a new cli flag manifest-file=manifests-by-task.json where the file would be downloaded and made available. What we need is the task label, then we can gain access to the list of paths.

A task label isn't easily available, here I propose adding it as an environment variable TASKCLUSTER_LABEL, this should suffice so we can gather the label and list of tests which would then replace the need for the MOZHARNESS_TEST_PATHS

:gbrown, :ahal- do either of you have thoughts on this or concerns?

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

the only thing I don't like is this would require changes to every harness

Comment 2

[Geoff Brown [:gbrown]]

As a possible alternative, could you "shield" Xsession from MOZHARNESS_TEST_PATHS with something like:

(
    unset MOZHARNESS_TEST_PATHS
    /etc/X11/Xsession 2>&1 &
)

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

we could only do that once the python harness started and we read the values of MOZHARNESS_TEST_PATHS (which is mozharness :: desktop_unittest.py). xsession is failing at test_linux.sh, which is prior to calling mozharness.

This does bring up a point- there might be ways to help self recover.

Comment 4

[Geoff Brown [:gbrown]]

I meant, use the code block in comment 2 in test_linux.sh: Then MOZHARNESS_TEST_PATHS would not be set in Xsession's environment, but would continue to be available in the rest of test_linux.sh (so it would still be available to call mozharness). ...I think!

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: Bug 1896221 - unset MOZHARNESS_TEST_PATHS before calling xsession. r=gbrown

Comment 6

[Andrew Halberstadt [:ahal]]

Good find! Sounds like we have a workaround, though fwiw I like the original suggested approach better than stuffing everything into an env.

What we need is the task label, then we can gain access to the list of paths.

I assume you mean task id? In Taskgraph, we have access to the Decision task id when generating tasks, so we can simply forward that down as an environment variable to the test tasks. Then something (perhaps mozharness) can handle downloading the artifact.

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/35a601e0655a
unset MOZHARNESS_TEST_PATHS before calling xsession. r=gbrown DONTBUILD

Comment 8

[Serban Stanca [:SerbanS]]

https://hg.mozilla.org/mozilla-central/rev/35a601e0655a