1700763 - Run tests with isolated process on in automation

Status: RESOLVED FIXED

(GeckoView Graveyard :: Sandboxing, task, P1)

Description

[[:owlish] 🦉 PST]

To make sure we don't regress the ones we already got passing, and later to run the whole suite of tests

This needs a new geckoview-junit job similar to what we did for multi-e10s and fission

Comment 1

[[:owlish] 🦉 PST]

Attachment: WIP: Bug 1700763 - New build experiment

Comment 2

[[:owlish] 🦉 PST]

Hey Nick, a question for you as a Build Master: what am I missing, trying to setup a new build variant in automation to run tests on? This is what I have so far: https://phabricator.services.mozilla.com/D113726
And when I try ./mach try fuzzy -q "android", I get this error which I'm not sure how to resolve:

Exception: No test-platform matching 'android-x86_64/debug-sandboxed' nor 'default' found while determining item `mozharness.config` in `mochitest-plain`

Full output: https://paste.mozilla.org/3rf1TWvP

Comment 3

[Nick Alexander :nalexander [he/him]]

(In reply to [:owlish] 🦉 PST from comment #2)

Hey Nick, a question for you as a Build Master: what am I missing, trying to setup a new build variant in automation to run tests on? This is what I have so far: https://phabricator.services.mozilla.com/D113726
And when I try ./mach try fuzzy -q "android", I get this error which I'm not sure how to resolve:

Exception: No test-platform matching 'android-x86_64/debug-sandboxed' nor 'default' found while determining item `mozharness.config` in `mochitest-plain`

Full output: https://paste.mozilla.org/3rf1TWvP

Irene! So sorry to be late replying. The issue is that the platform: ... lines probably should remain as build-platform: android-x86_64/{opt,debug}. It's a hassle to add new TreeHerder platforms; see https://github.com/mozilla/treeherder/blob/master/ui/helpers/constants.js, so don't bother. Good luck!

Comment 4

[[:owlish] 🦉 PST]

Attachment: Bug 1700763 - [Part 1] Add new build type with isolated process on

Comment 5

[[:owlish] 🦉 PST]

Attachment: Bug 1700763 - [Part 2] Adjust test tiers for the isolated process build (make failing ones tier 3)

Depends on D115389

Comment 6

[[:owlish] 🦉 PST]

Attachment: Bug 1700763 - [Part 3] Disable specific junit tests that fail in isolated process build

Depends on D115390

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/998fc8ecd2ae
WIP: Bug 1700763 - [Part 1] Add new build type with isolated process on r=geckoview-reviewers,aklotz

Comment 8

[Natalia Csoregi [:nataliaCs]]

Backed out for causing gecko decision task failure.

backout: https://hg.mozilla.org/integration/autoland/rev/52f11a103c34eb2f765c8576762f38503ddb7055

push: https://treeherder.mozilla.org/jobs?repo=autoland&selectedTaskRun=QT8vsTszRC63svquab-A4w.0&revision=998fc8ecd2ae2dbb4753bd4089006c8c8ae0c44b

failure log: https://treeherder.mozilla.org/logviewer?job_id=340757593&repo=autoland&lineNumber=3336

[task 2021-05-25T17:48:17.087Z] Generating tasks for test jittest on platform android-hw-p2-8-0-android-aarch64/debug
[task 2021-05-25T17:48:29.874Z] Error loading tasks for kind test:
[task 2021-05-25T17:48:29.875Z] PERFHERDER_DATA: {"framework": {"name": "build_metrics"}, "suites": [{"name": "bugbug_push_schedules_time", "value": 0.3562767050002549, "lowerIsBetter": true, "shouldAlert": false, "subtests": []}, {"name": "bugbug_push_schedules_retries", "value": 0, "lowerIsBetter": true, "shouldAlert": false, "subtests": []}]}
[task 2021-05-25T17:48:29.875Z] Traceback (most recent call last):
[task 2021-05-25T17:48:29.875Z] File "/builds/worker/checkouts/gecko/taskcluster/mach_commands.py", line 354, in taskgraph_decision
[task 2021-05-25T17:48:29.875Z] ret = taskgraph.decision.taskgraph_decision(options)
[task 2021-05-25T17:48:29.875Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/decision.py", line 238, in taskgraph_decision
[task 2021-05-25T17:48:29.875Z] full_task_json = tgg.full_task_graph.to_json()
[task 2021-05-25T17:48:29.875Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/generator.py", line 174, in full_task_graph
[task 2021-05-25T17:48:29.875Z] return self._run_until("full_task_graph")
[task 2021-05-25T17:48:29.875Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/generator.py", line 435, in _run_until
[task 2021-05-25T17:48:29.875Z] k, v = next(self._run)
[task 2021-05-25T17:48:29.875Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/generator.py", line 312, in _run
[task 2021-05-25T17:48:29.875Z] self._write_artifacts,
[task 2021-05-25T17:48:29.875Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/generator.py", line 92, in load_tasks
[task 2021-05-25T17:48:29.875Z] for task_dict in transforms(trans_config, inputs)
[task 2021-05-25T17:48:29.875Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/generator.py", line 80, in <listcomp>
[task 2021-05-25T17:48:29.875Z] Task(
[task 2021-05-25T17:48:29.875Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/task.py", line 2112, in check_run_task_caches
[task 2021-05-25T17:48:29.875Z] for task in tasks:
[task 2021-05-25T17:48:29.875Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/task.py", line 2020, in check_task_dependencies
[task 2021-05-25T17:48:29.875Z] for task in tasks:
[task 2021-05-25T17:48:29.875Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/task.py", line 2006, in check_task_identifiers
[task 2021-05-25T17:48:29.875Z] for task in tasks:
[task 2021-05-25T17:48:29.875Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/task.py", line 1987, in chain_of_trust
[task 2021-05-25T17:48:29.875Z] for task in tasks:
[task 2021-05-25T17:48:29.875Z] File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/task.py", line 1821, in build_task
[task 2021-05-25T17:48:29.875Z] raise Exception(UNKNOWN_GROUP_NAME.format(groupSymbol, path))
[task 2021-05-25T17:48:29.875Z] Exception: Treeherder group W-1proc (from taskcluster/ci/test/web-platform.yml) has no name; add it to taskcluster/ci/config.yml
[taskcluster 2021-05-25 17:48:30.887Z] === Task Finished ===
[taskcluster 2021-05-25 17:48:39.827Z] Unsuccessful task run with exit code: 1 completed in 89.708 seconds

Comment 9

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/treeherder/pull/7156

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/570a1c32aff5
[Part 1] Add new build type with isolated process on r=geckoview-reviewers,aklotz

Comment 11

[Atila Butkovits]

Backed out for causing Gecko Decision Task bustage.

Backout link: https://hg.mozilla.org/integration/autoland/rev/1968536bb6682c56934d7b85dc8c6e67587c15ae

Push with failures: https://treeherder.mozilla.org/jobs?repo=autoland&selectedTaskRun=cdapi85KSaexvZC-IcY3rg.0&revision=570a1c32aff511f2414c215f428f9556ac648a7e&searchStr=gecko%2Cdecision%2Ctask%2Copt%2Cgecko%2Cdecision%2Ctask%2Cd

Failure log: https://treeherder.mozilla.org/logviewer?job_id=340988785&repo=autoland&lineNumber=3329

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/5e3b8ef79886
[Part 1] Add new build type with isolated process on r=geckoview-reviewers,aklotz

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ca1eb7f9f89e
[Part 2] Adjust test tiers for the isolated process build (make failing ones tier 3) r=aklotz,jmaher,nalexander

Comment 14

[[:owlish] 🦉 PST]

Attachment: Bug 1700763 - fix linting error

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/0c758d163326
[Part 3] Disable specific junit tests that fail in isolated process build r=geckoview-reviewers,aklotz

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/05c958569e6d
Shorten the lines to fix linting error r=jmaher

Comment 17

[[:owlish] 🦉 PST]

Attachment: Bug 1700763 - Disable intermittent failure in isolated process builds

Comment 18

[Phabricator Automation]

Comment on attachment 9224018 [details]
Bug 1700763 - Disable intermittent failure in isolated process builds

Revision D116249 was moved to bug 1713237. Setting attachment 9224018 [details] to obsolete.

Comment 19

[Alexandru Michis [:malexandru]]

https://hg.mozilla.org/mozilla-central/rev/5e3b8ef79886
https://hg.mozilla.org/mozilla-central/rev/ca1eb7f9f89e
https://hg.mozilla.org/mozilla-central/rev/0c758d163326
https://hg.mozilla.org/mozilla-central/rev/05c958569e6d

Comment 20

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Many tasks of the isolated config are failing frequently or permanently.

1. Is there a bug which tracks disabling the failing tests?
2. Should these "tier 3" tasks run on autoland?

Comment 21

[[:owlish] 🦉 PST]

(In reply to Sebastian Hengst [:aryx] (needinfo on intermittent or backout) from comment #20)

1. Is there a bug which tracks disabling the failing tests?

There is a series of bugs that tracks fixing those tests so that they don't fail and then making them tier1, like in our other builds. They won't get worked on for a couple of months, however.

2. Should these "tier 3" tasks run on autoland?

No, I'll turn them off for autoland

Comment 22

[Chris Peterson [:cpeterson]]

Moving isolated process bugs to the new GeckoView::Sandboxing component.