Closed Bug 1408403 Opened 7 years ago Closed 7 years ago

Permaorange CCov PROCESS-CRASH | tps | application crashed [@ js::jit::ReleaseProcessExecutableMemory]

Categories

(Testing :: Talos, defect, P5)

Version 3
defect

Tracking

(firefox-esr52 unaffected, firefox56 unaffected, firefox57 unaffected, firefox58 fixed)

RESOLVED FIXED
mozilla58
Tracking Status
firefox-esr52 --- unaffected
firefox56 --- unaffected
firefox57 --- unaffected
firefox58 --- fixed

People

(Reporter: intermittent-bug-filer, Assigned: jmaher)

References

Details

(Keywords: crash, intermittent-failure, Whiteboard: [stockwell disabled])

Crash Data

Attachments

(1 file, 1 obsolete file)

Summary: Intermittent PROCESS-CRASH | tps | application crashed [@ js::jit::ReleaseProcessExecutableMemory] → Permaorange CCov PROCESS-CRASH | tps | application crashed [@ js::jit::ReleaseProcessExecutableMemory]
ideally this will be fixed in bug 1407593.
See Also: → 1407593
Whiteboard: [stockwell needswork]
Attached patch disable g2 job on ccov builds (obsolete) — Splinter Review
Attachment #8920545 - Flags: review?(rwood)
Attachment #8920545 - Flags: review?(rwood) → review+
Pushed by jmaher@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/b8a85f1cb39e
disable g2 for linux64-ccov builds. r=rwood
https://hg.mozilla.org/mozilla-central/rev/b8a85f1cb39e
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla58
Reopening cause these fails keep appearing even after the fix:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-searchStr=7960b9ffca4bb0c8a47f589b03beb93a2d9047c3&selectedJob=138658711
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Waiting progress on bug 1407593
See Also: → 1411216
There are 46 failures related to this bug for the last 7 days.

Still waiting on updates on bug 1407593.
:rwood is there anything else we can do regarding this?
Flags: needinfo?(rwood)
Hmm, for some reason disabling g2 on linux x64 ccov didn't work:

https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=a9131757dfdd4f2820c397523b54386f9675d607&filter-searchStr=talos

:jmaher, any idea why the fix in comment 5 didn't take? I don't see anything obvious
Flags: needinfo?(rwood) → needinfo?(jmaher)
odd, I had tried to disable the g2 job from running on mozilla-central, but it failed:
http://searchfox.org/mozilla-central/source/taskcluster/ci/test/talos.yml#102

:dustin, do you know why this would continue to run on mozilla-central after doing some logic to restrict it to try only?
Flags: needinfo?(jmaher) → needinfo?(dustin)
Looking at the decision task, I see the task label (test-linux64-ccov/opt-talos-g2-e10s) in https://public-artifacts.taskcluster.net/HGmBpJnOSS6dtLCE3edLGQ/0/public/target-tasks.json, so it is being selected as a target task.  Looking in full-task-graph.json, I see

  "test-linux64-ccov/opt-talos-g2-e10s": {
    "attributes": {
      "build_platform": "linux64-ccov",
      "build_type": "opt",
      "e10s": true,
      "kind": "test",
      "run_on_projects": [
        "mozilla-central"
      ],
      "talos_try_name": "g2-e10s",
      "test_chunk": "1",
      "test_platform": "linux64-ccov/opt",
      "unittest_flavor": "talos",
      "unittest_suite": "talos"
    },

so something is setting run_on_projects differently.  Probably a transform.  Looking for `mozilla-taskcluster` in taskcluster/taskgraph/transforms/test.py shows

@transforms.add
def enable_code_coverage(config, tests):
    """Enable code coverage for the linux64-ccov/opt & linux64-jsdcov/opt build-platforms"""
    for test in tests:
        if test['build-platform'] == 'linux64-ccov/opt' and \
                not test['test-name'].startswith('test-verify'):
            test['mozharness'].setdefault('extra-options', []).append('--code-coverage')
            test['when'] = {}
            test['instance-size'] = 'xlarge'
            test['run-on-projects'] = ['mozilla-central']

            if test['test-name'].startswith('talos'):
                test['max-run-time'] = 7200
                test['docker-image'] = {"in-tree": "desktop1604-test"}
                test['mozharness']['config'] = ['talos/linux64_config_taskcluster.py']
                test['mozharness']['extra-options'].append('--add-option')
                test['mozharness']['extra-options'].append('--cycles,1')
                test['mozharness']['extra-options'].append('--add-option')
                test['mozharness']['extra-options'].append('--tppagecycles,1')
                test['mozharness']['extra-options'].append('--add-option')
                test['mozharness']['extra-options'].append('--no-upload-results')
                test['mozharness']['extra-options'].append('--add-option')
                test['mozharness']['extra-options'].append('--tptimeout,15000')
        elif test['build-platform'] == 'linux64-jsdcov/opt':
            test['run-on-projects'] = ['mozilla-central']
        yield test

so it looks like that's why!  It might be better for that transform to either only set run-on-projects if it's not already set to something, or to error out if it is set to sometihng incorrect.
Flags: needinfo?(dustin)
Assignee: nobody → jmaher
Attachment #8920545 - Attachment is obsolete: true
Attachment #8926917 - Flags: review?(dustin)
Attachment #8926917 - Flags: review?(dustin) → review+
Pushed by jmaher@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/ab2ece0e5ef7
disable talos-g2-e10s on ccov builds. r=dustin
https://hg.mozilla.org/mozilla-central/rev/ab2ece0e5ef7
https://hg.mozilla.org/mozilla-central/rev/af1af3c260f5
Status: REOPENED → RESOLVED
Closed: 7 years ago7 years ago
Resolution: --- → FIXED
Whiteboard: [stockwell needswork] → [stockwell disabled]
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: