1918285 - Perma Gtest gtest TEST-UNEXPECTED-FAIL | gtest | test failed with return code -11 | TinderboxPrint: gtest-gtest<br/><em class="testfail">T-FAIL</em>

Status: RESOLVED FIXED

(Core :: mozglue, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: nerli [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=473932861&repo=mozilla-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/doiw5Yu4QWukkQFjaLAjJg/runs/0/artifacts/public/logs/live_backing.log

---

[task 2024-09-11T23:53:11.181Z] 23:53:11     INFO -  TEST-START | Hashtables/BaseHashtableTest/DefaultConstructible_DefaultConstructible.GetGeneration
[task 2024-09-11T23:53:11.182Z] 23:53:11     INFO -  gtest INFO | gtest | process wait complete, returncode=-11
[task 2024-09-11T23:53:11.182Z] 23:53:11     INFO -  mozcrash checking /builds/worker/workspace/build/tests/gtest for minidumps...
[task 2024-09-11T23:53:11.182Z] 23:53:11  WARNING -  gtest TEST-UNEXPECTED-FAIL | gtest | test failed with return code -11
[task 2024-09-11T23:53:11.182Z] 23:53:11     INFO -  gtest INFO | rungtests.py exits with code 1
[task 2024-09-11T23:53:11.200Z] 23:53:11     INFO - Return code: 1
[task 2024-09-11T23:53:11.200Z] 23:53:11    ERROR - No tests run or test summary not found
[task 2024-09-11T23:53:11.200Z] 23:53:11     INFO - TinderboxPrint: gtest-gtest<br/><em class="testfail">T-FAIL</em>
[task 2024-09-11T23:53:11.200Z] 23:53:11  WARNING - setting return code to 2
[task 2024-09-11T23:53:11.200Z] 23:53:11     INFO - The gtest suite: gtest ran with return status: FAILURE
[task 2024-09-11T23:53:11.201Z] 23:53:11     INFO - Running post-action listener: _package_coverage_data
[task 2024-09-11T23:53:11.201Z] 23:53:11     INFO - Running post-action listener: _resource_record_post_action
[task 2024-09-11T23:53:11.201Z] 23:53:11     INFO - Running post-action listener: process_java_coverage_data
[task 2024-09-11T23:53:11.201Z] 23:53:11     INFO - [mozharness: 2024-09-11 23:53:11.201238Z] Finished run-tests step (success)
[task 2024-09-11T23:53:11.201Z] 23:53:11     INFO - [mozharness: 2024-09-11 23:53:11.201286Z] Running uninstall step.
[task 2024-09-11T23:53:11.201Z] 23:53:11     INFO - Running pre-action listener: _resource_record_pre_action
[task 2024-09-11T23:53:11.201Z] 23:53:11     INFO - Running main action method: uninstall
[task 2024-09-11T23:53:11.201Z] 23:53:11     INFO - Skipping uninstall for non-MSIX test
[task 2024-09-11T23:53:11.201Z] 23:53:11     INFO - Running post-action listener: _resource_record_post_action
[task 2024-09-11T23:53:11.201Z] 23:53:11     INFO - [mozharness: 2024-09-11 23:53:11.201551Z] Finished uninstall step (success)
[task 2024-09-11T23:53:11.201Z] 23:53:11     INFO - Running post-run listener: _resource_record_post_run
[task 2024-09-11T23:53:11.262Z] 23:53:11     INFO - Validating Perfherder data against /builds/worker/workspace/mozharness/external_tools/performance-artifact-schema.json
[task 2024-09-11T23:53:11.264Z] 23:53:11     INFO - PERFHERDER_DATA: {"framework": {"name": "job_resource_usage"}, "suites": [{"name": "gtest.gtest.overall", "extraOptions": ["taskcluster-projects/887720501152/machineTypes/c2-standard-4"], "subtests": [{"name": "cpu_percent", "value": 9.157902298850582}, {"name": "io_write_bytes", "value": 588095488}, {"name": "io.read_bytes", "value": 0}, {"name": "io_write_time", "value": 518272}, {"name": "io_read_time", "value": 0}]}, {"name": "gtest.gtest.start-pulseaudio", "subtests": [{"name": "time", "value": 0.013596271999972487}, {"name": "cpu_percent", "value": 0}]}, {"name": "gtest.gtest.install", "subtests": [{"name": "time", "value": 11.460005616000046}, {"name": "cpu_percent", "value": 25.16969026548673}]}, {"name": "gtest.gtest.stage-files", "subtests": [{"name": "time", "value": 0.1238416379999876}, {"name": "cpu_percent", "value": 0}]}, {"name": "gtest.gtest.run-tests", "subtests": [{"name": "time", "value": 58.07492522000001}, {"name": "cpu_percent", "value": 5.928577586206897}]}, {"name": "gtest.gtest.uninstall", "subtests": [{"name": "time", "value": 0.00018680299996276517}, {"name": "cpu_percent", "value": 0}]}]}
[task 2024-09-11T23:53:11.264Z] 23:53:11     INFO - Total resource usage - Wall time: 70s; CPU: Can't collect data; Read bytes: 0; Write bytes: 588095488; Read time: 0; Write time: 518272
[task 2024-09-11T23:53:11.264Z] 23:53:11     INFO - TinderboxPrint: I/O read bytes / time<br/>0 / 0
[task 2024-09-11T23:53:11.265Z] 23:53:11     INFO - TinderboxPrint: I/O write bytes / time<br/>588,095,488 / 518,272
[task 2024-09-11T23:53:11.265Z] 23:53:11     INFO - TinderboxPrint: CPU idle<br/>248.8 (89.6%)
[task 2024-09-11T23:53:11.265Z] 23:53:11     INFO - TinderboxPrint: CPU iowait<br/>3.3 (1.2%)
[task 2024-09-11T23:53:11.265Z] 23:53:11     INFO - TinderboxPrint: CPU system<br/>6.9 (2.5%)
[task 2024-09-11T23:53:11.266Z] 23:53:11     INFO - TinderboxPrint: CPU user<br/>18.6 (6.7%)
[task 2024-09-11T23:53:11.266Z] 23:53:11     INFO - TinderboxPrint: Swap in / out<br/>0 / 0
[task 2024-09-11T23:53:11.266Z] 23:53:11     INFO - start-pulseaudio - Wall time: 0s; CPU: Can't collect data; Read bytes: 0; Write bytes: 0; Read time: 0; Write time: 0
[task 2024-09-11T23:53:11.266Z] 23:53:11     INFO - install - Wall time: 11s; CPU: 25%; Read bytes: 0; Write bytes: 2207744; Read time: 0; Write time: 196
[task 2024-09-11T23:53:11.266Z] 23:53:11     INFO - stage-files - Wall time: 0s; CPU: Can't collect data; Read bytes: 0; Write bytes: 0; Read time: 0; Write time: 0
[task 2024-09-11T23:53:11.269Z] 23:53:11     INFO - run-tests - Wall time: 58s; CPU: 6%; Read bytes: 0; Write bytes: 585887744; Read time: 0; Write time: 518076
[task 2024-09-11T23:53:11.269Z] 23:53:11     INFO - uninstall - Wall time: 0s; CPU: Can't collect data; Read bytes: 0; Write bytes: 0; Read time: 0; Write time: 0
[task 2024-09-11T23:53:11.392Z] 23:53:11  WARNING - returning nonzero exit status 2
[task 2024-09-11T23:53:11.417Z] cleanup
[task 2024-09-11T23:53:11.417Z] + cleanup
[task 2024-09-11T23:53:11.417Z] + local rv=2
[task 2024-09-11T23:53:11.417Z] + [[ -s /builds/worker/.xsession-errors ]]
[task 2024-09-11T23:53:11.417Z] + cp /builds/worker/.xsession-errors /builds/worker/artifacts/public/xsession-errors.log
[task 2024-09-11T23:53:11.419Z] + '[' ']'
[task 2024-09-11T23:53:11.419Z] + true
[task 2024-09-11T23:53:11.419Z] + cleanup_xvfb
[task 2024-09-11T23:53:11.419Z] ++ pidof Xvfb
[task 2024-09-11T23:53:11.422Z] + local xvfb_pid=50
[task 2024-09-11T23:53:11.422Z] + local vnc=false
[task 2024-09-11T23:53:11.422Z] + local interactive=false
[task 2024-09-11T23:53:11.422Z] + '[' -n 50 ']'
[task 2024-09-11T23:53:11.422Z] + [[ false == false ]]
[task 2024-09-11T23:53:11.422Z] + [[ false == false ]]
[task 2024-09-11T23:53:11.422Z] + kill 50
[task 2024-09-11T23:53:11.422Z] + screen -XS xvfb quit
[task 2024-09-11T23:53:11.443Z] No screen session found.
[task 2024-09-11T23:53:11.443Z] + true
[task 2024-09-11T23:53:11.444Z] + exit 2
[taskcluster 2024-09-11 23:53:11.927Z] === Task Finished ===
[taskcluster 2024-09-11 23:53:12.715Z] Unsuccessful task run with exit code: 2 completed in 321.256 seconds

Comment 1

[Noemi Erli[:noemi_erli]]

This started to permafail on central from this push

Comment 2

[Intermittent Failures Robot]

16 failures in 903 pushes (0.018 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 16

Platform and build breakdown:

• linux1804-64-shippable-qr: 10
  • opt: 10
• linux1804-32-shippable-qr: 6
  • opt: 6

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1918285&startday=2024-09-11&endday=2024-09-11&tree=all

Comment 3

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Backfills identified the change from bug 1917739 as the regressor.

Comment 4

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1917739

Comment 5

[Intermittent Failures Robot]

42 failures in 4448 pushes (0.009 failures/push) were associated with this bug in the last 7 days.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 37
• try: 5

Platform and build breakdown:

• linux1804-32-shippable-qr: 18
  • opt: 18
• linux1804-64-shippable-qr: 24
  • opt: 24

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1918285&startday=2024-09-09&endday=2024-09-15&tree=all

Comment 6

[Mike Hommey [:glandium]]

Attachment: Bug 1918285 - Avoid lz4frame symbols being exported from gtest libxul.

The lz4 symbols were never meant to be exposed, the API in Compression.h
being the official way to use it. This is we had LZ4LIB_VISIBILITY set to
nothing. Unfortunately, that wasn't enough, because there is another
similar define for lz4frame: LZ4FLIB_VISIBILITY.

So we had been exporting those lz4frame symbols from firefox-bin
forever, without noticing, but it didn't cause problems until the
symbols were moved to libxul. With them moved to libxul, we end up
with the situation where we might actually end up using the symbols
from the system liblz4, which is pulled indirectly through other
dependencies (through libsystemd, which comes through libdbus).

This is all fine-ish on a "normal" opt build, but with LTO, things
end up such that some calls go through our copy of lz4frame and others
through the system one, and the discrepancy causes a crash.

The symbols file for non-gtest libxul, that hides all symbols but a few,
was saving the non-gtest case, fortunately.

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/0006f81acc5b
Avoid lz4frame symbols being exported from gtest libxul. r=firefox-build-system-reviewers,sergesanspaille

Comment 8

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/0006f81acc5b