1790872 - Report telemetry about IPC messages sent/received

Assignee

Description

•

3 years ago

Ideally this telemetry would also include information about whether the process is a foreground or background content process, so that we can identify messages which are waking up background processes in the wild.

This would be useful as part of the ongoing power usage reduction work, as it appears that a significant percentage of thread wake-ups are currently on the IPC I/O thread, and insight into what those messages are could potentially allow us to make future power usage optimizations to our IPC layer and reduce thread wakeups.

Nika Layzell [:nika] (ni? for response)

Assignee

Comment 1

•

3 years ago

After some discussion with :chutten, it seems like perhaps glean wouldn't be the best fit for this telemetry. In order to use glean, we'd currently need to enumerate every IPC message in a metrics.yaml file, which would be difficult, as we can't know the full set of IPC messages until we've run the IPDL compiler. :chutten linked me bug 1657473 as a future change to glean which may make this work easier.

It may also be possible to use legacy telemetry, as we could use keyed histograms with string keys based on message types, which we already have readily available in the IPC code.

Chris H-C :chutten

Updated

•

3 years ago

Comment 2

•

3 years ago

Thanks for filing this Nika! I almost started on this a few months ago. Bug 1772163 happened after some discussion with :chutten, when I was preparing to collect this data.

(In reply to Nika Layzell [:nika] (ni? for response) from comment #1)

After some discussion with :chutten, it seems like perhaps glean wouldn't be the best fit for this telemetry. In order to use glean, we'd currently need to enumerate every IPC message in a metrics.yaml file, which would be difficult, as we can't know the full set of IPC messages until we've run the IPDL compiler.

The idea I had was to generate a list of IPC messages once and put that list in the metrics.yaml file. The list will get out of date relatively quickly, but that shouldn't cause significant issues. We would have a slowly increasing percentage of IPCs in the other bucket. Once that reaches a high enough value, we can update the list.

This is of course only a temporary solution until something better (probably the bug :chutten mentioned) could be implemented, but it's good enough to give us a quick sense of where most of the IPC traffic goes.

I used the same approach to instrument the names of threads in bug 1763474, I just built a list and had to accept it would get out of date. In both cases, I expect the busiest threads and the most frequent IPC messages to remain the same for a long while.

Nika Layzell [:nika] (ni? for response)

Assignee

Comment 3

•

3 years ago

Attached file data-review — Details

Attachment #9295627 - Flags: data-review?(chutten)

Nika Layzell [:nika] (ni? for response)

Assignee

Comment 4

•

3 years ago

Attached file Bug 1790872 - Part 1: Report glean telemetry about which IPC messages are being sent, r=florian!,#ipc-reviewers! (obsolete) — Details

This will provide us with more information with which to discover what
IPC messages are being sent extremely frequently, especially from
background processes, and what we can do to avoid unnecessary wakeups.

This is intended for power usage work, and is being tracked in a similar
way to the thread wakeups work, based on process types, such that the
data can be compared between the two.

Roger Yang [:royang]

Updated

•

3 years ago

Attachment #9295627 - Flags: data-review?(chutten) → data-review?(royang)

Roger Yang [:royang]

Comment 5

•

3 years ago

Comment on attachment 9295627 [details]
data-review

DATA COLLECTION REVIEW RESPONSE:

Is there or will there be documentation that describes the schema for the ultimate data set in a public, complete, and accurate way?

Yes

Is there a control mechanism that allows the user to turn the data collection on and off?

Yes, These collections are Glean. The opt-out can be found in the product's preferences.

If the request is for permanent data collection, is there someone who will monitor the data over time?

nika@mozilla.com will be responsible for the permanent collections.

Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 1, Technical

Is the data collection request for default-on or default-off?

The collection will be default-on in Nightly

Does the instrumentation include the addition of any new identifiers?

No

Is the data collection covered by the existing Firefox privacy notice?

Yes

Does the data collection use a third-party collection tool?

No

Result: data-review+

Attachment #9295627 - Flags: data-review?(royang) → data-review+

Phabricator Automation

Updated

•

3 years ago

Attachment #9295628 - Attachment description: Bug 1790872 - Report glean telemetry about which IPC messages are being sent, r=florian!,#ipc-reviewers! → Bug 1790872 - Part 1: Report glean telemetry about which IPC messages are being sent, r=florian!,#ipc-reviewers!

Nika Layzell [:nika] (ni? for response)

Assignee

Comment 6

•

3 years ago

Attached file Bug 1790872 - Part 2: Make the process mode available to IPDL gtests, r=#ipc-reviewers (obsolete) — Details

This will be useful in the next part to write a test with different behaviour
depending on whether it's cross-process.

Depends on D157855

Nika Layzell [:nika] (ni? for response)

Assignee

Comment 7

•

3 years ago

Attached file Bug 1790872 - Part 3: Add a unit test for the IPC sent/recvd message telemetry, r=#ipc-reviewers (obsolete) — Details

This is done using the gtest framework, and adds a new protocol which has its
messages counted. As the IPDLUnitTest process doesn't count IPC messages with
glean, this only counts messages on the parent process side.

This test also checks that we don't count this information outside of nightly,
and that it isn't counted for messages sent within the same process.

Depends on D158274

Pulsebot

Comment 8

•

3 years ago

Pushed by nlayzell@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/3aac3b7cbe18 Part 1: Report glean telemetry about which IPC messages are being sent, r=florian,ipc-reviewers,mccr8 https://hg.mozilla.org/integration/autoland/rev/500681c9ebc6 Part 2: Make the process mode available to IPDL gtests, r=ipc-reviewers,mccr8 https://hg.mozilla.org/integration/autoland/rev/b88674165106 Part 3: Add a unit test for the IPC sent/recvd message telemetry, r=ipc-reviewers,mccr8

Atila Butkovits

Comment 9

•

3 years ago

Backed out for causing gtest failures.

Backout link: https://hg.mozilla.org/integration/autoland/rev/4421458d07c5441e2bfef79d5e2012f61c64c919

Push with failures: https://treeherder.mozilla.org/jobs?repo=autoland&selectedTaskRun=OiJNS0o8SUu4kRuQwFTU3w.0&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel&revision=b886741651069bf877c5c2b186bdba25bc2d1203

Failure log:
https://treeherder.mozilla.org/logviewer?job_id=391857726&repo=autoland&lineNumber=12373
https://treeherder.mozilla.org/logviewer?job_id=391858541&repo=autoland&lineNumber=8926

Flags: needinfo?(nika)

Nika Layzell [:nika] (ni? for response)

Assignee

Updated

•

3 years ago

Flags: needinfo?(nika)

Sandor Molnar[:smolnar]

Comment 10

•

3 years ago

Backed out for causing browser-chrome failures in toolkit/components/processtools/tests/browser/browser_test_powerMetrics.js

Backout link: https://hg.mozilla.org/integration/autoland/rev/8558bf806b95cba4ee8f25b1ce40ad23e797edda

Push with failures

Failure log

INFO - TEST-PASS | toolkit/components/processtools/tests/browser/browser_test_powerMetrics.js | The sum of CPU time used by all process types should match totalCpuTimeMs - 6497 == 6497 - 
[task 2022-10-03T21:04:10.947Z] 21:04:10     INFO - Buffered messages finished
[task 2022-10-03T21:04:10.948Z] 21:04:10     INFO - TEST-UNEXPECTED-FAIL | toolkit/components/processtools/tests/browser/browser_test_powerMetrics.js | reported parent cpu time should be at least what the first requestProcInfo returned - 3016 >= 4484 - JS frame :: chrome://mochitests/content/browser/toolkit/components/processtools/tests/browser/browser_test_powerMetrics.js :: <TOP_LEVEL> :: line 148

Flags: needinfo?(nika)

Pulsebot

Comment 11

•

3 years ago

Pushed by smolnar@mozilla.com: https://hg.mozilla.org/mozilla-central/rev/8e79c5a0eaf2 Part 1: Report glean telemetry about which IPC messages are being sent, r=florian,ipc-reviewers,mccr8 https://hg.mozilla.org/mozilla-central/rev/37335775c392 Part 2: Make the process mode available to IPDL gtests, r=ipc-reviewers,mccr8

Nika Layzell [:nika] (ni? for response)

Assignee

Comment 12

•

3 years ago

It appears this is caused by overloading the pre-init queue with IPC message telemetry, causing us to drop the power metrics telemetry during startup, making it effectively another instance of bug 1764549.

Flags: needinfo?(nika)

Chris H-C :chutten

Comment 13

•

3 years ago

Work is ongoing in bug 1780035 to investigate what to do with this on the Glean side.

Comment 14

•

3 years ago

There are some r+ patches which didn't land and no activity in this bug for 2 weeks.
:nika, could you have a look please?
If you still have some work to do, you can add an action "Plan Changes" in Phabricator.
For more information, please visit auto_nag documentation.

Flags: needinfo?(nika)

Flags: needinfo?(continuation)

Andrew McCreight [:mccr8]

Updated

•

3 years ago

Flags: needinfo?(continuation)

Nika Layzell [:nika] (ni? for response)

Assignee

Comment 15

•

3 years ago

Still waiting on glean work in bug 1780035

Flags: needinfo?(nika)

Chris H-C :chutten

Updated

•

3 years ago

Updated

•

3 years ago

Depends on: 1797363

Chris H-C :chutten

Comment 16

•

3 years ago

Preinit queue overflows are mostly gone thanks to the bump to 10^6 tasks. Should be clear to try this one again.

Pulsebot

Comment 17

•

3 years ago

Pushed by nlayzell@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/575d2e57ab31 Part 1: Report glean telemetry about which IPC messages are being sent, r=florian,ipc-reviewers,mccr8 https://hg.mozilla.org/integration/autoland/rev/c7a5a591b885 Part 2: Make the process mode available to IPDL gtests, r=ipc-reviewers,mccr8 https://hg.mozilla.org/integration/autoland/rev/4f712ee19854 Part 3: Add a unit test for the IPC sent/recvd message telemetry, r=ipc-reviewers,mccr8

Iulian Moraru

Comment 18

•

3 years ago

Backed out for causing build bustages with this line of failure Automation Error: mozprocess timed out after 2400 seconds running ['/builds/worker/workspace/obj-build/_virtualenvs/common/bin/python', 'mach', '--log-no-times', 'build', '-v']

The only affected platform, so far, is Linux x64 debug.

Push with failures

Failure log

Backout link

[task 2022-11-04T21:30:39.833Z] 21:30:39     INFO -  gmake[4]: Entering directory '/builds/worker/workspace/obj-build/gfx/thebes'
[task 2022-11-04T21:30:39.838Z] 21:30:39     INFO -  /builds/worker/fetches/clang/bin/clang++ --sysroot /builds/worker/fetches/sysroot-x86_64-linux-gnu -Qunused-arguments -std=gnu++17 -o Unified_cpp_gfx_thebes1.o -c  -I/builds/worker/workspace/obj-build/dist/stl_wrappers -I/builds/worker/workspace/obj-build/dist/system_wrappers -include /builds/worker/checkouts/gecko/config/gcc_hidden.h -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -fstack-clash-protection -ftrivial-auto-var-init=pattern -DDEBUG=1 -DOS_POSIX=1 -DOS_LINUX=1 -DMOZ_ENABLE_FREETYPE -DGRAPHITE2_STATIC -DMOZ_HAS_MOZGLUE -DMOZILLA_INTERNAL_API -DIMPL_LIBXUL -DSTATIC_EXPORTABLE_JS_API -I/builds/worker/checkouts/gecko/gfx/thebes -I/builds/worker/workspace/obj-build/gfx/thebes -I/builds/worker/workspace/obj-build/ipc/ipdl/_ipdlheaders -I/builds/worker/checkouts/gecko/ipc/chromium/src -I/builds/worker/workspace/obj-build/security/rlbox -I/builds/worker/checkouts/gecko/dom/base -I/builds/worker/checkouts/gecko/dom/media/platforms/apple -I/builds/worker/checkouts/gecko/dom/xml -I/builds/worker/checkouts/gecko/gfx/cairo/cairo/src -I/builds/worker/checkouts/gecko/widget/gtk -I/builds/worker/checkouts/gecko/gfx/skia -I/builds/worker/checkouts/gecko/gfx/skia/skia -I/builds/worker/workspace/obj-build/dist/include -I/builds/worker/workspace/obj-build/dist/include/nspr -I/builds/worker/workspace/obj-build/dist/include/nss -DMOZILLA_CLIENT -include /builds/worker/workspace/obj-build/mozilla-config.h -fno-sized-deallocation -fno-aligned-new -fno-exceptions -fPIC -fno-rtti -fno-exceptions -fno-math-errno -pthread -pipe -gdwarf-4 -Os -fno-omit-frame-pointer -funwind-tables -Werror -Wall -Wbitfield-enum-conversion -Wdeprecated-this-capture -Wempty-body -Wformat-type-confusion -Wignored-qualifiers -Wpointer-arith -Wshadow-field-in-constructor-modified -Wsign-compare -Wtype-limits -Wno-error=tautological-type-limit-compare -Wunreachable-code -Wunreachable-code-return -Wunused-but-set-parameter -Wno-invalid-offsetof -Wclass-varargs -Wempty-init-stmt -Wfloat-overflow-conversion -Wfloat-zero-conversion -Wloop-analysis -Wno-range-loop-analysis -Wc++2a-compat -Wenum-compare-conditional -Wenum-float-conversion -Wno-ambiguous-reversed-operator -Wno-error=deprecated -Wno-error=deprecated-anon-enum-enum-conversion -Wno-error=deprecated-enum-enum-conversion -Wno-error=deprecated-pragma -Wno-error=deprecated-this-capture -Wno-error=deprecated-volatile -Wcomma -Wimplicit-fallthrough -Wstring-conversion -Wno-inline-new-delete -Wno-error=deprecated-declarations -Wno-error=array-bounds -Wno-error=free-nonheap-object -Wno-error=atomic-alignment -Wno-error=deprecated-builtins -Wformat -Wformat-security -Wno-psabi -Wthread-safety -Wno-unknown-warning-option -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/include/gtk-3.0 -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/include/pango-1.0 -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/include/glib-2.0 -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/lib/x86_64-linux-gnu/glib-2.0/include -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/include -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/include/cairo -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/include/pixman-1 -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/include/freetype2 -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/include/libpng12 -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/include/gdk-pixbuf-2.0 -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/include/gio-unix-2.0/ -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/include/harfbuzz -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/include/atk-1.0 -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/include/at-spi2-atk/2.0 -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/include/at-spi-2.0 -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/include/dbus-1.0 -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/lib/x86_64-linux-gnu/dbus-1.0/include -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/include/gtk-3.0/unix-print -pthread -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/include/pango-1.0 -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/include/glib-2.0 -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/lib/x86_64-linux-gnu/glib-2.0/include -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/include -pthread -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/include/gtk-3.0 -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/include/pango-1.0 -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/include/glib-2.0 -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/lib/x86_64-linux-gnu/glib-2.0/include -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/include -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/include/cairo -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/include/pixman-1 -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/include/freetype2 -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/include/libpng12 -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/include/gdk-pixbuf-2.0 -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/include/gio-unix-2.0/ -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/include/harfbuzz -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/include/atk-1.0 -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/include/at-spi2-atk/2.0 -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/include/at-spi-2.0 -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/include/dbus-1.0 -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/lib/x86_64-linux-gnu/dbus-1.0/include -pthread -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/include/libdrm -Werror=switch -fno-strict-aliasing -ffp-contract=off  -MD -MP -MF .deps/Unified_cpp_gfx_thebes1.o.pp   Unified_cpp_gfx_thebes1.cpp
[task 2022-11-04T21:30:39.838Z] 21:30:39     INFO -  gmake[4]: Leaving directory '/builds/worker/workspace/obj-build/gfx/thebes'
[task 2022-11-04T22:10:39.865Z] 22:10:39     INFO - Automation Error: mozprocess timed out after 2400 seconds running ['/builds/worker/workspace/obj-build/_virtualenvs/common/bin/python', 'mach', '--log-no-times', 'build', '-v']
[task 2022-11-04T22:10:39.866Z] 22:10:39    ERROR - timed out after 2400 seconds of no output
[task 2022-11-04T22:10:39.866Z] 22:10:39    ERROR - Return code: -15
[task 2022-11-04T22:10:39.866Z] 22:10:39  WARNING - setting return code to 2
[task 2022-11-04T22:10:39.866Z] 22:10:39    FATAL - 'mach build -v' did not run successfully. Please check log for errors.
[task 2022-11-04T22:10:39.866Z] 22:10:39    FATAL - Running post_fatal callback...
[task 2022-11-04T22:10:39.866Z] 22:10:39    FATAL - Exiting -1
[task 2022-11-04T22:10:39.866Z] 22:10:39     INFO - [mozharness: 2022-11-04 22:10:39.866308Z] Finished build step (failed)
[task 2022-11-04T22:10:39.866Z] 22:10:39     INFO - Running post-run listener: _parse_build_tests_ccov
[task 2022-11-04T22:10:39.866Z] 22:10:39     INFO - Running command: ['/builds/worker/workspace/obj-build/_virtualenvs/common/bin/python', 'mach', 'python', 'testing/parse_build_tests_ccov.py'] in /builds/worker/checkouts/gecko
[task 2022-11-04T22:10:39.866Z] 22:10:39     INFO - Copy/paste: /builds/worker/workspace/obj-build/_virtualenvs/common/bin/python mach python testing/parse_build_tests_ccov.py
[task 2022-11-04T22:10:39.867Z] 22:10:39     INFO - Using env: {'ACCEPTED_MAR_CHANNEL_IDS': 'firefox-mozilla-central',
[task 2022-11-04T22:10:39.867Z] 22:10:39     INFO -  'DEBIAN_FRONTEND': 'noninteractive',
[task 2022-11-04T22:10:39.867Z] 22:10:39     INFO -  'DISPLAY': ':2',
[task 2022-11-04T22:10:39.867Z] 22:10:39     INFO -  'EXTRA_MOZHARNESS_CONFIG': '{"disable_package_metrics": true, '
[task 2022-11-04T22:10:39.867Z] 22:10:39     INFO -                             '"mozconfig_variant": "plain-debug", "objdir": '
[task 2022-11-04T22:10:39.867Z] 22:10:39     INFO -                             '"obj-build"}',
[task 2022-11-04T22:10:39.867Z] 22:10:39     INFO -  'GECKO_BASE_REPOSITORY': 'https://hg.mozilla.org/mozilla-unified',
[task 2022-11-04T22:10:39.867Z] 22:10:39     INFO -  'GECKO_HEAD_REPOSITORY': 'https://hg.mozilla.org/integration/autoland',
[task 2022-11-04T22:10:39.867Z] 22:10:39     INFO -  'GECKO_HEAD_REV': '4f712ee198549ced74ccb3c256aec9a30305f500',
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -  'GECKO_PATH': '/builds/worker/checkouts/gecko',
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -  'GRCOV_PATH': '/builds/worker/fetches/grcov',
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -  'HG_SHARE_BASE_DIR': '/builds/hg-shared',
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -  'HG_STORE_PATH': '/builds/worker/checkouts/hg-store',
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -  'HOME': '/builds/worker',
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -  'HOSTNAME': 'taskcluster-worker',
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -  'LC_ALL': 'C',
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -  'LC_CTYPE': 'C.UTF-8',
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -  'LOGNAME': 'worker',
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -  'MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE': 'system',
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -  'MACH_MAIN_PID': '31',
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -  'MACH_SYSTEM_ASSERTED_COMPATIBLE_WITH_COMMON_SITE': '1',
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -  'MACH_SYSTEM_ASSERTED_COMPATIBLE_WITH_MACH_SITE': '1',
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -  'MAR_CHANNEL_ID': 'firefox-mozilla-central',
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -  'MH_BRANCH': 'autoland',
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -  'MH_BUILD_POOL': 'taskcluster',
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -  'MOZBUILD_STATE_PATH': '/builds/worker/.mozbuild',
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -  'MOZHARNESS_ACTIONS': 'build',
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -  'MOZHARNESS_CONFIG': 'builds/releng_base_firefox.py '
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -                       'builds/releng_base_linux_64_builds.py',
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -  'MOZHARNESS_SCRIPT': 'mozharness/scripts/fx_desktop_build.py',
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -  'MOZ_AUTOMATION': '1',
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -  'MOZ_BUILD_DATE': '20221104170532',
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -  'MOZ_CRASHREPORTER_NO_REPORT': '1',
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -  'MOZ_FETCHES': '[{"artifact": "public/build/binutils.tar.zst", "extract": '
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -                 'true, "task": "B6a2dApVS_i8EYT4AmLyIw"}, {"artifact": '
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -                 '"public/build/cbindgen.tar.zst", "extract": true, "task": '
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -                 '"P85YOslvQ1mIl8yTSR7bxQ"}, {"artifact": '
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -                 '"public/build/clang.tar.zst", "extract": true, "task": '
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -                 '"cQ80kb1eSpOF-ipi5N39ag"}, {"artifact": '
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -                 '"public/build/dump_syms.tar.zst", "extract": true, "task": '
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -                 '"JTBjqzf9Su6AjxCwQYMZeg"}, {"artifact": '
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -                 '"public/build/nasm.tar.zst", "extract": true, "task": '
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -                 '"MbpriPLdQtqg6dNIz3-DRQ"}, {"artifact": '
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -                 '"public/build/node.tar.zst", "extract": true, "task": '
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -                 '"E-oYJkvsQt-tXX6EavJT8Q"}, {"artifact": '
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -                 '"public/build/pkgconf.tar.zst", "extract": true, "task": '
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -                 '"eV25uGFpRqaN-85X0plV0g"}, {"artifact": '
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -                 '"public/build/rustc.tar.zst", "extract": true, "task": '
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -                 '"SuK-d4hOTMiC4ZqZqCMs2g"}, {"artifact": '
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -                 '"public/build/sysroot-wasm32-wasi.tar.zst", "extract": true, '
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -                 '"task": "UpYz0AS3ReKIAmvO78i_iw"}, {"artifact": '
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -                 '"public/build/sysroot-x86_64-linux-gnu.tar.zst", "extract": '
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -                 'true, "task": "ZBAxiavCT_OQC3U8JZIBvQ"}]',
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -  'MOZ_FETCHES_DIR': '/builds/worker/fetches',
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -  'MOZ_OBJDIR': '/builds/worker/workspace/obj-build',
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -  'MOZ_SCM_LEVEL': '3',
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -  'MOZ_SIMPLE_PACKAGE_NAME': 'target',
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -  'MOZ_SOURCE_CHANGESET': '4f712ee198549ced74ccb3c256aec9a30305f500',
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -  'MOZ_SOURCE_REPO': 'https://hg.mozilla.org/integration/autoland',
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -  'OLDPWD': '/builds/worker',
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -  'PATH': '/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin',
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -  'PERFHERDER_EXTRA_OPTIONS': 'plain',
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -  'PWD': '/builds/worker',
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -  'PYTHONDONTWRITEBYTECODE': '1',
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -  'PYTHONUNBUFFERED': '1',
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -  'REQUIRE_GPU': '0',
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -  'RUN_ID': '0',
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -  'SCCACHE_DISABLE': '1',
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -  'SHELL': '/bin/bash',
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -  'SHLVL': '1',
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -  'TASKCLUSTER_INSTANCE_TYPE': 'projects/970387039909/machineTypes/n2-custom-16-73728',
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -  'TASKCLUSTER_NAME': '/dazzling_tereshkova/taskcluster',
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -  'TASKCLUSTER_PORT': 'tcp://172.17.0.3:80',
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -  'TASKCLUSTER_PORT_80_TCP': 'tcp://172.17.0.3:80',
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -  'TASKCLUSTER_PORT_80_TCP_ADDR': '172.17.0.3',
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -  'TASKCLUSTER_PORT_80_TCP_PORT': '80',
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -  'TASKCLUSTER_PORT_80_TCP_PROTO': 'tcp',
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -  'TASKCLUSTER_PROXY_URL': 'http://taskcluster',
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -  'TASKCLUSTER_PUBLIC_IP': '34.83.76.42',
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -  'TASKCLUSTER_ROOT_URL': 'https://firefox-ci-tc.services.mozilla.com',
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -  'TASKCLUSTER_WORKER_GROUP': 'us-west1',
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -  'TASKCLUSTER_WORKER_LOCATION': '{"cloud":"google","region":"us-west1","zone":"us-west1-b"}',
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -  'TASKCLUSTER_WORKER_POOL': 'gecko-3/b-linux-gcp',
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -  'TASKCLUSTER_WORKER_TYPE': 'b-linux-gcp',
[task 2022-11-04T22:10:39.870Z] 22:10:39     INFO -  'TASK_ID': 'IUEUEDHiSOKE4btwl3Uguw',
[task 2022-11-04T22:10:39.870Z] 22:10:39     INFO -  'TERM': 'xterm',
[task 2022-11-04T22:10:39.870Z] 22:10:39     INFO -  'TINDERBOX_OUTPUT': '1',
[task 2022-11-04T22:10:39.870Z] 22:10:39     INFO -  'TOOLTOOL_CACHE': '/builds/worker/tooltool-cache',
[task 2022-11-04T22:10:39.870Z] 22:10:39     INFO -  'TOOLTOOL_HOME': '/builds',
[task 2022-11-04T22:10:39.870Z] 22:10:39     INFO -  'UPLOAD_DIR': '/builds/worker/artifacts',
[task 2022-11-04T22:10:39.870Z] 22:10:39     INFO -  'USER': 'worker',
[task 2022-11-04T22:10:39.870Z] 22:10:39     INFO -  'WORKSPACE': '/builds/worker/workspace',
[task 2022-11-04T22:10:39.870Z] 22:10:39     INFO -  'XZ_OPT': '-T0',
[task 2022-11-04T22:10:39.870Z] 22:10:39     INFO -  '_': '/builds/worker/checkouts/gecko/mach'}
[task 2022-11-04T22:10:40.913Z] 22:10:40     INFO - Return code: 0
[task 2022-11-04T22:10:40.913Z] 22:10:40     INFO - Running post-run listener: _summarize
[task 2022-11-04T22:10:40.913Z] 22:10:40    ERROR - # TBPL FAILURE #
[task 2022-11-04T22:10:40.913Z] 22:10:40     INFO - [mozharness: 2022-11-04 22:10:40.913227Z] FxDesktopBuild summary:
[task 2022-11-04T22:10:40.913Z] 22:10:40    ERROR - # TBPL FAILURE #
[taskcluster 2022-11-04 22:10:41.348Z] === Task Finished ===
[taskcluster 2022-11-04 22:10:41.583Z] Unsuccessful task run with exit code: 255 completed in 3746.168 seconds

Flags: needinfo?(nika)

Nika Layzell [:nika] (ni? for response)

Assignee

Comment 19

•

3 years ago

This is an odd error, and seems like it might be infrastructure related, as there's nothing in here which should be able to cause the build process to hang.

Flags: needinfo?(nika)

Pulsebot

Comment 20

•

3 years ago

Pushed by nlayzell@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/15a5a67c1cee Part 1: Report glean telemetry about which IPC messages are being sent, r=florian,ipc-reviewers,mccr8 https://hg.mozilla.org/integration/autoland/rev/99c09dfae9a7 Part 2: Make the process mode available to IPDL gtests, r=ipc-reviewers,mccr8 https://hg.mozilla.org/integration/autoland/rev/e4a4e7668efb Part 3: Add a unit test for the IPC sent/recvd message telemetry, r=ipc-reviewers,mccr8

Cosmin Sabou [:CosminS]

Comment 21

•

3 years ago

•

Edited

Backed out for causing linux build timeouts.

Push with failures: https://treeherder.mozilla.org/jobs?repo=autoland&resultStatus=testfailed%2Cbusted%2Cexception%2Csuccess%2Cusercancel%2Crunning%2Cpending%2Crunnable&searchStr=linux%2Cx64%2Cdebug%2Cbuild-linux64-plain%2Fdebug%2Cbp&fromchange=e4a4e7668efb5db36c39f2125508e074987fd9d0&tochange=a62fef9a10c5c100844e7596f97300b2f529495a&selectedTaskRun=WzDrg-aDRPKketqmhSgGsA.0

Failure log: https://treeherder.mozilla.org/logviewer?job_id=395874391&repo=autoland

Backout link: https://hg.mozilla.org/integration/autoland/rev/a62fef9a10c5c100844e7596f97300b2f529495a

Flags: needinfo?(nika)

Chris H-C :chutten

Updated

•

3 years ago

Depends on: 1799751

Alex Finder

Comment 22

•

3 years ago

(In reply to Iulian Moraru from comment #18)

Backed out for causing build bustages with this line of failure Automation Error: mozprocess timed out after 2400 seconds running ['/builds/worker/workspace/obj-build/_virtualenvs/common/bin/python', 'mach', '--log-no-times', 'build', '-v']

The only affected platform, so far, is Linux x64 debug.

Push with failures

Failure log

Backout link

[task 2022-11-04T21:30:39.833Z] 21:30:39     INFO -  gmake[4]: Entering directory '/builds/worker/workspace/obj-build/gfx/thebes'
[task 2022-11-04T21:30:39.838Z] 21:30:39     INFO -  /builds/worker/fetches/clang/bin/clang++ --sysroot /builds/worker/fetches/sysroot-x86_64-linux-gnu -Qunused-arguments -std=gnu++17 -o Unified_cpp_gfx_thebes1.o -c  -I/builds/worker/workspace/obj-build/dist/stl_wrappers -I/builds/worker/workspace/obj-build/dist/system_wrappers -include /builds/worker/checkouts/gecko/config/gcc_hidden.h -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -fstack-clash-protection -ftrivial-auto-var-init=pattern -DDEBUG=1 -DOS_POSIX=1 -DOS_LINUX=1 -DMOZ_ENABLE_FREETYPE -DGRAPHITE2_STATIC -DMOZ_HAS_MOZGLUE -DMOZILLA_INTERNAL_API -DIMPL_LIBXUL -DSTATIC_EXPORTABLE_JS_API -I/builds/worker/checkouts/gecko/gfx/thebes -I/builds/worker/workspace/obj-build/gfx/thebes -I/builds/worker/workspace/obj-build/ipc/ipdl/_ipdlheaders -I/builds/worker/checkouts/gecko/ipc/chromium/src -I/builds/worker/workspace/obj-build/security/rlbox -I/builds/worker/checkouts/gecko/dom/base -I/builds/worker/checkouts/gecko/dom/media/platforms/apple -I/builds/worker/checkouts/gecko/dom/xml -I/builds/worker/checkouts/gecko/gfx/cairo/cairo/src -I/builds/worker/checkouts/gecko/widget/gtk -I/builds/worker/checkouts/gecko/gfx/skia -I/builds/worker/checkouts/gecko/gfx/skia/skia -I/builds/worker/workspace/obj-build/dist/include -I/builds/worker/workspace/obj-build/dist/include/nspr -I/builds/worker/workspace/obj-build/dist/include/nss -DMOZILLA_CLIENT -include /builds/worker/workspace/obj-build/mozilla-config.h -fno-sized-deallocation -fno-aligned-new -fno-exceptions -fPIC -fno-rtti -fno-exceptions -fno-math-errno -pthread -pipe -gdwarf-4 -Os -fno-omit-frame-pointer -funwind-tables -Werror -Wall -Wbitfield-enum-conversion -Wdeprecated-this-capture -Wempty-body -Wformat-type-confusion -Wignored-qualifiers -Wpointer-arith -Wshadow-field-in-constructor-modified -Wsign-compare -Wtype-limits -Wno-error=tautological-type-limit-compare -Wunreachable-code -Wunreachable-code-return -Wunused-but-set-parameter -Wno-invalid-offsetof -Wclass-varargs -Wempty-init-stmt -Wfloat-overflow-conversion -Wfloat-zero-conversion -Wloop-analysis -Wno-range-loop-analysis -Wc++2a-compat -Wenum-compare-conditional -Wenum-float-conversion -Wno-ambiguous-reversed-operator -Wno-error=deprecated -Wno-error=deprecated-anon-enum-enum-conversion -Wno-error=deprecated-enum-enum-conversion -Wno-error=deprecated-pragma -Wno-error=deprecated-this-capture -Wno-error=deprecated-volatile -Wcomma -Wimplicit-fallthrough -Wstring-conversion -Wno-inline-new-delete -Wno-error=deprecated-declarations -Wno-error=array-bounds -Wno-error=free-nonheap-object -Wno-error=atomic-alignment -Wno-error=deprecated-builtins -Wformat -Wformat-security -Wno-psabi -Wthread-safety -Wno-unknown-warning-option -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/include/gtk-3.0 -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/include/pango-1.0 -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/include/glib-2.0 -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/lib/x86_64-linux-gnu/glib-2.0/include -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/include -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/include/cairo -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/include/pixman-1 -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/include/freetype2 -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/include/libpng12 -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/include/gdk-pixbuf-2.0 -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/include/gio-unix-2.0/ -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/include/harfbuzz -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/include/atk-1.0 -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/include/at-spi2-atk/2.0 -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/include/at-spi-2.0 -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/include/dbus-1.0 -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/lib/x86_64-linux-gnu/dbus-1.0/include -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/include/gtk-3.0/unix-print -pthread -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/include/pango-1.0 -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/include/glib-2.0 -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/lib/x86_64-linux-gnu/glib-2.0/include -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/include -pthread -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/include/gtk-3.0 -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/include/pango-1.0 -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/include/glib-2.0 -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/lib/x86_64-linux-gnu/glib-2.0/include -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/include -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/include/cairo -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/include/pixman-1 -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/include/freetype2 -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/include/libpng12 -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/include/gdk-pixbuf-2.0 -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/include/gio-unix-2.0/ -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/include/harfbuzz -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/include/atk-1.0 -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/include/at-spi2-atk/2.0 -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/include/at-spi-2.0 -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/include/dbus-1.0 -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/lib/x86_64-linux-gnu/dbus-1.0/include -pthread -I/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/include/libdrm -Werror=switch -fno-strict-aliasing -ffp-contract=off  -MD -MP -MF .deps/Unified_cpp_gfx_thebes1.o.pp   Unified_cpp_gfx_thebes1.cpp
[task 2022-11-04T21:30:39.838Z] 21:30:39     INFO -  gmake[4]: Leaving directory '/builds/worker/workspace/obj-build/gfx/thebes'
[task 2022-11-04T22:10:39.865Z] 22:10:39     INFO - Automation Error: mozprocess timed out after 2400 seconds running ['/builds/worker/workspace/obj-build/_virtualenvs/common/bin/python', 'mach', '--log-no-times', 'build', '-v']
[task 2022-11-04T22:10:39.866Z] 22:10:39    ERROR - timed out after 2400 seconds of no output
[task 2022-11-04T22:10:39.866Z] 22:10:39    ERROR - Return code: -15
[task 2022-11-04T22:10:39.866Z] 22:10:39  WARNING - setting return code to 2
[task 2022-11-04T22:10:39.866Z] 22:10:39    FATAL - 'mach build -v' did not run successfully. Please check log for errors.
[task 2022-11-04T22:10:39.866Z] 22:10:39    FATAL - Running post_fatal callback...
[task 2022-11-04T22:10:39.866Z] 22:10:39    FATAL - Exiting -1
[task 2022-11-04T22:10:39.866Z] 22:10:39     INFO - [mozharness: 2022-11-04 22:10:39.866308Z] Finished build step (failed)
[task 2022-11-04T22:10:39.866Z] 22:10:39     INFO - Running post-run listener: _parse_build_tests_ccov
[task 2022-11-04T22:10:39.866Z] 22:10:39     INFO - Running command: ['/builds/worker/workspace/obj-build/_virtualenvs/common/bin/python', 'mach', 'python', 'testing/parse_build_tests_ccov.py'] in /builds/worker/checkouts/gecko
[task 2022-11-04T22:10:39.866Z] 22:10:39     INFO - Copy/paste: /builds/worker/workspace/obj-build/_virtualenvs/common/bin/python mach python testing/parse_build_tests_ccov.py
[task 2022-11-04T22:10:39.867Z] 22:10:39     INFO - Using env: {'ACCEPTED_MAR_CHANNEL_IDS': 'firefox-mozilla-central',
[task 2022-11-04T22:10:39.867Z] 22:10:39     INFO -  'DEBIAN_FRONTEND': 'noninteractive',
[task 2022-11-04T22:10:39.867Z] 22:10:39     INFO -  'DISPLAY': ':2',
[task 2022-11-04T22:10:39.867Z] 22:10:39     INFO -  'EXTRA_MOZHARNESS_CONFIG': '{"disable_package_metrics": true, '
[task 2022-11-04T22:10:39.867Z] 22:10:39     INFO -                             '"mozconfig_variant": "plain-debug", "objdir": '
[task 2022-11-04T22:10:39.867Z] 22:10:39     INFO -                             '"obj-build"}',
[task 2022-11-04T22:10:39.867Z] 22:10:39     INFO -  'GECKO_BASE_REPOSITORY': 'https://hg.mozilla.org/mozilla-unified',
[task 2022-11-04T22:10:39.867Z] 22:10:39     INFO -  'GECKO_HEAD_REPOSITORY': 'https://hg.mozilla.org/integration/autoland',
[task 2022-11-04T22:10:39.867Z] 22:10:39     INFO -  'GECKO_HEAD_REV': '4f712ee198549ced74ccb3c256aec9a30305f500',
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -  'GECKO_PATH': '/builds/worker/checkouts/gecko',
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -  'GRCOV_PATH': '/builds/worker/fetches/grcov',
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -  'HG_SHARE_BASE_DIR': '/builds/hg-shared',
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -  'HG_STORE_PATH': '/builds/worker/checkouts/hg-store',
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -  'HOME': '/builds/worker',
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -  'HOSTNAME': 'taskcluster-worker',
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -  'LC_ALL': 'C',
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -  'LC_CTYPE': 'C.UTF-8',
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -  'LOGNAME': 'worker',
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -  'MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE': 'system',
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -  'MACH_MAIN_PID': '31',
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -  'MACH_SYSTEM_ASSERTED_COMPATIBLE_WITH_COMMON_SITE': '1',
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -  'MACH_SYSTEM_ASSERTED_COMPATIBLE_WITH_MACH_SITE': '1',
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -  'MAR_CHANNEL_ID': 'firefox-mozilla-central',
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -  'MH_BRANCH': 'autoland',
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -  'MH_BUILD_POOL': 'taskcluster',
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -  'MOZBUILD_STATE_PATH': '/builds/worker/.mozbuild',
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -  'MOZHARNESS_ACTIONS': 'build',
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -  'MOZHARNESS_CONFIG': 'builds/releng_base_firefox.py '
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -                       'builds/releng_base_linux_64_builds.py',
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -  'MOZHARNESS_SCRIPT': 'mozharness/scripts/fx_desktop_build.py',
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -  'MOZ_AUTOMATION': '1',
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -  'MOZ_BUILD_DATE': '20221104170532',
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -  'MOZ_CRASHREPORTER_NO_REPORT': '1',
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -  'MOZ_FETCHES': '[{"artifact": "public/build/binutils.tar.zst", "extract": '
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -                 'true, "task": "B6a2dApVS_i8EYT4AmLyIw"}, {"artifact": '
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -                 '"public/build/cbindgen.tar.zst", "extract": true, "task": '
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -                 '"P85YOslvQ1mIl8yTSR7bxQ"}, {"artifact": '
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -                 '"public/build/clang.tar.zst", "extract": true, "task": '
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -                 '"cQ80kb1eSpOF-ipi5N39ag"}, {"artifact": '
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -                 '"public/build/dump_syms.tar.zst", "extract": true, "task": '
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -                 '"JTBjqzf9Su6AjxCwQYMZeg"}, {"artifact": '
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -                 '"public/build/nasm.tar.zst", "extract": true, "task": '
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -                 '"MbpriPLdQtqg6dNIz3-DRQ"}, {"artifact": '
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -                 '"public/build/node.tar.zst", "extract": true, "task": '
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -                 '"E-oYJkvsQt-tXX6EavJT8Q"}, {"artifact": '
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -                 '"public/build/pkgconf.tar.zst", "extract": true, "task": '
[task 2022-11-04T22:10:39.868Z] 22:10:39     INFO -                 '"eV25uGFpRqaN-85X0plV0g"}, {"artifact": '
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -                 '"public/build/rustc.tar.zst", "extract": true, "task": '
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -                 '"SuK-d4hOTMiC4ZqZqCMs2g"}, {"artifact": '
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -                 '"public/build/sysroot-wasm32-wasi.tar.zst", "extract": true, '
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -                 '"task": "UpYz0AS3ReKIAmvO78i_iw"}, {"artifact": '
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -                 '"public/build/sysroot-x86_64-linux-gnu.tar.zst", "extract": '
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -                 'true, "task": "ZBAxiavCT_OQC3U8JZIBvQ"}]',
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -  'MOZ_FETCHES_DIR': '/builds/worker/fetches',
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -  'MOZ_OBJDIR': '/builds/worker/workspace/obj-build',
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -  'MOZ_SCM_LEVEL': '3',
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -  'MOZ_SIMPLE_PACKAGE_NAME': 'target',
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -  'MOZ_SOURCE_CHANGESET': '4f712ee198549ced74ccb3c256aec9a30305f500',
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -  'MOZ_SOURCE_REPO': 'https://hg.mozilla.org/integration/autoland',
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -  'OLDPWD': '/builds/worker',
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -  'PATH': '/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin',
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -  'PERFHERDER_EXTRA_OPTIONS': 'plain',
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -  'PWD': '/builds/worker',
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -  'PYTHONDONTWRITEBYTECODE': '1',
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -  'PYTHONUNBUFFERED': '1',
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -  'REQUIRE_GPU': '0',
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -  'RUN_ID': '0',
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -  'SCCACHE_DISABLE': '1',
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -  'SHELL': '/bin/bash',
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -  'SHLVL': '1',
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -  'TASKCLUSTER_INSTANCE_TYPE': 'projects/970387039909/machineTypes/n2-custom-16-73728',
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -  'TASKCLUSTER_NAME': '/dazzling_tereshkova/taskcluster',
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -  'TASKCLUSTER_PORT': 'tcp://172.17.0.3:80',
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -  'TASKCLUSTER_PORT_80_TCP': 'tcp://172.17.0.3:80',
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -  'TASKCLUSTER_PORT_80_TCP_ADDR': '172.17.0.3',
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -  'TASKCLUSTER_PORT_80_TCP_PORT': '80',
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -  'TASKCLUSTER_PORT_80_TCP_PROTO': 'tcp',
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -  'TASKCLUSTER_PROXY_URL': 'http://taskcluster',
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -  'TASKCLUSTER_PUBLIC_IP': '34.83.76.42',
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -  'TASKCLUSTER_ROOT_URL': 'https://firefox-ci-tc.services.mozilla.com',
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -  'TASKCLUSTER_WORKER_GROUP': 'us-west1',
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -  'TASKCLUSTER_WORKER_LOCATION': '{"cloud":"google","region":"us-west1","zone":"us-west1-b"}',
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -  'TASKCLUSTER_WORKER_POOL': 'gecko-3/b-linux-gcp',
[task 2022-11-04T22:10:39.869Z] 22:10:39     INFO -  'TASKCLUSTER_WORKER_TYPE': 'b-linux-gcp',
[task 2022-11-04T22:10:39.870Z] 22:10:39     INFO -  'TASK_ID': 'IUEUEDHiSOKE4btwl3Uguw',
[task 2022-11-04T22:10:39.870Z] 22:10:39     INFO -  'TERM': 'xterm',
[task 2022-11-04T22:10:39.870Z] 22:10:39     INFO -  'TINDERBOX_OUTPUT': '1',
[task 2022-11-04T22:10:39.870Z] 22:10:39     INFO -  'TOOLTOOL_CACHE': '/builds/worker/tooltool-cache',
[task 2022-11-04T22:10:39.870Z] 22:10:39     INFO -  'TOOLTOOL_HOME': '/builds',
[task 2022-11-04T22:10:39.870Z] 22:10:39     INFO -  'UPLOAD_DIR': '/builds/worker/artifacts',
[task 2022-11-04T22:10:39.870Z] 22:10:39     INFO -  'USER': 'worker',
[task 2022-11-04T22:10:39.870Z] 22:10:39     INFO -  'WORKSPACE': '/builds/worker/workspace',
[task 2022-11-04T22:10:39.870Z] 22:10:39     INFO -  'XZ_OPT': '-T0',
[task 2022-11-04T22:10:39.870Z] 22:10:39     INFO -  '_': '/builds/worker/checkouts/gecko/mach'}
[task 2022-11-04T22:10:40.913Z] 22:10:40     INFO - Return code: 0
[task 2022-11-04T22:10:40.913Z] 22:10:40     INFO - Running post-run listener: _summarize
[task 2022-11-04T22:10:40.913Z] 22:10:40    ERROR - # TBPL FAILURE #
[task 2022-11-04T22:10:40.913Z] 22:10:40     INFO - [mozharness: 2022-11-04 22:10:40.913227Z] FxDesktopBuild summary:
[task 2022-11-04T22:10:40.913Z] 22:10:40    ERROR - # TBPL FAILURE #
[taskcluster 2022-11-04 22:10:41.348Z] === Task Finished ===
[taskcluster 2022-11-04 22:10:41.583Z] Unsuccessful task run with exit code: 255 completed in 3746.168 seconds

== Change summary for alert #35977 (as of Fri, 04 Nov 2022 17:47:53 GMT) ==

Regressions:

Ratio	Test	Platform	Options	Absolute values (old vs new)
1%	installer size	osx-cross		82,499,679.92 -> 83,494,936.67

For up to date results, see: https://treeherder.mozilla.org/perfherder/alerts?id=35977

FYI: the regression was caused by revision 4f712ee19854 backed out by 66114e463, not the backout itself, as visible in this graph.

Alex Finder

Comment 23

•

3 years ago

•

Edited

(In reply to Cosmin Sabou [:CosminS] from comment #21)

Backed out for causing linux build timeouts.

Push with failures: https://treeherder.mozilla.org/jobs?repo=autoland&resultStatus=testfailed%2Cbusted%2Cexception%2Csuccess%2Cusercancel%2Crunning%2Cpending%2Crunnable&searchStr=linux%2Cx64%2Cdebug%2Cbuild-linux64-plain%2Fdebug%2Cbp&fromchange=e4a4e7668efb5db36c39f2125508e074987fd9d0&tochange=a62fef9a10c5c100844e7596f97300b2f529495a&selectedTaskRun=WzDrg-aDRPKketqmhSgGsA.0

Failure log: https://treeherder.mozilla.org/logviewer?job_id=395874391&repo=autoland

Backout link: https://hg.mozilla.org/integration/autoland/rev/a62fef9a10c5c100844e7596f97300b2f529495a

== Change summary for alert #36027 (as of Thu, 10 Nov 2022 06:57:39 GMT) ==

Regressions:

Ratio	Test	Platform	Options	Absolute values (old vs new)
1%	installer size	osx-aarch64-shippable	aarch64 nightly	83,336,469.71 -> 83,869,279.42

For up to date results, see: https://treeherder.mozilla.org/perfherder/alerts?id=36027

FYI: the regression was caused by revision e4a4e7668efb5 backed out by a62fef9a10c5c, not the backout itself, as visible in this graph.

Nika Layzell [:nika] (ni? for response)

Assignee

Comment 24

•

3 years ago

Hopefully the binary size regression issue will have been largely mitigated by the fix in bug 1799751, though I believe the regression still isn't amazing even after that change.

Flags: needinfo?(nika)

Sandor Molnar[:smolnar]

Comment 25

•

3 years ago

bugherder

https://hg.mozilla.org/mozilla-central/rev/6b72e6c8e3bc
https://hg.mozilla.org/mozilla-central/rev/3e53a1922022
https://hg.mozilla.org/mozilla-central/rev/862b578d462e

Status: NEW → RESOLVED

Closed: 3 years ago

status-firefox109: --- → fixed

Resolution: --- → FIXED

Target Milestone: --- → 109 Branch

Andrej (:aglavic)

Updated

•

3 years ago

Regressions: 1803798

Nika Layzell [:nika] (ni? for response)

Assignee

Updated

•

3 years ago

Regressions: 1803500

Olli Pettay [:smaug][bugs@pettay.fi]

Comment 26

•

3 years ago

I wonder if we could have the code at least behind a pref, so that one could run the benchmarks on Nightly without too much random overhead.
We have similar issue in https://bugzilla.mozilla.org/show_bug.cgi?id=1796080, in that case toolkit.content-background-hang-monitor.disabled=true helps. (Of course it is a bit annoying if one needs remember to change more and more pref values in order to get reasonable performance profiles)

Alex Finder

Updated

•

3 years ago

Regressions: 1804420

Alex Finder

Updated

•

3 years ago

Regressions: 1804482

Nika Layzell [:nika] (ni? for response)

Assignee

Comment 27

•

3 years ago

(In reply to Olli Pettay [:smaug][bugs@pettay.fi] from comment #26)

I wonder if we could have the code at least behind a pref, so that one could run the benchmarks on Nightly without too much random overhead.
We have similar issue in https://bugzilla.mozilla.org/show_bug.cgi?id=1796080, in that case toolkit.content-background-hang-monitor.disabled=true helps. (Of course it is a bit annoying if one needs remember to change more and more pref values in order to get reasonable performance profiles)

The overhead here is unfortunately in areas which are not easy to put behind a pref. The runtime perf stuff could be gated to avoid the runtime overhead, but things like the memory usage errors are happening deep in glean, and aren't configurable on either a per-release or per-execution basis.

I'm currently getting this backed out until glean improves.

Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)

Comment 28

•

3 years ago

Backed out on request of nika for causing regression:
https://hg.mozilla.org/integration/autoland/rev/31dfb9281c042ced6fbffac0a48afa7ba8d58c13

Status: RESOLVED → REOPENED

status-firefox109: fixed → ---

Flags: needinfo?(nika)

Resolution: FIXED → ---

Target Milestone: 109 Branch → ---

Norisz Fay [:noriszfay]

Comment 29

•

3 years ago

Backout merged to central: https://hg.mozilla.org/mozilla-central/rev/31dfb9281c04

Acasandrei Beatrice (needinfo me)

Comment 30

•

3 years ago

(In reply to Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout) from comment #28)

Backed out on request of nika for causing regression:
https://hg.mozilla.org/integration/autoland/rev/31dfb9281c042ced6fbffac0a48afa7ba8d58c13

== Change summary for alert #36404 (as of Sun, 11 Dec 2022 18:19:45 GMT) ==

Improvements:

Ratio	Test	Platform	Options	Absolute values (old vs new)
20%	Base Content Heap Unclassified	windows10-64-2004-shippable-qr	fission	2,126,904.33 -> 1,706,050.67
16%	Base Content Heap Unclassified	linux1804-64-shippable-qr	fission	2,542,005.67 -> 2,132,005.67
13%	Base Content Heap Unclassified	macosx1015-64-shippable-qr	fission	2,908,229.33 -> 2,539,302.33
5%	Base Content Explicit	linux1804-64-shippable-qr	fission	8,038,554.67 -> 7,636,634.67
5%	Heap Unclassified	windows10-64-2004-shippable-qr	fission tp6	103,816,141.84 -> 98,841,285.59
...	...	...	...	...
4%	Base Content Resident Unique Memory	windows10-64-2004-shippable-qr	fission	9,117,354.67 -> 8,753,834.67

For up to date results, see: https://treeherder.mozilla.org/perfherder/alerts?id=36404

Acasandrei Beatrice (needinfo me)

Updated

•

3 years ago

Keywords: perf-alert

Andrej (:aglavic)

Comment 31

•

3 years ago

(In reply to Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout) from comment #28)

Backed out on request of nika for causing regression:
https://hg.mozilla.org/integration/autoland/rev/31dfb9281c042ced6fbffac0a48afa7ba8d58c13

== Change summary for alert #36405 (as of Mon, 12 Dec 2022 01:30:32 GMT) ==

Improvements:

Ratio	Test	Platform	Options	Absolute values (old vs new)
12%	a11yr dhtml.html	windows10-64-shippable-qr	e10s fission stylo webrender-sw	933.19 -> 824.46
10%	cross_origin_pageload	linux1804-64-shippable-qr	e10s fission stylo webrender-sw	276.94 -> 250.35
9%	cross_origin_pageload	linux1804-64-shippable-qr	e10s fission stylo webrender	291.99 -> 266.10
3%	tp5o_webext	linux1804-64-shippable-qr	e10s fission stylo webrender-sw	270.91 -> 263.60
2%	tp5o_webext	linux1804-64-shippable-qr	e10s fission stylo webrender	281.08 -> 275.42

For up to date results, see: https://treeherder.mozilla.org/perfherder/alerts?id=36405

Chris H-C :chutten

Updated

•

3 years ago

Comment 32

•

3 years ago

(In reply to Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout) from comment #28)

Backed out on request of nika for causing regression:
https://hg.mozilla.org/integration/autoland/rev/31dfb9281c042ced6fbffac0a48afa7ba8d58c13

== Change summary for alert #36396 (as of Sat, 10 Dec 2022 05:04:25 GMT) ==

Improvements:

Ratio	Test	Platform	Options	Absolute values (old vs new)	Performance Profiles
2%	outlook loadtime	macosx1015-64-shippable-qr	fission warm webrender	322.55 -> 315.88	Before/After

For up to date results, see: https://treeherder.mozilla.org/perfherder/alerts?id=36396

Andrew McCreight [:mccr8]

Updated

•

3 years ago

Severity: -- → S3

Nika Layzell [:nika] (ni? for response)

Assignee

Updated

•

3 years ago

Depends on: 1805427

Flags: needinfo?(nika)

Ted Campbell [:tcampbell]

Comment 33

•

3 years ago

(In reply to Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout) from comment #28)

Backed out on request of nika for causing regression:
https://hg.mozilla.org/integration/autoland/rev/31dfb9281c042ced6fbffac0a48afa7ba8d58c13

Backing this our also improve the Win64 Speedometer2 score by almost 2%.
https://treeherder.mozilla.org/perfherder/graphs?highlightAlerts=1&highlightChangelogData=1&highlightCommonAlerts=0&series=autoland,3912818,1,13&timerange=7776000&zoom=1670477073565,1670572480332,118.95565198437565,128.83272741570272

Phabricator Automation

Updated

•

3 months ago

Attachment #9295628 - Attachment is obsolete: true

Phabricator Automation

Updated

•

3 months ago

Attachment #9296514 - Attachment is obsolete: true

Phabricator Automation

Updated

•

3 months ago

Attachment #9296515 - Attachment is obsolete: true

data-review 3 years ago Nika Layzell [:nika] (ni? for response) 4.48 KB, text/plain	royang : data-review+	Details
Bug 1790872 - Part 1: Report glean telemetry about which IPC messages are being sent, r=florian!,#ipc-reviewers! 3 years ago Nika Layzell [:nika] (ni? for response) 48 bytes, text/x-phabricator-request		Details \| Review
Bug 1790872 - Part 2: Make the process mode available to IPDL gtests, r=#ipc-reviewers 3 years ago Nika Layzell [:nika] (ni? for response) 48 bytes, text/x-phabricator-request		Details \| Review
Bug 1790872 - Part 3: Add a unit test for the IPC sent/recvd message telemetry, r=#ipc-reviewers 3 years ago Nika Layzell [:nika] (ni? for response) 48 bytes, text/x-phabricator-request		Details \| Review