Closed Bug 1547289 Opened 5 years ago Closed 5 years ago

Perma subprocess.CalledProcessError: Command '[u'/builds/worker/workspace/build/src/clang/bin/llvm-profdata', 'merge', '-o', 'merged.profdata', '/builds/worker/fetches/default.profraw']' returned non-zero exit status 1 when Gecko 68 merges to Beta on 2019

Categories

(Firefox for Android Graveyard :: General, defect)

Product:
Firefox for Android Graveyard
Component:
General
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(firefox-esr60 unaffected, firefox66 unaffected, firefox67 unaffected, firefox68- wontfix)

Status:
RESOLVED WORKSFORME
Tracking Flags:
Tracking Status
firefox-esr60 --- unaffected
firefox66 --- unaffected
firefox67 --- unaffected
firefox68 - wontfix

People

(Reporter: rmaries, Unassigned)

References

(Regression)

Details

(Keywords: regression)

[Tracking Requested - why for this release]:

https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=242894427&repo=try&lineNumber=1619

https://treeherder.mozilla.org/#/jobs?repo=try&resultStatus=testfailed%2Cbusted%2Cexception&classifiedState=unclassified&revision=890c1689ab3fe50aa04bf16110a68c892d990f26&selectedJob=242894427&searchStr=android%2C4.0%2Capi16%2B%2Copt%2Cbuild-android-api-16-nightly%2Fopt%2C%28n%29

[task 2019-04-26T15:15:41.941Z] 15:15:41 ERROR - Traceback (most recent call last):
[task 2019-04-26T15:15:41.941Z] 15:15:41 INFO - File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
[task 2019-04-26T15:15:41.941Z] 15:15:41 INFO - "main", fname, loader, pkg_name)
[task 2019-04-26T15:15:41.941Z] 15:15:41 INFO - File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
[task 2019-04-26T15:15:41.941Z] 15:15:41 INFO - exec code in run_globals
[task 2019-04-26T15:15:41.941Z] 15:15:41 INFO - File "/builds/worker/workspace/build/src/python/mozbuild/mozbuild/action/file_generate.py", line 120, in <module>
[task 2019-04-26T15:15:41.941Z] 15:15:41 INFO - sys.exit(main(sys.argv[1:]))
[task 2019-04-26T15:15:41.941Z] 15:15:41 INFO - File "/builds/worker/workspace/build/src/python/mozbuild/mozbuild/action/file_generate.py", line 71, in main
[task 2019-04-26T15:15:41.941Z] 15:15:41 INFO - ret = module.dict[method](output, *args.additional_arguments, **kwargs)
[task 2019-04-26T15:15:41.942Z] 15:15:41 INFO - File "/builds/worker/workspace/build/src/build/merge_profdata.py", line 11, in main
[task 2019-04-26T15:15:41.942Z] 15:15:41 INFO - '-o', 'merged.profdata', profile_file])
[task 2019-04-26T15:15:41.942Z] 15:15:41 INFO - File "/usr/lib/python2.7/subprocess.py", line 186, in check_call
[task 2019-04-26T15:15:41.942Z] 15:15:41 INFO - raise CalledProcessError(retcode, cmd)
[task 2019-04-26T15:15:41.942Z] 15:15:41 INFO - subprocess.CalledProcessError: Command '[u'/builds/worker/workspace/build/src/clang/bin/llvm-profdata', 'merge', '-o', 'merged.profdata', '/builds/worker/fetches/default.profraw']' returned non-zero exit status 1
[task 2019-04-26T15:15:41.942Z] 15:15:41 INFO - backend.mk:43: recipe for target '.deps/merged.profdata.stub.stub' failed
[task 2019-04-26T15:15:41.942Z] 15:15:41 ERROR - make[2]: *** [.deps/merged.profdata.stub.stub] Error 1
[task 2019-04-26T15:15:41.942Z] 15:15:41 INFO - make[2]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox'
[task 2019-04-26T15:15:41.942Z] 15:15:41 INFO - make[2]: *** Waiting for unfinished jobs....
[task 2019-04-26T15:15:41.944Z] 15:15:41 INFO - make[2]: Entering directory '/builds/worker/workspace/build/src/obj-firefox'
[task 2019-04-26T15:15:41.945Z] 15:15:41 INFO - ./buildid.h.stub
[task 2019-04-26T15:15:41.945Z] 15:15:41 INFO - /builds/worker/workspace/build/src/obj-firefox/_virtualenvs/init/bin/python -m mozbuild.action.file_generate /builds/worker/workspace/build/src/build/variables.py buildid_header buildid.h .deps/buildid.h.pp .deps/buildid.h.stub
[task 2019-04-26T15:15:41.945Z] 15:15:41 INFO - make[2]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox'
[task 2019-04-26T15:15:41.945Z] 15:15:41 INFO - make[2]: Entering directory '/builds/worker/workspace/build/src/obj-firefox'
[task 2019-04-26T15:15:41.945Z] 15:15:41 INFO - ./mozilla-config.h.stub
[task 2019-04-26T15:15:41.945Z] 15:15:41 INFO - /builds/worker/workspace/build/src/obj-firefox/_virtualenvs/init/bin/python -m mozbuild.action.file_generate /builds/worker/workspace/build/src/python/mozbuild/mozbuild/action/process_define_files.py process_define_file mozilla-config.h .deps/mozilla-config.h.pp .deps/mozilla-config.h.stub /builds/worker/workspace/build/src/mozilla-config.h.in
[task 2019-04-26T15:15:41.945Z] 15:15:41 INFO - make[2]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox'
[task 2019-04-26T15:15:41.945Z] 15:15:41 INFO - make[2]: Entering directory '/builds/worker/workspace/build/src/obj-firefox'
[task 2019-04-26T15:15:41.945Z] 15:15:41 INFO - ./source-repo.h.stub
[task 2019-04-26T15:15:41.945Z] 15:15:41 INFO - /builds/worker/workspace/build/src/obj-firefox/_virtualenvs/init/bin/python -m mozbuild.action.file_generate /builds/worker/workspace/build/src/build/variables.py source_repo_header source-repo.h .deps/source-repo.h.pp .deps/source-repo.h.stub
[task 2019-04-26T15:15:41.946Z] 15:15:41 INFO - make[2]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox'
[task 2019-04-26T15:15:41.946Z] 15:15:41 INFO - /builds/worker/workspace/build/src/config/rules.mk:400: recipe for target 'default' failed
[task 2019-04-26T15:15:41.946Z] 15:15:41 ERROR - make[1]: *** [default] Error 2
[task 2019-04-26T15:15:41.946Z] 15:15:41 INFO - client.mk:125: recipe for target 'build' failed
[task 2019-04-26T15:15:41.946Z] 15:15:41 INFO - make: *** [build] Error 2
[task 2019-04-26T15:15:41.982Z] 15:15:41 INFO - 0 compiler warnings present.
[task 2019-04-26T15:15:42.029Z] 15:15:42 ERROR - Return code: 2
[task 2019-04-26T15:15:42.029Z] 15:15:42 WARNING - setting return code to 2
[task 2019-04-26T15:15:42.029Z] 15:15:42 FATAL - 'mach build -v' did not run successfully. Please check log for errors.
[task 2019-04-26T15:15:42.029Z] 15:15:42 FATAL - Running post_fatal callback...
[task 2019-04-26T15:15:42.029Z] 15:15:42 FATAL - Exiting -1
[task 2019-04-26T15:15:42.029Z] 15:15:42 INFO - [mozharness: 2019-04-26 15:15:42.029849Z] Finished build step (failed)
[task 2019-04-26T15:15:42.029Z] 15:15:42 INFO - Running post-run listener: _parse_build_tests_ccov
[task 2019-04-26T15:15:42.030Z] 15:15:42 INFO - Update channel set to: beta
[task 2019-04-26T15:15:42.030Z] 15:15:42 INFO - Running command: ['/builds/worker/workspace/build/src/obj-x86_64-pc-linux-gnu/_virtualenvs/init/bin/python', 'mach', 'python', 'testing/parse_build_tests_ccov.py'] in /builds/worker/workspace/build/src
[task 2019-04-26T15:15:42.030Z] 15:15:42 INFO - Copy/paste: /builds/worker/workspace/build/src/obj-x86_64-pc-linux-gnu/virtualenvs/init/bin/python mach python testing/parse_build_tests_ccov.py
[task 2019-04-26T15:15:42.032Z] 15:15:42 INFO - Using env: {'DEBIAN_FRONTEND': 'noninteractive',
[task 2019-04-26T15:15:42.032Z] 15:15:42 INFO - 'DISPLAY': ':2',
[task 2019-04-26T15:15:42.032Z] 15:15:42 INFO - 'EXTRA_MOZHARNESS_CONFIG': '{"update_channel": "beta"}',
[task 2019-04-26T15:15:42.032Z] 15:15:42 INFO - 'GECKO_BASE_REPOSITORY': 'https://hg.mozilla.org/mozilla-unified',
[task 2019-04-26T15:15:42.032Z] 15:15:42 INFO - 'GECKO_HEAD_REPOSITORY': 'https://hg.mozilla.org/try',
[task 2019-04-26T15:15:42.032Z] 15:15:42 INFO - 'GECKO_HEAD_REV': '890c1689ab3fe50aa04bf16110a68c892d990f26',
[task 2019-04-26T15:15:42.032Z] 15:15:42 INFO - 'GECKO_PATH': '/builds/worker/workspace/build/src',
[task 2019-04-26T15:15:42.032Z] 15:15:42 INFO - 'GRADLE_USER_HOME': '/builds/worker/workspace/build/src/mobile/android/gradle/dotgradle-offline',
[task 2019-04-26T15:15:42.032Z] 15:15:42 INFO - 'GRCOV_PATH': '/builds/worker/fetches/grcov',
[task 2019-04-26T15:15:42.032Z] 15:15:42 INFO - 'HG_SHARE_BASE_DIR': '/builds/hg-shared',
[task 2019-04-26T15:15:42.032Z] 15:15:42 INFO - 'HG_STORE_PATH': '/builds/worker/checkouts/hg-store',
[task 2019-04-26T15:15:42.032Z] 15:15:42 INFO - 'HOME': '/builds/worker',
[task 2019-04-26T15:15:42.032Z] 15:15:42 INFO - 'HOSTNAME': 'taskcluster-worker',
[task 2019-04-26T15:15:42.032Z] 15:15:42 INFO - 'LC_ALL': 'C',
[task 2019-04-26T15:15:42.032Z] 15:15:42 INFO - 'LIBRARY_PATH': ':/builds/worker/workspace/src/obj-firefox:/builds/worker/workspace/src/gcc/lib64',
[task 2019-04-26T15:15:42.032Z] 15:15:42 INFO - 'LOGNAME': 'worker',
[task 2019-04-26T15:15:42.032Z] 15:15:42 INFO - 'MACH_MAIN_PID': '33',
[task 2019-04-26T15:15:42.032Z] 15:15:42 INFO - 'MH_BRANCH': 'try',
[task 2019-04-26T15:15:42.032Z] 15:15:42 INFO - 'MH_BUILD_POOL': 'taskcluster',
[task 2019-04-26T15:15:42.032Z] 15:15:42 INFO - 'MH_CUSTOM_BUILD_VARIANT_CFG': 'api-16-profile-use',
[task 2019-04-26T15:15:42.032Z] 15:15:42 INFO - 'MOZBUILD_STATE_PATH': '/builds/worker/.mozbuild',
[task 2019-04-26T15:15:42.032Z] 15:15:42 INFO - 'MOZHARNESS_ACTIONS': 'get-secrets build multi-l10n',
[task 2019-04-26T15:15:42.032Z] 15:15:42 INFO - 'MOZHARNESS_CONFIG': 'builds/releng_base_android_64_builds.py taskcluster_nightly.py',
[task 2019-04-26T15:15:42.032Z] 15:15:42 INFO - 'MOZHARNESS_SCRIPT': 'mozharness/scripts/fx_desktop_build.py',
[task 2019-04-26T15:15:42.032Z] 15:15:42 INFO - 'MOZ_AUTOMATION': '1',
[task 2019-04-26T15:15:42.033Z] 15:15:42 INFO - 'MOZ_BUILD_DATE': '20190426122010',
[task 2019-04-26T15:15:42.033Z] 15:15:42 INFO - 'MOZ_CRASHREPORTER_NO_REPORT': '1',
[task 2019-04-26T15:15:42.033Z] 15:15:42 INFO - 'MOZ_FETCHES': '[{"artifact": "public/build/profdata.tar.xz", "extract": true, "task": "dNgt2SKcQuWrNuYc4aXjEQ"}]',
[task 2019-04-26T15:15:42.033Z] 15:15:42 INFO - 'MOZ_FETCHES_DIR': '/builds/worker/fetches',
[task 2019-04-26T15:15:42.033Z] 15:15:42 INFO - 'MOZ_OBJDIR': '/builds/worker/workspace/build/src/obj-firefox',
[task 2019-04-26T15:15:42.033Z] 15:15:42 INFO - 'MOZ_SCM_LEVEL': '1',
[task 2019-04-26T15:15:42.033Z] 15:15:42 INFO - 'MOZ_SIMPLE_PACKAGE_NAME': 'target',
[task 2019-04-26T15:15:42.033Z] 15:15:42 INFO - 'MOZ_SOURCE_CHANGESET': '890c1689ab3fe50aa04bf16110a68c892d990f26',
[task 2019-04-26T15:15:42.033Z] 15:15:42 INFO - 'MOZ_SOURCE_REPO': 'https://hg.mozilla.org/try',
[task 2019-04-26T15:15:42.033Z] 15:15:42 INFO - 'MOZ_TOOLCHAINS': 'public/build/android-gradle-dependencies.tar.xz@GRm3louDSM-UepytOLhtTw project/gecko/android-ndk/android-ndk.tar.xz@K6gwESP3SUiBmmUjSfHJeA project/gecko/android-sdk/android-sdk-linux.tar.xz@HMqXRoraTgqC6lZYa6tHBg public/build/clang.tar.xz@IBrD876tTvWCfw0veoCnQw public/build/rustc.tar.xz@EI2hOpqkTRaj6cWFMNeNkg public/build/rust-size.tar.xz@IlyF0MjpRDOhehriA_ZOnQ public/build/cbindgen.tar.xz@bPEue9VVQ_eRjPuRyJhmOA public/build/nasm.tar.bz2@cpcO-FU_QLGuIOWcHJuReQ public/build/node.tar.xz@F3EyjOKoQSy1Hfu4UC4kQw',
[task 2019-04-26T15:15:42.033Z] 15:15:42 INFO - 'MOZ_UPDATE_CHANNEL': 'beta',
[task 2019-04-26T15:15:42.033Z] 15:15:42 INFO - 'NEED_XVFB': 'false',
[task 2019-04-26T15:15:42.033Z] 15:15:42 INFO - 'NEXUS_SHA1SUM': '1a9aaad8414baffe0a2fd46eed1f41b85f4049e6',
[task 2019-04-26T15:15:42.033Z] 15:15:42 INFO - 'NEXUS_VERSION': '2.12.0-01',
[task 2019-04-26T15:15:42.033Z] 15:15:42 INFO - 'NEXUS_WORK': '/builds/worker/workspace/nexus',
[task 2019-04-26T15:15:42.033Z] 15:15:42 INFO - 'OLDPWD': '/',
[task 2019-04-26T15:15:42.033Z] 15:15:42 INFO - 'PATH': '/usr/local/bin:/bin:/usr/bin',
[task 2019-04-26T15:15:42.033Z] 15:15:42 INFO - 'PWD': '/builds/worker',
[task 2019-04-26T15:15:42.033Z] 15:15:42 INFO - 'PYTHONDONTWRITEBYTECODE': '1',
[task 2019-04-26T15:15:42.033Z] 15:15:42 INFO - 'PYTHONUNBUFFERED': '1',
[task 2019-04-26T15:15:42.033Z] 15:15:42 INFO - 'RELENGAPI_NAME': '/stoic_boyd/relengapi',
[task 2019-04-26T15:15:42.033Z] 15:15:42 INFO - 'RELENGAPI_PORT': 'tcp://172.17.0.3:80',
[task 2019-04-26T15:15:42.033Z] 15:15:42 INFO - 'RELENGAPI_PORT_80_TCP': 'tcp://172.17.0.3:80',
[task 2019-04-26T15:15:42.033Z] 15:15:42 INFO - 'RELENGAPI_PORT_80_TCP_ADDR': '172.17.0.3',
[task 2019-04-26T15:15:42.033Z] 15:15:42 INFO - 'RELENGAPI_PORT_80_TCP_PORT': '80',
[task 2019-04-26T15:15:42.033Z] 15:15:42 INFO - 'RELENGAPI_PORT_80_TCP_PROTO': 'tcp',
[task 2019-04-26T15:15:42.033Z] 15:15:42 INFO - 'RUN_ID': '6',
[task 2019-04-26T15:15:42.033Z] 15:15:42 INFO - 'SCCACHE_DISABLE': '1',
[task 2019-04-26T15:15:42.033Z] 15:15:42 INFO - 'SHELL': '/bin/bash',
[task 2019-04-26T15:15:42.033Z] 15:15:42 INFO - 'SHIP_LICENSED_FONTS': '1',
[task 2019-04-26T15:15:42.033Z] 15:15:42 INFO - 'SHLVL': '1',
[task 2019-04-26T15:15:42.033Z] 15:15:42 INFO - 'TASKCLUSTER_INSTANCE_TYPE': 'c5d.4xlarge',
[task 2019-04-26T15:15:42.033Z] 15:15:42 INFO - 'TASKCLUSTER_NAME': '/stoic_boyd/taskcluster',
[task 2019-04-26T15:15:42.033Z] 15:15:42 INFO - 'TASKCLUSTER_PORT': 'tcp://172.17.0.2:80',
[task 2019-04-26T15:15:42.033Z] 15:15:42 INFO - 'TASKCLUSTER_PORT_80_TCP': 'tcp://172.17.0.2:80',
[task 2019-04-26T15:15:42.033Z] 15:15:42 INFO - 'TASKCLUSTER_PORT_80_TCP_ADDR': '172.17.0.2',
[task 2019-04-26T15:15:42.033Z] 15:15:42 INFO - 'TASKCLUSTER_PORT_80_TCP_PORT': '80',
[task 2019-04-26T15:15:42.033Z] 15:15:42 INFO - 'TASKCLUSTER_PORT_80_TCP_PROTO': 'tcp',
[task 2019-04-26T15:15:42.033Z] 15:15:42 INFO - 'TASKCLUSTER_PROXY_URL': 'http://taskcluster',
[task 2019-04-26T15:15:42.033Z] 15:15:42 INFO - 'TASKCLUSTER_PUBLIC_IP': '34.228.247.90',
[task 2019-04-26T15:15:42.033Z] 15:15:42 INFO - 'TASKCLUSTER_ROOT_URL': 'https://taskcluster.net',
[task 2019-04-26T15:15:42.033Z] 15:15:42 INFO - 'TASKCLUSTER_WORKER_GROUP': 'us-east-1',
[task 2019-04-26T15:15:42.033Z] 15:15:42 INFO - 'TASKCLUSTER_WORKER_TYPE': 'gecko-1-b-linux',
[task 2019-04-26T15:15:42.034Z] 15:15:42 INFO - 'TASK_ID': 'eLU_lg5yS5ecQXWERzaLeA',
[task 2019-04-26T15:15:42.034Z] 15:15:42 INFO - 'TERM': 'xterm',
[task 2019-04-26T15:15:42.034Z] 15:15:42 INFO - 'TINDERBOX_OUTPUT': '1',
[task 2019-04-26T15:15:42.034Z] 15:15:42 INFO - 'TOOLTOOL_CACHE': '/builds/worker/tooltool-cache',
[task 2019-04-26T15:15:42.034Z] 15:15:42 INFO - 'TOOLTOOL_HOME': '/builds',
[task 2019-04-26T15:15:42.034Z] 15:15:42 INFO - 'TOOLTOOL_MANIFEST': 'mobile/android/config/tooltool-manifests/android/releng.manifest',
[task 2019-04-26T15:15:42.034Z] 15:15:42 INFO - 'TRY_COMMIT_MSG': '',
[task 2019-04-26T15:15:42.034Z] 15:15:42 INFO - 'USER': 'worker',
[task 2019-04-26T15:15:42.034Z] 15:15:42 INFO - '': '/builds/worker/workspace/build/src/mach'}
[task 2019-04-26T15:15:42.438Z] 15:15:42 INFO - Return code: 0
[task 2019-04-26T15:15:42.438Z] 15:15:42 INFO - Running post-run listener: _shutdown_sccache
[task 2019-04-26T15:15:42.438Z] 15:15:42 INFO - Running post-run listener: _summarize
[task 2019-04-26T15:15:42.438Z] 15:15:42 ERROR - # TBPL FAILURE #
[task 2019-04-26T15:15:42.438Z] 15:15:42 INFO - [mozharness: 2019-04-26 15:15:42.438442Z] FxDesktopBuild summary:
[task 2019-04-26T15:15:42.438Z] 15:15:42 ERROR - # TBPL FAILURE #
[fetches 2019-04-26T15:15:42.464Z] removing /builds/worker/fetches
[fetches 2019-04-26T15:15:42.464Z] finished
[taskcluster 2019-04-26 15:15:42.816Z] === Task Finished ===
[taskcluster 2019-04-26 15:15:42.944Z] Artifact "public/build/geckoview_example.apk" not found at "/builds/worker/workspace/build/src/obj-firefox/gradle/build/mobile/android/geckoview_example/outputs/apk/withGeckoBinaries/debug/geckoview_example-withGeckoBinaries-debug.apk"
[taskcluster 2019-04-26 15:15:43.124Z] Artifact "public/android/maven" not found at "/builds/worker/workspace/build/src/obj-firefox/gradle/build/mobile/android/geckoview/maven/"
[taskcluster 2019-04-26 15:15:43.228Z] Artifact "public/build/target.maven.zip" not found at "/builds/worker/workspace/build/src/obj-firefox/gradle/build/mobile/android/geckoview/target.maven.zip"
[taskcluster 2019-04-26 15:15:43.340Z] Artifact "public/build/geckoview-androidTest.apk" not found at "/builds/worker/workspace/build/src/obj-firefox/gradle/build/mobile/android/geckoview/outputs/apk/androidTest/withGeckoBinaries/debug/geckoview-withGeckoBinaries-debug-androidTest.apk"
[taskcluster 2019-04-26 15:15:44.047Z] Unsuccessful task run with exit code: 255 completed in 355.58 seconds

Nathan can this be from bug 1540082?

Flags: needinfo?(nfroyd)
Regressed by: 1540082
Has Regression Range: --- → yes
Keywords: regression

I don't see why, as those changes don't affect android builds; the toolchain they created doesn't get used by anything. Mike, Chris, do you know why the Android PGO builds would be failing on beta?

Flags: needinfo?(nfroyd)
Flags: needinfo?(mshal)
Flags: needinfo?(cmanchester)

Does this happen on every push? Both the 'Android 4.0 API16+ opt N' and 'Android 4.0 API16+ pgo Bpgo(B)' builds consume the data from the Bpgo(run) task. Although the run task completed successfully, it looks like the emulator may have been killed while writing out the profile data, leaving a corrupt artifact. I'm wondering if a new Bpgo(run) task will have the same corruption or if it will fix both the N and Bpgo(B) builds. In either case, it sounds like we'll want to have some way of verifying that the profile data was successfully written out in the run task. I'll take a look at that.

Flags: needinfo?(mshal)
Flags: needinfo?(cmanchester)

Andreea - Can we close this one out based on Comment 4?

Flags: needinfo?(apavel)

Not occurred for the last 3 days.

Status: NEW → RESOLVED
Closed: 5 years ago
Flags: needinfo?(apavel)
Resolution: --- → WORKSFORME
Product: Firefox for Android → Firefox for Android Graveyard
You need to log in before you can comment on or make changes to this bug.