1763343 - Perma android Bgv builds gmake[1]: *** [/builds/worker/checkouts/gecko/build/moz-automation.mk:110: automation/buildsymbols] Error 2

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, defect, P1)

Description

[Treeherder Bug Filer]

Filed by: imoraru [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=373569303&repo=mozilla-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/H9Op82kcQZGZ692d4dfSFg/runs/0/artifacts/public/logs/live_backing.log

---

[task 2022-04-06T07:18:52.424Z] 07:18:52     INFO -  package> gmake[5]: Leaving directory '/builds/worker/workspace/obj-build/mobile/android/installer'
[task 2022-04-06T07:18:52.450Z] 07:18:52     INFO -  buildsymbols> gmake[2]: Entering directory '/builds/worker/workspace/obj-build'
[task 2022-04-06T07:18:52.451Z] 07:18:52     INFO -  buildsymbols> rm -f 'dist/target.crashreporter-symbols-full.tar.zst'
[task 2022-04-06T07:18:52.451Z] 07:18:52     INFO -  buildsymbols> /builds/worker/workspace/obj-build/_virtualenvs/build/bin/python -m mozbuild.action.symbols_archive 'dist/target.crashreporter-symbols-full.tar.zst' /builds/worker/workspace/obj-build/dist/crashreporter-symbols --full-archive
[task 2022-04-06T07:18:52.451Z] 07:18:52    ERROR -  buildsymbols> Traceback (most recent call last):
[task 2022-04-06T07:18:52.451Z] 07:18:52     INFO -  buildsymbols>   File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
[task 2022-04-06T07:18:52.451Z] 07:18:52     INFO -  buildsymbols>     return _run_code(code, main_globals, None,
[task 2022-04-06T07:18:52.451Z] 07:18:52     INFO -  buildsymbols>   File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
[task 2022-04-06T07:18:52.451Z] 07:18:52     INFO -  buildsymbols>     exec(code, run_globals)
[task 2022-04-06T07:18:52.452Z] 07:18:52     INFO -  buildsymbols>   File "/builds/worker/checkouts/gecko/python/mozbuild/mozbuild/action/symbols_archive.py", line 90, in <module>
[task 2022-04-06T07:18:52.452Z] 07:18:52     INFO -  buildsymbols>     main(sys.argv[1:])
[task 2022-04-06T07:18:52.452Z] 07:18:52     INFO -  buildsymbols>   File "/builds/worker/checkouts/gecko/python/mozbuild/mozbuild/action/symbols_archive.py", line 86, in main
[task 2022-04-06T07:18:52.452Z] 07:18:52     INFO -  buildsymbols>     make_archive(args.archive, args.base, excludes, includes)
[task 2022-04-06T07:18:52.452Z] 07:18:52     INFO -  buildsymbols>   File "/builds/worker/checkouts/gecko/python/mozbuild/mozbuild/action/symbols_archive.py", line 47, in make_archive
[task 2022-04-06T07:18:52.452Z] 07:18:52     INFO -  buildsymbols>     import zstandard
[task 2022-04-06T07:18:52.452Z] 07:18:52     INFO -  buildsymbols> ModuleNotFoundError: No module named 'zstandard'
[task 2022-04-06T07:18:52.452Z] 07:18:52    ERROR -  buildsymbols> gmake[2]: *** [Makefile:186: symbolsfullarchive] Error 1
[task 2022-04-06T07:18:52.452Z] 07:18:52     INFO -  buildsymbols> gmake[2]: Leaving directory '/builds/worker/workspace/obj-build'
[task 2022-04-06T07:18:52.453Z] 07:18:52     INFO -  buildsymbols> gmake[2]: *** Waiting for unfinished jobs....
[task 2022-04-06T07:18:52.520Z] 07:18:52     INFO -  package> gmake[5]: Entering directory '/builds/worker/workspace/obj-build/mobile/android/installer'
<...>
[task 2022-04-06T07:18:58.001Z] 07:18:58     INFO -  buildsymbols> gmake[2]: Entering directory '/builds/worker/workspace/obj-build'
[task 2022-04-06T07:18:58.001Z] 07:18:58     INFO -  buildsymbols> rm -f 'dist/target.crashreporter-symbols.zip'
[task 2022-04-06T07:18:58.001Z] 07:18:58     INFO -  buildsymbols> /builds/worker/workspace/obj-build/_virtualenvs/build/bin/python -m mozbuild.action.symbols_archive 'dist/target.crashreporter-symbols.zip' /builds/worker/workspace/obj-build/dist/crashreporter-symbols
[task 2022-04-06T07:18:58.001Z] 07:18:58     INFO -  buildsymbols>   Adding to "target.crashreporter-symbols.zip":
[task 2022-04-06T07:18:58.001Z] 07:18:58     INFO -  buildsymbols> 	"BadCertAndPinningServer/2C225DCE214123E982C7C1F2F2FF2D900/BadCertAndPinningServer.sym"
[task 2022-04-06T07:18:58.002Z] 07:18:58     INFO -  buildsymbols>   Adding to "target.crashreporter-symbols.zip":
[task 2022-04-06T07:18:58.002Z] 07:18:58     INFO -  buildsymbols> 	"DelegatedCredentialsServer/1F90AB49A0DA1BE751777B642B4C76E50/DelegatedCredentialsServer.sym"
[task 2022-04-06T07:18:58.002Z] 07:18:58     INFO -  buildsymbols>   Adding to "target.crashreporter-symbols.zip":
[task 2022-04-06T07:18:58.002Z] 07:18:58     INFO -  buildsymbols> 	"EncryptedClientHelloServer/87E188904901EA6B0701780CAA7B439F0/EncryptedClientHelloServer.sym"
[task 2022-04-06T07:18:58.002Z] 07:18:58     INFO -  buildsymbols>   Adding to "target.crashreporter-symbols.zip":
<...>
[task 2022-04-06T07:18:58.018Z] 07:18:58     INFO -  buildsymbols>   Adding to "target.crashreporter-symbols.zip":
[task 2022-04-06T07:18:58.018Z] 07:18:58     INFO -  buildsymbols> 	"xpcshell/F19D07953BDBAAFE1351EAED70D659380/xpcshell.sym"
[task 2022-04-06T07:18:58.019Z] 07:18:58     INFO -  buildsymbols> gmake[2]: Leaving directory '/builds/worker/workspace/obj-build'
[task 2022-04-06T07:18:58.019Z] 07:18:58    ERROR -  gmake[1]: *** [/builds/worker/checkouts/gecko/build/moz-automation.mk:110: automation/buildsymbols] Error 2
[task 2022-04-06T07:18:58.019Z] 07:18:58     INFO -  gmake: *** [client.mk:67: build] Error 2
[task 2022-04-06T07:18:58.036Z] 07:18:58     INFO -  367 compiler warnings present.
[task 2022-04-06T07:18:58.207Z] 07:18:58    ERROR - Return code: 2
[task 2022-04-06T07:18:58.207Z] 07:18:58  WARNING - setting return code to 2
[task 2022-04-06T07:18:58.207Z] 07:18:58    FATAL - 'mach build -v' did not run successfully. Please check log for errors.
[task 2022-04-06T07:18:58.207Z] 07:18:58    FATAL - Running post_fatal callback...
[task 2022-04-06T07:18:58.207Z] 07:18:58    FATAL - Exiting -1
[task 2022-04-06T07:18:58.207Z] 07:18:58     INFO - [mozharness: 2022-04-06 07:18:58.207610Z] Finished build step (failed)

Comment 1

[Iulian Moraru]

Hi Mitchell! Can you please take a look at this?
Thank you!

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1755515

Comment 3

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1763343: Android AAR builds should use system Python packages

These tasks:

• Are builds: so pip install is frowned upon
• Use zstandard: so it better be installed in the system environment

Opt into using the system python packages accordingly.

Comment 4

[Mitchell Hentges [:mhentges] 🦀]

You got it, look successful :)

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/098884d5de4d
Android AAR builds should use system Python packages r=firefox-build-system-reviewers,nalexander

Comment 6

[Ryan VanderMeulen [:RyanVM]]

Do we understand why this broke on mozilla-central but not on autoland? I see shippable fat-AAR builds there, but they never started failing during all of this.

Comment 7

[Mitchell Hentges [:mhentges] 🦀]

Do we understand why this broke on mozilla-central but not on autoland? I see shippable fat-AAR builds there, but they never started failing during all of this.

Ah, it's because this failure is only encountered when generating "full" crashreporter symbols.
Looking at an autoland fat AAR build log, we can't find MOZ_ENABLE_FULL_SYMBOLS, but we do see it defined in this green central build log.
It gets fuzzier for how the tasks are defined differently, but I'm guessing that perhaps this transform has something to do with it.

Comment 8

[Norisz Fay [:noriszfay]]

https://hg.mozilla.org/mozilla-central/rev/098884d5de4d

Comment 9

[Intermittent Failures Robot]

10 failures in 2922 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 10

Platform and build breakdown:

• android-5-0-geckoview-fat-aar-shippable: 5
  • opt: 5
• android-4-1-geckoview-fat-aar-shippable: 5
  • opt: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1763343&startday=2022-04-04&endday=2022-04-10&tree=all