1680675 - Perma win aarch shippable build: ModuleNotFoundError: No module named 'zstandard'

Status: RESOLVED FIXED

(Firefox Build System :: General, defect, P1)

Description

[Treeherder Bug Filer]

Filed by: apavel [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=323542863&repo=mozilla-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/QXYLf7zXQSSk4JQ6TlpCaw/runs/0/artifacts/public/logs/live_backing.log

---

[task 2020-12-04T05:26:13.118Z] 05:26:13     INFO -  Error running mach:
[task 2020-12-04T05:26:13.119Z] 05:26:13     INFO -      ['--log-no-times', 'artifact', 'install', '--symbols=full', '--host-bins', '-v']
[task 2020-12-04T05:26:13.119Z] 05:26:13     INFO -  The error occurred in code that was called by the mach command. This is either
[task 2020-12-04T05:26:13.119Z] 05:26:13     INFO -  a bug in the called code itself or in the way that mach is calling it.
[task 2020-12-04T05:26:13.119Z] 05:26:13     INFO -  You can invoke |./mach busted| to check if this issue is already on file. If it
[task 2020-12-04T05:26:13.119Z] 05:26:13     INFO -  isn't, please use |./mach busted file artifact| to report it. If |./mach busted| is
[task 2020-12-04T05:26:13.119Z] 05:26:13     INFO -  misbehaving, you can also inspect the dependencies of bug 1543241.
[task 2020-12-04T05:26:13.119Z] 05:26:13     INFO -  If filing a bug, please include the full output of mach, including this error
[task 2020-12-04T05:26:13.120Z] 05:26:13     INFO -  message.
[task 2020-12-04T05:26:13.120Z] 05:26:13     INFO -  The details of the failure are as follows:
[task 2020-12-04T05:26:13.120Z] 05:26:13     INFO -  ModuleNotFoundError: No module named 'zstandard'
[task 2020-12-04T05:26:13.120Z] 05:26:13     INFO -    File "/builds/worker/checkouts/gecko/python/mozbuild/mozbuild/artifact_commands.py", line 204, in artifact_install
[task 2020-12-04T05:26:13.120Z] 05:26:13     INFO -      return artifacts.install_from(source, distdir or self.distdir)
[task 2020-12-04T05:26:13.120Z] 05:26:13     INFO -    File "/builds/worker/checkouts/gecko/python/mozbuild/mozbuild/artifacts.py", line 1520, in install_from
[task 2020-12-04T05:26:13.120Z] 05:26:13     INFO -      return self.install_from_task(os.environ[var], distdir)
[task 2020-12-04T05:26:13.121Z] 05:26:13     INFO -    File "/builds/worker/checkouts/gecko/python/mozbuild/mozbuild/artifacts.py", line 1498, in install_from_task
[task 2020-12-04T05:26:13.121Z] 05:26:13     INFO -      if self.install_from_url(url, distdir):
[task 2020-12-04T05:26:13.121Z] 05:26:13     INFO -    File "/builds/worker/checkouts/gecko/python/mozbuild/mozbuild/artifacts.py", line 1387, in install_from_url
[task 2020-12-04T05:26:13.121Z] 05:26:13     INFO -      return self.install_from_file(filename, distdir)
[task 2020-12-04T05:26:13.121Z] 05:26:13     INFO -    File "/builds/worker/checkouts/gecko/python/mozbuild/mozbuild/artifacts.py", line 1346, in install_from_file
[task 2020-12-04T05:26:13.121Z] 05:26:13     INFO -      self._artifact_job.process_artifact(filename, processed_filename)
[task 2020-12-04T05:26:13.121Z] 05:26:13     INFO -    File "/builds/worker/checkouts/gecko/python/mozbuild/mozbuild/artifacts.py", line 227, in process_artifact
[task 2020-12-04T05:26:13.122Z] 05:26:13     INFO -      return self.process_symbols_archive(filename, processed_filename)
[task 2020-12-04T05:26:13.122Z] 05:26:13     INFO -    File "/builds/worker/checkouts/gecko/python/mozbuild/mozbuild/artifacts.py", line 370, in process_symbols_archive
[task 2020-12-04T05:26:13.122Z] 05:26:13     INFO -      for filename, entry in self.iter_artifact_archive(filename):
[task 2020-12-04T05:26:13.122Z] 05:26:13     INFO -    File "/builds/worker/checkouts/gecko/python/mozbuild/mozbuild/artifacts.py", line 403, in iter_artifact_archive
[task 2020-12-04T05:26:13.122Z] 05:26:13     INFO -      import zstandard
[task 2020-12-04T05:26:13.122Z] 05:26:13    ERROR -  make[3]: *** [Makefile:126: recurse_artifact] Error 1
[task 2020-12-04T05:26:13.122Z] 05:26:13     INFO -  make[3]: Leaving directory '/builds/worker/workspace/obj-build'
[task 2020-12-04T05:26:13.123Z] 05:26:13    ERROR -  make[2]: *** [/builds/worker/checkouts/gecko/config/recurse.mk:34: artifact] Error 2
[task 2020-12-04T05:26:13.123Z] 05:26:13    ERROR -  make[1]: *** [/builds/worker/checkouts/gecko/config/rules.mk:355: default] Error 2
[task 2020-12-04T05:26:13.123Z] 05:26:13     INFO -  make: *** [client.mk:89: build] Error 2
[task 2020-12-04T05:26:13.123Z] 05:26:13     INFO -  0 compiler warnings present.
[task 2020-12-04T05:26:13.183Z] 05:26:13    ERROR - Return code: 2
[task 2020-12-04T05:26:13.183Z] 05:26:13  WARNING - setting return code to 2
[task 2020-12-04T05:26:13.183Z] 05:26:13    FATAL - 'mach build -v' did not run successfully. Please check log for errors.
[task 2020-12-04T05:26:13.183Z] 05:26:13    FATAL - Running post_fatal callback...
[task 2020-12-04T05:26:13.183Z] 05:26:13    FATAL - Exiting -1
[task 2020-12-04T05:26:13.184Z] 05:26:13     INFO - [mozharness: 2020-12-04 05:26:13.184002Z] Finished build step (failed)
[task 2020-12-04T05:26:13.184Z] 05:26:13     INFO - Running post-run listener: _parse_build_tests_ccov
[task 2020-12-04T05:26:13.184Z] 05:26:13     INFO - Update channel set to: nightly```

Comment 1

[Natalia Csoregi [:nataliaCs]]

:mhentges seems to have started once Bug 1661153 was merged to mozilla-central.
Could you take a look please?
Thank you.

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1661153

Comment 3

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1680675: Re-adds manual zstandard installation for CI

We run ./mach artifact install in CI, and the CI environment doesn't
bootstrap/create-mach-environment, so native modules (such as
zstandard) aren't installed.
Since we need zstandard for this command, we must install it ad-hoc.

Comment 4

[Intermittent Failures Robot]

19 failures in 3712 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 11
• try: 8

Platform breakdown:

• windows2012-aarch64-shippable: 15
• windows2012-aarch64-devedition: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1680675&startday=2020-11-30&endday=2020-12-06&tree=all

Comment 5

[BugBot [:suhaib / :marco/ :calixte]]

Changing the priority to p2 as the bug is tracked by a release manager for the current nightly.
See What Do You Triage for more information

Comment 6

[Julien Cristau [:jcristau]]

This will block shipping betas next week.

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a812a52adea8
Re-adds manual zstandard installation for CI r=firefox-build-system-reviewers,dmajor

Comment 8

[Dorel Luca [:dluca]]

https://hg.mozilla.org/mozilla-central/rev/a812a52adea8

Comment 9

[Intermittent Failures Robot]

5 failures in 4060 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 3
• try: 2

Platform breakdown:

• windows2012-aarch64-shippable: 4
• windows2012-aarch64-devedition: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1680675&startday=2020-12-07&endday=2020-12-13&tree=all

Comment 10

[Intermittent Failures Robot]

1 failures in 3683 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 1

Platform and build breakdown:

• android-5-0-aarch64-shippable: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1680675&startday=2021-03-01&endday=2021-03-07&tree=all