./mach bootstrap fails if download cache is corrupted

RESOLVED FIXED in Firefox 66

Status

defect
--
major
RESOLVED FIXED
5 months ago
3 months ago

People

(Reporter: emk, Assigned: tomprince)

Tracking

({regression})

unspecified
mozilla67

Firefox Tracking Flags

(firefox-esr60 unaffected, firefox65 unaffected, firefox66 fixed, firefox67 fixed)

Details

Attachments

(1 attachment, 1 obsolete attachment)

(Reporter)

Description

5 months ago
$ ./mach bootstrap
mach bootstrap is not fully implemented in MozillaBuild

Note on Artifact Mode:

Artifact builds download prebuilt C++ components rather than building
them locally. Artifact builds are faster!

Artifact builds are recommended for people working on Firefox or
Firefox for Android frontends. They are unsuitable for those working
on C++ code. For more information see:
https://developer.mozilla.org/en-US/docs/Artifact_builds.

Please choose the version of Firefox you want to build:
1. Firefox for Desktop Artifact Mode
2. Firefox for Desktop
3. Firefox for Android Artifact Mode
4. Firefox for Android
Your choice: 2
Running pip to ensure Mercurial is up-to-date...
Requirement already up-to-date: Mercurial in d:\mozilla-build\python\lib\site-packages (4.8.1)
You are using pip version 10.0.1, however version 18.1 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.
Your version of Python (2.7.15) is new enough.
Your version of Rust (1.31.1) is new enough.
Rust supports i686-pc-windows-msvc, x86_64-pc-windows-msvc targets.

Mozilla recommends a number of changes to Mercurial to enhance your
experience with it.

Would you like to run a configuration wizard to ensure Mercurial is
optimally configured?

  1. Yes
  2. No

Please enter your reply: 2
 0:20.98 Downloading clang.tar.bz2
 0:20.98 Downloading to temporary location c:\Users\kimu\.mozbuild\toolchains\5a8e6758878eade2-clang.tar.bz2
 0:20.98 Downloaded artifact to c:\Users\kimu\.mozbuild\toolchains\5a8e6758878eade2-clang.tar.bz2
 0:24.23 rm tree: c:\Users\kimu\.mozbuild\clang
 0:24.23 untarring "c:\Users\kimu\.mozbuild\clang.tar.bz2"
Error running mach:

    ['artifact', 'toolchain', '--from-build', 'win64-clang-cl']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

IOError: invalid data stream

  File "e:\m\mozilla-unified\python/mozbuild/mozbuild/mach_commands.py", line 1563, in artifact_toolchain
    unpack_file(local, record.setup)
  File "e:\m\mozilla-unified\python/mozbuild\mozbuild\action\tooltool.py", line 533, in unpack_file
    tar.extractall()
  File "d:\mozilla-build\python\lib\tarfile.py", line 2081, in extractall
    self.extract(tarinfo, path)
  File "d:\mozilla-build\python\lib\tarfile.py", line 2118, in extract
    self._extract_member(tarinfo, os.path.join(path, tarinfo.name))
  File "d:\mozilla-build\python\lib\tarfile.py", line 2194, in _extract_member
    self.makefile(tarinfo, targetpath)
  File "d:\mozilla-build\python\lib\tarfile.py", line 2235, in makefile
    copyfileobj(source, target)
  File "d:\mozilla-build\python\lib\tarfile.py", line 266, in copyfileobj
    shutil.copyfileobj(src, dst)
  File "d:\mozilla-build\python\lib\shutil.py", line 63, in copyfileobj
    buf = fsrc.read(length)
  File "d:\mozilla-build\python\lib\tarfile.py", line 831, in read
    buf += self.fileobj.read(size - len(buf))
  File "d:\mozilla-build\python\lib\tarfile.py", line 743, in read
    return self.readnormal(size)
  File "d:\mozilla-build\python\lib\tarfile.py", line 758, in readnormal
    return self.__read(size)
  File "d:\mozilla-build\python\lib\tarfile.py", line 748, in __read
    buf = self.fileobj.read(size)
Error running mach:

    ['bootstrap']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

CalledProcessError: Command '['d:\\mozilla-build\\python\\python2.7.exe', u'e:\\m\\mozilla-unified\\mach', u'artifact', u'toolchain', u'--from-build', u'win64-clang-cl']' returned non-zero exit status 1

  File "e:\m\mozilla-unified\python/mozboot/mozboot/mach_commands.py", line 43, in bootstrap
    bootstrapper.bootstrap()
  File "e:\m\mozilla-unified\python/mozboot\mozboot\bootstrap.py", line 507, in bootstrap
    checkout_root)
  File "e:\m\mozilla-unified\python/mozboot\mozboot\bootstrap.py", line 383, in maybe_install_private_packages_or_exit
    self.instance.ensure_stylo_packages(state_dir, checkout_root)
  File "e:\m\mozilla-unified\python/mozboot\mozboot\mozillabuild.py", line 53, in ensure_stylo_packages
    self.install_toolchain_artifact(state_dir, checkout_root, stylo.WINDOWS_CLANG)
  File "e:\m\mozilla-unified\python/mozboot\mozboot\base.py", line 312, in install_toolchain_artifact
    subprocess.check_call(cmd, cwd=state_dir)
  File "d:\mozilla-build\python\lib\subprocess.py", line 190, in check_call
    raise CalledProcessError(retcode, cmd)
(Reporter)

Comment 1

5 months ago
I can no longer build Firefox at all because clang-cl is not installed.
(Reporter)

Comment 2

5 months ago
Manually extracting %userprofile%\.mozbuild\clang.tar.bz2 also failed (CRC error). is 5a8e6758878eade2-clang.tar.bz2 broken?
(Reporter)

Comment 5

4 months ago
Reverting bug 1512188 fixed the issue. Please backout.
I'll backout by myself if nobody responds.
(Reporter)

Updated

4 months ago
Assignee: nobody → VYV03354
Status: NEW → ASSIGNED
(Reporter)

Updated

4 months ago
Assignee: VYV03354 → nobody
Status: ASSIGNED → NEW
Severity: normal → major

I'm away so I'll leave this for @tomprince to look at.

Flags: needinfo?(dustin)
(Assignee)

Comment 7

4 months ago

I'm fairly certain that you have have a corrupt download at c:\Users\kimu.mozbuild\toolchains\5a8e6758878eade2-clang.tar.bz2 and removing it will cause it to be redownloaded an succeed. The artifact itself is being used in automation builds, so isn't corrupted.

Flags: needinfo?(mozilla)
(Assignee)

Updated

4 months ago
Attachment #9035003 - Flags: review-
(Assignee)

Comment 9

4 months ago

I've attached a patch that should catch the corruption and attempt to download again.

(Reporter)

Comment 10

4 months ago

Comment on attachment 9035003 [details] [diff] [review]
Backed out 8 changesets (bug 1512285, bug 1512188) for breaking ./mach bootstrap on Windows msys

(In reply to Tom Prince [:tomprince] from comment #7)

I'm fairly certain that you have have a corrupt download at c:\Users\kimu.mozbuild\toolchains\5a8e6758878eade2-clang.tar.bz2 and removing it will cause it to be redownloaded an succeed. The artifact itself is being used in automation builds, so isn't corrupted.

You are right, removing cache fixed the issue. Thank you!

Attachment #9035003 - Attachment is obsolete: true
(Reporter)

Comment 11

4 months ago
Summary: ./mach bootstrap fails → ./mach bootstrap fails if download cache is corrupted
Attachment #9035078 - Attachment description: Bug 1517177: Fix hash verification of downloaded toolchains; → Bug 1517177: Fix hash verification of downloaded toolchains; r?glandium

Comment 12

4 months ago
Pushed by mozilla@hocat.ca:
https://hg.mozilla.org/integration/autoland/rev/e793c88c1fd8
Fix hash verification of downloaded toolchains; r=glandium

Comment 13

4 months ago
bugherder
Status: NEW → RESOLVED
Last Resolved: 4 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla67
Assignee: nobody → mozilla

Not sure how many people are going to try to bootstrap off a release branch, but I could see it being possible for someone wanting to build 66 once it goes to release. Might as well uplift this a=NPOTB.

Whiteboard: [checkin-needed-beta]
Whiteboard: [checkin-needed-beta]
You need to log in before you can comment on or make changes to this bug.