Closed Bug 1517177 Opened 2 years ago Closed 2 years ago

./mach bootstrap fails if download cache is corrupted

Categories

(Firefox Build System :: Bootstrap Configuration, defect)

defect
Not set
major

Tracking

(firefox-esr60 unaffected, firefox65 unaffected, firefox66 fixed, firefox67 fixed)

RESOLVED FIXED
mozilla67
Tracking Status
firefox-esr60 --- unaffected
firefox65 --- unaffected
firefox66 --- fixed
firefox67 --- fixed

People

(Reporter: emk, Assigned: tomprince)

References

Details

(Keywords: regression)

Attachments

(1 file, 1 obsolete file)

$ ./mach bootstrap
mach bootstrap is not fully implemented in MozillaBuild

Note on Artifact Mode:

Artifact builds download prebuilt C++ components rather than building
them locally. Artifact builds are faster!

Artifact builds are recommended for people working on Firefox or
Firefox for Android frontends. They are unsuitable for those working
on C++ code. For more information see:
https://developer.mozilla.org/en-US/docs/Artifact_builds.

Please choose the version of Firefox you want to build:
1. Firefox for Desktop Artifact Mode
2. Firefox for Desktop
3. Firefox for Android Artifact Mode
4. Firefox for Android
Your choice: 2
Running pip to ensure Mercurial is up-to-date...
Requirement already up-to-date: Mercurial in d:\mozilla-build\python\lib\site-packages (4.8.1)
You are using pip version 10.0.1, however version 18.1 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.
Your version of Python (2.7.15) is new enough.
Your version of Rust (1.31.1) is new enough.
Rust supports i686-pc-windows-msvc, x86_64-pc-windows-msvc targets.

Mozilla recommends a number of changes to Mercurial to enhance your
experience with it.

Would you like to run a configuration wizard to ensure Mercurial is
optimally configured?

  1. Yes
  2. No

Please enter your reply: 2
 0:20.98 Downloading clang.tar.bz2
 0:20.98 Downloading to temporary location c:\Users\kimu\.mozbuild\toolchains\5a8e6758878eade2-clang.tar.bz2
 0:20.98 Downloaded artifact to c:\Users\kimu\.mozbuild\toolchains\5a8e6758878eade2-clang.tar.bz2
 0:24.23 rm tree: c:\Users\kimu\.mozbuild\clang
 0:24.23 untarring "c:\Users\kimu\.mozbuild\clang.tar.bz2"
Error running mach:

    ['artifact', 'toolchain', '--from-build', 'win64-clang-cl']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

IOError: invalid data stream

  File "e:\m\mozilla-unified\python/mozbuild/mozbuild/mach_commands.py", line 1563, in artifact_toolchain
    unpack_file(local, record.setup)
  File "e:\m\mozilla-unified\python/mozbuild\mozbuild\action\tooltool.py", line 533, in unpack_file
    tar.extractall()
  File "d:\mozilla-build\python\lib\tarfile.py", line 2081, in extractall
    self.extract(tarinfo, path)
  File "d:\mozilla-build\python\lib\tarfile.py", line 2118, in extract
    self._extract_member(tarinfo, os.path.join(path, tarinfo.name))
  File "d:\mozilla-build\python\lib\tarfile.py", line 2194, in _extract_member
    self.makefile(tarinfo, targetpath)
  File "d:\mozilla-build\python\lib\tarfile.py", line 2235, in makefile
    copyfileobj(source, target)
  File "d:\mozilla-build\python\lib\tarfile.py", line 266, in copyfileobj
    shutil.copyfileobj(src, dst)
  File "d:\mozilla-build\python\lib\shutil.py", line 63, in copyfileobj
    buf = fsrc.read(length)
  File "d:\mozilla-build\python\lib\tarfile.py", line 831, in read
    buf += self.fileobj.read(size - len(buf))
  File "d:\mozilla-build\python\lib\tarfile.py", line 743, in read
    return self.readnormal(size)
  File "d:\mozilla-build\python\lib\tarfile.py", line 758, in readnormal
    return self.__read(size)
  File "d:\mozilla-build\python\lib\tarfile.py", line 748, in __read
    buf = self.fileobj.read(size)
Error running mach:

    ['bootstrap']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

CalledProcessError: Command '['d:\\mozilla-build\\python\\python2.7.exe', u'e:\\m\\mozilla-unified\\mach', u'artifact', u'toolchain', u'--from-build', u'win64-clang-cl']' returned non-zero exit status 1

  File "e:\m\mozilla-unified\python/mozboot/mozboot/mach_commands.py", line 43, in bootstrap
    bootstrapper.bootstrap()
  File "e:\m\mozilla-unified\python/mozboot\mozboot\bootstrap.py", line 507, in bootstrap
    checkout_root)
  File "e:\m\mozilla-unified\python/mozboot\mozboot\bootstrap.py", line 383, in maybe_install_private_packages_or_exit
    self.instance.ensure_stylo_packages(state_dir, checkout_root)
  File "e:\m\mozilla-unified\python/mozboot\mozboot\mozillabuild.py", line 53, in ensure_stylo_packages
    self.install_toolchain_artifact(state_dir, checkout_root, stylo.WINDOWS_CLANG)
  File "e:\m\mozilla-unified\python/mozboot\mozboot\base.py", line 312, in install_toolchain_artifact
    subprocess.check_call(cmd, cwd=state_dir)
  File "d:\mozilla-build\python\lib\subprocess.py", line 190, in check_call
    raise CalledProcessError(retcode, cmd)
I can no longer build Firefox at all because clang-cl is not installed.
Manually extracting %userprofile%\.mozbuild\clang.tar.bz2 also failed (CRC error). is 5a8e6758878eade2-clang.tar.bz2 broken?
Reverting bug 1512188 fixed the issue. Please backout.
I'll backout by myself if nobody responds.
Assignee: nobody → VYV03354
Status: NEW → ASSIGNED
Assignee: VYV03354 → nobody
Status: ASSIGNED → NEW
Severity: normal → major

I'm away so I'll leave this for @tomprince to look at.

Flags: needinfo?(dustin)

I'm fairly certain that you have have a corrupt download at c:\Users\kimu.mozbuild\toolchains\5a8e6758878eade2-clang.tar.bz2 and removing it will cause it to be redownloaded an succeed. The artifact itself is being used in automation builds, so isn't corrupted.

Flags: needinfo?(mozilla)
Attachment #9035003 - Flags: review-

I've attached a patch that should catch the corruption and attempt to download again.

Comment on attachment 9035003 [details] [diff] [review]
Backed out 8 changesets (bug 1512285, bug 1512188) for breaking ./mach bootstrap on Windows msys

(In reply to Tom Prince [:tomprince] from comment #7)

I'm fairly certain that you have have a corrupt download at c:\Users\kimu.mozbuild\toolchains\5a8e6758878eade2-clang.tar.bz2 and removing it will cause it to be redownloaded an succeed. The artifact itself is being used in automation builds, so isn't corrupted.

You are right, removing cache fixed the issue. Thank you!

Attachment #9035003 - Attachment is obsolete: true
Summary: ./mach bootstrap fails → ./mach bootstrap fails if download cache is corrupted
Attachment #9035078 - Attachment description: Bug 1517177: Fix hash verification of downloaded toolchains; → Bug 1517177: Fix hash verification of downloaded toolchains; r?glandium
Pushed by mozilla@hocat.ca:
https://hg.mozilla.org/integration/autoland/rev/e793c88c1fd8
Fix hash verification of downloaded toolchains; r=glandium
Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla67
Assignee: nobody → mozilla

Not sure how many people are going to try to bootstrap off a release branch, but I could see it being possible for someone wanting to build 66 once it goes to release. Might as well uplift this a=NPOTB.

Whiteboard: [checkin-needed-beta]
Whiteboard: [checkin-needed-beta]
You need to log in before you can comment on or make changes to this bug.