1676107 - ERROR!!!!!! Could not find artifacts for a toolchain build named `win32-minidump-stackwalk` with latest central

Status: RESOLVED WORKSFORME

(Firefox Build System :: Bootstrap Configuration, defect)

Description

[Jean-Yves Avenard [:jya]]

Just pulled from mozilla-central, and ./mach bootstrap gives me the error:

 0:04.00 untarring "c:\Users\jyavenard\.mozbuild\fix-stacks.tar.bz2"
 0:03.38 ERROR!!!!!! Could not find artifacts for a toolchain build named `win32-minidump-stackwalk`. Local commits, dirty/stale files, and other changes in your checkout may cause this error. Make sure you are on a fresh, current checkout of mozilla-central. Beware that commands like `mach bootstrap` and `mach artifact` are unlikely to work on any versions of the code besides recent revisions of mozilla-central.
 0:04.67 Hint: consider reverting your local changes to the following files: ['build/common_virtualenv_packages.txt', 'dom/media/Benchmark.cpp', 'dom/media/MediaData.cpp', 'dom/media/MediaFormatReader.cpp', 'dom/media/MediaFormatReader.h', 'dom/media/ipc/IRemoteDecoderChild.h', 'dom/media/ipc/PRemoteDecoder.ipdl', 'dom/media/ipc/PRemoteDecoderManager.ipdl', 'dom/media/ipc/RDDProcessManager.cpp', 'dom/media/ipc/RDDProcessManager.h', 'dom/media/ipc/RemoteAudioDecoder.cpp', 'dom/media/ipc/RemoteAudioDecoder.h', 'dom/media/ipc/RemoteDecoderChild.cpp', 'dom/media/ipc/RemoteDecoderChild.h', 'dom/media/ipc/RemoteDecoderManagerChild.cpp', 'dom/media/ipc/RemoteDecoderManagerChild.h', 'dom/media/ipc/RemoteDecoderManagerParent.cpp', 'dom/media/ipc/RemoteDecoderManagerParent.h', 'dom/media/ipc/RemoteDecoderModule.cpp', 'dom/media/ipc/RemoteDecoderModule.h', 'dom/media/ipc/RemoteDecoderParent.cpp', 'dom/media/ipc/RemoteDecoderParent.h', 'dom/media/ipc/RemoteMediaDataDecoder.cpp', 'dom/media/ipc/RemoteMediaDataDecoder.h', 'dom/media/ipc/RemoteVideoDecoder.cpp', 'dom/media/ipc/RemoteVideoDecoder.h', 'dom/media/ipc/moz.build', 'dom/media/mp3/MP3Demuxer.cpp', 'dom/media/platforms/AllocationPolicy.cpp', 'dom/media/platforms/PDMFactory.cpp', 'dom/media/platforms/PDMFactory.h', 'dom/media/platforms/PlatformDecoderModule.cpp', 'dom/media/platforms/PlatformDecoderModule.h', 'dom/media/platforms/agnostic/AOMDecoder.h', 'dom/media/platforms/agnostic/AgnosticDecoderModule.cpp', 'dom/media/platforms/agnostic/DAV1DDecoder.h', 'dom/media/platforms/agnostic/OpusDecoder.cpp', 'dom/media/platforms/agnostic/OpusDecoder.h', 'dom/media/platforms/agnostic/TheoraDecoder.h', 'dom/media/platforms/agnostic/VPXDecoder.h', 'dom/media/platforms/agnostic/VorbisDecoder.cpp', 'dom/media/platforms/agnostic/VorbisDecoder.h', 'dom/media/platforms/agnostic/WAVDecoder.cpp', 'dom/media/platforms/agnostic/WAVDecoder.h', 'dom/media/platforms/agnostic/eme/EMEDecoderModule.cpp', 'dom/media/platforms/agnostic/eme/EMEDecoderModule.h', 'dom/media/platforms/agnostic/eme/SamplesWaitingForKey.cpp', 'dom/media/platforms/agnostic/eme/SamplesWaitingForKey.h', 'dom/media/platforms/agnostic/gmp/GMPVideoDecoder.cpp', 'dom/media/platforms/agnostic/gmp/GMPVideoDecoder.h', 'dom/media/platforms/android/AndroidDecoderModule.cpp', 'dom/media/platforms/android/RemoteDataDecoder.h', 'dom/media/platforms/apple/AppleATDecoder.cpp', 'dom/media/platforms/apple/AppleATDecoder.h', 'dom/media/platforms/apple/AppleDecoderModule.cpp', 'dom/media/platforms/ffmpeg/FFmpegAudioDecoder.cpp', 'dom/media/platforms/ffmpeg/FFmpegAudioDecoder.h', 'dom/media/platforms/ffmpeg/FFmpegDecoderModule.h', 'dom/media/platforms/moz.build', 'dom/media/platforms/omx/OmxDataDecoder.cpp', 'dom/media/platforms/wrappers/AudioTrimmer.cpp', 'dom/media/platforms/wrappers/MediaChangeMonitor.cpp', 'dom/media/platforms/wrappers/MediaChangeMonitor.h', 'dom/media/webaudio/MediaBufferDecoder.cpp', 'dom/media/webaudio/test/cropped_8000hz_8kbs_mono_lame3.100.abr.mp3', 'dom/media/webaudio/test/mochitest.ini', 'dom/media/webaudio/test/test_decoderDelay.html', 'dom/media/webaudio/test/v3.mp3', 'dom/media/webrtc/libwebrtcglue/MediaDataCodec.cpp', 'dom/media/webrtc/libwebrtcglue/WebrtcMediaDataDecoderCodec.cpp', 'dom/media/webrtc/libwebrtcglue/WebrtcMediaDataDecoderCodec.h', 'ipc/glue/BackgroundImpl.cpp', 'ipc/glue/BackgroundParentImpl.cpp', 'ipc/glue/BackgroundParentImpl.h', 'ipc/glue/GeckoChildProcessHost.cpp', 'ipc/glue/PBackground.ipdl', 'ipc/ipdl/sync-messages.ini', 'mfbt/WeakPtr.h', 'python/mozbuild/mozbuild/mach_commands.py', 'taskcluster/taskgraph/transforms/tests.py']
Error running mach:

    ['bootstrap']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke |./mach busted| to check if this issue is already on file. If it
isn't, please use |./mach busted file bootstrap| to report it. If |./mach busted| is
misbehaving, you can also inspect the dependencies of bug 1543241.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

subprocess.CalledProcessError: Command '['c:/Users/jyavenard/.mozbuild\\_virtualenvs\\mach\\Scripts\\python.exe', 'c:\\Users\\jyavenard\\Work\\Mozilla\\gecko\\mach', 'artifact', 'toolchain', '--bootstrap', '--from-build', 'win32-minidump-stackwalk']' returned non-zero exit status 1.

  File "c:\Users\jyavenard\Work\Mozilla\gecko\python/mozboot/mozboot/mach_commands.py", line 58, in bootstrap
    bootstrapper.bootstrap()
  File "c:\Users\jyavenard\Work\Mozilla\gecko\python/mozboot\mozboot\bootstrap.py", line 451, in bootstrap
    self.maybe_install_private_packages_or_exit(state_dir, checkout_root)
  File "c:\Users\jyavenard\Work\Mozilla\gecko\python/mozboot\mozboot\bootstrap.py", line 315, in maybe_install_private_packages_or_exit
    self.instance.ensure_minidump_stackwalk_packages(state_dir, checkout_root)
  File "c:\Users\jyavenard\Work\Mozilla\gecko\python/mozboot\mozboot\mozillabuild.py", line 295, in ensure_minidump_stackwalk_packages
    state_dir, checkout_root, minidump_stackwalk.WINDOWS_MINIDUMP_STACKWALK
  File "c:\Users\jyavenard\Work\Mozilla\gecko\python/mozboot\mozboot\base.py", line 398, in install_toolchain_artifact
    subprocess.check_call(cmd, cwd=state_dir)
  File "c:\mozilla-build\python3\lib\subprocess.py", line 347, in check_call
    raise CalledProcessError(retcode, cmd)

I'm configured to build a x864_64 build, not sure where this win32 is coming from

Comment 1

[Mike Hommey [:glandium]]

Hint: consider reverting your local changes to the following files:

Have you looked at this list of files? Does any of them contain changes? Are some of them not tracked by VCS? If the answer to both questions is no, then I bet it's a CRLF problem.

Comment 2

[Jean-Yves Avenard [:jya]]

(In reply to Mike Hommey [:glandium] from comment #1)

Hint: consider reverting your local changes to the following files:

Have you looked at this list of files? Does any of them contain changes? Are some of them not tracked by VCS? If the answer to both questions is no, then I bet it's a CRLF problem.

yes, those are the files that have been modified in my various patches.
https://treeherder.mozilla.org/jobs?repo=try&selectedTaskRun=IpSnT0XUQ9ab_oX4we6MqQ.0&revision=ac0181f69f287ce09d18730d66b0337d0d01d57d

There's no error about having CRLF in there.

You seem to be right (as usual), a checkout of mozilla-central unmodified and it's working.

Comment 3

[Mike Hommey [:glandium]]

That list of file is weird, though... many of them are not in https://searchfox.org/mozilla-central/rev/353f334d13359f7cb879b3cb38a94294231c2aa8/taskcluster/ci/toolchain/minidump_stackwalk.yml#15-24 (but one is, the file in mfbt)
Ricky, can you look into making the list more useful?

Comment 4

[Mike Hommey [:glandium]]

Actually, there seems to be two problems:

• the reported list is too broad and contains files that are not part of the hash
• the list of resource files for minidump-stackwalk actually might be too broad too (containing files/directories that don't impact the toolchain).

Comment 5

[Jean-Yves Avenard [:jya]]

Yes, so this is this commit:

https://hg.mozilla.org/try/rev/c5de619bab102b9d0b9e8fe06b6bb5fae5077201

it's a one-liner, it doesn't change the line ending.

--- a/mfbt/WeakPtr.h
+++ b/mfbt/WeakPtr.h
@@ -289,16 +289,18 @@ class WeakPtr {
 #ifdef MOZILLA_INTERNAL_API
     if (Destruct == detail::WeakPtrDestructorBehavior::ProxyToMainThread) {
       MOZ_ASSERT(NS_IsMainThread(),
                  "MainThreadWeakPtr makes no sense on non-main threads");
     }
 #endif
   }
 
+  explicit WeakPtr(const RefPtr<T>& aOther) { *this = aOther.get(); }
+
   // Ensure that mRef is dereferenceable in the uninitialized state.
   WeakPtr() : mRef(new WeakReference(nullptr)) {}
 
   explicit operator bool() const { return mRef->get(); }
   T* get() const { return static_cast<T*>(mRef->get()); }
   operator T*() const { return get(); }
   T& operator*() const { return *get(); }
   T* operator->() const MOZ_NO_ADDREF_RELEASE_ON_RETURN { return get(); }

There's no CRLF in this file.

So puzzling, how could adding one line in a header file break bootstrap?

Comment 6

[Mitchell Hentges [:mhentges] 🦀]

Workaround

Commit or shelve your changes, check out central, then bootstrap there. Once done, go back on top of your stack and continue work.

Why is this failing?

When generating each artifact, we store them on Taskcluster and make them findable by indexing them with a hash of all input files that affect the outputted artifact.
Then, when you run bootstrap on your machine, it finds the artifact by hashing those same input files and using that hash in the lookup.

Since there was a change to WeakPtr.h, the hash changed, thereby making the lookup of fix-stacks fail.

Note that, as mentioned by glandium, it appears that there may be files in that "input files list" that shouldn't be there, making this "hash change" more likely than it needs to be.

Comment 7

[Ricky Stewart]

I've said I would be in favor of refactoring to make the error message more informative, but that'll require some refactoring and I'm not familiar enough with the taskgraph code to speak to how that would best be done.

Comment 8

[Alex Hochheiden [:ahochheiden] [he/him]]

I tried to reproduce this and was not able to. I think this was resolved by other work.

Closing for now, feel free to re-open if this is incorrect.