1916985 - "tarfile.StreamError: seeking backwards is not allowed" error during Windows bootstrap

Status: RESOLVED FIXED

(Firefox Build System :: Toolchains, defect)

Description

[Ryan VanderMeulen [:RyanVM]]

With MozillaBuild 4.1, I'm seeing this when trying to build today:

 0:12.70 Installing bootstrapped toolchain in C:/Users/Ryan/.mozbuild/clang
 0:14.77 Setting up artifact clang.tar.zst
 0:14.77 Using artifact from local cache: C:\Users\Ryan\.mozbuild\toolchains\cd6531dfc6f033fd-clang.tar.zst
 0:15.72 untarring "C:\Users\Ryan\.mozbuild\clang.tar.zst"
 0:20.46 Error running mach:
 0:20.46
 0:20.46     mach --log-no-times artifact toolchain --from-task IHqLzZq2TsKGzKF_EjUDbQ:public/build/clang.tar.zst
 0:20.46
 0:20.46 The error occurred in code that was called by the mach command. This is either
 0:20.46 a bug in the called code itself or in the way that mach is calling it.
 0:20.46 You can invoke ``./mach busted`` to check if this issue is already on file. If it
 0:20.46 isn't, please use ``./mach busted file artifact`` to report it. If ``./mach busted`` is
 0:20.46 misbehaving, you can also inspect the dependencies of bug 1543241.
 0:20.46
 0:20.46 If filing a bug, please include the full output of mach, including this error
 0:20.46 message.
 0:20.46
 0:20.46 The details of the failure are as follows:
 0:20.46
 0:20.46 tarfile.StreamError: seeking backwards is not allowed
 0:20.46
 0:20.46   File "C:\Users\Ryan\repos\mozilla\python\mozbuild\mozbuild\artifact_commands.py", line 590, in artifact_toolchain
 0:20.46     unpack_file(local)
 0:20.46   File "c:\users\ryan\repos\mozilla\python\mozbuild\mozbuild\action\tooltool.py", line 996, in unpack_file
 0:20.47     safe_extract(tar)
 0:20.47   File "c:\users\ryan\repos\mozilla\python\mozbuild\mozbuild\action\tooltool.py", line 972, in safe_extract
 0:20.47     tar.extractall(path, members=_files(tar, path), numeric_owner=numeric_owner)
 0:20.47   File "C:\mozilla-build\python3\Lib\tarfile.py", line 2264, in extractall
 0:20.47     self._extract_one(tarinfo, path, set_attrs=not tarinfo.isdir(),
 0:20.47   File "C:\mozilla-build\python3\Lib\tarfile.py", line 2327, in _extract_one
 0:20.47     self._extract_member(tarinfo, os.path.join(path, tarinfo.name),
 0:20.47   File "C:\mozilla-build\python3\Lib\tarfile.py", line 2418, in _extract_member
 0:20.47     self.makelink(tarinfo, targetpath)
 0:20.47   File "C:\mozilla-build\python3\Lib\tarfile.py", line 2519, in makelink
 0:20.47     self._extract_member(self._find_link_target(tarinfo),
 0:20.47   File "C:\mozilla-build\python3\Lib\tarfile.py", line 2410, in _extract_member
 0:20.47     self.makefile(tarinfo, targetpath)
 0:20.47   File "C:\mozilla-build\python3\Lib\tarfile.py", line 2453, in makefile
 0:20.47     source.seek(tarinfo.offset_data)
 0:20.47   File "C:\mozilla-build\python3\Lib\tarfile.py", line 520, in seek
 0:20.47     raise StreamError("seeking backwards is not allowed")

Bisected to https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=54e0d92a05c75f62a0583169a4880343844c77b7&tochange=380dabc0efdcf3092241276fcb22955e6d83ecba

Comment 1

[BugBot [:suhaib / :marco]]

Set release status flags based on info from the regressing bug 1915778

:glandium, since you are the author of the regressor, bug 1915778, could you take a look? Also, could you set the severity field?

For more information, please visit BugBot documentation.

Comment 2

[Mike Hommey [:glandium]]

Attachment: Bug 1916985 - Work around tarfile.StreamError in tooltool.

Before bug 1915568, we didn't have symbolic links in the clang artifact.
That was an unexpected side effect, but supporting unpacking toolchain
artifacts with symbolic links has been a longstanding thing I've wanted
to fix, unlocking Linux and Mac cross-compiles on Windows (e.g. sysroot
artifacts contain symbolic links).

The clang situation is a good occasion to finally do it. It's also
somehow desirable to keep clang with symbolic links, as it makes the
toolchain artifact smaller.

When a tarball contains symbolic links, the python tarfile module ends
up trying to seek in the tar, which doesn't quite work when the tar is
a stream (which it is when decompressing at the same time).

We wrap TarFile so as to handle the symbolic link situation ourselves,
and when we can't create symbolic links, we instead create a copy of
the file the link points to.

Comment 3

[James Teh [:Jamie]]

FWIW, I applied this patch, but now I get this error:

ValueError: too many values to unpack (expected 3)                                                                                                                                                        
  File "C:\Users\jamie\src\gecko\python\mozbuild\mozbuild\artifact_commands.py", line 590, in artifact_toolchain                                                                                          
    unpack_file(local)                                                                                                                                                                                    
  File "c:\users\jamie\src\gecko\python\mozbuild\mozbuild\action\tooltool.py", line 1043, in unpack_file                                                                                                  
    safe_extract(tar)                                                                                                                                                                                     
  File "c:\users\jamie\src\gecko\python\mozbuild\mozbuild\action\tooltool.py", line 1019, in safe_extract                                                                                                 
    tar.extractall(path, members=_files(tar, path), numeric_owner=numeric_owner)                                                                                                                          
  File "c:\users\jamie\src\gecko\python\mozbuild\mozbuild\action\tooltool.py", line 1007, in extractall                                                                                                   
    for tarinfo, linkpath, numeric_owner in self._defered_links:                                                                                                                                          

Comment 4

[Mike Hommey [:glandium]]

(In reply to James Teh [:Jamie] from comment #3)

FWIW, I applied this patch, but now I get this error:

ValueError: too many values to unpack (expected 3)                                                                                                                                                        
  File "C:\Users\jamie\src\gecko\python\mozbuild\mozbuild\artifact_commands.py", line 590, in artifact_toolchain                                                                                          
    unpack_file(local)                                                                                                                                                                                    
  File "c:\users\jamie\src\gecko\python\mozbuild\mozbuild\action\tooltool.py", line 1043, in unpack_file                                                                                                  
    safe_extract(tar)                                                                                                                                                                                     
  File "c:\users\jamie\src\gecko\python\mozbuild\mozbuild\action\tooltool.py", line 1019, in safe_extract                                                                                                 
    tar.extractall(path, members=_files(tar, path), numeric_owner=numeric_owner)                                                                                                                          
  File "c:\users\jamie\src\gecko\python\mozbuild\mozbuild\action\tooltool.py", line 1007, in extractall                                                                                                   
    for tarinfo, linkpath, numeric_owner in self._defered_links:                                                                                                                                          

When getting what toolchain?

Comment 5

[James Teh [:Jamie]]

Sorry:

Using artifact from local cache: C:\Users\jamie.mozbuild\toolchains\cd6531dfc6f033fd-clang.tar.zst
untarring "C:\Users\jamie.mozbuild\clang.tar.zst"

Comment 6

[Mike Hommey [:glandium]]

(In reply to James Teh [:Jamie] from comment #5)

Sorry:

Using artifact from local cache: C:\Users\jamie.mozbuild\toolchains\cd6531dfc6f033fd-clang.tar.zst
untarring "C:\Users\jamie.mozbuild\clang.tar.zst"

Try the last version of the patch.

Comment 7

[James Teh [:Jamie]]

Working now, thank you.

Comment 9

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

Odd, I still see the bug even applying the latest patch.]

untarring "C:\Users\toybox\.mozbuild\clang.tar.zst"
Error running mach:

    mach --log-no-times artifact toolchain --from-task IHqLzZq2TsKGzKF_EjUDbQ:public/build/clang.tar.zst

<snip>

tarfile.StreamError: seeking backwards is not allowed

  File "M:\src\python\mozbuild\mozbuild\artifact_commands.py", line 590, in artifact_toolchain
    unpack_file(local)
  File "m:\src\python\mozbuild\mozbuild\action\tooltool.py", line 1046, in unpack_file
    safe_extract(tar)
  File "m:\src\python\mozbuild\mozbuild\action\tooltool.py", line 1022, in safe_extract
    tar.extractall(path, members=_files(tar, path), numeric_owner=numeric_owner)
  File "m:\src\python\mozbuild\mozbuild\action\tooltool.py", line 1008, in extractall
    super(TarFile, self).extractall(*args, **kwargs)
  File "C:\mozilla-build\python3\Lib\tarfile.py", line 2264, in extractall
    self._extract_one(tarinfo, path, set_attrs=not tarinfo.isdir(),
  File "C:\mozilla-build\python3\Lib\tarfile.py", line 2327, in _extract_one
    self._extract_member(tarinfo, os.path.join(path, tarinfo.name),
  File "C:\mozilla-build\python3\Lib\tarfile.py", line 2418, in _extract_member
    self.makelink(tarinfo, targetpath)
  File "C:\mozilla-build\python3\Lib\tarfile.py", line 2519, in makelink
    self._extract_member(self._find_link_target(tarinfo),
  File "C:\mozilla-build\python3\Lib\tarfile.py", line 2410, in _extract_member
    self.makefile(tarinfo, targetpath)
  File "C:\mozilla-build\python3\Lib\tarfile.py", line 2453, in makefile
    source.seek(tarinfo.offset_data)
  File "C:\mozilla-build\python3\Lib\tarfile.py", line 520, in seek
    raise StreamError("seeking backwards is not allowed")

Comment 10

[Mike Hommey [:glandium]]

File "C:\mozilla-build\python3\Lib\tarfile.py", line 2264, in extractall
    self._extract_one(tarinfo, path, set_attrs=not tarinfo.isdir(),

gah, looks like python changed its implementation of extractall not to use extract... are you using 3.12?

Comment 11

[Mike Hommey [:glandium]]

huh, yeah, it was changed in https://github.com/python/cpython/commit/af530469954e8ad49f1e071ef31c844b9bfda414, in 3.12 :(

Comment 12

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

No, ./mach python3 shows 3.11.7. Did I change the version by myself??

Comment 13

[Mike Hommey [:glandium]]

Can you try this patch on top?

diff --git a/python/mozbuild/mozbuild/action/tooltool.py b/python/mozbuild/mozbuild/action/tooltool.py
index adf6422c2ac58..6f506dedf5df2 100755
--- a/python/mozbuild/mozbuild/action/tooltool.py
+++ b/python/mozbuild/mozbuild/action/tooltool.py
@@ -964,7 +964,7 @@ def validate_tar_member(member, path):
 
 
 class TarFile(tarfile.TarFile):
-    def extract(self, member, path="", set_attrs=True, numeric_owner=False, **kwargs):
+    def _extract_one(self, member, path="", set_attrs=True, numeric_owner=False, **kwargs):
         defered_links = getattr(self, "_defered_links", None)
         if not isinstance(member, tarfile.TarInfo):
             member = self.getmember(member)
@@ -1002,6 +1002,8 @@ class TarFile(tarfile.TarFile):
                 self.chown(tarinfo, linkpath, numeric_owner)
             self._extracted_members.add(targetpath)
 
+    extract = _extract_one
+
     def extractall(self, *args, **kwargs):
         self._defered_links = {}
         self._extracted_members = set()

Comment 14

[Mike Hommey [:glandium]]

(In reply to Masayuki Nakano [:masayuki] (he/him)(JST, +0900) from comment #12)

No, ./mach python3 shows 3.11.7. Did I change the version by myself??

It was backported to 3.11 too https://github.com/python/cpython/commit/241f2e54a6a2801a1d2022f0fa56309e124866c3

Comment 15

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

Hmm, I got "maximum recursion depth exceeded in comparison" error.

 0:11.42   File "m:\src\python\mozbuild\mozbuild\action\tooltool.py", line 1048, in unpack_file
 0:11.42     safe_extract(tar)
 0:11.42   File "m:\src\python\mozbuild\mozbuild\action\tooltool.py", line 1024, in safe_extract
 0:11.42     tar.extractall(path, members=_files(tar, path), numeric_owner=numeric_owner)
 0:11.42   File "m:\src\python\mozbuild\mozbuild\action\tooltool.py", line 1010, in extractall
 0:11.42     super(TarFile, self).extractall(*args, **kwargs)
 0:11.42   File "C:\mozilla-build\python3\Lib\tarfile.py", line 2264, in extractall
 0:11.43     self._extract_one(tarinfo, path, set_attrs=not tarinfo.isdir(),
 0:11.43   File "m:\src\python\mozbuild\mozbuild\action\tooltool.py", line 996, in _extract_one
 0:11.43     super(TarFile, self).extract(
 0:11.43   File "C:\mozilla-build\python3\Lib\tarfile.py", line 2297, in extract
 0:11.43     self._extract_one(tarinfo, path, set_attrs, numeric_owner)
 0:11.43   File "m:\src\python\mozbuild\mozbuild\action\tooltool.py", line 996, in _extract_one
 0:11.43     super(TarFile, self).extract(

Comment 16

[Mike Hommey [:glandium]]

 0:11.42   File "C:\mozilla-build\python3\Lib\tarfile.py", line 2264, in extractall
 0:11.43     self._extract_one(tarinfo, path, set_attrs=not tarinfo.isdir(),
 0:11.43   File "m:\src\python\mozbuild\mozbuild\action\tooltool.py", line 996, in _extract_one
 0:11.43     super(TarFile, self).extract(
 0:11.43   File "C:\mozilla-build\python3\Lib\tarfile.py", line 2297, in extract
 0:11.43     self._extract_one(tarinfo, path, set_attrs, numeric_owner)

facepalm, of course...

Comment 17

[Mike Hommey [:glandium]]

Latest version on phabricator should work, now.

Comment 18

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)]

Yeah, it works! Thank you!

Comment 19

[Ryan VanderMeulen [:RyanVM]]

Latest patch also worked for me, thanks!

Comment 22

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/e74c8074fe52
Work around tarfile.StreamError in tooltool. r=releng-reviewers,jcristau

Comment 23

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/e74c8074fe52

Comment 25

[tszentpeteri]

Backed out as requested by Glandium wrong patch was landed.

• Backout link

Comment 26

[Jared Hirsch [:jhirsch] (he/him) (Needinfo please)]

Just confirming that the latest patch in https://phabricator.services.mozilla.com/D221282, when applied locally, does fix the bug for me.

Comment 27

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/28d39f6b7fca
Work around tarfile.StreamError in tooltool. r=releng-reviewers,jcristau

Comment 28

[Norisz Fay [:noriszfay]]

https://hg.mozilla.org/mozilla-central/rev/28d39f6b7fca