1640815 - mach bootstrap fails if you run from an hg repository initialized from a bundle

Philippe VIGNEAU

Reporter

Description

•

3 years ago

User Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:76.0) Gecko/20100101 Firefox/76.0

Steps to reproduce:

Note on Artifact Mode:

Artifact builds download prebuilt C++ components rather than building
them locally. Artifact builds are faster!

Artifact builds are recommended for people working on Firefox or
Firefox for Android frontends, or the GeckoView Java API. They are unsuitable
for those working on C++ code. For more information see:
https://developer.mozilla.org/en-US/docs/Artifact_builds.

Please choose the version of Firefox you want to build:

Firefox for Desktop Artifact Mode
Firefox for Desktop
GeckoView/Firefox for Android Artifact Mode
GeckoView/Firefox for Android
Your choice: 2
Executing as root: sudo apt-get install autoconf2.13 build-essential nodejs python-setuptools unzip uuid zip python2.7 python2.7-dev
[sudo] Mot de passe de philou :
Lecture des listes de paquets... Fait
Construction de l'arbre des dépendances
Lecture des informations d'état... Fait
build-essential est déjà la version la plus récente (12.8ubuntu1).
unzip est déjà la version la plus récente (6.0-25ubuntu1).
zip est déjà la version la plus récente (3.0-11build1).
autoconf2.13 est déjà la version la plus récente (2.13-68).
nodejs est déjà la version la plus récente (10.19.0dfsg-3ubuntu1).
python-setuptools est déjà la version la plus récente (44.0.0-2).
python2.7 est déjà la version la plus récente (2.7.18rc1-2).
python2.7-dev est déjà la version la plus récente (2.7.18~rc1-2).
uuid est déjà la version la plus récente (1.6.2-1.5build7).
0 mis à jour, 0 nouvellement installés, 0 à enlever et 0 non mis à jour.
Executing as root: sudo apt-get install libasound2-dev libcurl4-openssl-dev libdbus-1-dev libdbus-glib-1-dev libgtk-3-dev libgtk2.0-dev libpulse-dev libx11-xcb-dev libxt-dev xvfb yasm
Lecture des listes de paquets... Fait
Construction de l'arbre des dépendances
Lecture des informations d'état... Fait
libasound2-dev est déjà la version la plus récente (1.2.2-2.1).
libcurl4-openssl-dev est déjà la version la plus récente (7.68.0-1ubuntu2).
libdbus-1-dev est déjà la version la plus récente (1.12.16-2ubuntu2).
libdbus-glib-1-dev est déjà la version la plus récente (0.110-5fakssync1).
libgtk-3-dev est déjà la version la plus récente (3.24.18-1ubuntu1).
libx11-xcb-dev est déjà la version la plus récente (2:1.6.9-2ubuntu1).
libxt-dev est déjà la version la plus récente (1:1.1.5-1).
libgtk2.0-dev est déjà la version la plus récente (2.24.32-4ubuntu4).
xvfb est déjà la version la plus récente (2:1.20.8-2ubuntu2).
yasm est déjà la version la plus récente (1.3.0-2ubuntu1).
libpulse-dev est déjà la version la plus récente (1:13.99.1-1ubuntu3.2).
0 mis à jour, 0 nouvellement installés, 0 à enlever et 0 non mis à jour.
Your version of Mercurial (5.4) is sufficiently modern.
Your version of Python (2.7.18rc1) is new enough.
Your version of Rust (1.43.1) is new enough.
Rust supports x86_64-unknown-linux-gnu targets.
Error running mach:

['bootstrap']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke |./mach busted| to check if this issue is already on file. If it
isn't, please use |./mach busted file| to report it. If |./mach busted| is
misbehaving, you can also inspect the dependencies of bug 1543241.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

ValueError: invalid literal for int() with base 10: ''

File "/mnt/3a52a748-a69c-41e0-8ded-35963b1cf0c7/mozilla-central/python/mozboot/mozboot/mach_commands.py", line 44, in bootstrap
bootstrapper.bootstrap()
File "/mnt/3a52a748-a69c-41e0-8ded-35963b1cf0c7/mozilla-central/python/mozboot/mozboot/bootstrap.py", line 475, in bootstrap
r = current_firefox_checkout(check_output=self.instance.check_output,
File "/mnt/3a52a748-a69c-41e0-8ded-35963b1cf0c7/mozilla-central/python/mozboot/mozboot/bootstrap.py", line 699, in current_firefox_checkout
_warn_if_risky_revision(path)
File "/mnt/3a52a748-a69c-41e0-8ded-35963b1cf0c7/mozilla-central/python/mozboot/mozboot/bootstrap.py", line 831, in _warn_if_risky_revision
if (time.time() - repo.get_commit_time()) >= NUM_SECONDS_IN_MONTH:
File "/mnt/3a52a748-a69c-41e0-8ded-35963b1cf0c7/mozilla-central/python/mozversioncontrol/mozversioncontrol/init.py", line 325, in get_commit_time
return int(self._run(

Actual results:

fails

Expected results:

works

Nathan Froyd [:froydnj]

Comment 1

•

3 years ago

Does applying this patch and then bootstrapping work any better:

diff --git a/python/mozversioncontrol/mozversioncontrol/__init__.py b/python/mozversioncontrol/mozversioncontrol/__init__.py
index 60e48ce1..8fde388 100644
--- a/python/mozversioncontrol/mozversioncontrol/__init__.py
+++ b/python/mozversioncontrol/mozversioncontrol/__init__.py
@@ -282,6 +282,7 @@ class HgRepository(Repository):
         self._client = hglib.client.hgclient(self.path, encoding=b'UTF-8',
                                              configs=None, connect=False)
 
+        self._client._env['HGPLAIN'] = '1'
         # Work around py3 compat issues in python-hglib
         self._client._env = ensure_subprocess_env(self._client._env)

? (not sure if I got the binary/text bits right...)

Flags: needinfo?(cardbook.thunderbird)

Philippe VIGNEAU

Reporter

Comment 2

•

3 years ago

here is what is in my python/mozversioncontrol/mozversioncontrol/init.py :

266 class HgRepository(Repository):
267     '''An implementation of `Repository` for Mercurial repositories.'''
268     def __init__(self, path, hg='hg'):
269         import hglib.client
270 
271         super(HgRepository, self).__init__(path, tool=hg)
272         self._env[b'HGPLAIN'] = b'1'
273 
274         # Setting this modifies a global variable and makes all future hglib
275         # instances use this binary. Since the tool path was validated, this
276         # should be OK. But ideally hglib would offer an API that defines
277         # per-instance binaries.
278         hglib.HGPATH = self._tool
279 
280         # Without connect=False this spawns a persistent process. We want
281         # the process lifetime tied to a context manager.
282         self._client = hglib.client.hgclient(self.path, encoding=b'UTF-8',
283                                              configs=None, connect=False)
284 
285         # Work around py3 compat issues in python-hglib
286         self._client._env = ensure_subprocess_env(self._client._env)
287

Flags: needinfo?(cardbook.thunderbird)

Nathan Froyd [:froydnj]

Comment 3

•

3 years ago

Right, does inserting:

        self._client._env['HGPLAIN'] = '1'

so that starting at line 285, it reads:

        self._client._env['HGPLAIN'] = '1'
        # Work around py3 compat issues in python-hglib
        self._client._env = ensure_subprocess_env(self._client._env)

and then running ./mach bootstrap again work any better for you?

Flags: needinfo?(cardbook.thunderbird)

Philippe VIGNEAU

Reporter

Comment 4

•

3 years ago

after having inserted the line
self._client._env['HGPLAIN'] = '1'

it's the same :
The details of the failure are as follows:

ValueError: invalid literal for int() with base 10: ''

File "/mnt/3a52a748-a69c-41e0-8ded-35963b1cf0c7/mozilla-central/python/mozboot/mozboot/mach_commands.py", line 44, in bootstrap
bootstrapper.bootstrap()
File "/mnt/3a52a748-a69c-41e0-8ded-35963b1cf0c7/mozilla-central/python/mozboot/mozboot/bootstrap.py", line 475, in bootstrap
r = current_firefox_checkout(check_output=self.instance.check_output,
File "/mnt/3a52a748-a69c-41e0-8ded-35963b1cf0c7/mozilla-central/python/mozboot/mozboot/bootstrap.py", line 699, in current_firefox_checkout
_warn_if_risky_revision(path)
File "/mnt/3a52a748-a69c-41e0-8ded-35963b1cf0c7/mozilla-central/python/mozboot/mozboot/bootstrap.py", line 831, in _warn_if_risky_revision
if (time.time() - repo.get_commit_time()) >= NUM_SECONDS_IN_MONTH:
File "/mnt/3a52a748-a69c-41e0-8ded-35963b1cf0c7/mozilla-central/python/mozversioncontrol/mozversioncontrol/init.py", line 326, in get_commit_time
return int(self._run(

Flags: needinfo?(cardbook.thunderbird)

Nathan Froyd [:froydnj]

Comment 5

•

3 years ago

Hm, not sure what's going on, then. The failing line is doing:

        return int(self._run(
            'log', '--rev', 'heads(ancestors(.) and not draft())',
            '--template', '{word(0, date|hgdate)}', '--limit', '1').strip())

and one might think that a custom log command is getting in the way. But HgRepository already sets HGPLAIN and the hgclient server apparently sets HGPLAIN:

https://www.mercurial-scm.org/repo/python-hglib/file/tip/hglib/client.py#l61

so I'm not sure what's happening. Unless it's locale-depdendent? Or we need to be converting subprocess.check_output back into a string?

Ricky Stewart

Comment 6

•

3 years ago

The relevant error message is:

ValueError: invalid literal for int() with base 10: ''

So we're calling int() but on an empty string, which is going to fail (that makes sense). What doesn't make sense to me is that the hg log isn't producing any output (I'm sure if I knew anything about Mercurial, then this might make sense to me, but I don't :) )

What version of the code are you synced to? Why does hg log not produce any non-draft commits?

Ricky Stewart

Comment 7

•

3 years ago

This patch should probably get your tree into a working state, but for the actual fix I would like to address this more rigorously if possible.

diff --git a/python/mozversioncontrol/mozversioncontrol/__init__.py b/python/mozversioncontrol/mozversioncontrol/__init__.py
index 60e48ce15286..99dbedd5e4e6 100644
--- a/python/mozversioncontrol/mozversioncontrol/__init__.py
+++ b/python/mozversioncontrol/mozversioncontrol/__init__.py
@@ -324,7 +324,8 @@ class HgRepository(Repository):
     def get_commit_time(self):
         return int(self._run(
             'log', '--rev', 'heads(ancestors(.) and not draft())',
-            '--template', '{word(0, date|hgdate)}', '--limit', '1').strip())
+            '--template', '{word(0, date|hgdate)}', '--limit', '1').strip()
+                   or '0')
 
     def sparse_checkout_present(self):
         # We assume a sparse checkout is enabled if the .hg/sparse file

Philippe VIGNEAU

Reporter

Comment 8

•

3 years ago

issue solved with your correction, thanks :O)

Ricky Stewart

Comment 9

•

3 years ago

How did you get your copy of the code, Philippe? I'm wondering how you got a workspace with no non-draft commits. I assume you didn't just do a normal hg clone.

Flags: needinfo?(cardbook.thunderbird)

Philippe VIGNEAU

Reporter

Comment 10

•

3 years ago

as I've got a slow connection I've downloaded a bundle (using this https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Source_Code/Mercurial/Bundles)

Flags: needinfo?(cardbook.thunderbird)

Ricky Stewart

Comment 11

•

3 years ago

•

Edited

Interesting, so if you clone using a bundle from those directions then hg log --rev "heads(ancestors(.) and not draft())" --limit 1 produces no output. (Apparently every commit in the repo is considered a "draft" commit.) The revset would probably have to be updated, again, to be inclusive of this case.

Ricky Stewart

Comment 12

•

3 years ago

Doing an hg pull (followed by an hg update) should fix this for you.

Ricky Stewart

Updated

•

3 years ago

Severity: -- → S3

Keywords: in-triage

Priority: -- → P3

Summary: mach bootstrap fails with ValueError: invalid literal for int() with base 10 → mach bootstrap fails if you run from an hg repository initialized from a bundle

Philippe VIGNEAU

Reporter

Comment 13

•

3 years ago

thanks a lot :O) I will redo it all...

my goal is to build Thunderbird, is it the procedure or are there additionnal steps ?

Severity: S3 → --

Keywords: in-triage

Priority: P3 → --

Summary: mach bootstrap fails if you run from an hg repository initialized from a bundle → mach bootstrap fails with ValueError: invalid literal for int() with base 10

Ricky Stewart

Updated

•

3 years ago

Summary: mach bootstrap fails with ValueError: invalid literal for int() with base 10 → mach bootstrap fails if you run from an hg repository initialized from a bundle

Ricky Stewart

Updated

•

3 years ago

Severity: -- → S3

Keywords: in-triage

Priority: -- → P3

Ricky Stewart

Updated

•

3 years ago

Assignee: nobody → mhentges

Mitchell Hentges [:mhentges] 🦀

Assignee

Comment 14

•

3 years ago

thanks a lot :O) I will redo it all...

You shouldn't need to re-do - you can use the bundle you already downloaded, then either use the suggestion Ricky provided (which it sounds like is working?) or re-extract the bundle and do hg pull && hg update # && ./mach bootstrap. Both should work :)

my goal is to build Thunderbird, is it the procedure or are there additionnal steps ?

I think you're on the right track, you want to clone the repository, bootstrap, then build. This case that you ran into here is an issue, but hopefully it's "smooth sailing" from here. You should be able to follow the build documentation as normal from here.

Mitchell Hentges [:mhentges] 🦀

Assignee

Comment 15

•

3 years ago

Attached file Bug 1640815: Raise helpful error if no non-public commits on bootstrap r?rstewart — Details

Pulsebot

Comment 16

•

3 years ago

Pushed by mhentges@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/a7395b8a7706
Raise helpful error if no non-public commits on bootstrap r=rstewart

Alexandru Michis [:malexandru]

Comment 17

•

3 years ago

bugherder

https://hg.mozilla.org/mozilla-central/rev/a7395b8a7706

Status: UNCONFIRMED → RESOLVED

Closed: 3 years ago

status-firefox78: --- → fixed

Resolution: --- → FIXED

Target Milestone: --- → mozilla78

Bugzilla

Quick Search

mach bootstrap fails if you run from an hg repository initialized from a bundle

Categories

(Firefox Build System :: Bootstrap Configuration, defect, P3)

Tracking

(firefox78 fixed)

People

(Reporter: cardbook.thunderbird, Assigned: mhentges)

References

Details

(Keywords: in-triage)

Crash Data

Security

(public)

User Story

Attachments

(1 file)

Description

Comment 1

Comment 2

Comment 3

Comment 4

Comment 5

Comment 6

Comment 7

Comment 8

Comment 9

Comment 10

Comment 11

Comment 12

Updated

Comment 13

Updated

Updated

Updated

Comment 14

Comment 15

Comment 16

Comment 17