Closed Bug 1640815 Opened 4 months ago Closed 4 months ago

mach bootstrap fails if you run from an hg repository initialized from a bundle

Categories

(Firefox Build System :: Bootstrap Configuration, defect, P3)

76 Branch
defect

Tracking

(firefox78 fixed)

RESOLVED FIXED
mozilla78
Tracking Status
firefox78 --- fixed

People

(Reporter: cardbook.thunderbird, Assigned: mhentges)

Details

(Keywords: in-triage)

Attachments

(1 file)

User Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:76.0) Gecko/20100101 Firefox/76.0

Steps to reproduce:

Note on Artifact Mode:

Artifact builds download prebuilt C++ components rather than building
them locally. Artifact builds are faster!

Artifact builds are recommended for people working on Firefox or
Firefox for Android frontends, or the GeckoView Java API. They are unsuitable
for those working on C++ code. For more information see:
https://developer.mozilla.org/en-US/docs/Artifact_builds.

Please choose the version of Firefox you want to build:

  1. Firefox for Desktop Artifact Mode
  2. Firefox for Desktop
  3. GeckoView/Firefox for Android Artifact Mode
  4. GeckoView/Firefox for Android
    Your choice: 2
    Executing as root: sudo apt-get install autoconf2.13 build-essential nodejs python-setuptools unzip uuid zip python2.7 python2.7-dev
    [sudo] Mot de passe de philou :
    Lecture des listes de paquets... Fait
    Construction de l'arbre des dépendances
    Lecture des informations d'état... Fait
    build-essential est déjà la version la plus récente (12.8ubuntu1).
    unzip est déjà la version la plus récente (6.0-25ubuntu1).
    zip est déjà la version la plus récente (3.0-11build1).
    autoconf2.13 est déjà la version la plus récente (2.13-68).
    nodejs est déjà la version la plus récente (10.19.0dfsg-3ubuntu1).
    python-setuptools est déjà la version la plus récente (44.0.0-2).
    python2.7 est déjà la version la plus récente (2.7.18
    rc1-2).
    python2.7-dev est déjà la version la plus récente (2.7.18~rc1-2).
    uuid est déjà la version la plus récente (1.6.2-1.5build7).
    0 mis à jour, 0 nouvellement installés, 0 à enlever et 0 non mis à jour.
    Executing as root: sudo apt-get install libasound2-dev libcurl4-openssl-dev libdbus-1-dev libdbus-glib-1-dev libgtk-3-dev libgtk2.0-dev libpulse-dev libx11-xcb-dev libxt-dev xvfb yasm
    Lecture des listes de paquets... Fait
    Construction de l'arbre des dépendances
    Lecture des informations d'état... Fait
    libasound2-dev est déjà la version la plus récente (1.2.2-2.1).
    libcurl4-openssl-dev est déjà la version la plus récente (7.68.0-1ubuntu2).
    libdbus-1-dev est déjà la version la plus récente (1.12.16-2ubuntu2).
    libdbus-glib-1-dev est déjà la version la plus récente (0.110-5fakssync1).
    libgtk-3-dev est déjà la version la plus récente (3.24.18-1ubuntu1).
    libx11-xcb-dev est déjà la version la plus récente (2:1.6.9-2ubuntu1).
    libxt-dev est déjà la version la plus récente (1:1.1.5-1).
    libgtk2.0-dev est déjà la version la plus récente (2.24.32-4ubuntu4).
    xvfb est déjà la version la plus récente (2:1.20.8-2ubuntu2).
    yasm est déjà la version la plus récente (1.3.0-2ubuntu1).
    libpulse-dev est déjà la version la plus récente (1:13.99.1-1ubuntu3.2).
    0 mis à jour, 0 nouvellement installés, 0 à enlever et 0 non mis à jour.
    Your version of Mercurial (5.4) is sufficiently modern.
    Your version of Python (2.7.18rc1) is new enough.
    Your version of Rust (1.43.1) is new enough.
    Rust supports x86_64-unknown-linux-gnu targets.
    Error running mach:
['bootstrap']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke |./mach busted| to check if this issue is already on file. If it
isn't, please use |./mach busted file| to report it. If |./mach busted| is
misbehaving, you can also inspect the dependencies of bug 1543241.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

ValueError: invalid literal for int() with base 10: ''

File "/mnt/3a52a748-a69c-41e0-8ded-35963b1cf0c7/mozilla-central/python/mozboot/mozboot/mach_commands.py", line 44, in bootstrap
bootstrapper.bootstrap()
File "/mnt/3a52a748-a69c-41e0-8ded-35963b1cf0c7/mozilla-central/python/mozboot/mozboot/bootstrap.py", line 475, in bootstrap
r = current_firefox_checkout(check_output=self.instance.check_output,
File "/mnt/3a52a748-a69c-41e0-8ded-35963b1cf0c7/mozilla-central/python/mozboot/mozboot/bootstrap.py", line 699, in current_firefox_checkout
_warn_if_risky_revision(path)
File "/mnt/3a52a748-a69c-41e0-8ded-35963b1cf0c7/mozilla-central/python/mozboot/mozboot/bootstrap.py", line 831, in _warn_if_risky_revision
if (time.time() - repo.get_commit_time()) >= NUM_SECONDS_IN_MONTH:
File "/mnt/3a52a748-a69c-41e0-8ded-35963b1cf0c7/mozilla-central/python/mozversioncontrol/mozversioncontrol/init.py", line 325, in get_commit_time
return int(self._run(

Actual results:

fails

Expected results:

works

Does applying this patch and then bootstrapping work any better:

diff --git a/python/mozversioncontrol/mozversioncontrol/__init__.py b/python/mozversioncontrol/mozversioncontrol/__init__.py
index 60e48ce1..8fde388 100644
--- a/python/mozversioncontrol/mozversioncontrol/__init__.py
+++ b/python/mozversioncontrol/mozversioncontrol/__init__.py
@@ -282,6 +282,7 @@ class HgRepository(Repository):
         self._client = hglib.client.hgclient(self.path, encoding=b'UTF-8',
                                              configs=None, connect=False)
 
+        self._client._env['HGPLAIN'] = '1'
         # Work around py3 compat issues in python-hglib
         self._client._env = ensure_subprocess_env(self._client._env)
 

? (not sure if I got the binary/text bits right...)

Flags: needinfo?(cardbook.thunderbird)

here is what is in my python/mozversioncontrol/mozversioncontrol/init.py :

266 class HgRepository(Repository):
267     '''An implementation of `Repository` for Mercurial repositories.'''
268     def __init__(self, path, hg='hg'):
269         import hglib.client
270 
271         super(HgRepository, self).__init__(path, tool=hg)
272         self._env[b'HGPLAIN'] = b'1'
273 
274         # Setting this modifies a global variable and makes all future hglib
275         # instances use this binary. Since the tool path was validated, this
276         # should be OK. But ideally hglib would offer an API that defines
277         # per-instance binaries.
278         hglib.HGPATH = self._tool
279 
280         # Without connect=False this spawns a persistent process. We want
281         # the process lifetime tied to a context manager.
282         self._client = hglib.client.hgclient(self.path, encoding=b'UTF-8',
283                                              configs=None, connect=False)
284 
285         # Work around py3 compat issues in python-hglib
286         self._client._env = ensure_subprocess_env(self._client._env)
287
Flags: needinfo?(cardbook.thunderbird)

Right, does inserting:

        self._client._env['HGPLAIN'] = '1'

so that starting at line 285, it reads:

        self._client._env['HGPLAIN'] = '1'
        # Work around py3 compat issues in python-hglib
        self._client._env = ensure_subprocess_env(self._client._env)

and then running ./mach bootstrap again work any better for you?

Flags: needinfo?(cardbook.thunderbird)

after having inserted the line
self._client._env['HGPLAIN'] = '1'

it's the same :
The details of the failure are as follows:

ValueError: invalid literal for int() with base 10: ''

File "/mnt/3a52a748-a69c-41e0-8ded-35963b1cf0c7/mozilla-central/python/mozboot/mozboot/mach_commands.py", line 44, in bootstrap
bootstrapper.bootstrap()
File "/mnt/3a52a748-a69c-41e0-8ded-35963b1cf0c7/mozilla-central/python/mozboot/mozboot/bootstrap.py", line 475, in bootstrap
r = current_firefox_checkout(check_output=self.instance.check_output,
File "/mnt/3a52a748-a69c-41e0-8ded-35963b1cf0c7/mozilla-central/python/mozboot/mozboot/bootstrap.py", line 699, in current_firefox_checkout
_warn_if_risky_revision(path)
File "/mnt/3a52a748-a69c-41e0-8ded-35963b1cf0c7/mozilla-central/python/mozboot/mozboot/bootstrap.py", line 831, in _warn_if_risky_revision
if (time.time() - repo.get_commit_time()) >= NUM_SECONDS_IN_MONTH:
File "/mnt/3a52a748-a69c-41e0-8ded-35963b1cf0c7/mozilla-central/python/mozversioncontrol/mozversioncontrol/init.py", line 326, in get_commit_time
return int(self._run(

Flags: needinfo?(cardbook.thunderbird)

Hm, not sure what's going on, then. The failing line is doing:

        return int(self._run(
            'log', '--rev', 'heads(ancestors(.) and not draft())',
            '--template', '{word(0, date|hgdate)}', '--limit', '1').strip())

and one might think that a custom log command is getting in the way. But HgRepository already sets HGPLAIN and the hgclient server apparently sets HGPLAIN:

https://www.mercurial-scm.org/repo/python-hglib/file/tip/hglib/client.py#l61

so I'm not sure what's happening. Unless it's locale-depdendent? Or we need to be converting subprocess.check_output back into a string?

The relevant error message is:

ValueError: invalid literal for int() with base 10: ''

So we're calling int() but on an empty string, which is going to fail (that makes sense). What doesn't make sense to me is that the hg log isn't producing any output (I'm sure if I knew anything about Mercurial, then this might make sense to me, but I don't :) )

What version of the code are you synced to? Why does hg log not produce any non-draft commits?

This patch should probably get your tree into a working state, but for the actual fix I would like to address this more rigorously if possible.

diff --git a/python/mozversioncontrol/mozversioncontrol/__init__.py b/python/mozversioncontrol/mozversioncontrol/__init__.py
index 60e48ce15286..99dbedd5e4e6 100644
--- a/python/mozversioncontrol/mozversioncontrol/__init__.py
+++ b/python/mozversioncontrol/mozversioncontrol/__init__.py
@@ -324,7 +324,8 @@ class HgRepository(Repository):
     def get_commit_time(self):
         return int(self._run(
             'log', '--rev', 'heads(ancestors(.) and not draft())',
-            '--template', '{word(0, date|hgdate)}', '--limit', '1').strip())
+            '--template', '{word(0, date|hgdate)}', '--limit', '1').strip()
+                   or '0')
 
     def sparse_checkout_present(self):
         # We assume a sparse checkout is enabled if the .hg/sparse file

issue solved with your correction, thanks :O)

How did you get your copy of the code, Philippe? I'm wondering how you got a workspace with no non-draft commits. I assume you didn't just do a normal hg clone.

Flags: needinfo?(cardbook.thunderbird)

as I've got a slow connection I've downloaded a bundle (using this https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Source_Code/Mercurial/Bundles)

Flags: needinfo?(cardbook.thunderbird)

Interesting, so if you clone using a bundle from those directions then hg log --rev "heads(ancestors(.) and not draft())" --limit 1 produces no output. (Apparently every commit in the repo is considered a "draft" commit.) The revset would probably have to be updated, again, to be inclusive of this case.

Doing an hg pull (followed by an hg update) should fix this for you.

Severity: -- → S3
Keywords: in-triage
Priority: -- → P3
Summary: mach bootstrap fails with ValueError: invalid literal for int() with base 10 → mach bootstrap fails if you run from an hg repository initialized from a bundle

thanks a lot :O) I will redo it all...

my goal is to build Thunderbird, is it the procedure or are there additionnal steps ?

Severity: S3 → --
Keywords: in-triage
Priority: P3 → --
Summary: mach bootstrap fails if you run from an hg repository initialized from a bundle → mach bootstrap fails with ValueError: invalid literal for int() with base 10
Summary: mach bootstrap fails with ValueError: invalid literal for int() with base 10 → mach bootstrap fails if you run from an hg repository initialized from a bundle
Severity: -- → S3
Keywords: in-triage
Priority: -- → P3
Assignee: nobody → mhentges

thanks a lot :O) I will redo it all...

You shouldn't need to re-do - you can use the bundle you already downloaded, then either use the suggestion Ricky provided (which it sounds like is working?) or re-extract the bundle and do hg pull && hg update # && ./mach bootstrap. Both should work :)

my goal is to build Thunderbird, is it the procedure or are there additionnal steps ?

I think you're on the right track, you want to clone the repository, bootstrap, then build. This case that you ran into here is an issue, but hopefully it's "smooth sailing" from here. You should be able to follow the build documentation as normal from here.

Pushed by mhentges@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/a7395b8a7706
Raise helpful error if no non-public commits on bootstrap r=rstewart
Status: UNCONFIRMED → RESOLVED
Closed: 4 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla78
You need to log in before you can comment on or make changes to this bug.