Closed Bug 1199356 Opened 9 years ago Closed 7 years ago

abort: error fetching bundle: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581) with bundleclone

Categories

(Firefox Build System :: MozillaBuild, task)

task
Not set
normal

Tracking

(Not tracked)

RESOLVED INCOMPLETE

People

(Reporter: gps, Unassigned)

References

Details

Mossop encountered the error in the summary line when attempting to use bundleclone on Windows. Attempted to fetch from https://s3-us-west-2.amazonaws.com/moz-hg-bundles-us-west-2/integration/fx-team/0cc66c142126b36fe80947c199bb6d903f52817c.gzip.hg.

I suspect the Mercurial SSL CA store isn't defined properly or is missing whatever root cert Amazon is using. Either way, it sounds like there is a bug here.
Mossop says he isn't using the MozillaBuild Mercurial, so who knows what's happening.
FWIW, we see this in the logs in automation too.
Link to automation log, please.
Flags: needinfo?(ryanvm)
My bad, "Failed to log stats. Exception = [Errno 1] _ssl.c:504: error:1407742E:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert protocol version" is the error we see in automation.
Flags: needinfo?(ryanvm)
Also, we use the NSS 3.19.1 CA Certs in MozillaBuild, FWIW.
I'm going to tentatively mark this as blocking bug 1177788, but I'm also not going to spend much time investigating this if it doesn't reproduce with the version of Mercurial we ship with MozillaBuild.
Blocks: 1177788
(In reply to Ryan VanderMeulen [:RyanVM UTC-4] from comment #6)
> I'm going to tentatively mark this as blocking bug 1177788, but I'm also not
> going to spend much time investigating this if it doesn't reproduce with the
> version of Mercurial we ship with MozillaBuild.

It does.  I'm running MozillaBuild 2.0.0.  I don't have any other Mercurial
versions installed on this Win2k8R2 server.  

I cloned comm-central off https://hg.mozilla.org/comm-central.

Then I went into the source.

then I did |python client.py checkout|

that's when it gave me the certificate verify error.

so.. STR:

1) hg clone https://hg.mozilla.org/comm-central cc
2) cd cc
3) python client.py checkout

certificate error.
I suspect client.py is picking up an hg executable not from MozillaBuild.

Try running `python client.py --hg /path/to/mozilla-build/.../hg.exe` (I think we ship a hg.exe shim in MozillaBuild).

I'd love to see this reproduced by calling `hg clone https://hg.mozilla.org/mozilla-central` directly so we can isolate the role of client.py.
After trying comment #8,  I found out that in d:\mozilla-build\python\Scripts,
there are two hg instances.  hg and hg.exe.

python client.py checkout uses the first instance.
I was unable to reproduce this on my host Windows 10 system. Using a 2.1 test build (http://people.mozilla.org/~rvandermeulen/MozillaBuildSetup2.1.0pre.exe), I followed the STR from comment 7 and I was able to successfully clone m-c from client.py (though annoyingly, bundleclone was apparently not used for it - I guess fodder for another bug possibly).

HOWEVER, I *can* reproduce this issue in a Windows Server 2k3 VM. Some kind of OS-level certificate store issue maybe? Do we know for sure what certificate store python is using?

---------------------------------------------------

$ hg clone https://hg.mozilla.org/comm-central cc
downloading bundle https://hg.cdn.mozilla.net/comm-central/8a2f9daeede4045fb30e4de43ab738db51d1da9d.gzip.hg
adding changesets
adding manifests
adding file changes
added 18422 changesets with 80039 changes to 14699 files (+5 heads)
finishing applying bundle; pulling
searching for changes
adding changesets
adding manifests
adding file changes
added 2 changesets with 3 changes to 3 files
updating to branch default
8798 files updated, 0 files merged, 0 files removed, 0 files unresolved

$ cd cc

$ python client.py checkout
Executing command: ['hg', 'pull', '-R', '.\\.']
pulling from https://hg.mozilla.org/comm-central
searching for changes
no changes found
Executing command: ['hg', 'update', '-r', 'default', '-R', '.\\.']
0 files updated, 0 files merged, 0 files removed, 0 files unresolved
Updated to revision b345d1330615e893b054e38b37c10abb61626818.
Executing command: ['hg', 'clone', 'https://hg.mozilla.org/mozilla-central/', '.\\mozilla']
requesting all changes
adding changesets
adding manifests
adding file changes
added 264053 changesets with 1490535 changes to 222395 files
updating to branch default
127293 files updated, 0 files merged, 0 files removed, 0 files unresolved
Executing command: ['hg', 'update', '-r', 'default', '-R', '.\\mozilla']
0 files updated, 0 files merged, 0 files removed, 0 files unresolved
Updated to revision f1dffc8682fbba463cb4bb305f293ddcccbc20b4.
Executing command: ['hg', 'clone', 'https://hg.mozilla.org/chatzilla/', '.\\mozilla\\extensions\\irc']
requesting all changes
adding changesets
adding manifests
adding file changes
added 1963 changesets with 4019 changes to 390 files (+99 heads)
updating to branch default
179 files updated, 0 files merged, 0 files removed, 0 files unresolved
Executing command: ['hg', 'update', '-r', 'default', '-R', '.\\mozilla\\extensions\\irc']
0 files updated, 0 files merged, 0 files removed, 0 files unresolved
Updated to revision dbeacae3a1f8438945a1340706b9eba55f4bc918.
Executing command: ['hg', 'clone', 'https://hg.mozilla.org/dom-inspector/', '.\\mozilla\\extensions\\inspector']
requesting all changes
adding changesets
adding manifests
adding file changes
added 1878 changesets with 5501 changes to 784 files (+83 heads)
updating to branch default
557 files updated, 0 files merged, 0 files removed, 0 files unresolved
Executing command: ['hg', 'update', '-r', 'default', '-R', '.\\mozilla\\extensions\\inspector']
0 files updated, 0 files merged, 0 files removed, 0 files unresolved
Updated to revision 708a9ed41eac545a5f5b3e71cb96088bf537ba51.

---------------------------------------------------

(In reply to Edmund Wong (:ewong) from comment #9)
> After trying comment #8,  I found out that in
> d:\mozilla-build\python\Scripts,
> there are two hg instances.  hg and hg.exe.
> 
> python client.py checkout uses the first instance.

That's interesting. During the 2.0 development cycle, we had to resort to building the hg.exe stub specifically to address issues with client.py not finding a valid hg executable otherwise.
FWIW, I'm pretty sure that this is a regression from python 2.7.9's stricter certificate handling based on the Googling around I've done.
(In reply to Edmund Wong (:ewong) from comment #7)
> It does.  I'm running MozillaBuild 2.0.0.  I don't have any other Mercurial
> versions installed on this Win2k8R2 server.  
> 
> I cloned comm-central off https://hg.mozilla.org/comm-central.
> 
> Then I went into the source.
> 
> then I did |python client.py checkout|
> 
> that's when it gave me the certificate verify error.
> 
> so.. STR:
> 
> 1) hg clone https://hg.mozilla.org/comm-central cc
> 2) cd cc
> 3) python client.py checkout
> 
> certificate error.

I am unable to reproduce this on MozillaBuild 2.0.0 with Python 2.7.10 on Windows Server 2012 R2. I activated bundleclone using:

bundleclone = ~/version-control-tools/hgext/bundleclone/__init__.py

(after cloning https://hg.mozilla.org/hgcustom/version-control-tools/ and updating to tip)

then tried:

hg clone https://hg.mozilla.org/mozilla-central m-c

and got:

$ hg clone https://hg.mozilla.org/mozilla-central/ mozilla-central
downloading bundle https://hg.cdn.mozilla.net/mozilla-central/6256ec9113c115141aab089c45ee69438884b680.gzip.hg
adding changesets
...
/snip
$ which hg
/c/mozilla-build/python/Scripts/hg
So I should probably recheck using comm-central.....
I cloned comm-central via `hg clone https://hg.mozilla.org/comm-central comm-central`, cd'ed into it, then ran:

$ python client.py checkout
Executing command: ['hg', 'pull', '-R', '.\\.']
pulling from https://hg.mozilla.org/comm-central
searching for changes
no changes found
Executing command: ['hg', 'update', '-r', 'default', '-R', '.\\.']
0 files updated, 0 files merged, 0 files removed, 0 files unresolved
Updated to revision 27c91093129e7ab4411fbe2e28bcfc90f3d54f35.
Executing command: ['hg', 'clone', 'https://hg.mozilla.org/mozilla-central/', '.
\\mozilla']
downloading bundle https://hg.cdn.mozilla.net/mozilla-central/6256ec9113c115141a
ab089c45ee69438884b680.gzip.hg
adding changesets
...
/snip

As you can see, bundleclone is being used.
So I retested with MozillaBuild 2.1.0pre in bug 1201520 comment 2, installing to a new folder and commenting out bundleclone in ~/.hgrc and I verified that bundleclone was not running in 2.0.0.

I cloned comm-central, it was using bundleclone.
I next ran `python client.py checkout`, and bundleclone was *not* used.

RyanVM had similar findings over IRC.
RyanVM's and my issue from comment 10 - 16 don't seem related to the original issue in comment 0. They are now summarised in bug 1177788 comment 4.
(In reply to Gary Kwong [:gkw] [:nth10sd] from comment #16)

I've filed bug 1209289 for the client.py bundleclone oddities.
(In reply to Ryan VanderMeulen [:RyanVM UTC-4] from comment #10)
> HOWEVER, I *can* reproduce this issue in a Windows Server 2k3 VM. Some kind
> of OS-level certificate store issue maybe? Do we know for sure what
> certificate store python is using?

Is still very-much relevant to this bug.
hg will call https://docs.python.org/2/library/ssl.html#ssl.SSLContext.load_default_certs() unless the hgrc specifies its own path.

You should be able to play around in a Python REPL to see which certs it loads.
MozillaBuild 3.0 is being restricted to Win7 and newer. Also, I don't intend to spend any more time investigating as there's been no ongoing complaints about it since.
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → INCOMPLETE
Product: mozilla.org → Firefox Build System
You need to log in before you can comment on or make changes to this bug.