Closed
Bug 1199356
Opened 9 years ago
Closed 7 years ago
abort: error fetching bundle: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581) with bundleclone
Categories
(Firefox Build System :: MozillaBuild, task)
Firefox Build System
MozillaBuild
Tracking
(Not tracked)
RESOLVED
INCOMPLETE
People
(Reporter: gps, Unassigned)
References
Details
Mossop encountered the error in the summary line when attempting to use bundleclone on Windows. Attempted to fetch from https://s3-us-west-2.amazonaws.com/moz-hg-bundles-us-west-2/integration/fx-team/0cc66c142126b36fe80947c199bb6d903f52817c.gzip.hg. I suspect the Mercurial SSL CA store isn't defined properly or is missing whatever root cert Amazon is using. Either way, it sounds like there is a bug here.
Reporter | ||
Comment 1•9 years ago
|
||
Mossop says he isn't using the MozillaBuild Mercurial, so who knows what's happening.
Comment 2•9 years ago
|
||
FWIW, we see this in the logs in automation too.
Comment 4•9 years ago
|
||
My bad, "Failed to log stats. Exception = [Errno 1] _ssl.c:504: error:1407742E:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert protocol version" is the error we see in automation.
Flags: needinfo?(ryanvm)
Comment 5•9 years ago
|
||
Also, we use the NSS 3.19.1 CA Certs in MozillaBuild, FWIW.
Comment 6•9 years ago
|
||
I'm going to tentatively mark this as blocking bug 1177788, but I'm also not going to spend much time investigating this if it doesn't reproduce with the version of Mercurial we ship with MozillaBuild.
Blocks: 1177788
Comment 7•9 years ago
|
||
(In reply to Ryan VanderMeulen [:RyanVM UTC-4] from comment #6) > I'm going to tentatively mark this as blocking bug 1177788, but I'm also not > going to spend much time investigating this if it doesn't reproduce with the > version of Mercurial we ship with MozillaBuild. It does. I'm running MozillaBuild 2.0.0. I don't have any other Mercurial versions installed on this Win2k8R2 server. I cloned comm-central off https://hg.mozilla.org/comm-central. Then I went into the source. then I did |python client.py checkout| that's when it gave me the certificate verify error. so.. STR: 1) hg clone https://hg.mozilla.org/comm-central cc 2) cd cc 3) python client.py checkout certificate error.
Reporter | ||
Comment 8•9 years ago
|
||
I suspect client.py is picking up an hg executable not from MozillaBuild. Try running `python client.py --hg /path/to/mozilla-build/.../hg.exe` (I think we ship a hg.exe shim in MozillaBuild). I'd love to see this reproduced by calling `hg clone https://hg.mozilla.org/mozilla-central` directly so we can isolate the role of client.py.
Comment 9•9 years ago
|
||
After trying comment #8, I found out that in d:\mozilla-build\python\Scripts, there are two hg instances. hg and hg.exe. python client.py checkout uses the first instance.
Comment 10•9 years ago
|
||
I was unable to reproduce this on my host Windows 10 system. Using a 2.1 test build (http://people.mozilla.org/~rvandermeulen/MozillaBuildSetup2.1.0pre.exe), I followed the STR from comment 7 and I was able to successfully clone m-c from client.py (though annoyingly, bundleclone was apparently not used for it - I guess fodder for another bug possibly). HOWEVER, I *can* reproduce this issue in a Windows Server 2k3 VM. Some kind of OS-level certificate store issue maybe? Do we know for sure what certificate store python is using? --------------------------------------------------- $ hg clone https://hg.mozilla.org/comm-central cc downloading bundle https://hg.cdn.mozilla.net/comm-central/8a2f9daeede4045fb30e4de43ab738db51d1da9d.gzip.hg adding changesets adding manifests adding file changes added 18422 changesets with 80039 changes to 14699 files (+5 heads) finishing applying bundle; pulling searching for changes adding changesets adding manifests adding file changes added 2 changesets with 3 changes to 3 files updating to branch default 8798 files updated, 0 files merged, 0 files removed, 0 files unresolved $ cd cc $ python client.py checkout Executing command: ['hg', 'pull', '-R', '.\\.'] pulling from https://hg.mozilla.org/comm-central searching for changes no changes found Executing command: ['hg', 'update', '-r', 'default', '-R', '.\\.'] 0 files updated, 0 files merged, 0 files removed, 0 files unresolved Updated to revision b345d1330615e893b054e38b37c10abb61626818. Executing command: ['hg', 'clone', 'https://hg.mozilla.org/mozilla-central/', '.\\mozilla'] requesting all changes adding changesets adding manifests adding file changes added 264053 changesets with 1490535 changes to 222395 files updating to branch default 127293 files updated, 0 files merged, 0 files removed, 0 files unresolved Executing command: ['hg', 'update', '-r', 'default', '-R', '.\\mozilla'] 0 files updated, 0 files merged, 0 files removed, 0 files unresolved Updated to revision f1dffc8682fbba463cb4bb305f293ddcccbc20b4. Executing command: ['hg', 'clone', 'https://hg.mozilla.org/chatzilla/', '.\\mozilla\\extensions\\irc'] requesting all changes adding changesets adding manifests adding file changes added 1963 changesets with 4019 changes to 390 files (+99 heads) updating to branch default 179 files updated, 0 files merged, 0 files removed, 0 files unresolved Executing command: ['hg', 'update', '-r', 'default', '-R', '.\\mozilla\\extensions\\irc'] 0 files updated, 0 files merged, 0 files removed, 0 files unresolved Updated to revision dbeacae3a1f8438945a1340706b9eba55f4bc918. Executing command: ['hg', 'clone', 'https://hg.mozilla.org/dom-inspector/', '.\\mozilla\\extensions\\inspector'] requesting all changes adding changesets adding manifests adding file changes added 1878 changesets with 5501 changes to 784 files (+83 heads) updating to branch default 557 files updated, 0 files merged, 0 files removed, 0 files unresolved Executing command: ['hg', 'update', '-r', 'default', '-R', '.\\mozilla\\extensions\\inspector'] 0 files updated, 0 files merged, 0 files removed, 0 files unresolved Updated to revision 708a9ed41eac545a5f5b3e71cb96088bf537ba51. --------------------------------------------------- (In reply to Edmund Wong (:ewong) from comment #9) > After trying comment #8, I found out that in > d:\mozilla-build\python\Scripts, > there are two hg instances. hg and hg.exe. > > python client.py checkout uses the first instance. That's interesting. During the 2.0 development cycle, we had to resort to building the hg.exe stub specifically to address issues with client.py not finding a valid hg executable otherwise.
Comment 11•9 years ago
|
||
FWIW, I'm pretty sure that this is a regression from python 2.7.9's stricter certificate handling based on the Googling around I've done.
(In reply to Edmund Wong (:ewong) from comment #7) > It does. I'm running MozillaBuild 2.0.0. I don't have any other Mercurial > versions installed on this Win2k8R2 server. > > I cloned comm-central off https://hg.mozilla.org/comm-central. > > Then I went into the source. > > then I did |python client.py checkout| > > that's when it gave me the certificate verify error. > > so.. STR: > > 1) hg clone https://hg.mozilla.org/comm-central cc > 2) cd cc > 3) python client.py checkout > > certificate error. I am unable to reproduce this on MozillaBuild 2.0.0 with Python 2.7.10 on Windows Server 2012 R2. I activated bundleclone using: bundleclone = ~/version-control-tools/hgext/bundleclone/__init__.py (after cloning https://hg.mozilla.org/hgcustom/version-control-tools/ and updating to tip) then tried: hg clone https://hg.mozilla.org/mozilla-central m-c and got: $ hg clone https://hg.mozilla.org/mozilla-central/ mozilla-central downloading bundle https://hg.cdn.mozilla.net/mozilla-central/6256ec9113c115141aab089c45ee69438884b680.gzip.hg adding changesets ... /snip
$ which hg /c/mozilla-build/python/Scripts/hg
So I should probably recheck using comm-central.....
I cloned comm-central via `hg clone https://hg.mozilla.org/comm-central comm-central`, cd'ed into it, then ran: $ python client.py checkout Executing command: ['hg', 'pull', '-R', '.\\.'] pulling from https://hg.mozilla.org/comm-central searching for changes no changes found Executing command: ['hg', 'update', '-r', 'default', '-R', '.\\.'] 0 files updated, 0 files merged, 0 files removed, 0 files unresolved Updated to revision 27c91093129e7ab4411fbe2e28bcfc90f3d54f35. Executing command: ['hg', 'clone', 'https://hg.mozilla.org/mozilla-central/', '. \\mozilla'] downloading bundle https://hg.cdn.mozilla.net/mozilla-central/6256ec9113c115141a ab089c45ee69438884b680.gzip.hg adding changesets ... /snip As you can see, bundleclone is being used.
So I retested with MozillaBuild 2.1.0pre in bug 1201520 comment 2, installing to a new folder and commenting out bundleclone in ~/.hgrc and I verified that bundleclone was not running in 2.0.0. I cloned comm-central, it was using bundleclone. I next ran `python client.py checkout`, and bundleclone was *not* used. RyanVM had similar findings over IRC.
RyanVM's and my issue from comment 10 - 16 don't seem related to the original issue in comment 0. They are now summarised in bug 1177788 comment 4.
Comment 18•9 years ago
|
||
(In reply to Gary Kwong [:gkw] [:nth10sd] from comment #16) I've filed bug 1209289 for the client.py bundleclone oddities.
Comment 19•9 years ago
|
||
(In reply to Ryan VanderMeulen [:RyanVM UTC-4] from comment #10) > HOWEVER, I *can* reproduce this issue in a Windows Server 2k3 VM. Some kind > of OS-level certificate store issue maybe? Do we know for sure what > certificate store python is using? Is still very-much relevant to this bug.
Reporter | ||
Comment 20•9 years ago
|
||
hg will call https://docs.python.org/2/library/ssl.html#ssl.SSLContext.load_default_certs() unless the hgrc specifies its own path. You should be able to play around in a Python REPL to see which certs it loads.
Comment 21•7 years ago
|
||
MozillaBuild 3.0 is being restricted to Win7 and newer. Also, I don't intend to spend any more time investigating as there's been no ongoing complaints about it since.
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → INCOMPLETE
Updated•1 year ago
|
Product: mozilla.org → Firefox Build System
You need to log in
before you can comment on or make changes to this bug.
Description
•