1261202 - AWS builds on various trees failing with abort: certificate for s3-external-1.amazonaws.com has unexpected fingerprint c7:60:58:e5:d0:3f:1c:71:88:f8:e4:1b:04:05:28:0a:fb:1f:0f:40

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Closing beta for this:
https://treeherder.mozilla.org/logviewer.html#?job_id=960470&repo=mozilla-beta

Comment 1

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

 downloading bundle https://s3-external-1.amazonaws.com/moz-hg-bundles-us-east-1/build/tools/e3b9880209de5af9f905134b4e7a00cc58a20160.stream-legacy.hg
 Traceback (most recent call last):
   File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/dispatch.py", line 140, in _runcatch
     return _dispatch(req)
   File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/dispatch.py", line 850, in _dispatch
     cmdpats, cmdoptions)
   File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/dispatch.py", line 611, in runcommand
     ret = _runcommand(ui, options, cmd, d)
   File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/dispatch.py", line 941, in _runcommand
     return checkargs()
   File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/dispatch.py", line 912, in checkargs
     return cmdfunc()
   File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/dispatch.py", line 847, in <lambda>
     d = lambda: util.checksignature(func)(ui, *args, **cmdoptions)
   File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/util.py", line 677, in check
     return func(*args, **kwargs)
   File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/commands.py", line 1371, in clone
     branch=opts.get('branch'))
   File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/hg.py", line 423, in clone
     destpeer.local().clone(srcpeer, heads=revs, stream=stream)
   File "/usr/local/lib/hgext/bundleclone.py", line 847, in clone
     fh = hgurl.open(self.ui, url)
   File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/url.py", line 509, in open
     return opener(ui, authinfo).open(url_, data)
   File "/tools/python27/lib/python2.7/urllib2.py", line 400, in open
     response = self._open(req, data)
   File "/tools/python27/lib/python2.7/urllib2.py", line 418, in _open
     '_open', req)
   File "/tools/python27/lib/python2.7/urllib2.py", line 378, in _call_chain
     result = func(*args)
   File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/url.py", line 371, in https_open
     return self.do_open(self._makeconnection, req)
   File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/keepalive.py", line 252, in do_open
     self._start_transaction(h, req)
   File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/url.py", line 357, in _start_transaction
     return keepalive.KeepAliveHandler._start_transaction(self, h, req)
   File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/keepalive.py", line 350, in _start_transaction
     h.endheaders()
   File "/tools/python27/lib/python2.7/httplib.py", line 954, in endheaders
     self._send_output(message_body)
   File "/tools/python27/lib/python2.7/httplib.py", line 814, in _send_output
     self.send(msg)
   File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/url.py", line 144, in _sendfile
     orgsend(self, data)
   File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/keepalive.py", line 530, in safesend
     self.connect()
   File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/url.py", line 346, in connect
     sslutil.validator(self.ui, host)(self.sock)
   File "/tools/python27-mercurial/lib/python2.7/site-packages/mercurial/sslutil.py", line 166, in __call__
     hint=_('check hostfingerprint configuration'))
 Abort: certificate for s3-external-1.amazonaws.com has unexpected fingerprint c7:60:58:e5:d0:3f:1c:71:88:f8:e4:1b:04:05:28:0a:fb:1f:0f:40
 abort: certificate for s3-external-1.amazonaws.com has unexpected fingerprint c7:60:58:e5:d0:3f:1c:71:88:f8:e4:1b:04:05:28:0a:fb:1f:0f:40
 (check hostfingerprint configuration)
 program finished with exit code 255
 elapsedTime=0.920490

Comment 2

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Seeing Linux desktop builds failing with similar errors on inbound now.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [puppet] Remove host sig for us-east-1 S3

Bug 1259457 but for us-east-1.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8736962 [details] [diff] [review]
[puppet] Remove host sig for us-east-1 S3

https://hg.mozilla.org/build/puppet/rev/697a976a1c44
https://hg.mozilla.org/build/puppet/rev/16ecf6dce0d5

Will regen golden images soon.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

A new bld-linux64-ec2-golden was generated in us-east-1 (we hit bug 1259399 so it wasn't copied to us-west-2, but I did that manually because we need it for another bug). There are about 190 instances still running on the older image, but we seem to having some successful builds anyway. The older images should self-destruct in a couple of hours or so anyway.