1066403 - blocklist update failing to wget from amo

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Edmund Wong (:ewong)]

the blocklist update step for c-c is failing:

--2014-09-11 15:52:51--  https://addons.mozilla.org/blocklist/3/%7B92650c4d-4b8e-4d2a-b7eb-24ecf4f6b63a%7D/2.32a1/SeaMonkey/20090105024647/blocklist-sync/en-US/nightly/blocklist-sync/default/default/
ERROR: Failed to open cert /etc/ssl/certs/make-dummy-cert: (-34).
ERROR: Failed to open cert /etc/ssl/certs/Makefile: (-34).
ERROR: Failed to open cert /etc/ssl/certs/ca-bundle.trust.crt: (-34).
Resolving addons.mozilla.org... 63.245.216.132, 2620:101:8020:5::2:132
Connecting to addons.mozilla.org|63.245.216.132|:443... connected.
GnuTLS: A TLS fatal alert has been received.
GnuTLS: received alert [47]: Illegal parameter
Unable to establish SSL connection.
ERROR wget exited with a non-zero exit code: 4
Blocklist files differ, updating hg.

slave is using wget version 1.15.

Comment 1

[Justin Wood (:Callek)]

Any rough ideas here?

The job for us on Aug 30 was green, while the job on Sep 6 was broken (I don't see a log for that one to know why), the log in c#0 was from Sep 11

Fwiw the file SeaMonkey is using here, is http://hg.mozilla.org/users/Callek_gmail.com/tools/file/d26103a49cef/scripts/blocklist/sync-hg-blocklist.sh

While the file m-c is using (afaict) is http://mxr.mozilla.org/build/source/tools/scripts/periodic_file_updates/periodic_file_updates.sh

the blocklist pieces look near-identical to me. I'm still looking to see if the Firefox run itself was green.

Comment 2

[Justin Wood (:Callek)]

So, I just had to trigger a new moco, m-c job of this, and we're getting the same errors for Firefox.

Relevent (shown) log:

INFO: New HPKP preload lists differs from what is in-tree.
ERROR: Failed to open cert /etc/ssl/certs/make-dummy-cert: (-34).
ERROR: Failed to open cert /etc/ssl/certs/Makefile: (-34).
ERROR: Failed to open cert /etc/ssl/certs/ca-bundle.trust.crt: (-34).
GnuTLS: A TLS fatal alert has been received.
GnuTLS: received alert [47]: Illegal parameter
Unable to establish SSL connection.
ERROR: wget exited with a non-zero exit code: 4


The info line about HPKP is explicitly so you can tell where in the file we are.

Jorgev any clue what changed with AMO recently, and why a wget 1.15 would fail (even with --no-check-certificate).

Comment 3

[Justin Wood (:Callek)]

coop,

So the interesting bit here is that despite the failing blocklist update this builder has been succeeding, and we had no (in my mind) way of noticing that they were failing.  I'm not sure what/if there is a proper path forward here for our end. Thoughts? [n-i for you on this piece since it was you who did the script in Bug 869051]

Comment 4

[Jorge Villalobos [:jorgev] (he/him)]

I know that there were some SSL errors on AMO early yesterday, but they were temporary. Maybe something was changed in the back end.

I think Jason might know what's up.

Comment 5

[Jason Thomas [:jason]]

The last major change we made with SSL termination was when we upgraded our LB to latest release on Aug 27th (bug 1041759).

Also AMO blocklist has moved to blocklist.addons.mozilla.org. Bug 1006815 to update sync-hg-blocklist.sh. Please update scripts to the new endpoint.

It looks like wget is complied with GnuTLS. What version of GnuTLS are you using? I was able to replicate this issue with libgnutls26 2.12.20-8+deb7u2 and gnutls-cli but not with a newer version:

libgnutls26-2.12.20-8+deb7u2:
root@ssj1:~# gnutls-cli -p 443 addons.mozilla.org -d 3
Resolving 'addons.mozilla.org'...
Connecting to '63.245.216.132:443'...
|<2>| ASSERT: gnutls_constate.c:695
|<3>| HSK[0xd18c30]: Keeping ciphersuite: DHE_RSA_AES_128_CBC_SHA1
|<3>| HSK[0xd18c30]: Keeping ciphersuite: DHE_RSA_AES_128_CBC_SHA256
|<3>| HSK[0xd18c30]: Keeping ciphersuite: DHE_RSA_CAMELLIA_128_CBC_SHA1
|<3>| HSK[0xd18c30]: Keeping ciphersuite: DHE_RSA_AES_256_CBC_SHA1
|<3>| HSK[0xd18c30]: Keeping ciphersuite: DHE_RSA_AES_256_CBC_SHA256
|<3>| HSK[0xd18c30]: Keeping ciphersuite: DHE_RSA_CAMELLIA_256_CBC_SHA1
|<3>| HSK[0xd18c30]: Keeping ciphersuite: DHE_RSA_3DES_EDE_CBC_SHA1
|<3>| HSK[0xd18c30]: Keeping ciphersuite: DHE_DSS_AES_128_CBC_SHA1
|<3>| HSK[0xd18c30]: Keeping ciphersuite: DHE_DSS_AES_128_CBC_SHA256
|<3>| HSK[0xd18c30]: Keeping ciphersuite: DHE_DSS_CAMELLIA_128_CBC_SHA1
|<3>| HSK[0xd18c30]: Keeping ciphersuite: DHE_DSS_AES_256_CBC_SHA1
|<3>| HSK[0xd18c30]: Keeping ciphersuite: DHE_DSS_AES_256_CBC_SHA256
|<3>| HSK[0xd18c30]: Keeping ciphersuite: DHE_DSS_CAMELLIA_256_CBC_SHA1
|<3>| HSK[0xd18c30]: Keeping ciphersuite: DHE_DSS_3DES_EDE_CBC_SHA1
|<3>| HSK[0xd18c30]: Keeping ciphersuite: DHE_DSS_ARCFOUR_SHA1
|<3>| HSK[0xd18c30]: Keeping ciphersuite: RSA_AES_128_CBC_SHA1
|<3>| HSK[0xd18c30]: Keeping ciphersuite: RSA_AES_128_CBC_SHA256
|<3>| HSK[0xd18c30]: Keeping ciphersuite: RSA_CAMELLIA_128_CBC_SHA1
|<3>| HSK[0xd18c30]: Keeping ciphersuite: RSA_AES_256_CBC_SHA1
|<3>| HSK[0xd18c30]: Keeping ciphersuite: RSA_AES_256_CBC_SHA256
|<3>| HSK[0xd18c30]: Keeping ciphersuite: RSA_CAMELLIA_256_CBC_SHA1
|<3>| HSK[0xd18c30]: Keeping ciphersuite: RSA_3DES_EDE_CBC_SHA1
|<3>| HSK[0xd18c30]: Keeping ciphersuite: RSA_ARCFOUR_SHA1
|<3>| HSK[0xd18c30]: Keeping ciphersuite: RSA_ARCFOUR_MD5
|<2>| EXT[0xd18c30]: Sending extension SERVER NAME (23 bytes)
|<2>| EXT[0xd18c30]: Sending extension SAFE RENEGOTIATION (1 bytes)
|<2>| EXT[0xd18c30]: Sending extension SESSION TICKET (0 bytes)
|<2>| EXT[SIGA]: sent signature algo (4.2) DSA-SHA256
|<2>| EXT[SIGA]: sent signature algo (4.1) RSA-SHA256
|<2>| EXT[SIGA]: sent signature algo (2.1) RSA-SHA1
|<2>| EXT[SIGA]: sent signature algo (2.2) DSA-SHA1
|<2>| EXT[0xd18c30]: Sending extension SIGNATURE ALGORITHMS (10 bytes)
|<3>| HSK[0xd18c30]: CLIENT HELLO was sent [143 bytes]
|<3>| HSK[0xd18c30]: SERVER HELLO was received [81 bytes]
|<3>| HSK[0xd18c30]: Server's version: 3.3
|<3>| HSK[0xd18c30]: SessionID length: 32
|<3>| HSK[0xd18c30]: SessionID: a30e11fa40f2a168aec7564f4cd1faba5094493bd98706c2196465f62965c301
|<3>| HSK[0xd18c30]: Selected cipher suite: RSA_AES_128_CBC_SHA1
|<2>| EXT[0xd18c30]: Parsing extension 'SAFE RENEGOTIATION/65281' (1 bytes)
|<3>| HSK[0xd18c30]: Safe renegotiation succeeded
|<3>| HSK[0xd18c30]: CERTIFICATE was received [3646 bytes]
|<2>| ASSERT: ext_signature.c:393
|<2>| ASSERT: ext_signature.c:393
|<3>| HSK[0xd18c30]: SERVER HELLO DONE was received [4 bytes]
|<2>| ASSERT: gnutls_handshake.c:1369
|<3>| HSK[0xd18c30]: CLIENT KEY EXCHANGE was sent [262 bytes]
|<3>| REC[0xd18c30]: Sent ChangeCipherSpec
|<3>| HSK[0xd18c30]: Cipher Suite: RSA_AES_128_CBC_SHA1
|<3>| HSK[0xd18c30]: Initializing internal [write] cipher sessions
|<3>| HSK[0xd18c30]: recording tls-unique CB (send)
|<3>| HSK[0xd18c30]: FINISHED was sent [16 bytes]
|<2>| ASSERT: gnutls_record.c:726
|<2>| ASSERT: gnutls_record.c:1122
|<2>| ASSERT: gnutls_handshake.c:2933
|<2>| ASSERT: gnutls_handshake.c:3139
*** Fatal error: A TLS fatal alert has been received.
*** Received alert [47]: Illegal parameter
|<2>| errno: 104
|<2>| ASSERT: gnutls_buffers.c:431
|<2>| ASSERT: gnutls_buffers.c:755
|<2>| ASSERT: gnutls_record.c:491
*** Handshake has failed
GnuTLS error: A TLS fatal alert has been received.

# gnutls-3.1.2
gnutls-cli -p 443 addons.mozilla.org -v 3
Processed 153 CA certificate(s).
Resolving 'addons.mozilla.org'...
Connecting to '63.245.216.132:443'...
- Certificate type: X.509
- Got a certificate list of 2 certificates.
- Certificate[0] info:
 - subject `businessCategory=Private Organization,jurisdictionOfIncorporationCountryName=US,jurisdictionOfIncorporationStateOrProvinceName=California,serialNumber=C2543436,street=650 Castro St Ste 300,postalCode=94041,C=US,ST=CA,L=Mountain View,O=Mozilla Foundation,CN=addons.mozilla.org', issuer `C=US,O=DigiCert Inc,OU=www.digicert.com,CN=DigiCert High Assurance EV CA-1', RSA key 2048 bits, signed using RSA-SHA1, activated `2013-08-19 00:00:00 UTC', expires `2015-08-24 12:00:00 UTC', SHA-1 fingerprint `13f033f430ba4c7918b472f9846a0a96a83508dd'
	Public Key Id:
		a5290dae6f0ccdacd9d3fbc9fe369f8e19168c01
	Public key's random art:
		+--[ RSA 2048]----+
		|        E        |
		|         .       |
		|      .   o      |
		|     . o + +     |
		|     +o S . o    |
		|    ..+.     .   |
		|    .* .    o    |
		|    o.= .. oo+ . |
		|     ....+=o+++  |
		+-----------------+

- Certificate[1] info:
 - subject `C=US,O=DigiCert Inc,OU=www.digicert.com,CN=DigiCert High Assurance EV CA-1', issuer `C=US,O=DigiCert Inc,OU=www.digicert.com,CN=DigiCert High Assurance EV Root CA', RSA key 2048 bits, signed using RSA-SHA1, activated `2007-11-09 12:00:00 UTC', expires `2021-11-10 00:00:00 UTC', SHA-1 fingerprint `dbc7e90b0da5d88a5535430eeb665d077859e8e8'
- Status: The certificate is trusted.
- Description: (TLS1.2-PKIX)-(RSA)-(AES-128-CBC)-(SHA1)
- Session ID: 7A:E2:7D:2A:FD:1C:79:67:1D:79:DF:1B:F2:6C:11:5D:6A:7C:17:4B:50:06:C4:73:6B:33:5A:B3:56:79:C4:01
- Version: TLS1.2
- Key Exchange: RSA
- Cipher: AES-128-CBC
- MAC: SHA1
- Compression: NULL
- Handshake was completed

- Simple Client Mode:

Comment 6

[Chris Cooper [:coop] (he/him)]

We also seem to have an empty blocklist in-tree right now:

http://hg.mozilla.org/mozilla-central/rev/5fd6428ae179

We should protect against that.

Comment 7

[Chris Cooper [:coop] (he/him)]

All the train branches except for release were affected, so I landed a blocklist update for all of them:

https://hg.mozilla.org/mozilla-central/rev/989c1c75889c
https://hg.mozilla.org/releases/mozilla-aurora/rev/d3b5bd6415de
https://hg.mozilla.org/releases/mozilla-beta/rev/06300676d4cd

I'm kicking off our periodic updates builder on each of those branches now to see if I can generate an actionable log.

However, I don't think the above is even related to comment #0. If Seamonkey is using the old script, I'm not sure what we can do.

Comment 8

[Chris Cooper [:coop] (he/him)]

We were failing to exit on wget failures, which caused us to push empty blocklist files.

https://hg.mozilla.org/build/tools/rev/9b79689ee12f

Seamonkey may want to check the script they're using to see if similar errors are present.

Comment 9

[Edmund Wong (:ewong)]

(In reply to Chris Cooper [:coop] from comment #8)
> We were failing to exit on wget failures, which caused us to push empty
> blocklist files.
> 
> https://hg.mozilla.org/build/tools/rev/9b79689ee12f
> 
> Seamonkey may want to check the script they're using to see if similar
> errors are present.

Yes, we got the same errors as stated in comment #0.  So the solution is
to upgrade wget w/ a newer GnuTLS lib?

Comment 10

[Edmund Wong (:ewong)]

(In reply to Edmund Wong (:ewong) from comment #9)
> (In reply to Chris Cooper [:coop] from comment #8)
> > We were failing to exit on wget failures, which caused us to push empty
> > blocklist files.
> > 
> > https://hg.mozilla.org/build/tools/rev/9b79689ee12f
> > 
> > Seamonkey may want to check the script they're using to see if similar
> > errors are present.
> 
> Yes, we got the same errors as stated in comment #0.  So the solution is
> to upgrade wget w/ a newer GnuTLS lib?

(*embarrassed look*) Of course we get the same error as comment #0... I posted 
comment #0. ;/

will converse with Callek to see the best approach..  is there a mock
package update for wget/libgnutls?

Comment 11

[Justin Wood (:Callek)]

(In reply to Chris Cooper [:coop] from comment #7)
> All the train branches except for release were affected, so I landed a
> blocklist update for all of them:
> 
> https://hg.mozilla.org/mozilla-central/rev/989c1c75889c
> https://hg.mozilla.org/releases/mozilla-aurora/rev/d3b5bd6415de
> https://hg.mozilla.org/releases/mozilla-beta/rev/06300676d4cd
> 
> I'm kicking off our periodic updates builder on each of those branches now
> to see if I can generate an actionable log.
> 
> However, I don't think the above is even related to comment #0. If Seamonkey
> is using the old script, I'm not sure what we can do.

The blank blocklist was not a sympted as affected by SeaMonkey.

However the underlying issue, the wget failure, is indeed affecting seamonkey despite the different script. I'm expecting the remediation identified here for Firefox/MoCo to be directly translateable to either an automatic-for-free fix for SeaMonkey, or an action item we'll take in a new bug.

The "protect against a blank blocklist" is not "enough" to solve this bug, but "continue to properly update the blocklist" is what we need.

Comment 12

[Chris Cooper [:coop] (he/him)]

Found bug 967452 where the new version was deployed. Going to see if I can roll a new package with an updated GnuTLS using those instructions.

Comment 13

[Chris Cooper [:coop] (he/him)]

(In reply to Chris Cooper [:coop] from comment #12)
> Found bug 967452 where the new version was deployed. Going to see if I can
> roll a new package with an updated GnuTLS using those instructions.

Our build environment is CentOS 6.2, so finding off-the-shelf packages for GnuTLS 3+ is unlikely.

A simpler change would be to change the compile options to specify "--with-ssl=openssl" instead. I've tested this quickly and know that the compiled version of wget works against blocklist.addons.mozilla.org

cc-ing Dustin as the person who deployed the current version of wget in bug 967452.

Comment 14

[Dustin J. Mitchell [:dustin] (he/him)]

I feel like we deliberately switched from OpenSSL to GnuTLS in that bug.  I think that's what I meant in bug 967452 comment 1: the version of OpenSSL available doesn't recognize alt names.

Anyway, if OpenSSL works, I don't have (or can't remember) any reason not to rebuild wget to use that.

Comment 15

[Chris Cooper [:coop] (he/him)]

Attachment: Point to a new version of wget compiled with openssl

I tested this on b-linux64-ix-0006. The periodic file updates works fine with the openssl version, and it can also access ftp-ssl.mozilla.org.

rpms are available here:

http://puppetagain.pub.build.mozilla.org/data/repos/yum/releng/public/CentOS/6/i386/wget-1.15-2.el6.i686.rpm
http://puppetagain.pub.build.mozilla.org/data/repos/yum/releng/public/CentOS/6/noarch/wget-1.15-2.el6.src.rpm
http://puppetagain.pub.build.mozilla.org/data/repos/yum/releng/public/CentOS/6/x86_64/wget-1.15-2.el6.x86_64.rpm

Comment 16

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8491753 [details] [diff] [review]
Point to a new version of wget compiled with openssl

You should update the %changelog at the bottom, too, indicating why we changed.  You probably also want to bump the repoflag.

Comment 17

[Chris Cooper [:coop] (he/him)]

Comment on attachment 8491753 [details] [diff] [review]
Point to a new version of wget compiled with openssl

Review of attachment 8491753 [details] [diff] [review]:
-----------------------------------------------------------------

Landed with Dustin suggestions:

https://hg.mozilla.org/build/puppet/rev/64c8641aa23f

Comment 18

[Chris Cooper [:coop] (he/him)]

Comment on attachment 8491753 [details] [diff] [review]
Point to a new version of wget compiled with openssl

Review of attachment 8491753 [details] [diff] [review]:
-----------------------------------------------------------------

Backed out to regenerate repo indices, and then re-landed:

http://hg.mozilla.org/build/puppet/rev/b3537935f103

Comment 19

[Chris Cooper [:coop] (he/him)]

Builder succeeded this past weekend:

http://buildbot-master86.srv.releng.scl3.mozilla.com:8001/builders/Linux%20x86-64%20mozilla-central%20periodic%20file%20update/builds/0/steps/run_script/logs/stdio

Comment 20

[Nick Thomas [:nthomas] (UTC+12)]

The wget upgrade also resolved this output on each request:
 ERROR: Failed to open cert /etc/ssl/certs/Makefile: (-34).
 ERROR: Failed to open cert /etc/ssl/certs/make-dummy-cert: (-34).
 ERROR: Failed to open cert /etc/ssl/certs/ca-bundle.trust.crt: (-34).

Comment 21

[Elbart]

The blocklist is still empty in 31ESR:
https://hg.mozilla.org/releases/mozilla-esr31/log/tip/browser/app/blocklist.xml

31.1.0 was the last good, 31.1.1 was the first bad version.

Comment 22

[Benjamin Kerensa [:bkerensa]]

Reopening this as ESR seems impacted. 31.2.0ESR users are reporting lack of block list.

Comment 23

[Chris Cooper [:coop] (he/him)]

(In reply to Benjamin Kerensa [:bkerensa] from comment #22)
> Reopening this as ESR seems impacted. 31.2.0ESR users are reporting lack of
> block list.

Re-running against esr31 now.

Comment 24

[Chris Cooper [:coop] (he/him)]

(In reply to Chris Cooper [:coop] from comment #23) 
> Re-running against esr31 now.

Done:

http://hg.mozilla.org/releases/mozilla-esr31/rev/b2f23baf9863