blocklist update failing to wget from amo

RESOLVED FIXED

Status

defect
RESOLVED FIXED
5 years ago
Last year

People

(Reporter: ewong, Assigned: coop)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

the blocklist update step for c-c is failing:

--2014-09-11 15:52:51--  https://addons.mozilla.org/blocklist/3/%7B92650c4d-4b8e-4d2a-b7eb-24ecf4f6b63a%7D/2.32a1/SeaMonkey/20090105024647/blocklist-sync/en-US/nightly/blocklist-sync/default/default/
ERROR: Failed to open cert /etc/ssl/certs/make-dummy-cert: (-34).
ERROR: Failed to open cert /etc/ssl/certs/Makefile: (-34).
ERROR: Failed to open cert /etc/ssl/certs/ca-bundle.trust.crt: (-34).
Resolving addons.mozilla.org... 63.245.216.132, 2620:101:8020:5::2:132
Connecting to addons.mozilla.org|63.245.216.132|:443... connected.
GnuTLS: A TLS fatal alert has been received.
GnuTLS: received alert [47]: Illegal parameter
Unable to establish SSL connection.
ERROR wget exited with a non-zero exit code: 4
Blocklist files differ, updating hg.

slave is using wget version 1.15.
Summary: blocklist update failing → SeaMonkey comm-central blocklist update failing
Any rough ideas here?

The job for us on Aug 30 was green, while the job on Sep 6 was broken (I don't see a log for that one to know why), the log in c#0 was from Sep 11

Fwiw the file SeaMonkey is using here, is http://hg.mozilla.org/users/Callek_gmail.com/tools/file/d26103a49cef/scripts/blocklist/sync-hg-blocklist.sh

While the file m-c is using (afaict) is http://mxr.mozilla.org/build/source/tools/scripts/periodic_file_updates/periodic_file_updates.sh

the blocklist pieces look near-identical to me. I'm still looking to see if the Firefox run itself was green.
So, I just had to trigger a new moco, m-c job of this, and we're getting the same errors for Firefox.

Relevent (shown) log:

INFO: New HPKP preload lists differs from what is in-tree.
ERROR: Failed to open cert /etc/ssl/certs/make-dummy-cert: (-34).
ERROR: Failed to open cert /etc/ssl/certs/Makefile: (-34).
ERROR: Failed to open cert /etc/ssl/certs/ca-bundle.trust.crt: (-34).
GnuTLS: A TLS fatal alert has been received.
GnuTLS: received alert [47]: Illegal parameter
Unable to establish SSL connection.
ERROR: wget exited with a non-zero exit code: 4


The info line about HPKP is explicitly so you can tell where in the file we are.

Jorgev any clue what changed with AMO recently, and why a wget 1.15 would fail (even with --no-check-certificate).
Component: Release Engineering → General Automation
Flags: needinfo?(jorge)
Product: SeaMonkey → Release Engineering
QA Contact: catlee
Summary: SeaMonkey comm-central blocklist update failing → blocklist update failing to wget from amo
Version: SeaMonkey 2.29 Branch → other
coop,

So the interesting bit here is that despite the failing blocklist update this builder has been succeeding, and we had no (in my mind) way of noticing that they were failing.  I'm not sure what/if there is a proper path forward here for our end. Thoughts? [n-i for you on this piece since it was you who did the script in Bug 869051]
Flags: needinfo?(coop)
I know that there were some SSL errors on AMO early yesterday, but they were temporary. Maybe something was changed in the back end.

I think Jason might know what's up.
Flags: needinfo?(jorge) → needinfo?(jthomas)
The last major change we made with SSL termination was when we upgraded our LB to latest release on Aug 27th (bug 1041759).

Also AMO blocklist has moved to blocklist.addons.mozilla.org. Bug 1006815 to update sync-hg-blocklist.sh. Please update scripts to the new endpoint.

It looks like wget is complied with GnuTLS. What version of GnuTLS are you using? I was able to replicate this issue with libgnutls26 2.12.20-8+deb7u2 and gnutls-cli but not with a newer version:

libgnutls26-2.12.20-8+deb7u2:
root@ssj1:~# gnutls-cli -p 443 addons.mozilla.org -d 3
Resolving 'addons.mozilla.org'...
Connecting to '63.245.216.132:443'...
|<2>| ASSERT: gnutls_constate.c:695
|<3>| HSK[0xd18c30]: Keeping ciphersuite: DHE_RSA_AES_128_CBC_SHA1
|<3>| HSK[0xd18c30]: Keeping ciphersuite: DHE_RSA_AES_128_CBC_SHA256
|<3>| HSK[0xd18c30]: Keeping ciphersuite: DHE_RSA_CAMELLIA_128_CBC_SHA1
|<3>| HSK[0xd18c30]: Keeping ciphersuite: DHE_RSA_AES_256_CBC_SHA1
|<3>| HSK[0xd18c30]: Keeping ciphersuite: DHE_RSA_AES_256_CBC_SHA256
|<3>| HSK[0xd18c30]: Keeping ciphersuite: DHE_RSA_CAMELLIA_256_CBC_SHA1
|<3>| HSK[0xd18c30]: Keeping ciphersuite: DHE_RSA_3DES_EDE_CBC_SHA1
|<3>| HSK[0xd18c30]: Keeping ciphersuite: DHE_DSS_AES_128_CBC_SHA1
|<3>| HSK[0xd18c30]: Keeping ciphersuite: DHE_DSS_AES_128_CBC_SHA256
|<3>| HSK[0xd18c30]: Keeping ciphersuite: DHE_DSS_CAMELLIA_128_CBC_SHA1
|<3>| HSK[0xd18c30]: Keeping ciphersuite: DHE_DSS_AES_256_CBC_SHA1
|<3>| HSK[0xd18c30]: Keeping ciphersuite: DHE_DSS_AES_256_CBC_SHA256
|<3>| HSK[0xd18c30]: Keeping ciphersuite: DHE_DSS_CAMELLIA_256_CBC_SHA1
|<3>| HSK[0xd18c30]: Keeping ciphersuite: DHE_DSS_3DES_EDE_CBC_SHA1
|<3>| HSK[0xd18c30]: Keeping ciphersuite: DHE_DSS_ARCFOUR_SHA1
|<3>| HSK[0xd18c30]: Keeping ciphersuite: RSA_AES_128_CBC_SHA1
|<3>| HSK[0xd18c30]: Keeping ciphersuite: RSA_AES_128_CBC_SHA256
|<3>| HSK[0xd18c30]: Keeping ciphersuite: RSA_CAMELLIA_128_CBC_SHA1
|<3>| HSK[0xd18c30]: Keeping ciphersuite: RSA_AES_256_CBC_SHA1
|<3>| HSK[0xd18c30]: Keeping ciphersuite: RSA_AES_256_CBC_SHA256
|<3>| HSK[0xd18c30]: Keeping ciphersuite: RSA_CAMELLIA_256_CBC_SHA1
|<3>| HSK[0xd18c30]: Keeping ciphersuite: RSA_3DES_EDE_CBC_SHA1
|<3>| HSK[0xd18c30]: Keeping ciphersuite: RSA_ARCFOUR_SHA1
|<3>| HSK[0xd18c30]: Keeping ciphersuite: RSA_ARCFOUR_MD5
|<2>| EXT[0xd18c30]: Sending extension SERVER NAME (23 bytes)
|<2>| EXT[0xd18c30]: Sending extension SAFE RENEGOTIATION (1 bytes)
|<2>| EXT[0xd18c30]: Sending extension SESSION TICKET (0 bytes)
|<2>| EXT[SIGA]: sent signature algo (4.2) DSA-SHA256
|<2>| EXT[SIGA]: sent signature algo (4.1) RSA-SHA256
|<2>| EXT[SIGA]: sent signature algo (2.1) RSA-SHA1
|<2>| EXT[SIGA]: sent signature algo (2.2) DSA-SHA1
|<2>| EXT[0xd18c30]: Sending extension SIGNATURE ALGORITHMS (10 bytes)
|<3>| HSK[0xd18c30]: CLIENT HELLO was sent [143 bytes]
|<3>| HSK[0xd18c30]: SERVER HELLO was received [81 bytes]
|<3>| HSK[0xd18c30]: Server's version: 3.3
|<3>| HSK[0xd18c30]: SessionID length: 32
|<3>| HSK[0xd18c30]: SessionID: a30e11fa40f2a168aec7564f4cd1faba5094493bd98706c2196465f62965c301
|<3>| HSK[0xd18c30]: Selected cipher suite: RSA_AES_128_CBC_SHA1
|<2>| EXT[0xd18c30]: Parsing extension 'SAFE RENEGOTIATION/65281' (1 bytes)
|<3>| HSK[0xd18c30]: Safe renegotiation succeeded
|<3>| HSK[0xd18c30]: CERTIFICATE was received [3646 bytes]
|<2>| ASSERT: ext_signature.c:393
|<2>| ASSERT: ext_signature.c:393
|<3>| HSK[0xd18c30]: SERVER HELLO DONE was received [4 bytes]
|<2>| ASSERT: gnutls_handshake.c:1369
|<3>| HSK[0xd18c30]: CLIENT KEY EXCHANGE was sent [262 bytes]
|<3>| REC[0xd18c30]: Sent ChangeCipherSpec
|<3>| HSK[0xd18c30]: Cipher Suite: RSA_AES_128_CBC_SHA1
|<3>| HSK[0xd18c30]: Initializing internal [write] cipher sessions
|<3>| HSK[0xd18c30]: recording tls-unique CB (send)
|<3>| HSK[0xd18c30]: FINISHED was sent [16 bytes]
|<2>| ASSERT: gnutls_record.c:726
|<2>| ASSERT: gnutls_record.c:1122
|<2>| ASSERT: gnutls_handshake.c:2933
|<2>| ASSERT: gnutls_handshake.c:3139
*** Fatal error: A TLS fatal alert has been received.
*** Received alert [47]: Illegal parameter
|<2>| errno: 104
|<2>| ASSERT: gnutls_buffers.c:431
|<2>| ASSERT: gnutls_buffers.c:755
|<2>| ASSERT: gnutls_record.c:491
*** Handshake has failed
GnuTLS error: A TLS fatal alert has been received.

# gnutls-3.1.2
gnutls-cli -p 443 addons.mozilla.org -v 3
Processed 153 CA certificate(s).
Resolving 'addons.mozilla.org'...
Connecting to '63.245.216.132:443'...
- Certificate type: X.509
- Got a certificate list of 2 certificates.
- Certificate[0] info:
 - subject `businessCategory=Private Organization,jurisdictionOfIncorporationCountryName=US,jurisdictionOfIncorporationStateOrProvinceName=California,serialNumber=C2543436,street=650 Castro St Ste 300,postalCode=94041,C=US,ST=CA,L=Mountain View,O=Mozilla Foundation,CN=addons.mozilla.org', issuer `C=US,O=DigiCert Inc,OU=www.digicert.com,CN=DigiCert High Assurance EV CA-1', RSA key 2048 bits, signed using RSA-SHA1, activated `2013-08-19 00:00:00 UTC', expires `2015-08-24 12:00:00 UTC', SHA-1 fingerprint `13f033f430ba4c7918b472f9846a0a96a83508dd'
	Public Key Id:
		a5290dae6f0ccdacd9d3fbc9fe369f8e19168c01
	Public key's random art:
		+--[ RSA 2048]----+
		|        E        |
		|         .       |
		|      .   o      |
		|     . o + +     |
		|     +o S . o    |
		|    ..+.     .   |
		|    .* .    o    |
		|    o.= .. oo+ . |
		|     ....+=o+++  |
		+-----------------+

- Certificate[1] info:
 - subject `C=US,O=DigiCert Inc,OU=www.digicert.com,CN=DigiCert High Assurance EV CA-1', issuer `C=US,O=DigiCert Inc,OU=www.digicert.com,CN=DigiCert High Assurance EV Root CA', RSA key 2048 bits, signed using RSA-SHA1, activated `2007-11-09 12:00:00 UTC', expires `2021-11-10 00:00:00 UTC', SHA-1 fingerprint `dbc7e90b0da5d88a5535430eeb665d077859e8e8'
- Status: The certificate is trusted.
- Description: (TLS1.2-PKIX)-(RSA)-(AES-128-CBC)-(SHA1)
- Session ID: 7A:E2:7D:2A:FD:1C:79:67:1D:79:DF:1B:F2:6C:11:5D:6A:7C:17:4B:50:06:C4:73:6B:33:5A:B3:56:79:C4:01
- Version: TLS1.2
- Key Exchange: RSA
- Cipher: AES-128-CBC
- MAC: SHA1
- Compression: NULL
- Handshake was completed

- Simple Client Mode:
Flags: needinfo?(jthomas)
We also seem to have an empty blocklist in-tree right now:

http://hg.mozilla.org/mozilla-central/rev/5fd6428ae179

We should protect against that.
Flags: needinfo?(coop)
All the train branches except for release were affected, so I landed a blocklist update for all of them:

https://hg.mozilla.org/mozilla-central/rev/989c1c75889c
https://hg.mozilla.org/releases/mozilla-aurora/rev/d3b5bd6415de
https://hg.mozilla.org/releases/mozilla-beta/rev/06300676d4cd

I'm kicking off our periodic updates builder on each of those branches now to see if I can generate an actionable log.

However, I don't think the above is even related to comment #0. If Seamonkey is using the old script, I'm not sure what we can do.
We were failing to exit on wget failures, which caused us to push empty blocklist files.

https://hg.mozilla.org/build/tools/rev/9b79689ee12f

Seamonkey may want to check the script they're using to see if similar errors are present.
(In reply to Chris Cooper [:coop] from comment #8)
> We were failing to exit on wget failures, which caused us to push empty
> blocklist files.
> 
> https://hg.mozilla.org/build/tools/rev/9b79689ee12f
> 
> Seamonkey may want to check the script they're using to see if similar
> errors are present.

Yes, we got the same errors as stated in comment #0.  So the solution is
to upgrade wget w/ a newer GnuTLS lib?
(In reply to Edmund Wong (:ewong) from comment #9)
> (In reply to Chris Cooper [:coop] from comment #8)
> > We were failing to exit on wget failures, which caused us to push empty
> > blocklist files.
> > 
> > https://hg.mozilla.org/build/tools/rev/9b79689ee12f
> > 
> > Seamonkey may want to check the script they're using to see if similar
> > errors are present.
> 
> Yes, we got the same errors as stated in comment #0.  So the solution is
> to upgrade wget w/ a newer GnuTLS lib?

(*embarrassed look*) Of course we get the same error as comment #0... I posted 
comment #0. ;/

will converse with Callek to see the best approach..  is there a mock
package update for wget/libgnutls?
(In reply to Chris Cooper [:coop] from comment #7)
> All the train branches except for release were affected, so I landed a
> blocklist update for all of them:
> 
> https://hg.mozilla.org/mozilla-central/rev/989c1c75889c
> https://hg.mozilla.org/releases/mozilla-aurora/rev/d3b5bd6415de
> https://hg.mozilla.org/releases/mozilla-beta/rev/06300676d4cd
> 
> I'm kicking off our periodic updates builder on each of those branches now
> to see if I can generate an actionable log.
> 
> However, I don't think the above is even related to comment #0. If Seamonkey
> is using the old script, I'm not sure what we can do.

The blank blocklist was not a sympted as affected by SeaMonkey.

However the underlying issue, the wget failure, is indeed affecting seamonkey despite the different script. I'm expecting the remediation identified here for Firefox/MoCo to be directly translateable to either an automatic-for-free fix for SeaMonkey, or an action item we'll take in a new bug.

The "protect against a blank blocklist" is not "enough" to solve this bug, but "continue to properly update the blocklist" is what we need.
Found bug 967452 where the new version was deployed. Going to see if I can roll a new package with an updated GnuTLS using those instructions.
(In reply to Chris Cooper [:coop] from comment #12)
> Found bug 967452 where the new version was deployed. Going to see if I can
> roll a new package with an updated GnuTLS using those instructions.

Our build environment is CentOS 6.2, so finding off-the-shelf packages for GnuTLS 3+ is unlikely.

A simpler change would be to change the compile options to specify "--with-ssl=openssl" instead. I've tested this quickly and know that the compiled version of wget works against blocklist.addons.mozilla.org

cc-ing Dustin as the person who deployed the current version of wget in bug 967452.
I feel like we deliberately switched from OpenSSL to GnuTLS in that bug.  I think that's what I meant in bug 967452 comment 1: the version of OpenSSL available doesn't recognize alt names.

Anyway, if OpenSSL works, I don't have (or can't remember) any reason not to rebuild wget to use that.
Attachment #8491753 - Flags: review?(bugspam.Callek) → review+
Comment on attachment 8491753 [details] [diff] [review]
Point to a new version of wget compiled with openssl

You should update the %changelog at the bottom, too, indicating why we changed.  You probably also want to bump the repoflag.
Comment on attachment 8491753 [details] [diff] [review]
Point to a new version of wget compiled with openssl

Review of attachment 8491753 [details] [diff] [review]:
-----------------------------------------------------------------

Landed with Dustin suggestions:

https://hg.mozilla.org/build/puppet/rev/64c8641aa23f
Attachment #8491753 - Flags: checked-in+
Comment on attachment 8491753 [details] [diff] [review]
Point to a new version of wget compiled with openssl

Review of attachment 8491753 [details] [diff] [review]:
-----------------------------------------------------------------

Backed out to regenerate repo indices, and then re-landed:

http://hg.mozilla.org/build/puppet/rev/b3537935f103
The wget upgrade also resolved this output on each request:
 ERROR: Failed to open cert /etc/ssl/certs/Makefile: (-34).
 ERROR: Failed to open cert /etc/ssl/certs/make-dummy-cert: (-34).
 ERROR: Failed to open cert /etc/ssl/certs/ca-bundle.trust.crt: (-34).
The blocklist is still empty in 31ESR:
https://hg.mozilla.org/releases/mozilla-esr31/log/tip/browser/app/blocklist.xml

31.1.0 was the last good, 31.1.1 was the first bad version.
Reopening this as ESR seems impacted. 31.2.0ESR users are reporting lack of block list.
Status: RESOLVED → REOPENED
Flags: needinfo?(coop)
Resolution: FIXED → ---
(In reply to Benjamin Kerensa [:bkerensa] from comment #22)
> Reopening this as ESR seems impacted. 31.2.0ESR users are reporting lack of
> block list.

Re-running against esr31 now.
Flags: needinfo?(coop)
(In reply to Chris Cooper [:coop] from comment #23) 
> Re-running against esr31 now.

Done:

http://hg.mozilla.org/releases/mozilla-esr31/rev/b2f23baf9863
Status: REOPENED → RESOLVED
Closed: 5 years ago5 years ago
Resolution: --- → FIXED
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.