880127 - fhr.data.mozilla.com SSL certificate domain name causes Android error

Status: VERIFIED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task, P1)

Description

[Nick Alexander :nalexander [he/him]]

There are a bunch of ways to work around this on the client, but SSL certificate checking seems to vary quite a bit between Android versions and devices.  It's not wise to tempt fate by pushing workarounds onto the client if this can be addressed easily on the server.

Android details below:

E GeckoHealth(21823)          javax.net.ssl.SSLException: hostname in certificate didn't match: <fhr.data.mozilla.com> != <data.mozilla.com> OR <data.mozilla.com>
E GeckoHealth(21823)          	at ch.boye.httpclientandroidlib.conn.ssl.AbstractVerifier.verify(AbstractVerifier.java:228)
E GeckoHealth(21823)          	at ch.boye.httpclientandroidlib.conn.ssl.BrowserCompatHostnameVerifier.verify(BrowserCompatHostnameVerifier.java:54)
E GeckoHealth(21823)          	at ch.boye.httpclientandroidlib.conn.ssl.AbstractVerifier.verify(AbstractVerifier.java:149)
E GeckoHealth(21823)          	at ch.boye.httpclientandroidlib.conn.ssl.AbstractVerifier.verify(AbstractVerifier.java:130)
E GeckoHealth(21823)          	at ch.boye.httpclientandroidlib.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:397)
E GeckoHealth(21823)          	at ch.boye.httpclientandroidlib.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:148)
E GeckoHealth(21823)          	at ch.boye.httpclientandroidlib.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:149)
E GeckoHealth(21823)          	at ch.boye.httpclientandroidlib.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:121)
E GeckoHealth(21823)          	at ch.boye.httpclientandroidlib.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:573)
E GeckoHealth(21823)          	at ch.boye.httpclientandroidlib.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:425)
E GeckoHealth(21823)          	at ch.boye.httpclientandroidlib.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:818)
E GeckoHealth(21823)          	at ch.boye.httpclientandroidlib.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:752)
E GeckoHealth(21823)          	at org.mozilla.gecko.sync.net.BaseResource.execute(BaseResource.java:229)
E GeckoHealth(21823)          	at org.mozilla.gecko.sync.net.BaseResource.retryRequest(BaseResource.java:268)
E GeckoHealth(21823)          	at org.mozilla.gecko.sync.net.BaseResource.execute(BaseResource.java:239)
E GeckoHealth(21823)          	at org.mozilla.gecko.sync.net.BaseResource.go(BaseResource.java:296)
E GeckoHealth(21823)          	at org.mozilla.gecko.sync.net.BaseResource.post(BaseResource.java:326)
E GeckoHealth(21823)          	at org.mozilla.gecko.background.bagheera.BagheeraClient.uploadJSONDocument(BagheeraClient.java:130)
E GeckoHealth(21823)          	at org.mozilla.gecko.background.healthreport.upload.HealthReportUploadService.doUpload(HealthReportUploadService.java:220)
E GeckoHealth(21823)          	at org.mozilla.gecko.background.healthreport.upload.HealthReportUploadService.onHandleIntent(HealthReportUploadService.java:250)
E GeckoHealth(21823)          	at android.app.IntentService$ServiceHandler.handleMessage(IntentService.java:65)
E GeckoHealth(21823)          	at android.os.Handler.dispatchMessage(Handler.java:99)
E GeckoHealth(21823)          	at android.os.Looper.loop(Looper.java:137)
E GeckoHealth(21823)          	at android.os.HandlerThread.run(HandlerThread.java:60)

Comment 1

[Richard Newman [:rnewman]]

(Data Collection ain't the right place, so let's put it here.)

Daniel, Laura: who's the right person to help Nick debug this?

When I fetch the cert, it looks sane:

CN = fhr.data.mozilla.com
O = Mozilla Corporation
L = Mountain View
ST = California
C = US
Object Identifier (2 5 4 5) = yot6usuksLbJLlulKvlKPDMhigm6XIeI


Nick, what URI are you making requests to?

Comment 2

[Nick Alexander :nalexander [he/him]]

(In reply to Richard Newman [:rnewman] from comment #1)
> (Data Collection ain't the right place, so let's put it here.)
> 
> Daniel, Laura: who's the right person to help Nick debug this?

I was hoping Data Collection would hit the metrics server ops people :)
 
> Nick, what URI are you making requests to?

Something like:

D GeckoHealth(21823)          fennec_ncalexan :: BaseResource :: HTTP POST https://fhr.data.mozilla.com/1.0/submit/metrics/553cd947-f5cb-4255-ae62-ec90dc49d838

I'll try this on a few devices today, see if it looks Android or device specific.  As I say, there are client side work-arounds.

Comment 3

[Daniel Einspanjer [:dre] [:deinspanjer]]

The SSL cert is actually on the Zeus load balancers and would be handled by someone in the normal IT team.  Going to try to toss it over there to see if someone can pick it up to help.

Note that this is one of the things that I was afraid of when mconnor suggested using a CNAME to create an alias for this.  I don't know if we can fix it on the server side without removing the CNAME, but that is a question that IT should answer.

Comment 4

[Chris Turra [:cturra]]

the setup for fhr.data.mozilla.com is covered in bug 867718. zeus uses Server Name Indication (SNI) to assosiate this hostname with the data.mozilla.org virtual server. more on tls 1.0 sni:

  http://en.wikipedia.org/wiki/Server_Name_Indication


*you can also test this with openssl using the following command:

  $ openssl s_client -servername fhr.data.mozilla.com -connect fhr.data.mozilla.com:443

Comment 5

[Richard Newman [:rnewman]]

We have an existing bug on file about SNI on Android Sync. It doesn't work, and our advice to users running their own Sync servers is to not use it. 

Drat.

Comment 6

[Chris Turra [:cturra]]

^ that was my gut feeling about this - i will spin up a new traffic ip group/virtual server setup on our load balancers dedicated to fhr.data to solve this for you guys. stay tuned.

Comment 7

[Richard Newman [:rnewman]]

Thanks, Chris.

For reference: Bug 765064.

Comment 8

[Chris Turra [:cturra]]

fhr.data.mozilla.com now has a dedicated load balancer setup as promised :) it could take a little longer for DNS propagation to complete, but should all be sorted in the next couple hours.

 $ dig +short @ns3.mozilla.com fhr.data.mozilla.com
 63.245.215.95

 $ host 63.245.215.95
 95.215.245.63.in-addr.arpa domain name pointer fhr-data-zlb.vips.scl3.mozilla.com.

Comment 9

[Nick Alexander :nalexander [he/him]]

D GeckoHealth(442)            fennec_ncalexan :: BaseResource :: HTTP POST https://fhr.data.mozilla.com/1.0/submit/metrics/2fbf5d50-2a49-43ea-8c2f-7c88485cdf30
D GeckoHealth(442)            fennec_ncalexan :: BaseResource :: Response: HTTP/1.1 201 Created
I GeckoHealth(442)            fennec_ncalexan :: HealthReportUploadPolicy :: Uploaded new health report and obsoleted old document.

Thanks!