fhr.data.mozilla.com SSL certificate domain name causes Android error



Infrastructure & Operations
WebOps: Other
5 years ago
4 years ago


(Reporter: nalexander, Assigned: cturra)





5 years ago
There are a bunch of ways to work around this on the client, but SSL certificate checking seems to vary quite a bit between Android versions and devices.  It's not wise to tempt fate by pushing workarounds onto the client if this can be addressed easily on the server.

Android details below:

E GeckoHealth(21823)          javax.net.ssl.SSLException: hostname in certificate didn't match: <fhr.data.mozilla.com> != <data.mozilla.com> OR <data.mozilla.com>
E GeckoHealth(21823)          	at ch.boye.httpclientandroidlib.conn.ssl.AbstractVerifier.verify(AbstractVerifier.java:228)
E GeckoHealth(21823)          	at ch.boye.httpclientandroidlib.conn.ssl.BrowserCompatHostnameVerifier.verify(BrowserCompatHostnameVerifier.java:54)
E GeckoHealth(21823)          	at ch.boye.httpclientandroidlib.conn.ssl.AbstractVerifier.verify(AbstractVerifier.java:149)
E GeckoHealth(21823)          	at ch.boye.httpclientandroidlib.conn.ssl.AbstractVerifier.verify(AbstractVerifier.java:130)
E GeckoHealth(21823)          	at ch.boye.httpclientandroidlib.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:397)
E GeckoHealth(21823)          	at ch.boye.httpclientandroidlib.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:148)
E GeckoHealth(21823)          	at ch.boye.httpclientandroidlib.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:149)
E GeckoHealth(21823)          	at ch.boye.httpclientandroidlib.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:121)
E GeckoHealth(21823)          	at ch.boye.httpclientandroidlib.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:573)
E GeckoHealth(21823)          	at ch.boye.httpclientandroidlib.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:425)
E GeckoHealth(21823)          	at ch.boye.httpclientandroidlib.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:818)
E GeckoHealth(21823)          	at ch.boye.httpclientandroidlib.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:752)
E GeckoHealth(21823)          	at org.mozilla.gecko.sync.net.BaseResource.execute(BaseResource.java:229)
E GeckoHealth(21823)          	at org.mozilla.gecko.sync.net.BaseResource.retryRequest(BaseResource.java:268)
E GeckoHealth(21823)          	at org.mozilla.gecko.sync.net.BaseResource.execute(BaseResource.java:239)
E GeckoHealth(21823)          	at org.mozilla.gecko.sync.net.BaseResource.go(BaseResource.java:296)
E GeckoHealth(21823)          	at org.mozilla.gecko.sync.net.BaseResource.post(BaseResource.java:326)
E GeckoHealth(21823)          	at org.mozilla.gecko.background.bagheera.BagheeraClient.uploadJSONDocument(BagheeraClient.java:130)
E GeckoHealth(21823)          	at org.mozilla.gecko.background.healthreport.upload.HealthReportUploadService.doUpload(HealthReportUploadService.java:220)
E GeckoHealth(21823)          	at org.mozilla.gecko.background.healthreport.upload.HealthReportUploadService.onHandleIntent(HealthReportUploadService.java:250)
E GeckoHealth(21823)          	at android.app.IntentService$ServiceHandler.handleMessage(IntentService.java:65)
E GeckoHealth(21823)          	at android.os.Handler.dispatchMessage(Handler.java:99)
E GeckoHealth(21823)          	at android.os.Looper.loop(Looper.java:137)
E GeckoHealth(21823)          	at android.os.HandlerThread.run(HandlerThread.java:60)


5 years ago
Blocks: 828654
(Data Collection ain't the right place, so let's put it here.)

Daniel, Laura: who's the right person to help Nick debug this?

When I fetch the cert, it looks sane:

CN = fhr.data.mozilla.com
O = Mozilla Corporation
L = Mountain View
ST = California
C = US
Object Identifier (2 5 4 5) = yot6usuksLbJLlulKvlKPDMhigm6XIeI

Nick, what URI are you making requests to?
Component: Data Collection → Client: Android
Priority: -- → P1

Comment 2

5 years ago
(In reply to Richard Newman [:rnewman] from comment #1)
> (Data Collection ain't the right place, so let's put it here.)
> Daniel, Laura: who's the right person to help Nick debug this?

I was hoping Data Collection would hit the metrics server ops people :)
> Nick, what URI are you making requests to?

Something like:

D GeckoHealth(21823)          fennec_ncalexan :: BaseResource :: HTTP POST https://fhr.data.mozilla.com/1.0/submit/metrics/553cd947-f5cb-4255-ae62-ec90dc49d838

I'll try this on a few devices today, see if it looks Android or device specific.  As I say, there are client side work-arounds.
The SSL cert is actually on the Zeus load balancers and would be handled by someone in the normal IT team.  Going to try to toss it over there to see if someone can pick it up to help.

Note that this is one of the things that I was afraid of when mconnor suggested using a CNAME to create an alias for this.  I don't know if we can fix it on the server side without removing the CNAME, but that is a question that IT should answer.
Assignee: nobody → server-ops-webops
Component: Client: Android → Server Operations: Web Operations
Product: Firefox Health Report → mozilla.org
QA Contact: nmaul
Version: unspecified → other

Comment 4

5 years ago
the setup for fhr.data.mozilla.com is covered in bug 867718. zeus uses Server Name Indication (SNI) to assosiate this hostname with the data.mozilla.org virtual server. more on tls 1.0 sni:


*you can also test this with openssl using the following command:

  $ openssl s_client -servername fhr.data.mozilla.com -connect fhr.data.mozilla.com:443
We have an existing bug on file about SNI on Android Sync. It doesn't work, and our advice to users running their own Sync servers is to not use it. 


Comment 6

5 years ago
^ that was my gut feeling about this - i will spin up a new traffic ip group/virtual server setup on our load balancers dedicated to fhr.data to solve this for you guys. stay tuned.
Assignee: server-ops-webops → cturra
Thanks, Chris.

For reference: Bug 765064.
See Also: → bug 765064

Comment 8

5 years ago
fhr.data.mozilla.com now has a dedicated load balancer setup as promised :) it could take a little longer for DNS propagation to complete, but should all be sorted in the next couple hours.

 $ dig +short @ns3.mozilla.com fhr.data.mozilla.com

 $ host domain name pointer fhr-data-zlb.vips.scl3.mozilla.com.
Last Resolved: 5 years ago
Resolution: --- → FIXED

Comment 9

5 years ago
D GeckoHealth(442)            fennec_ncalexan :: BaseResource :: HTTP POST https://fhr.data.mozilla.com/1.0/submit/metrics/2fbf5d50-2a49-43ea-8c2f-7c88485cdf30
D GeckoHealth(442)            fennec_ncalexan :: BaseResource :: Response: HTTP/1.1 201 Created
I GeckoHealth(442)            fennec_ncalexan :: HealthReportUploadPolicy :: Uploaded new health report and obsoleted old document.

Component: Server Operations: Web Operations → WebOps: Other
Product: mozilla.org → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.