1354594 - New Relic Synthetics reports SSL negotiation time for stage/prod doubled on 2017-04-01

Status: RESOLVED INCOMPLETE

(Tree Management :: Treeherder: Infrastructure, defect, P2)

Description

[Ed Morley [:emorley]]

I happened to spot this whilst looking at something else:
https://synthetics.newrelic.com/accounts/677903/monitors/421266b8-056c-41e2-94f0-ced3f8f75dae/results?tw[start]=1490993463&tw[end]=1491018981

The "SSL negotiation" part of the response time increased from ~155ms to ~300ms, starting 2017-04-01 00:00 UTC.

This occurred across all Treeherder Synthetics checks (ie both stage and prod, and UI static asset vs API endpoint checks) at the same time.

There were no deploys during this time, and anyway SSL termination is not something handled by us, but by the Heroku routing layer.

Looks like a regression on their side. Not exactly critical, but worth reporting to them anyway.

Comment 1

[Ed Morley [:emorley]]

Filed:
https://help.heroku.com/tickets/467051

Comment 2

[Ed Morley [:emorley]]

Their response (in April):

"""
I did some checks on requests to https://treeherder.mozilla.org and I'm not seeing SSL negotiation times of 375-400ms. This is the breakdown I'm seeing from my workstation based in Dallas, TX:

        DNS Lookup: 5ms
    TCP Connection: 49ms
     SSL Handshake: 225ms
Time to First Byte: 58ms
       Server Time: 0ms
             Total: 337ms

and this is what I'm seeing when making the request from inside of a dyno:

        DNS Lookup: 61ms
    TCP Connection: 0ms
     SSL Handshake: 37ms
Time to First Byte: 7ms
       Server Time: 0ms
             Total: 105ms

Is there a particular URL that New Relic is monitoring for you? The 225ms in my first test above seems to be about what you were seeing before. I did some precursory looking to see how New Relic is performing the check, but didn't find anything. It's possible something may have changed on their end. If you have more details on how the check is being performed or what URL it's hitting exactly, I can try doing some more investigating. I'm definitely not seeing that much time spent in SSL negotiation though.
"""

I didn't manage to find any other cause, so I think this must have just been New Relic changing something. I didn't/don't have the cycles to file tickets against New Relic for this, so I think we should just ignore and move on.