1399857 - Accessing Balrog from outside America takes more than a second

Status: RESOLVED WONTFIX

(Release Engineering Graveyard :: Applications: Balrog (backend), enhancement)

Description

[Rémy Hubscher (:natim)]

While working on PollBot, I realize two requests on Balrog where taking more than 2 seconds.

After investigation with Ben Hearsum, we realized that from East North America it was about 600ms from West North America about 400ms, from Europe 1 second and from Asia 1.5 seconds.

Here is the tool I used to gather this information:

ubuntu@ip-172-31-8-206:~$ cat > curl-format.txt
    time_namelookup:  %{time_namelookup}\n
       time_connect:  %{time_connect}\n
    time_appconnect:  %{time_appconnect}\n
   time_pretransfer:  %{time_pretransfer}\n
      time_redirect:  %{time_redirect}\n
 time_starttransfer:  %{time_starttransfer}\n
                    ----------\n
         time_total:  %{time_total}\n
ubuntu@ip-172-31-8-206:~$ curl -w "@curl-format.txt" -o /dev/null -s https://aus-api.mozilla.org/api/v1/rules/firefox-nightly
    time_namelookup:  0.253
       time_connect:  0.415
    time_appconnect:  1.000
   time_pretransfer:  1.000
      time_redirect:  0.000
 time_starttransfer:  1.173
                    ----------
         time_total:  1.173


I believe it is about latency but I wonder would it help adding a global CloudFront Cache in front of it?

Can this be a problem?

Comment 1

[Benson Wong [:mostlygeek]]

Looks like of the 1173ms, 1000ms of that was spent making the TCP connection and doing the TLS handshake. Balrog itself only took 173ms to receive, process and generate the response.  That's not too shabby considering ELB -> EC2 -> Nginx -> balrog.

Comment 2

[Daniel Thorn [:relud]]

I'm going to take this as something I can look into in coming weeks. Balrog has some other more pressing work though, so this might take a bit to investigate.

Comment 3

[Daniel Thorn [:relud]]

This isn't something we're going to try and fix for now. We may loop back around to it once we take balrog-multi-region next year.