1437987 - balrog agent is broken in stage and dev

Status: RESOLVED FIXED

(Cloud Services :: Operations: Miscellaneous, task)

Description

[[e-mail] (:bhearsum)]

Not super urgent, but it reduces confidence when we land certain types of patches.

Comment 1

[Jeremy Orem [:oremj]]

Looks like it is running in dev:

# docker ps
CONTAINER ID        IMAGE                                                                 COMMAND                  CREATED             STATUS              PORTS               NAMES
bba55ad4cecc        mozilla/balrogagent:master-de243f74670c8e6bf8058cb72f482a3f856a294f   "/bin/bash /app/run.s"   6 hours ago         Up 6 hours                              balrog-agent
c7f0b105d7e9        mozilla/balrog:master-de243f74670c8e6bf8058cb72f482a3f856a294f        "/bin/bash /app/scrip"   6 hours ago         Up 6 hours                              balrog-admin


It also appears to be running in stage:

# docker ps
CONTAINER ID        IMAGE                                           COMMAND                  CREATED             STATUS              PORTS               NAMES
63572d5c85e3        hub.prod.mozaws.net/mozilla/balrogagent:v2.47   "/bin/bash /app/run.s"   3 hours ago         Up 3 hours                              balrog-agent
39f4af470e13        hub.prod.mozaws.net/mozilla/balrog:v2.47        "/bin/bash /app/scrip"   3 hours ago         Up 3 hours                              balrog-admin


Is there anything else you'd like me to do to help debug? I don't see any log output, for what it is worth.

Comment 2

[[e-mail] (:bhearsum)]

Hm, that's very strange. It looks like the agent doesn't normally generate output (except on error) with how we have logging configured, so that part is normal. It should be making a few requests to admin every ~30s though - can we have a look at the admin access log for that? I would expect to see things like:
agent 1518617021.631 "127.0.0.1" "GET /api/scheduled_changes/rules HTTP/1.1" 200 260 45 balrogagent 0.068 0.068 "Python/3.5 aiohttp/2.2.3" "-"

Comment 3

[[e-mail] (:bhearsum)]

This turned out to be in-app issues (I forgot to grant the balrogagent account the right ACLs after the latest database reset). Sorry about the noise.