1286824 - balrogVPNProxy is busted after balrog -> cloudops migration

Status: RESOLVED FIXED

(Taskcluster :: Workers, defect)

Description

[Treeherder Bug Filer]

Filed by: tomcat [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=3016793&repo=mozilla-aurora

https://queue.taskcluster.net/v1/task/B_8sQjvqTKO4LkleIU2wbg/runs/0/artifacts/public%2Flogs%2Flive_backing.log

Comment 1

[Carsten Book [:Tomcat]]

garndt, pete: can someone take a look at this, its very frequent in the update tests :)

Comment 2

[Pete Moore [:pmoore][:pete]]

I'm going to see if I can catch a smoking gun...

Comment 3

[Pete Moore [:pmoore][:pete]]

15:03 <bhearsum> pmoore: i think https://bugzilla.mozilla.org/show_bug.cgi?id=1286824 is caused by the balrog -> cloudops migration
15:03 <bhearsum> aus4-admin.mozilla.org changed IPs

Comment 4

[[e-mail] (:bhearsum)]

Updating summary now that we've diagnosed this a bit. It seems there's a hack in docker-worker that hardcodes the aus4-admin.mozilla.org IP address, so when it changed yesterday the proxy stopped working. I've updated that in https://github.com/taskcluster/docker-worker/pull/234, and Taskcluster folks are currently redeploying docker-worker. Hopefully it will work after that's done.

Comment 5

[Pete Moore [:pmoore][:pete]]

Code is merged in docker-worker - working on making a new AMI now. Spoke to garndt (woke him up) to get help with this.

Comment 6

[[e-mail] (:bhearsum)]

Even after the AMI was rebuilt we were having issues with the vpn proxy connectinng to the new production interface. After a ton of debugging (thanks to garndt and mostlygeek for that), we finally realized that the taskcluster-balrog LDAP account was missing an ACL. We looped jabba in, and he got it added to the vpn_balrog group, which got things fixed up.

All of the funsize balrog jobs that have started since the acl was added have succeeded. Capacity is still low (we stopped new instances from starting after the issue was discovered), but that will come back up over time.

Comment 7

[Intermittent Failures Robot]

2530 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-aurora: 2159
* mozilla-central: 371

Platform breakdown:
* linux64: 757
* linux32: 702
* osx-10-10: 566
* windowsxp: 325
* windows8-64: 180

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1286824&startday=2016-07-14&endday=2016-07-14&tree=all

Comment 8

[Intermittent Failures Robot]

2596 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-aurora: 2126
* mozilla-central: 470

Platform breakdown:
* linux64: 777
* linux32: 737
* osx-10-10: 548
* windowsxp: 363
* windows8-64: 171

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1286824&startday=2016-07-11&endday=2016-07-17&tree=all