1388367 - we need to revisit the balrog staging instance we use for staging releases

Status: RESOLVED DUPLICATE of bug 1409170

(Release Engineering :: Release Automation, enhancement, P2)

Description

[Mihai Tabara [:mtabara]⌚️GMT]

We tried last week a lot to get as far as we could with staging releases to prevent us from delaying beta. Yet, there were still a bunch of things that couldn't be caught due to miscommunication to balrog staging instance:

* updates builder fails after tagging, as it can't establish connection to balrog staging
* we don't know which credentials to use
* with updates builder not working, updates verification didn't run, hence we didn't catch some macosx/win32/win64 errors
* etc

We need to revisit this conversation and reache a point where we ship full staging releases from staging buckets, on all ends. This would help us anytime, catch unwanted issues.

Comment 1

[Aki Sasaki (not active)]

I *think* the reason we're using a dev instance is because of netflows, as in

- staging balrog is not accessible on the internet, only through our vpc
- releasetasks balrog runs on docker-worker, which is on the internet, not our vpc

We previously used a proxy which was error prone.

The real solution is probably to get all of releasetasks in-tree, so we can use balrog scriptworkers, which live in the vpc.
A shorter term solution may involve improving the dev docker instance to be able to pre-populate releases and partials? Or revisit the proxy, which isn't ideal. Or spin up a dev balrogworker pool that can only talk to a dev balrog instance.

Comment 2

[Mihai Tabara [:mtabara]⌚️GMT]

(In reply to Aki Sasaki [:aki] from comment #1)
> I *think* the reason we're using a dev instance is because of netflows, as in
> 
> - staging balrog is not accessible on the internet, only through our vpc
> - releasetasks balrog runs on docker-worker, which is on the internet, not
> our vpc
> 
> We previously used a proxy which was error prone.
> 
> The real solution is probably to get all of releasetasks in-tree, so we can
> use balrog scriptworkers, which live in the vpc.
> A shorter term solution may involve improving the dev docker instance to be
> able to pre-populate releases and partials? Or revisit the proxy, which
> isn't ideal. Or spin up a dev balrogworker pool that can only talk to a dev
> balrog instance.

Thanks a lot :aki for this, I missed the netflows logic. 
This is a good item to talk at the postmortem so I'll add it there and remove it as a hard-dependency for bug 1388357 as we already shipped that last night.

Will continue this conversation as I suppose we'll need staging releases sooner than we can finish up the in-tree releasetasks, which is quite a standalone project.