1417575 - Add balrog in-tree tasks for Firefox

Status: RESOLVED FIXED

(Release Engineering :: Release Automation, enhancement)

Comment 1

[Mihai Tabara [:mtabara]⌚️GMT]

First approach: https://hg.mozilla.org/projects/maple/rev/a377bd6ddded53dc7b12a956b2b6f99039081412

Comment 2

[Mihai Tabara [:mtabara]⌚️GMT]

Fix-up: https://hg.mozilla.org/projects/maple/json-rev/9ac7858294c2b3a9e54b970ccd7bd3b6f8a1f84e

Comment 3

[Mihai Tabara [:mtabara]⌚️GMT]

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #2)
> Fix-up:
> https://hg.mozilla.org/projects/maple/json-rev/
> 9ac7858294c2b3a9e54b970ccd7bd3b6f8a1f84e

Backed-it out https://hg.mozilla.org/projects/maple/rev/34df03a12c95.
Scope issues[1], too many "balrog-dev" scope used everywhere. I think it's easier to just tweak the dep relpro client. 

[1]: https://treeherder.mozilla.org/logviewer.html#?job_id=147717705&repo=maple&lineNumber=6465

Comment 4

[Mihai Tabara [:mtabara]⌚️GMT]

Okay, so there's a problem with the scopes.My best guess:

* while we developed date few months back for nightly, we used "balrogworker-dev" queue scopes, hence the ones in the client[1]
* the production ones are "balrogworker-v1"[2]
* when we started developing on maple, we amended the scope to "balrog-dev"[3], most likely propagated from the "beetmover-dev" node[4]

I tried amending the scope in tree to reflect the proper value but it seems it's too deepen down in the scope tree[5] so I think for now, our best bet is just to amend the TC client[1] used by dep balrogworker to encompass 

queue:claim-task:scriptworker-prov-v1/balrog-dev*
queue:claim-work:scriptworker-prov-v1/balrog-dev*

as well.

[1]: https://tools.taskcluster.net/auth/clients/project%2Freleng%2Fscriptworker%2Fbalrogworker-dev
[2]: https://hg.mozilla.org/mozilla-central/file/tip/taskcluster/taskgraph/transforms/balrog.py#l100
[3]: https://hg.mozilla.org/projects/maple/file/tip/taskcluster/taskgraph/transforms/balrog.py#l100
[4]: https://hg.mozilla.org/build/puppet/file/default/manifests/moco-nodes.pp#l1024
[5]: https://tools.taskcluster.net/auth/scopes/queue%3Acreate-task%3Alow%3Ascriptworker-prov-v1%2Fbalrog-dev
[6]:

Comment 5

[Mihai Tabara [:mtabara]⌚️GMT]

Okay, finally the balrog dep worker is claiming jobs and runs them. Next error is package distribution so I'll have a look at that now.

Comment 6

[Mihai Tabara [:mtabara]⌚️GMT]

Currently failing with:

(py27venv)[[e-mail] scriptworker]$ balrogscript help
Traceback (most recent call last):
  File "/builds/scriptworker/py27venv/bin/balrogscript", line 5, in <module>
    from pkg_resources import load_entry_point
  File "/builds/scriptworker/py27venv/lib/python2.7/site-packages/distribute-0.6.24-py2.7.egg/pkg_resources.py", line 2707, in <module>
    working_set.require(__requires__)
  File "/builds/scriptworker/py27venv/lib/python2.7/site-packages/distribute-0.6.24-py2.7.egg/pkg_resources.py", line 686, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/builds/scriptworker/py27venv/lib/python2.7/site-packages/distribute-0.6.24-py2.7.egg/pkg_resources.py", line 584, in resolve
    raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: asn1crypto>=0.21.0
(py27venv)[[e-mail] scriptworker]$ pip freeze
appdirs==1.4.3
arrow==0.10.0
asn1crypto==0.22.0
balrogclient==0.0.4
balrogscript==0.2.0-dev1
cffi==1.10.0
click==6.7
construct==2.8.11
cryptography==1.8.1
enum34==1.1.6
functools32==3.2.3-2
idna==2.5
ipaddress==1.0.18
jsonschema==2.6.0
mar==2.0
packaging==16.8
pyasn1==0.2.3
pycparser==2.17
pyparsing==2.2.0
python-dateutil==2.6.0
requests==2.13.0
six==1.10.0
wsgiref==0.1.2

After nuking the existing venvs and recreating from scratch, other libs fail as well. 
cryptography fails for missing pycparser, although few packages later on it installs successfully "pycparser==2.17]/returns: executed successfully". Afterwards, hangs again in jsonschema for missing vcversioner

Pip might be the culprit, we currently use 1.5.5.
This is raising serious flags as we might hit the same issues if we try to recreate the environments on production balrogworkers as well.

Comment 7

[Aki Sasaki (not active)]

https://github.com/mozilla/build-puppet/compare/master...escapewindow:pip?expand=1 gets us on virtualenv 15.1.0 and pip 9.1.0. However, this forces ssl verification, and the puppet ssl certs are self-signed, so pip dies.

I'm going to see if there's a version of virtualenv + pip that is more modern, but doesn't require ssl verification?

Comment 8

[Aki Sasaki (not active)]

https://github.com/mozilla/build-puppet/commit/9c1a7b73ed455a1ce0ee1bfb34730500334bb242 fixes it!
The key was a) virtualenv 15.0.2, and b) using http again, and c) --trusted-host options for each puppet server. I'd rather have valid ssl certs but this solves the issue.

Now I'm going to look at forking modules/python/ to modules/python27/ so we can upgrade individual modules rather than upgrading the world.

Comment 9

[Mihai Tabara [:mtabara]⌚️GMT]

(In reply to Aki Sasaki [:aki] from comment #8)
> https://github.com/mozilla/build-puppet/commit/
> 9c1a7b73ed455a1ce0ee1bfb34730500334bb242 fixes it!
> The key was a) virtualenv 15.0.2, and b) using http again, and c)
> --trusted-host options for each puppet server. I'd rather have valid ssl
> certs but this solves the issue.
> 
> Now I'm going to look at forking modules/python/ to modules/python27/ so we
> can upgrade individual modules rather than upgrading the world.

Thanks a lot for your help with this Aki!

Comment 10

[Mihai Tabara [:mtabara]⌚️GMT]

After bug 1421124 was solved, I redeployed environment with all changes up-to-date (puppet) and triggered a new staging release up-to-date maple-based.

Initially there were a couple of things concerning me:
a) jobs were running green but didn't submit any data to Balrog. Or at least that's how it looked
b) jobs were returning 0 although I was seeing plenty of errors like this in the logs:

2017-11-30 16:51:34,412 - ERROR - Caught HTTPError: {
  "detail": "[\"Couldn't find locale identified by: Firefox-58.0b10-build1, Darwin_x86_64-gcc3-u-i386-x86_64, az\"]",
  "status": 404,
  "title": "Not Found",
  "type": "about:blank"
}

c) releases were suffixed by "dummy"

===
After a bunch of in-tree fixings pushed by :aki and :bhearsum, everything seems to be fine as follows:

for a) looks like they were actually submitting data. I retriggered two tasks of the old one (a locale + a en-US) and worked smoothly pushing a staging release into Balrog

for b) this is expected I guess. nightly balrog jobs have it too

for c) pushed a fix into puppet and into my puppet PR deployed and pinned to my environment to fix this. 

The things I strongly suspect made things worked is https://hg.mozilla.org/projects/maple/rev/bdb3a072306c in which the shipping_phase + shipping_product was added per Balrog jobs. Aki disagrees and mentiones there were some scopes changes that unblocked this. Either way, last staging release[1] looks good with balrog staging release nicely in Balrog.

[1]: https://tools.taskcluster.net/groups/QfDsdXrkQHe6N7PceVoLCg

Comment 11

[Mihai Tabara [:mtabara]⌚️GMT]

Still to investigate the shipping-{phase,product} in order to correctly set the downstream dependencies (update verification and  such I suppose).

Comment 12

[Mihai Tabara [:mtabara]⌚️GMT]

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #11)
> Still to investigate the shipping-{phase,product} in order to correctly set
> the downstream dependencies (update verification and  such I suppose).

Solved in the meantime by Aki. All good here, we can close this.