1331098 - Use queue.claimWork in Scriptworker

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

This will allow the bridge to handle task priorities properly, which we need to support running tasks on hardware.

This will replace the existing queue.claimTask process.  It's already supported in the queue.

Don't deploy this until bug 1331094 is done, as Jonas will use that bug to monitor load on the queue as this new approach is deployed.

Comment 1

[Aki Sasaki (not active)]

This is also https://github.com/mozilla-releng/scriptworker/issues/56

Comment 2

[Greg Arndt [:garndt]]

Removing dependency on 1331094 as docker-worker has been entirely moved over which accounts for a majority of our load.  Tasks that have been created using a provisioner ID of scriptworker-prov-v1 accounts for about 1.7% of our tasks for the last 30 days so should be fine with moving over to claimWork.

Comment 3

[Aki Sasaki (not active)]

Attachment: claimWork

Comment 4

[Aki Sasaki (not active)]

This bug has a reviewed PR at https://github.com/mozilla-releng/scriptworker/pull/106 .
I'm currently holding off on merging until we get https://github.com/taskcluster/taskcluster-client.py/pull/67 merged first.

Comment 5

[Kim Moir [:kmoir] ET]

Aki:

Randi is asking the status of this bug. It looks like https://github.com/taskcluster/taskcluster-client.py/pull/67 has been merged. Does this mean https://github.com/mozilla-releng/scriptworker/pull/106  can be merged as well?

Comment 6

[Aki Sasaki (not active)]

We need a taskcluster-client.py release with https://github.com/taskcluster/taskcluster-client.py/pull/67 in it, and then we'll be fully unblocked.

Comment 7

[Greg Arndt [:garndt]]

John, can you create a new release of the client that picks up that change?

Comment 8

[John Ford [:jhford] CET/CEST Berlin Time]

Sure.  i'll do that today

Comment 9

[John Ford [:jhford] CET/CEST Berlin Time]

I'm having trouble accessing my desktop in Berlin to be able to generate a release.

Josh, are you aware of any trouble with the VPN setup that would be causing consistent connection drops?  I'm accessing my desktop (jhford.corp.ber1) from Teksavvy DSL in Toronto through our VPN.

Comment 10

[John Ford [:jhford] CET/CEST Berlin Time]

I've been able to get it to work long enough to generate the release.

Submitting dist/taskcluster-1.3.0.tar.gz to https://upload.pypi.org/legacy/
Server response (200): OK

Comment 11

[Aki Sasaki (not active)]

Thanks John!

I'm currently hitting integration test failures (locally; travis doesn't run integration tests) and flake8 failures [1]; I suspect these are due to new aiohttp and flake8 releases, but I'll have to narrow it down.  I'm hitting both types of test failures on both master and the claimWork branch.

[1] https://travis-ci.org/mozilla-releng/scriptworker/jobs/231238998

Comment 12

[Aki Sasaki (not active)]

Attachment: integration log

Comment 13

[Aki Sasaki (not active)]

Downgrading to taskcluster==1.2.0 fixes the integration tests, but we lose retries.  I'm going to see if there's something in the integration tests I can fix.

Comment 14

[Aki Sasaki (not active)]

I think the integration bustage has to do with this line:
https://github.com/taskcluster/taskcluster-client.py/commit/83c53dcea46619858f5bd195feafee7a9a081689#diff-e3b70cb7ee9cc15ea2c36a2de136df52R62

I added that line so we'd stop getting unclosed session errors.  Ideally this won't affect production, and I can change the tests to recreate the session(s) every time we need... probably by not passing in a session in the tests.  Otherwise we may need to remove it.

Comment 15

[Aki Sasaki (not active)]

Attachment: tc-client.py PR 71

taskcluster-client.py PR to fix the scriptworker integration tests.

Comment 16

[Aki Sasaki (not active)]

scriptworker 4.0.0 released.  I need to bump signing- beetmover- balrog- and pushapk- scriptworkers to 4.0.0, then all scriptworkers will be using claimWork.

Comment 17

[Aki Sasaki (not active)]

Attachment: bump_scriptworker.4.0.0.diff

Comment 18

[Mihai Tabara [:mtabara]⌚️GMT]

Comment on attachment 8867795 [details] [diff] [review]
bump_scriptworker.4.0.0.diff

Review of attachment 8867795 [details] [diff] [review]:
-----------------------------------------------------------------

All packages seem to be at their place in puppet.
Lgtm, dephash++!

Comment 19

[Aki Sasaki (not active)]

https://hg.mozilla.org/build/puppet/rev/e0130f389b941ac4f182defb97265fc5f0c207e7
bug 1331098 - scriptworker 4.0.0. r=mtabara

Comment 20

[Aki Sasaki (not active)]

https://hg.mozilla.org/build/puppet/rev/56d7fcebda3738e1a51651c61a3accce9216835c
bug 1331098 - add missing dep (pyparsing) to pushapk. r=bustage

Comment 21

[Aki Sasaki (not active)]

https://hg.mozilla.org/build/puppet/rev/7d91bd8604faf670b4e2c41be648398a7c101ced
bug 1331098 - fix beetmover bustage. r=bustage

Comment 22

[Aki Sasaki (not active)]

Attachment: PR 73

How does this PR look jhford?

Essentially, we were getting

May 15 11:42:31 signing-linux-1.srv.releng.use1.mozilla.com python: signing_scriptworker 2017-05-15T18:42:31  WARNING - <class 'taskcluster.exceptions.TaskclusterRestFailure'> Unknown Server Error

which isn't very informative.

This PR a) fixes the await json(), so we actually get the dict instead of a coroutine (this fixes the current issue), and b) adds some more info to the unknown server error message if we do hit it in the future.

After I patched a scriptworker with this patch locally, I got the missing scope error message logged, and I was able to fix the underlying problem.

Sorry for the churn; we're slowly but surely getting this function where it needs to be.

Comment 23

[Aki Sasaki (not active)]

I think we're good here... claimWork is rolled out.  I had to update the clientId scopes, and downgrade everything from chardet==3.0.2 back to 2.3.0 until a new chardet release is rolled out.

I'll resolve once we fix the above tc.py PR.

Comment 24

[John Ford [:jhford] CET/CEST Berlin Time]

Comment on attachment 8867862 [details] [review]
PR 73

Looks good!  I've landed this and released 1.3.2 with this patch.  Please open a quick PR to address the review comments from #73, but I didn't want to block this on the comment there.

Comment 25

[Aki Sasaki (not active)]

Thanks John!
https://github.com/taskcluster/taskcluster-client.py/pull/74 for max 1024 chars.

Comment 26

[Aki Sasaki (not active)]

Bumped to taskcluster 1.3.2 in bug 1365290 here: https://hg.mozilla.org/build/puppet/rev/7cfa62eda6d584c5fbda457300310e11b4e14f32

I'm planning to upgrade chardet to 3.0.3 and taskcluster to 1.3.3 tomorrow; just cleanup.

Comment 27

[Aki Sasaki (not active)]

Randi considers this a blocker, so I'm going to resolve and open a non-blocker for cleanup.