1214613 - [mozilla-taskcluster] Submit to Treeherder's API using Hawk credentials

Status: RESOLVED FIXED

(Taskcluster :: Services, defect)

Description

[Greg Arndt [:garndt]]

With treeherder moving over to using hawk instead of oauth, mozilla-taskcluster needs to be changed to use the new method and credentials.

Client ids are mozilla-taskcluster-prod for production and mozilla-taskcluster-staging for staging.

Comment 1

[Ed Morley [:emorley]]

Thank you for creating those/filing this.

I was thinking the usernames on stage/prod would typically be the same (we've used the same for treeherder's etl) - I'm open for either, but wondered if there was a use case I'd missed? :-)

Comment 2

[Greg Arndt [:garndt]]

My understanding was that we could track the requests being made on a per client-id basis.  I would think we would want to separate out the requests made between the prod and staging environments, but if that's not possible or not a good reason to have separate creds, I'm ok with just creating one for mozilla-taskcluster and use it for both.

Comment 3

[Greg Arndt [:garndt]]

Also, this is for submitting things to the API. how would this tie into the work camd was doing to ingest via pulse?

Comment 4

[Ed Morley [:emorley]]

(In reply to Greg Arndt [:garndt] from comment #2)
> My understanding was that we could track the requests being made on a per
> client-id basis.  I would think we would want to separate out the requests
> made between the prod and staging environments

Stage and prod are completely different environments, so requests made to one would never appear in the logs for the other, so we can already differentiate between the two, without using difference usernames for each.

(In reply to Greg Arndt [:garndt] from comment #3)
> Also, this is for submitting things to the API. how would this tie into the
> work camd was doing to ingest via pulse?

I'm not sure really, but the pulse work is a way out, whereas we need people to switch away from oauth sooner.

Comment 5

[Greg Arndt [:garndt]]

Thanks Ed, I have created 'mozilla-taskcluster' on both staging and prod.

Comment 6

[Ed Morley [:emorley]]

Thanks - have approved both.

For people using the Python client, we've already added handling, but for the nodejs client support needs adding:
https://github.com/hueniverse/hawk#usage-example

In the client example there, the credentials would be:
var credentials = {
    id: 'mozilla-taskcluster',
    key: '...',
    algorithm: 'sha256'
}

Where the key is the secret shown to you when the keys were requested, or can be looked up at:
Stage: https://treeherder.allizom.org/credentials/
Prod: https://treeherder.mozilla.org/credentials/
...and then select the credential to view the details including secret.

Comment 7

[Ed Morley [:emorley]]

It's worth also noting that the system clocks of the nodes making the submissions needs to be correct within 60s, otherwise authorisation will fail.

Comment 8

[Ed Morley [:emorley]]

Auth currently occurs in this file:
https://github.com/mozilla/treeherder-node/blob/master/project.js

It instead needs to use the hawk npm package instead (see https://github.com/hueniverse/hawk#usage-example and scroll down a bit to the client example).

Also now that the credentials are per-user rather than per-project, I imagine it makes sense to break auth out of project.js at some point (though that can always happen later).

Comment 9

[Greg Arndt [:garndt]]

merged https://github.com/taskcluster/mozilla-taskcluster/commit/b515744d17ecd5804769416d8e691b8f2b614295

Comment 10

[Ed Morley [:emorley]]

Awesome, thank you! :-)

Comment 11

[Ed Morley [:emorley]]

I'm seeing submissions using oauth on the /resultset/ endpoint on gaia and gaia-master, eg:

127.0.0.1 - - [10/Nov/2015:01:58:56 -0800] "POST /api/project/gaia/resultset/?oauth_body_hash=REDACTED&oauth_consumer_key=REDACTED&oauth_nonce=REDACTED&oauth_signature_method=HMAC-SHA1&oauth_timestamp=1447149535&oauth_token=&oauth_version=1.0&user=treeherder-node%200.4.2&oauth_signature=REDACTED HTTP/1.1" 200 37 "-" "-"

127.0.0.1 - - [10/Nov/2015:02:06:33 -0800] "POST /api/project/gaia-master/resultset/?oauth_body_hash=REDACTED&oauth_consumer_key=REDACTED&oauth_nonce=REDACTED&oauth_signature_method=HMAC-SHA1&oauth_timestamp=1447149993&oauth_token=&oauth_version=1.0&user=treeherder-node%200.4.2&oauth_signature=REDACTED HTTP/1.1" 200 37 "-" "-"

Do you know where this might be coming from? :-)

Comment 12

[Greg Arndt [:garndt]]

that is a different integration component than mozilla-taskcluster that's used for gaia integration.  That's probably being done by 'gaia-taskcluster' https://github.com/taskcluster/gaia-taskcluster

This bug [1] was created to switch over gaia-taskcluster.  No one really owns that component right now so I guess I'm going to be the winner of it :)


[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1221647

Comment 13

[Ed Morley [:emorley]]

Ah thank you - I had a rummage around the taskcluster github org, but there are so many repos it's hard to know where to start (or to search). (One advantage of unified repos at least :-))

I'll close this out in favour of bug 1221647.