Scheduler: Fix tests on travis after pulse migration

RESOLVED FIXED

Status

Taskcluster
Scheduler
RESOLVED FIXED
3 years ago
2 years ago

People

(Reporter: jonasfj, Assigned: pmoore)

Tracking

Details

Attachments

(1 attachment)

(Reporter)

Description

3 years ago
Also look at other components not configured correctly on travis after pulse migration...

Queue and Index, are two other likely candicates...
Maybe also Events...
(Assignee)

Updated

2 years ago
Component: TaskCluster → Scheduler
Product: Testing → Taskcluster
(Assignee)

Comment 1

2 years ago
I can reproduce travis errors locally. I'm able to fix by adding taskcluster credentials in my config.

I don't have a key to the decrpyt the config file used in travis (https://github.com/taskcluster/task-graph-scheduler/blob/master/task-graph-scheduler.conf.json.enc), my guess though is that it is missing the taskcluster creds.

The giveaway was:

https://github.com/taskcluster/task-graph-scheduler/blob/7fb20e08cccc515fba1f2ff00af35b07e89e51f7/test/helper.js#L62

I added the following to my local config:

{

  ....

  "taskcluster": {
    "credentials": {
      "accessToken": ".....",
      "clientId": "....."
    }
  }
}

I propose we:

a) Publish private key in our private repo so we can decrypt encrypted files used in travis (or gpg encrypt the decrypted form, with a README to say how to update files if needed)
b) Create some scopes just for running tests, and generate some (permanent) keys that have these scopes, and put the credentials inside the encrypted travis config.

Please note we have some contributors interested in running tests - I think we should try to find ways that tests can be run without needing to handover sensitive credentials. Maybe also a discussion for Berlin...
Flags: needinfo?(jopsen)
(Assignee)

Comment 2

2 years ago
s/handover/hand over/ :)
(Assignee)

Comment 3

2 years ago
What are the creds used for?

I see we deploy a mock auth server during testing - is the problem that we are using production auth rather than the mock server in the tests?
(Reporter)

Comment 4

2 years ago
credentials is used for creating tasks on production queue..
Flags: needinfo?(jopsen)
So, I think my question would be, is the current implementation support creating task without auth? i.e. for test mode or just a public service that everyone could utilize the scheduler?

Thanks.
If it does need auth to work, I will try to look into https://github.com/taskcluster/taskcluster-auth first and see how to make these 2 services work together.
(Assignee)

Comment 7

2 years ago
(In reply to Rudy Lu [:rudyl] (inactive after 2015/7/3) from comment #6)
> If it does need auth to work, I will try to look into
> https://github.com/taskcluster/taskcluster-auth first and see how to make
> these 2 services work together.

Hi Rudy,

I think the auth part is working ok and we have a mock auth server running for the tests. However, the tests submit jobs to the scheduler, and the scheduler then needs to submit them to "a taskcluster queue" and currently this is "the production taskcluster queue". So the TC credentials are needed for hitting the taskcluster queue APIs in production.

An option here is to mock the queue, in which case no credentials are needed. We can also add some encrypted credentials for travis-ci so that travis can run the tests, but that wouldn't help devs who want to run the tests locally or against their own github fork. For that we would need to mock the queue, or a dev would need to deploy the queue locally, and point their tests at that.

The task-graph-scheduler is going to be replaced by the big-graph scheduler at some point, therefore it is not really a focus of our attention at the moment, so probably we will not be mocking the queue in the task-graph-scheduler tests.

If you'd like to have a go at mocking it, that would be awesome. Otherwise, even though you can't run the tests, I can confirm the tests are passing with suitable credentials, if that is some consolation.

In any case, I'll encrypt some TC creds for travis, so you can see them passing in travis, even if you can't run locally.

Thanks!
Pete
Thanks for the clear explanation, it is very helpful!
(Assignee)

Comment 9

2 years ago
Created attachment 8658665 [details] [review]
Github Pull Request for taskcluster/task-graph-scheduler

This fixes the tests in travis by adding the required TC creds to the test config.

Since everything is encrpyted, there is nothing really to review, except to take note that the previously failing travis tests are now passing!

Please hit "merge" so we can close this bug. =)
Assignee: nobody → pmoore
Status: NEW → ASSIGNED
Attachment #8658665 - Flags: review?(jopsen)
(Assignee)

Comment 10

2 years ago
> The task-graph-scheduler is going to be replaced by the big-graph scheduler at some point

See bug 1178516
See Also: → bug 1178516
(Reporter)

Comment 11

2 years ago
Comment on attachment 8658665 [details] [review]
Github Pull Request for taskcluster/task-graph-scheduler

Thanks!
Attachment #8658665 - Flags: review?(jopsen) → review+
(Reporter)

Updated

2 years ago
Status: ASSIGNED → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.