Allow `./mach taskgraph .. --parameters P` to take a URL

RESOLVED FIXED in mozilla53

Status

Taskcluster
Task Configuration
RESOLVED FIXED
10 months ago
7 months ago

People

(Reporter: dustin, Assigned: HAMMAD AKHTAR, Mentored)

Tracking

({good-first-bug})

unspecified
mozilla53
good-first-bug

Details

MozReview Requests

Submitter Diff Changes Open Issues Last Updated
Loading...
Error loading review requests:

Attachments

(1 attachment, 3 obsolete attachments)

(Reporter)

Description

10 months ago
When experimenting with the task-graph generation, we encourage people to download a `parameters.yml` from a recent decision task and use that.

It would be easier if --parameters could download from a URL (so we can point it to an artifact URL) or even take a taskId and pick the parameters from the most recent run (`--parameters-from-task=<taskId>`).
(Reporter)

Updated

7 months ago
Assignee: nobody → hammad13060
(Assignee)

Comment 1

7 months ago
I have following concerns
- we can obtain parameters in the following ways
    - ./mach taskgraph tasks -p 'https://public-artifacts.taskcluster.net/{taskid}/0/public/parameters.yml'
    - ./mach taskgraph tasks -p '/path/to/file/in/local/directory/parameters.yml'
    - ./mach taskgraph tasks -p '--parameters-from-task=<taskId>' . I can parse the '--parameters' string and obtain the task-id and then fetch parameters by generating url listed in the first case.

- In order to differentiate b/w above cases we can use regular expressions

- function load_parameters_file(options) expects parameters either in a json format or in a yml format.

- This leads to a new problem. Suppose if I am given 'task-id' only, how do I know whether I should fetch a json or a yml ?
Flags: needinfo?(dustin)
(Reporter)

Comment 2

7 months ago
The artifact for a decision task is always a YAML filed named `public/parameters.yml`.

Rather than using regular expressions, I'd suggest first trying to `open(..)` the file, and if that fails, try to fetch it.
Flags: needinfo?(dustin)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
(Assignee)

Comment 6

7 months ago
Comment on attachment 8813546 [details]
Bug 1302765 - Allow `./mach taskgraph .. --parameters P` to take a URL -3;

Instead trying top open a file first and fetching from url only if open() fails, I am trying this approach. what do you think about it ?
Comment hidden (mozreview-request)
(Assignee)

Updated

7 months ago
Attachment #8813635 - Flags: review?(dustin)
(Assignee)

Updated

7 months ago
Attachment #8813521 - Flags: review?(dustin)
Attachment #8813522 - Flags: review?(dustin)
Attachment #8813546 - Flags: review?(dustin)
(Reporter)

Comment 8

7 months ago
mozreview-review
Comment on attachment 8813635 [details]
Bug 1302765 - Allow `./mach taskgraph .. --parameters P` to take a URL -4;

https://reviewboard.mozilla.org/r/95062/#review95300

I like this approach - it is simple and elegant!

From what I understand, the -P option could be

* -P parameters.yml  # just open a local file
* -P https://gist.githubusercontent.com/dustin/.../raw/...  # open a URL
* -P parameters-from-task=KGt8egfvRaqxczIRgOScaw

I'd suggest shortening the latter to `-P taskid=...` instead.

One issue here is that the URL you've constructed is the one that the queue redirects to for S3 artifacts.  If we started using some other service to store artifacts, it would stop working.  Instead, this should construct a URL for getLatestArtifact (https://docs.taskcluster.net/reference/platform/queue/api-docs#getLatestArtifact), which will always redirect to the right place.

This will also need some documentation under taskcluster/docs.  I think both the index.rst and how-to.rst will need attention.
Attachment #8813635 - Flags: review?(dustin) → review-
(Assignee)

Comment 9

7 months ago
Base URL : https://queue.taskcluster.net/v1
Route: /task/<taskId>/artifacts/<name>

so route for 'task-id=biR9rzgDT-eAoaLqIY1rWg'
becomes '/task/biR9rzgDT-eAoaLqIY1rWg/artifacts/public/parameters.yml'

right ?

Secondly I need to update the documentation for hacking into the taskgraph. right ?
Flags: needinfo?(dustin)
(Reporter)

Comment 10

7 months ago
that's right
Flags: needinfo?(dustin)
Comment hidden (mozreview-request)
(Assignee)

Updated

7 months ago
Attachment #8813522 - Attachment is obsolete: true
(Assignee)

Updated

7 months ago
Attachment #8813546 - Attachment is obsolete: true
(Assignee)

Updated

7 months ago
Attachment #8813635 - Attachment is obsolete: true
(Reporter)

Comment 12

7 months ago
mozreview-review
Comment on attachment 8813521 [details]
Bug 1302765 - Allow `./mach taskgraph .. --parameters P` to take a URL. URL building mechanism fixed;

https://reviewboard.mozilla.org/r/94972/#review95334

Awesome work!
Attachment #8813521 - Flags: review?(dustin) → review+

Comment 13

7 months ago
Pushed by dmitchell@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/55bbce8d1013
Allow `./mach taskgraph .. --parameters P` to take a URL. URL building mechanism fixed; r=dustin

Comment 14

7 months ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/55bbce8d1013
Status: NEW → RESOLVED
Last Resolved: 7 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla53
You need to log in before you can comment on or make changes to this bug.