1183001 - [taskcluster] Need a tool for downloading artifacts from Taskcluster

Status: RESOLVED FIXED

(Taskcluster :: General, defect)

Description

[Askeing Yen[:askeing][:fyen]]

For B2G test automation, we need a download tool (ex: taskcluster-cli, taskcluster-tools, or mozilla-download) which can download artifacts by namespace or taskID.

[1] mozilla-download only support b2g-desktop, no b2g device builds.

Comment 1

[Paul Yang [:pyang] (away)]

This will heavily block all our automation tests on Aries since build artifacts no longer exists in pvtbuild (previous build repository).  Can you help to raise this issue as blocker? thanks.

Comment 2

[Askeing Yen[:askeing][:fyen]]

Hi Jonas,
Could you please help us to take a look?
Thank you so much.

Comment 3

[Pete Moore [:pmoore][:pete]]

See the following taskcluster clients, available in node.js, python and go.

* Node.js: https://github.com/taskcluster/taskcluster-client
* Python: https://github.com/taskcluster/taskcluster-client.py
* Go: http://taskcluster.github.io/taskcluster-client-go

In particular, I think the API call you will probably want is:

* Node.js: http://docs.taskcluster.net/services/index/#findArtifactFromTask
* Python: https://github.com/taskcluster/taskcluster-client.py#get-artifact-from-indexed-task
* Go: https://godoc.org/github.com/taskcluster/taskcluster-client-go/index#Auth.FindArtifactFromTask

See docs under:

* http://docs.taskcluster.net/services/index

Possibly also:

* http://docs.taskcluster.net/queue/exchanges

Comment 4

[Pete Moore [:pmoore][:pete]]

Marking as resolved, since the client libraries already support downloading artifacts.

Please reopen if anything is not clear, or this does not meet your needs.

Comment 5

[Pete Moore [:pmoore][:pete]]

Resetting Version and Target Milestone that accidentally got changed...

Comment 6

[Askeing Yen[:askeing][:fyen]]

Implemented by Python
https://pypi.python.org/pypi/taskcluster_util

Comment 7

[Armen [:armenzg]]

Thanks for implementing this tool!

Is there anything of this new tool that could be ported back to the upstream taskcluster python client?
I'm just hoping that we can share as much code as possible since I *might* need this functionality in the near future.
Other projects will also be able to help your tool.

You might also be interested on visiting this thread:
https://groups.google.com/forum/#!topic/mozilla.tools/-USyAwHCJMY

Comment 8

[Askeing Yen[:askeing][:fyen]]

Hi Armem,
It depend on the Taskcluster team's decision.
I put the code on github [1]. If anything can help the python client, I am willing to merge the code.

[1] https://github.com/askeing/taskcluster-util-python

Comment 9

[John Ford [:jhford] CET/CEST Berlin Time]

That looks like a really cool use of the python client library!  I don't know if we should try to merge this code into the taskcluster-client.py repository.  I think it'll be easier to maintain both of these packages as independent repos because it decouples taskcluster-client.py releases from taskcluster_util releases.  On a more practical level, I don't want to introduce GUI dependencies or those which require GCC to be installed to build if possible.

Maybe it makes sense to move the repository on github to taskcluster/taskcluster-python-artifact-downloader?

What do you think?

Comment 10

[Armen [:armenzg]]

I'm OK with separate tools but I was just wondering if there was anything worth taking upstream.
mozci is already building on top of the client to interface with the APIs.
In other words, it is good to be decoupled :)

Comment 11

[Askeing Yen[:askeing][:fyen]]

(In reply to John Ford [:jhford] from comment #9)
> Maybe it makes sense to move the repository on github to
> taskcluster/taskcluster-python-artifact-downloader?
> 
> What do you think?

I am okay of that.
And if we move the repo, I think we also need the new working model for new repo :)