1518219 - Modify tc-lib-artifact-go tests to not require queue:create-artifact:..

Status: RESOLVED FIXED

(Taskcluster :: General, enhancement)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

This scope has been deprecated and in bug 1419127 we are in the process of removing it. It appears that tc-lib-artifact-go uses it for tests, though:

https://tools.taskcluster.net/groups/UAB-INoZTVudKuuY7HYuNQ/tasks/GY7GBzqdRCGR9U9aMhpT7g/runs/0/logs/public%2Flogs%2Flive.log
        main_test.go:175: Artifact Error:
              1. (internal) making createArtifact queue call during upload of ./testdata/small to IRm5ndBYS1a3C56T8TuuVg/0/public/small
              2. (*tcclient.APICallException) TC-Client Error: POST https://queue.taskcluster.net/v1/task/IRm5ndBYS1a3C56T8TuuVg/runs/0/artifacts/public%2Fsmall
              3. (httpbackoff.BadHttpResponseCode) (Permanent) HTTP response code 403
            HTTP/1.1 403 Forbidden
            Content-Length: 856
            Access-Control-Allow-Headers: X-Requested-With,Content-Type,Authorization,Accept,Origin,Cache-Control
            Access-Control-Allow-Methods: OPTIONS,GET,HEAD,POST,PUT,DELETE,TRACE,CONNECT
            Access-Control-Allow-Origin: *
            Access-Control-Max-Age: 900
            Access-Control-Request-Method: *
            Cache-Control: no-store no-cache must-revalidate
            Connection: keep-alive
            Content-Security-Policy: report-uri /__cspreport__;default-src 'none';frame-ancestors 'none';
            Content-Type: application/json; charset=utf-8
            Date: Mon, 07 Jan 2019 18:24:11 GMT
            Etag: W/"358-m21oi+oJLiiglwTixdwoWV4FOig"
            Server: Cowboy
            Via: 1.1 vegur
            X-Content-Type-Options: nosniff
            X-For-Request-Id: a84b4ef9-2a31-4c4f-8fde-66d9240ba6c6

Comment 1

[Chris Cooper [:coop] (he/him)]

A few questions here, in order of importance:

1. Is tc-lib-artifact-go still important? We haven't pulled it into the monorepo which makes me wonder.
2. Is the specific test still important? I'm guessing yes, but depends on #1
3. Is this mentorable? There are a few actions that need to happen here that make me think so, including the "simple* act of changing the scopes, but also updating the lib itself to use the go client version in the monorepo.

Thoughts?

Comment 2

[Pete Moore [:pmoore][:pete]]

(In reply to Chris Cooper [:coop] pronoun: he from comment #1)

A few questions here, in order of importance:

1. Is tc-lib-artifact-go still important? We haven't pulled it into the monorepo which makes me wonder.

tc-lib-artifact-go was pretty much finished, offered some benefits over the existing artifact system (which is why we implemented it) but I don't think we migrated any workers / utilities across to using it, since the module owner left the team. I don't think a new module owner was appointed, although I suspect the blob artifact ecosystem is fully functional, since we did migrate tests to start using it (like the test in this bug).

We probably should review the state of the blob artifact system, and decide whether to keep it and migrate tools/utilities/workers across to use it, or whether to abandon it, or iterate on it if there are any missing pieces.

2. Is the specific test still important? I'm guessing yes, but depends on #1

It is a CI test of tc-lib-artifact-go itself, so if we abandon the blob artifact system it isn't, but if we adopt it, yes we should adapt the CI to have more restricted scopes if possible.

3. Is this mentorable? There are a few actions that need to happen here that make me think so, including the "simple* act of changing the scopes, but also updating the lib itself to use the go client version in the monorepo.

The bigger piece of work is deciding if we're keeping the blob artifact system (and potentially removing the other artifact types) - changing the CI to use the different scope is probably a trivial change, quicker to research and implement than to mentor. Note, this test only runs when there is a change to tc-lib-artifact-go which hasn't happened in 8 months, since we're not actively working on it.

Thoughts?

Let's assign someone on the taskcluster services side, the task of assessing current state (i.e. if APIs are documented, functioning correctly, and are appropriate and preferable to previous artifact system), and then get rid of one system or the other. Rather than just making a decision though, a report would be useful that the whole team can read which summarises what the differences are, what the current state is, what the steps would be to migrate to use it, and what benefits/disadvantages there would be.

Comment 3

[Chris Cooper [:coop] (he/him)]

(In reply to Pete Moore [:pmoore][:pete] from comment #2)

Let's assign someone on the taskcluster services side, the task of assessing current state (i.e. if APIs are documented, functioning correctly, and are appropriate and preferable to previous artifact system), and then get rid of one system or the other. Rather than just making a decision though, a report would be useful that the whole team can read which summarises what the differences are, what the current state is, what the steps would be to migrate to use it, and what benefits/disadvantages there would be.

This would likely fall to :owlish as a prerequisite to starting the Object Service work. She has a bunch of provider stability work to get through still, so I'd guess we're looking at Q4.

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

I more-or-less did that analysis. Nothing was using either the 'blob' or 'azure' artifact types (except a task that bastien changed), and blob storage wasn't working. So, in anticipation of the object service and as an effort to simplify the current state, support for both was removed. So basically I've done what pete suggested, and will now archive this repo.

Owlish is aware and will refer to the library when developing the object service and -- if appropriate -- revive it and use it.