1649145 - taskclusteretl - Add a table for the artifact-created pulse exchange

Status: RESOLVED FIXED

(Data Platform and Tools :: General, enhancement, P1)

Comment 1

[Mike Trinkala [:trink]]

Joel, this message can be added to the pulse_task table in BQ as it has a lot overlap and only four new fields, opinions/preferences on a dedicated or combined table?

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

combined table sounds logical to me

Comment 3

[Mike Trinkala [:trink]]

Sadly this does not contain messages for the files we are interested in.

From the taskcluster docs:
NOTE: this message is currently only sent for reference and error artifacts. This will be remedied in a future version of Taskcluster.
I am only seeing the following

public/logs/live.log	reference
public/build	error
public/docker-contexts	error
public/logs	error
public/build/maven	error
public/build/geckoview_example.apk	error
public/build/geckoview-androidTest.apk	error
public	error
public/code-review/mozlint.json	error

Dustin, I assume this isn't happen any time soon and I would have to fall back to the listArtifacts API correct?

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

Yes -- that's the fix for the moment.

I expect sometime soon taskclusteretl could switch to doing ETL by shipping postgres backups or allowing you access to a read-only replica, which should help. One of the changes at this weekend's downtime is to convert the artifacts table into something with the expected columns like "task_id" and "name" and "storage_type".

We expect to have all of the DB tables in this "more normal" form in the next few weeks. Maybe after that point we can talk (with cloudops) about getting you closer access to that data, and how to manage compatibility as we migrate DB schemas (I have some vague ideas about more-stable views..)

Comment 5

[Mike Trinkala [:trink]]

the artifact_list table is in BigQuery as of 7/23/2020