1248192 - Add TaskCluster Index wildcard queries

Status: RESOLVED WONTFIX

(Taskcluster :: Services, defect)

Description

[Nick Alexander :nalexander [he/him]]

It would be very convenient for artifact build querying if the TaskCluster did more of the work.  |mach artifact| determines a list of Mercurial revision hashes, and wants to know what artifacts are available for those hashes.  In general, we're trying to be branch-agnostic.  Looking at https://github.com/taskcluster/taskcluster-client.py#list-namespaces, we have to do something like

>>> i.listNamespaces('buildbot.revisions.1b8a0036c771be89a2cc470ddb92616b10f96e40', {'limit':10})
{u'namespaces': [{u'expires': u'2017-02-11T00:00:00.000Z', u'namespace': u'buildbot.revisions.1b8a0036c771be89a2cc470ddb92616b10f96e40.fx-team', u'name': u'fx-team'}]}
>>> i.listTasks('buildbot.revisions.1b8a0036c771be89a2cc470ddb92616b10f96e40.fx-team', {'limit':10})
{u'tasks': [{u'data': {}, u'expires': u'2017-02-11T15:36:40.829Z', u'namespace': u'buildbot.revisions.1b8a0036c771be89a2cc470ddb92616b10f96e40.fx-team.android-api-15', u'rank': 1455286889, u'taskId': u'QMbkdc6CTSmoNUwNFnufcw'}, {u'data': {}, u'expires': u'2017-02-11T15:55:08.208Z', u'namespace': u'buildbot.revisions.1b8a0036c771be89a2cc470ddb92616b10f96e40.fx-team.android-api-15-b2gdroid', u'rank': 17581, u'taskId': u'X-xkKJt-Q0ils1aEmT0BRQ'}, {u'data': {}, u'expires': u'2017-02-11T15:42:29.692Z', u'namespace': u'buildbot.revisions.1b8a0036c771be89a2cc470ddb92616b10f96e40.fx-team.android-api-15-debug', u'rank': 1455286889, u'taskId': u'O7kcw676RICrhg6bjW9ycA'}, {u'data': {}, u'expires': u'2017-02-11T15:35:56.364Z', u'namespace': u'buildbot.revisions.1b8a0036c771be89a2cc470ddb92616b10f96e40.fx-team.android-api-9', u'rank': 1455286889, u'taskId': u'DRJt8-CSTY21yQvxB13d9g'}, {u'data': {}, u'expires': u'2017-02-11T15:26:21.109Z', u'namespace': u'buildbot.revisions.1b8a0036c771be89a2cc470ddb92616b10f96e40.fx-team.android-api-9-debug', u'rank': 1455286889, u'taskId': u'RQkcUQO2ThaIP3UUH4F1PA'}, {u'data': {}, u'expires': u'2017-02-11T15:38:02.811Z', u'namespace': u'buildbot.revisions.1b8a0036c771be89a2cc470ddb92616b10f96e40.fx-team.android-x86', u'rank': 1455286889, u'taskId': u'S6CyhyrjScaH9XjIWY18bQ'}, {u'data': {}, u'expires': u'2017-02-11T15:19:39.673Z', u'namespace': u'buildbot.revisions.1b8a0036c771be89a2cc470ddb92616b10f96e40.fx-team.emulator-ics', u'rank': 17581, u'taskId': u'SwVj2LloQKazuur1-9A8Cg'}, {u'data': {}, u'expires': u'2017-02-11T15:16:43.417Z', u'namespace': u'buildbot.revisions.1b8a0036c771be89a2cc470ddb92616b10f96e40.fx-team.emulator-kk', u'rank': 17581, u'taskId': u'AST8ua2zSXGjS1IfFPpiKw'}, {u'data': {}, u'expires': u'2017-02-11T19:14:09.709Z', u'namespace': u'buildbot.revisions.1b8a0036c771be89a2cc470ddb92616b10f96e40.fx-team.linux', u'rank': 1455286889, u'taskId': u'PImTzikfScySYJezH4NvQA'}, {u'data': {}, u'expires': u'2017-02-11T15:45:35.946Z', u'namespace': u'buildbot.revisions.1b8a0036c771be89a2cc470ddb92616b10f96e40.fx-team.linux-debug', u'rank': 1455286889, u'taskId': u'LsKbREddSJy88mVHLHok9A'}], u'continuationToken': u'1!172!ZmQ2MjgzMmNmZjA0Njc4ZjQzOTFiZTk1MzJiMjRjZjQ4NjUyOTQ3MDliZmUzNjU5ZWQ4MGM0OTlhZDhkMzk1Y2EyM2I4YjY2NGNlNjA4YmEyMDkzNmJlNDM0YTQyMDkwMmFjM2Y1MmM3Y2ZkNDk5M2E1MmRkMzM5MWNmMjI3MTk-~1!12!bGludXgzMg--'}

and then further disassemble the returned task routes to determine what type of tasks these are.

I'd like to be able to do something like:

i.listTasks('buildbot.revisions.{HASH1,HASH2}.*.{android-api-11,android-api-15}', {'limit':10})

And be able to get tasks with one request.  TaskCluster Index team, can this be done?  How hard might it be to do?

Comment 1

[Nick Alexander :nalexander [he/him]]

Build peers, is there some simpler interface we should be asking for?

Comment 2

[Jonas Finnemann Jensen (:jonasfj)]

I can't image that being possible...
Index is effectively a hierarchial key/value store.
This simplicity enables high-scalability at low cost and practically no maintaince or operational work.

One could do an attempt, but what if there is 10k subfolders to explore for android-11-api entries that won't work well.

I suspect doing so locally using some utility methods would be feasible.

Or we could move taskcluster-index to be backed by a relational database.
Which would result in less predictability of performance and less scalability.

That said, if this is a general issue. Maybe we could consider it. But it's no quick fix.

Note: nice thing with index being backed by azure table storage is that we don't have to worry about how much garbage goes into it :)
(In practice I guess an relational db would have no issues with the load though)

Comment 3

[Nick Alexander :nalexander [he/him]]

(In reply to Jonas Finnemann Jensen (:jonasfj) from comment #2)
> I can't image that being possible...
> Index is effectively a hierarchial key/value store.

Ah!  This is why I pre-flighted this request -- saves me reading the Index code to understand the model.

Let me change the question a bit, then: how expensive are redundant routes?  That is, it's clear that "a.*.b" is expensive, since you need to walk "a.*" in order to determine if "b" exists.  But is it expensive to store "a.b.c" and "a.c.b" for our load?  For artifacts we're more interested in the job type (android-api-15) than the branch (fx-team); could we store a redundant route like

build.revision.HASH.android-api-15.BRANCH

specifically for artifacts?

Comment 4

[Gregory Szorc [:gps]]

I don't think it is sane to count on routes being consistent or hierarchical in nature. There will always be ambiguous and conflicting naming rules in practice.

As Jonas said, we need some kind of index of the Index to facilitate rapid searching. If TaskCluster isn't able to provide one (I couldn't fault them for reaching that conclusion), I'd talk to the TreeHerder people - they already have a relational database of results and I /think/ they are the best positioned to implement this.

Comment 5

[Michael Shal [:mshal]]

There is an artificial limit of 15 (I believe) index entries, which we are close to for some build types, though we can bump that if necessary. Is there a reason you don't know the branch name or can't deduce it from hg?

If we want to, we can lump all the branches into the same revisions namespace, maybe with a fake branch name like "all" in the route. Then we could add something like this to testing/taskcluster/routes.json:

"{index}.gecko.v2.all.latest.{build_product}.{build_name}-{build_type}"

(This is the same as an existing route, but with "all" instead of {project}, where {project} is the branch name). So we'd still have the per-branch revisions from the existing route, and all revisions in a single tree for artifacts.

How do you determine which of the android-api-foo builds to use as the artifact to grab? Is that something the user specifies in the mozconfig? If there's a sane default, maybe that's something that we can annotate in the taskcluster configs somehow (ie: add something to say that android-api-15 is the default build for --enable-application=mobile/android, or whatever)

Also, I'd recommend using gecko.v2.* instead of buildbot.* or gecko.v1.*

Comment 6

[Gregory Szorc [:gps]]

IMO we should be using the same revisions namespace for all Firefox builds, as all the Firefox repositories really are the same logical repository, just with different DAG heads on different repositories. If we do this, we might have issues with some task names conflicting between repos. e.g. a "build" task on mozilla-central might conflict with a "build" task on beta.

Comment 7

[Jonas Finnemann Jensen (:jonasfj)]

index entries are cheap.

I don't know if we need better searching and wildcards (is it really a feature).
If we do we could move taskcluster-index to use a relational database, it's not a huge dataset.
(I can't give a time frame for when this would be possible, probably Q2)

@nalexander,
My fear is that if we give people ability to search they'll abuse it.
Imagine if we had given people the option of download an artifact based on a regular expression.
We would still have artifacts like:
     "public/build/firefox-25.4.tar.gz"

Forcing people to be explicit, also forces them to make things that are robust.


@nalexander,
what is the use-cases for globbing in automation; where globbing won't cause pain in the long run?

Comment 8

[Jonas Finnemann Jensen (:jonasfj)]

In the line of thinking that being explicit is always better in automation.

> "{index}.gecko.v2.all.latest.{build_product}.{build_name}-{build_type}"

Perhaps an "all" pattern like this isn't more than an efficient footgun.


 - Just something to think about. Before we wish for more flexibility.

Comment 9

[Nick Alexander :nalexander [he/him]]

(In reply to Jonas Finnemann Jensen (:jonasfj) from comment #7)
> index entries are cheap.
> 
> I don't know if we need better searching and wildcards (is it really a
> feature).
> If we do we could move taskcluster-index to use a relational database, it's
> not a huge dataset.
> (I can't give a time frame for when this would be possible, probably Q2)
> 
> @nalexander,
> My fear is that if we give people ability to search they'll abuse it.
> Imagine if we had given people the option of download an artifact based on a
> regular expression.
> We would still have artifacts like:
>      "public/build/firefox-25.4.tar.gz"
> 
> Forcing people to be explicit, also forces them to make things that are
> robust.
> 
> 
> @nalexander,
> what is the use-cases for globbing in automation; where globbing won't cause
> pain in the long run?

I have no use case in automation, at least not right now.  My use case is only for |mach artifact install|, which wants to find artifacts satisfying certain conditions using the Task Cluster Index.

(Clearing NI for glandium; gps made some valuable suggestions here.)

Comment 10

[Pete Moore [:pmoore][:pete] [PTO until 13 April 2026]]

Jonas, are we going to do this?

Comment 11

[Nick Alexander :nalexander [he/him]]

(In reply to Pete Moore [:pmoore][:pete] from comment #10)
> Jonas, are we going to do this?

Let's just say no -- we're doing fine without this, and it's been 1+ years.

Comment 12

[Gregory Szorc [:gps]]

It is still a performance concern. We are only going to lean on artifacts in local builds more and more. If we continue to require a separate HTTP request and round trip for each index entry, it will continue to make client-side operations take longer than they could, especially for people with slow internet connections.

There is definitely value in adding this feature. Even if it is a special service that lives in a datacenter next to the index so clients can send a "batch" request and those multiple index api requests are performed as fast as possible remotely due to lower round trip latency. But at the point you implement such as "batch" service, you might as well make that part of the official index api.

Anyway, we can continue working around this on the client side. But it is annoying.

Comment 13

[Pete Moore [:pmoore][:pete] [PTO until 13 April 2026]]

When the index service was built, it was envisioned that alternative indexing (or other) services might be desirable, so we didn't burn anything into the taskcluster platform that required indexes to work as they do now. This foresight then is now probably quite useful, as it allows us to stand up other (competing) services which can be deployed and operated independently of the taskcluster platform, and evaluated without impact to running services. The index that we have is in essence, the simplest it could be. It serves some use cases, but certainly not all.

The standard integration path for downstream processing is via the routes property of the task definition. The queue accepts arbitrary routes in task definitions, which it then CCs task creations, claims and resolutions to[1]. Integrations (such as taskcluster-pulse and the index) then bind with appropriate routing keys to the task exchanges[1] in order to process their message queues.

The first element of the route denotes the integration service, and the rest of the routing key is specific to that integration. We use 'index' to denote routes that should be consumed by the taskcluster index, 'tc-treeherder' for treeherder integration, etc.

Example:

  "routes": [
    "index.gecko.v2.mozilla-inbound.latest.firefox.macosx64-opt",
    "index.gecko.v2.mozilla-inbound.pushdate.2018.02.27.20180227110305.firefox.macosx64-opt",
    "index.gecko.v2.mozilla-inbound.pushlog-id.100242.firefox.macosx64-opt",
    "index.gecko.v2.mozilla-inbound.revision.724b51d7263fad10ec605a42db433fb784bba43e.firefox.macosx64-opt",
    "index.gecko.v2.trunk.revision.724b51d7263fad10ec605a42db433fb784bba43e.firefox.macosx64-opt",
    "tc-treeherder.v2.mozilla-inbound.724b51d7263fad10ec605a42db433fb784bba43e.100242"
  ],

The reason I mention this, is now might be an opportune time to think about what a more substantial index service ("mighty-index") might look like. It could be developed and tested in parallel to the existing index, and once we are happy that it fulfills a need that the current index lacks, we could progressively roll it out. It can also be maintained and developed by different teams if required, since we have the mechanics and authorization controls in place to enable arbitrary integrations via pulse notifications.

I think this would make most sense for a major overhaul/redesign. If we just want to tweak features, a progressive rollout to "index" might be better than creating a new integration.

----

[1] https://wiki.mozilla.org/Auto-tools/Projects/Pulse/Exchanges#TaskCluster