1200519 - bin/update-apis.js should not generate lib/apis.js, README.md, taskcluster-client.js in a random order

Status: RESOLVED FIXED

(Taskcluster :: Services, defect)

Description

[Pete Moore [:pmoore][:pete] [PTO until 13 April 2026]]

In order to update lib/apis.js, README.md, taskcluster-client.js, we run

bin/update-apis.js update
bin/update-apis.js docs
bin/update-apis.js browserify

However, the order in which the apis are processed is arbitrary, meaning that every time you run it, you can get updates to lib/apis.js, README.md, taskcluster-client.js even if there are no changes to the apis.

We should sort the apis, so that any changes to generated files reflect exactly the changes to the source APIs, without arbitrary reordering adding noise to the source control diffs.

This is already done in taskcluster-client-go and taskcluster-client-java, e.g.:
https://github.com/taskcluster/taskcluster-client-go/blob/55cedd1927aed5e90f4cc2c2720bb9dfaf54c752/codegenerator/model/model.go#L166-L171

Once this is done, we can have a periodic task to update taskcluster-client, like we do for taskcluster-client-go and taskcluster-client-java, that regenerates these files, and commits them if there are changes.

In taskcluster-client-go and taskcluster-client-java, we also fail the build in travis-ci if after regenerating the files there are any uncommitted changes. e.g. see https://github.com/taskcluster/taskcluster-client-go/blob/55cedd1927aed5e90f4cc2c2720bb9dfaf54c752/build.sh#L39-L42

This check guarantees that if there are changes, we will get a non-zero exit code, and the build will be marked as failed. This is a safety check to make sure the build will only pass if the generated code is up-to-date with the latest published apis.

I haven't checked how the python client handles this yet.

Comment 1

[Pete Moore [:pmoore][:pete] [PTO until 13 April 2026]]

P.S. The script which checks in updates to taskcluster-client-go and taskcluster-client-java is https://github.com/petemoore/myscrapbook/blob/master/update_clients.sh and runs on an hourly cron on my local computer. At some point this should move into taskcluster, when bug 1088350 is resolved.

Comment 2

[Pete Moore [:pmoore][:pete] [PTO until 13 April 2026]]

(In reply to Pete Moore [:pmoore][:pete] from comment #0)

> Once this is done, we can have a periodic task to update taskcluster-client,
> like we do for taskcluster-client-go and taskcluster-client-java, that
> regenerates these files, and commits them if there are changes.

We could alternatively consider using AWS Lambda in order to avoid polling, i.e. trigger taskcluster tasks to regenerate taskcluster client libraries upon an event notification of S3 bucket changes for:

  * http://references.taskcluster.net/manifest.json file
  * any of the references listed in the above manifest.json file
  * any of the schemas listed in any of the above references files

In practical terms, it may be simpler to just listen for any change to buckets references.taskcluster.net and schemas.taskcluster.net, triggering a taskcluster task to regenerate all clients upon any change event. A non-impactful change (such as a change to http://references.taskcluster.net/manage-manifest.sh) would result in a no-op, thus causing no harm.

Comment 3

[Pete Moore [:pmoore][:pete] [PTO until 13 April 2026]]

Attachment: Github Pull Request

Comment 4

[Jonas Finnemann Jensen (:jonasfj)]

Comment on attachment 8655464 [details]
Github Pull Request

r+ conditioned on the fact that circle-ci test pass.
See comments, I suggest you drop the magic stuff, it's not critical.

Comment 5

[Pete Moore [:pmoore][:pete] [PTO until 13 April 2026]]

Merged. Thanks Jonas!