Closed Bug 1513480 Opened 2 years ago Closed 2 years ago

Stop creating gzipped version of runnable-jobs.json once bug 1494750 deployed


(Firefox Build System :: Task Configuration, task)

Not set


(firefox-esr60 fixed, firefox66 fixed)

Tracking Status
firefox-esr60 --- fixed
firefox66 --- fixed


(Reporter: emorley, Assigned: dustin)




(1 file)

In bug 1423215 task configuration changes were made such that the runnable jobs file was now output in two formats, the original runnable-jobs.json.gz and a new runnable-jobs.json, to allow for easy transition to the non-gzipped version. These changes were also up

Then in bug 1494750 support was added to Treeherder for consuming the non-gzipped version. Once the changes in that bug are deployed to Treeherder production (and a few days passed to ensure it's not being backed out for any reason), the taskcluster configs can be adjusted to no longer output the old file:

(It will also be necessary to check that there are no other consumers of the file apart from Treeherder)
(In reply to Ed Morley [:emorley] from comment #0)
> These changes were also up

These changes were also uplifted to release/ESR.
Bug 1494750 was deployed some time ago, so this bug can now proceed.
> (It will also be necessary to check that there are no other consumers of the file apart from Treeherder)

This part concerns me -- I have no idea how to check that, but the file has existed for eons, so it doesn't seem unlikely.
I can't find any remaining usages via:

There is one remaining docs reference that will need updating as part of this bug however:

Checking queue's access logs using Papertrail and filtering just on GETs (given the POSTs are the uploads), there are still requests being made however:

Examples from the last 24 hours:

For the half of the above whose IPs I spot-checked, they all originated from
Sadly, most of the Internet is in us-west-2 these days!  I spot checked a few IP's plucked from fresh logs, and they are not TC instances.

I wonder if those requests are from scriptworker, validating the content matches its signature?  Johan, does CoT look at all artifacts on a decision task?
Flags: needinfo?(jlorenzo)
For sure it doesn't look at runnable-jobs.json:
Flags: needinfo?(jlorenzo)
On IRC, Dustin made a good call: even though runnable-jobs.json is not listed in scriptworker, the latter may download all artifacts of the decision task. I don't remember this behavior.

That said, XTAADI9bRG6yw78cqYmsyw[1] is a task mentioned in comment 4. This graph doesn't have any scriptworker jobs. Moreover, there has been no subgraph inheriting from XTAADI9bRG6yw78cqYmsyw[2]. Therefore, I don't think Chain of Trust is the origin of these requests. 

I'm sort of out of ideas then.  We could just stop generating these and see what breaks?
& thanks for looking Johan!
Catlee was also doing some tests to see artifact size via head requests, iirc
(In reply to Dustin J. Mitchell [:dustin] pronoun: he from comment #5)
> Sadly, most of the Internet is in us-west-2 these days!

Heroku's common runtime is us-east, so it at least means these came from something other than Treeherder/the Taskcluster services that run on Heroku.
I'd say let's try turning these off on mozilla-central and seeing if anything breaks.
There are zero code hits and it's a one-liner to restore.
Assignee: nobody → dustin
Pushed by
stop writing out deprecated runnable-jobs.json.gz r=emorley
Closed: 2 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla66
You need to log in before you can comment on or make changes to this bug.