1599883 - Spikes in requests for artifacts in firefox-ci-tc since 16:00 UTC today

Status: RESOLVED FIXED

(Cloud Services :: Operations: Taskcluster, defect)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: Screenshot_2019-11-27 Taskcluster Visuals - Grafana.png

The purples are 303's from queue, which is the response to getArtifact and getLatestArtifact. This looks like something automated..

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: image.png

https://earthangel-b40313e5.influxcloud.net/explore?orgId=1&left=%5B%22now-12h%22,%22now%22,%22svcops%22,%7B%22datasource%22:%22svcops%22,%22groupBy%22:%5B%7B%22params%22:%5B%22$interval%22%5D,%22type%22:%22time%22%7D,%7B%22params%22:%5B%22responsecode%22%5D,%22type%22:%22tag%22%7D,%7B%22params%22:%5B%22service%22%5D,%22type%22:%22tag%22%7D%5D,%22measurement%22:%22logging.googleapis.com%2Fuser%2Ftaskcluster%22,%22orderByTime%22:%22ASC%22,%22policy%22:%22default%22,%22resultFormat%22:%22time_series%22,%22select%22:%5B%5B%7B%22params%22:%5B%22nginx_counter%22%5D,%22type%22:%22field%22%7D,%7B%22params%22:%5B%5D,%22type%22:%22sum%22%7D%5D%5D,%22tags%22:%5B%7B%22key%22:%22cluster_name%22,%22operator%22:%22%3D~%22,%22value%22:%22%2F%5E$cluster_name$%2F%22%7D,%7B%22condition%22:%22AND%22,%22key%22:%22realm%22,%22operator%22:%22%3D~%22,%22value%22:%22%2F%5E$realm$%2F%22%7D,%7B%22condition%22:%22AND%22,%22key%22:%22service%22,%22operator%22:%22%3D%22,%22value%22:%22queue%22%7D,%7B%22condition%22:%22AND%22,%22key%22:%22responsecode%22,%22operator%22:%22%3D%22,%22value%22:%22303%22%7D%5D%7D,%7B%22mode%22:%22Metrics%22%7D,%7B%22ui%22:%5Btrue,true,true,%22none%22%5D%7D%5D

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

It looks like these were due to a staggering number of action tasks in https://treeherder.mozilla.org/#/jobs?repo=try&revision=a3ae0e62f7b57862fb0610a15335f268f6600dc1 for https://bugzilla.mozilla.org/show_bug.cgi?id=1587534. No real harm here except a few hours of my time tracing it down -- but this could easily have caused a treeclosure, and was easily avoided.

Comment 3

[Michal Novotny [:michal]]

I'm sorry about that. I was running just a single test and I tried to do it in batches. There is no other way I could find the cause than trying to reproduce it on try with added logging.

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

If you click on a job, then click on the "..." menu in the lower right and choose "custom action", you can adjust the "times" parameter for the retrigger, and create 8000 new test tasks with only one action :)

Also, I believe the mach try command has a way to specify a number of times to run each task, but my memory is pretty fuzzy.