1524845 - Download label-to-taskId's in parallel

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

https://tools.taskcluster.net/groups/exSDA0XCSgiBDbSGEn3T5w/tasks/Q5SzttalT-aS4Wnq-tnKkA/runs/0/logs/public%2Flogs%2Flive.log

...
[task 2019-02-03T12:45:42.059Z] fetching label-to-taskid.json for action task O3PiCoeUSzO1BIOJjk6ljw
[task 2019-02-03T12:45:42.487Z] fetching label-to-taskid.json for action task Q1Wxa9EkRCWezQcRB3nejA
[task 2019-02-03T12:45:42.855Z] fetching label-to-taskid.json for action task PODQp2v0TDu_FEZr_AM5Pg
[task 2019-02-03T12:46:01.365Z] fetching label-to-taskid.json for action task cFGuSW8HQuS5HQM7ntB0Gg
[task 2019-02-03T12:46:21.189Z] fetching label-to-taskid.json for action task NY00v69wSY6NBvxcVQ2mEQ
[task 2019-02-03T12:46:41.083Z] fetching label-to-taskid.json for action task EAIcB-1QQpS_t24RSF_WRg
[task 2019-02-03T12:47:00.956Z] fetching label-to-taskid.json for action task GnUAsx_WRc-p9KEYqPT2xQ
[task 2019-02-03T12:47:21.882Z] fetching label-to-taskid.json for action task c6UpHi3bQleoZqYyt8S71w
[task 2019-02-03T12:47:39.379Z] fetching label-to-taskid.json for action task NPN6bn5-SPWOsT1EL_s5wg
...

this task is trying to download a lot of label-to-taskid.json files, and as the first thing to download them is being forced to wait for a copy to the correct AWS region. We can get much better performance by running all of these downloads in parallel.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=5e0bdffda191a80026442ef70dac7f1d606356d4

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=3f6c549ad02a9f6feca3aaf910ce3455cde2c3ba

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

https://tools.taskcluster.net/groups/KvcTbxttRzyqd7C1EKjL_w/tasks/b-Alb8bmRy6cCKxfAeGIGw/runs/0/logs/public%2Flogs%2Flive_backing.log shows this patch successfully downloading a few artifacts in parallel.

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: Bug 1524845 - download data about previous crons, actions in parallel

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/45d561eacd00
download data about previous crons, actions in parallel r=aki

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

Joel, Sebastian -- this should help actions on pushes with lots of action tasks to succeed and run more quickly. Let me know if you see issues (and feel free to link this into the dependency tree of related bugs).

Comment 7

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/45d561eacd00