Closed Bug 1371489 Opened 5 years ago Closed 5 years ago

taskgraph uses new connections to index.taskcluster.net after a 404

Categories

(Taskcluster :: General, enhancement)

enhancement
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED
mozilla56

People

(Reporter: glandium, Assigned: glandium)

Details

Attachments

(1 file)

cf. the logs from a decision task on try:

task 2017-06-07T22:51:20.495815Z] "GET /index/v1/task/gecko.cache.level-3.toolchains.v1.linux64-clang.d616ce8d88e325714d2d490ed75333bf6772e6cc0b510769dd9226c717f940e2 HTTP/1.1" 404 449
[task 2017-06-07T22:51:20.497298Z] Starting new HTTP connection (2): taskcluster
[task 2017-06-07T22:51:20.696601Z] "GET /index/v1/task/gecko.cache.level-2.toolchains.v1.linux64-clang.d616ce8d88e325714d2d490ed75333bf6772e6cc0b510769dd9226c717f940e2 HTTP/1.1" 404 449
[task 2017-06-07T22:51:20.698184Z] Starting new HTTP connection (3): taskcluster
[task 2017-06-07T22:51:20.915775Z] "GET /index/v1/task/gecko.cache.level-1.toolchains.v1.linux64-clang.d616ce8d88e325714d2d490ed75333bf6772e6cc0b510769dd9226c717f940e2 HTTP/1.1" 200 239
[task 2017-06-07T22:51:20.916358Z] optimizing `toolchain-linux64-clang`, replacing with task `SHSZL0OYRdmlV-elrk5kxQ`
[task 2017-06-07T22:51:21.151152Z] "GET /index/v1/task/gecko.cache.level-3.toolchains.v1.linux64-clang-tidy.a64debfdbd29c9b355044a284de9486aa0ced259442db8b5febba277a674f247 HTTP/1.1" 404 454
[task 2017-06-07T22:51:21.152537Z] Starting new HTTP connection (4): taskcluster
[task 2017-06-07T22:51:21.396985Z] "GET /index/v1/task/gecko.cache.level-2.toolchains.v1.linux64-clang-tidy.a64debfdbd29c9b355044a284de9486aa0ced259442db8b5febba277a674f247 HTTP/1.1" 404 454
[task 2017-06-07T22:51:21.398548Z] Starting new HTTP connection (5): taskcluster
(...)
Comment on attachment 8875947 [details]
Bug 1371489 - Reuse HTTP connections after HTTP errors.

https://reviewboard.mozilla.org/r/147354/#review152356

::: taskcluster/taskgraph/util/taskcluster.py:52
(Diff revision 1)
>      For other types of content, a file-like object is returned.
>      """
>      response = _do_request(get_artifact_url(task_id, path, use_proxy))
> +    # Consume content before raise_for_status, so that the connection can be
> +    # reused.
> +    response.content

This function supports returning raw files, which won't work if we've already read the content. I don't think that functionality is used (at least, I don't see it used in https://hg.mozilla.org/mozilla-central/rev/c9dabe672fd8), so it could probably be removed.

In which case, why not just move `response.content` to `_do_request`?
Attachment #8875947 - Flags: review?(dustin) → review-
(In reply to Dustin J. Mitchell [:dustin] from comment #2)
> This function supports returning raw files, which won't work if we've
> already read the content. I don't think that functionality is used (at
> least, I don't see it used in
> https://hg.mozilla.org/mozilla-central/rev/c9dabe672fd8), so it could
> probably be removed.
> 
> In which case, why not just move `response.content` to `_do_request`?

In the end, I didn't want to decide whether to remove the raw file thing from get_artifact (possibly returning response.content, but then that'd be memory consuming and footgun'y), so I went with just doing it in case of errors, and moving it to `_do_request`.
Comment on attachment 8875947 [details]
Bug 1371489 - Reuse HTTP connections after HTTP errors.

https://reviewboard.mozilla.org/r/147354/#review155848

thanks!
Attachment #8875947 - Flags: review?(dustin) → review+
Pushed by mh@glandium.org:
https://hg.mozilla.org/integration/autoland/rev/9933a2e58516
Reuse HTTP connections after HTTP errors. r=dustin
https://hg.mozilla.org/mozilla-central/rev/9933a2e58516
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla56
You need to log in before you can comment on or make changes to this bug.