1247061 - Reusing connections to improve performance

Status: UNCONFIRMED

(Testing :: mozregression, defect, P5)

Description

[YF (Yang)]

I see repeated connections open and closed for fetch info from queue.taskcluster.net:443 and index.taskcluster.net:443 (and maybe more) by an proxy software, they transfer a small amount of information (every ~5KB), but open a TLS connection requires a complete synchronization delay for it.

At least, I see the packs download connections is persistent.

Comment 1

[Julien Pagès (:parkouss)]

(In reply to YF (Yang) from comment #0)
> I see repeated connections open and closed for fetch info from
> queue.taskcluster.net:443 and index.taskcluster.net:443 (and maybe more) by
> an proxy software, they transfer a small amount of information (every ~5KB),
> but open a TLS connection requires a complete synchronization delay for it.
> 
> At least, I see the packs download connections is persistent.


Those requests are done by the taskcluster-client.py module, that we use - other requests in mozregression are using a single http session (through requests.Session) and I believe they don't suffer this problem.

I don't see in taskcluster-client any way to specify a custom session, and as far as I can see there is no session concept in taskcluster-client, and there is no way for an user of the library to change that:

https://github.com/parkouss/taskcluster-client.py/blob/master/taskcluster/utils.py#L187

Jonas, I believe this can be fixed in taskcluster-client.py ? We would need to be able to allow to pass a custom requests.Session to use in Queue and Index, or at least it should be able to create a persistent connection (i.e., requests.Session again) for those - so multiple calls can benefits from it.

Thoughts ?

Comment 2

[Jonas Finnemann Jensen (:jonasfj)]

> Thoughts ?

Agree, we definitely should do this.. I defer to jhford who owns the taskcluster-client.py
This is a pretty serious bug, as one quickly runs out of available ports...

Comment 3

[John Ford [:jhford] CET/CEST Berlin Time]

https://github.com/taskcluster/taskcluster-client.py/pull/40

Comment 4

[YF (Yang)]

I still see some short-lived connections to index.taskcluster.net:443, queue.taskcluster.net:443, archive.mozilla.org:443 and others host in mozregression-gui 0.9.1 mozregression 2.3.1. I guess they should be persisted.