Closed Bug 1900789 Opened 2 years ago Closed 2 years ago

Use persistent S3 clients in thread-local storage

Categories

(Tecken :: General, defect, P2)

Tracking

(Not tracked)

VERIFIED FIXED

People

(Reporter: sven, Assigned: sven)

References

Details

Attachments

(1 external file)

Currently, the S3 client and the thread pool are regenerated in each upload request. Moreover, the boto3 session is shared across threads, while it is documented not to be thread safe 1.

We want to persist both the S3 client and the boto3 session across requests without sharing them across threads. This means these objects need to live in thread-local storage. We need some global object to store these objects. Since they are backend-specific, they should live in the storage interface we will introduce in bug 1564452. Before fully implementing the storage interface, we use the existing StorageBucket class (which will eventually become S3Bucket). Currently, StorageBucket instances are created on the fly in a few places, but some are also persisted inside SymbolDownloader, which is only used for downloads so far.

In summary, the plan for this ticket is this: Rename SymbolDownloader to SymbolStorage and make it a high-level interface to interact with the storage. All interactions with the storage backend should use global instances of the SymbolStorage class. These global instances will hold StorageBucket instances for each source, which in turn hold S3 client objects in thread-local storage.

Priority: -- → P2

This was deployed in bug 1904880 tag v2024.06.26.

This is looking good in production.

Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: