1671360 - improve diff upload

Status: RESOLVED FIXED

(Conduit :: moz-phab, defect)

Description

[Tarek Ziadé (:tarek)]

This api https://github.com/tarekziade/review/blob/master/mozphab/diff.py#L203

upload files one by one, then the diff. This adds a lot of overhead calls. We should

• zip all files in a single tarball, along with the diff
• send that tarball
• make bigger chunks.

I guess we need to adapt the server-side conduit api?

Comment 1

[:glob ✱]

(In reply to Tarek Ziadé (:tarek) from comment #0)

This api https://github.com/tarekziade/review/blob/master/mozphab/diff.py#L203

upload files one by one, then the diff. This adds a lot of overhead calls. We should

• zip all files in a single tarball, along with the diff
• send that tarball
• make bigger chunks.

I guess we need to adapt the server-side conduit api?

Be aware we don't own the server-side implementation; best we can do is request functionality from Phacility.

Comment 2

[Tarek Ziadé (:tarek)]

I see, I thought we owned the conduit thing for some reason. I guess one "free" improvement would be to upload files in parallel using a pool of threads, when we have more than one file. I guess it's not the most common use case, but it will speed up things a little bit on big changes.

here's one possible implementation: https://github.com/tarekziade/review/commit/f362551d73d9f2b6b633f9a9217a56242b5ea04e

I can push a clean patch if you think it's worth it. lmk

Comment 3

[:glob ✱]

Thanks - sounds like it's worth experimenting with threading.

Comment 4

[Tarek Ziadé (:tarek)]

Attachment: Bug 1671360 - improve diff upload r?glob

Comment 5

[Tarek Ziadé (:tarek)]

patch ready for your consideration

Comment 6

[:glob ✱]

https://github.com/mozilla-conduit/review/commit/7221faa1d71b63d245f6745e91e3e6566bc88fef