387368 - Joey Server - hashes to determine duplicate files between users

Status: RESOLVED WONTFIX

(Mozilla Labs :: Joey, defect)

Description

[Wil Clouser [:clouserw]]

I don't want to forget this idea, so I'm making a bug:

I think we're going to have a lot of duplicate media - once something gets popular on the internet, everyone wants a copy.

So, if we take hashes of files, we can just point users to files/previews/etc. that already exist - this would save us processing power (transcoding) and disk space.

I think the db can support relationships like this, but the file system probably can't (uploads go into a specific user's directory).

Comment 1

[Wil Clouser [:clouserw]]

We talked about this at the allhands.  Current plan is to store a hash with each file row.  When a new user uploads, we check the hash.  If it matches, we just add another row in the map table, otherwise we add a row and do the transcode.

This means we'll have to change the way we store files on disk.  Currently it's /$userid/.  We'd need to change that to /$hash/.

Comment 2

[Wil Clouser [:clouserw]]

From the whiteboard:

Remove uploads_users.owner and uploads.shared.

Comment 3

[Doug Turner (:dougt)]

Joey has been retired.  Thanks for participating.

http://joey.labs.mozilla.com/